WO2016113836A1

WO2016113836A1 - Manipulator control method, system, and manipulator

Info

Publication number: WO2016113836A1
Application number: PCT/JP2015/050616
Authority: WO
Inventors: 潔人伊藤; 宣隆木村; 敬介藤本
Original assignee: 株式会社日立製作所
Priority date: 2015-01-13
Filing date: 2015-01-13
Publication date: 2016-07-21
Also published as: JPWO2016113836A1; JP6328796B2

Abstract

Provided is a method for controlling the position of a mobile manipulator so as to avoid collisions with obstacles. The method is provided with a procedure for detecting the orientation of a mirror, and a procedure for detecting the orientation of an object from an image. The orientation of an object is detected from an image reflected in the mirror, and the manipulator is controlled on the basis of the detected orientation of the object.

Description

Manipulator control method, system, and manipulator

The present invention relates to a configuration and control method of a manipulator, and more particularly to a configuration and control method of a mobile manipulator using an image.

Mobile manipulators are expected to be put to practical use as alternatives to human activities in facilities such as warehouses and factories and in homes. In general, a moving manipulator includes a moving carriage unit that moves itself by moving forward, backward, and turning, an arm unit that has one or more joints attached to the moving carriage unit, and a gripper unit that is attached to the tip of the arm unit. And have. The gripper unit can hold or move a specific object. Such gripping and movement of the object by the gripper unit is referred to as an operation on the object.

A stationary manipulator fixed on a gantry in a factory or the like operates an object at a predetermined position in a specific work space. Accordingly, it is possible to design in advance a method for operating an object so that surrounding obstacles and the manipulator do not collide or interfere with each other.

On the other hand, unlike the stationary manipulator, the moving manipulator moves around and cannot limit the working space of the arm part and the gripper part. That is, in the mobile manipulator, an unknown space in which work objects and obstacles are cluttered is a work space. In such an unknown work space, the moving manipulator needs to operate the arm part, the gripper part, and the target object so as not to collide with or interfere with the obstacle. For this purpose, it is necessary to acquire the positional relationship between the manipulator itself, in particular, the arm part or gripper part that causes interaction with the object, and the target object or surrounding obstacles.

Conventionally, there is a technique for acquiring a positional relationship between a robot including a moving manipulator and an object or surrounding environment using a camera.

For example, Patent Document 1 discloses a technology in which a camera installed in a work space captures an image (monitoring image) of a working robot, and an operator performs an operation while viewing the first person image and the monitoring image. .

In Patent Document 2, a first camera is provided on the trunk of a robot having a manipulator, a second camera different from the first camera is provided in the manipulator, and an image of the first camera and the second camera are provided. An invention for synthesizing these images is disclosed. As a result, the area of the manipulator portion reflected in the first camera is complemented by the second camera, so that the blind spot due to the manipulator in the first camera is removed, and the position of the grasped object is specified by the first camera. is doing.

In Patent Document 3, an identification surface provided in a work space, a reflection mirror provided at a position facing the identification surface, and a camera that images the identification surface via the reflection mirror, the identification surface imaged by the camera Based on these images, a technique for detecting an intruder in the work space has been disclosed.

JP2001-062766 JP2010-131751 JP2013-52485A

However, in the technique described in Patent Document 1, a camera for monitoring images must be installed so that there is no blind spot in the work space. In this case, when the object is placed in a place such as a shelf, there is a problem that a blind spot is always formed.

Also, with the technique described in Patent Document 2, even if an arm camera is used, the blind spot behind the object cannot be photographed. In order to photograph the shadow of such an object, it is necessary to wrap the arm camera behind the object. If there is an obstacle in the path for moving the arm to wrap around the arm camera, the manipulator and the obstacle collide.

On the other hand, the technique described in Patent Document 3 is premised on that the position of the reflecting mirror and the position of the camera relative thereto are fixed in advance so as to have a predetermined positional relationship in the work space. For this reason, if the position of the camera or reflecting mirror is changed, the identification surface cannot be photographed. That is, when the work space is changed, it is necessary to re-install the reflecting mirror and the position of the camera relative to the reflecting mirror so as to have a predetermined positional relationship. Therefore, it cannot be applied to the case where the work area is arbitrarily changed, such as a moving manipulator.

In view of the above, the problem to be solved by the present invention is to capture the positional relationship between the gripper part and the object and the surrounding environment in the work space of the mobile manipulator without blind spots, and based on that, do not collide with an obstacle. The object is to provide a method for controlling the position of the gripper section.

A representative one of the inventions disclosed in the present application includes a procedure for detecting the attitude of a mirror from an image and a procedure for detecting the attitude of an object from an image, and the attitude is reflected in the detected mirror. In this method, the posture of an object is detected, and the manipulator is controlled based on the detected posture of the object.

Another aspect of the present invention is a manipulator control method for operating a movable manipulator in a work space where an object is arranged. In this method, a mirror placed in or near the work space is imaged with a camera mounted on a manipulator, and the mirror attitude is detected based on the camera attitude and the image captured by the camera. Then, a mirror region is detected from the image captured by the camera, and at least a partial image of the object is detected from the detected mirror region. Then, the posture of the target object in the work space is calculated based on the mirror posture and the target object image, and the manipulator is operated based on the calculated information on the target posture.

There are various methods for calculating the posture of the object in the work space based on the mirror posture and the object image. As an example, the mirror image viewpoint of the camera is calculated based on the camera posture and the mirror posture, and the object posture in the work space is calculated based on the camera mirror image viewpoint and the image of the object reflected in the mirror. . As another example, the virtual image of the object in the mirror image space is calculated based on the camera image and the image of the object reflected in the mirror, and the work space is calculated based on the virtual image attitude of the object and the mirror image. The posture of the object at is calculated.

Thus, by using the image reflected in the mirror, it is possible to obtain information that cannot be taken from the position of the camera, and it can be used for manipulator control.

To describe a more specific example, the object is placed on a shelf, the manipulator is moved to the vicinity of a predetermined shelf based on the shelf arrangement information, and the camera, mirror, etc. are set with reference to the origin fixed to the manipulator. , And the posture of the object may be calculated. Since the manipulator itself moves, control is facilitated by unifying the coordinate axes based on the manipulator at the time of work.

Also, as the image reflected in the mirror, not only the operation target object but also the image of the manipulator itself or other objects may be used. Further, more information can be obtained by integrating and using the image reflected in the mirror and the image of the area other than the mirror directly taken by the camera.

As a method for detecting the attitude of the mirror, for example, a marker can be arranged in the mirror or the work space, and the attitude of the mirror can be detected based on the marker information in the image captured by the camera.

Another aspect of the present invention is a manipulator control system for operating a movable manipulator in a work space where an object is arranged. In this system, the manipulator includes a moving mechanism that moves the manipulator, an operation mechanism that operates the object, a camera that images the work space, a camera control mechanism that changes the posture of the camera, and a control unit. The control unit has a function of detecting the posture of the camera and a function of detecting the posture of the mirror in the captured image based on the posture of the camera and the image captured by the camera. Further, based on the function of detecting a mirror region from the image captured by the camera and detecting at least a part of the image of the object from the detected mirror region, the posture of the mirror and the image of the object, the work space And a function for calculating the posture of the object. In this system, the operation mechanism is operated based on the calculated posture information of the object.

As an example of the configuration of the operation mechanism, an example having an arm unit connected by at least one joint mechanism and a gripper unit held by the arm unit and capable of grasping and moving an object can be considered. The camera can be arranged in the gripper part.

As an example of detecting the posture of the camera, the control unit controls the operation of the operation mechanism, stores geometric data of the shape of the operation mechanism, and uses the geometric data to determine the posture of the camera. Can be detected.

In a preferred embodiment, the work space in which the object is arranged can be defined with reference to the shelf on which the object is arranged, and can be configured to arrange a mirror on the shelf.

In a more preferred specific example, the manipulator is configured to be movable in the vicinity of an arbitrary shelf among the plurality of shelves. The control unit stores shelf arrangement information related to the arrangement of a plurality of shelves, shelf shape information related to the shape of the shelf, and object information related to the objects arranged on each shelf, and the control unit stores the shelf arrangement information and Based on the object information, the manipulator can be moved to the vicinity of a predetermined shelf, and the work space can be imaged with a camera.

As a specific example for making it easy to detect the posture of the mirror, a marker is arranged in the work space, and the relative positional relationship between the marker and the mirror is stored in the control unit. It is possible to detect the attitude of the mirror in the captured image based on the marker.

Still another aspect of the present invention is a manipulator that operates in a work space in which an object is arranged. The manipulator includes a moving mechanism for moving the manipulator, a gripper unit for operating the object, an arm unit for moving the gripper unit, a camera for imaging the work space, and a camera control mechanism for changing the posture of the camera. And having a control unit. The control unit has a function of detecting the posture of the camera and a function of detecting the posture of the mirror in the captured image based on the posture of the camera and the image captured by the camera. In addition, a function for detecting at least a part of an image of an object from an image captured by a camera, a function for calculating the attitude of an object in a workspace based on the attitude of a mirror and the image of the object, and a calculated object The gripper unit is operated based on the information on the posture of the object. As a more specific configuration example, in the function of having a marker arranged in the gripper unit and detecting the posture of the mirror, information on the posture of the marker and an image of the marker in the image captured by the camera are used. The attitude of the mirror in the captured image may be detected.

In the above, the configuration of the control unit may be configured by a single computer, or any part of the input device, output device, processing device, and storage device may be configured by another computer connected via a network. Also good. Specifically, the control unit may be configured by a computer mounted on a manipulator, or a part or all of the functions may be arranged on a server that can communicate with the manipulator. The function of the control unit can be realized by software executed by the processing device. Functions equivalent to those configured by software can also be realized by hardware such as FPGA (Field Programmable Gate Array) and ASIC (Application Specific Integrated Circuit). Such an embodiment is also included in the scope of the present invention.

According to the present invention, it is possible to control the mobile manipulator so as not to collide with an obstacle in the work space. Problems, configurations, and effects other than those described above will be clarified by the following description of embodiments.

It is the perspective view which showed typically the picking operation | movement of the target object by a movement manipulator. FIG. 2 is a perspective view schematically showing an example of a schematic configuration of a moving manipulator according to the first embodiment. 3 is a side view showing an example of a configuration of a gripper portion of the moving manipulator according to the first embodiment. FIG. It is the perspective view which showed typically an example of the structure of the shelf concerning Embodiment 1. FIG. 1 is a front view schematically showing an example of a configuration of a plane mirror according to a first exemplary embodiment. 1 is a block diagram illustrating an example of a configuration of a control system for a mobile manipulator according to a first embodiment. 3 is a flowchart illustrating an example of an operation procedure of the moving manipulator according to the first embodiment. It is the side view which showed typically an example of the mode seen from the side surface of the work space in step S702 of FIG. FIG. 9 is an image diagram illustrating an example of a photographed image 141 photographed by the camera 14 in FIG. 8. It is the side view which showed typically another example of the mode seen from the side surface of the work space in step S702 of FIG. It is the image figure which showed an example of the picked-up image 142 image | photographed with the camera 14 in FIG. FIG. 3 is a principle diagram schematically showing a method of calculating the attitude of a plane mirror according to the first embodiment. It is a side view explaining the mirror image viewpoint C 'with respect to the plane mirror of the camera concerning Embodiment 1. FIG. FIG. 6 is a side view for explaining a mirror image viewpoint C ″ with respect to the plane mirror of the camera according to the first embodiment; It is the side view which showed another example of a structure of the gripper part of the movement manipulator concerning Embodiment 1. FIG. It is the front view which showed typically another example of the structure of the plane mirror concerning Embodiment 1. FIG. 1 is a diagram illustrating a configuration of a checker pattern marker according to a first embodiment. FIG. 3 is a schematic diagram illustrating an example of a configuration of a convex mirror according to the first embodiment. It is the perspective view which showed another example of the structure of the shelf concerning Embodiment 1. FIG. It is the perspective view which showed typically the structure of the shelf concerning Embodiment 2. FIG. 10 is a flowchart illustrating an example of an operation procedure of the moving manipulator according to the second embodiment. FIG. 5 is a schematic diagram illustrating a configuration of a tray according to a second embodiment. It is the perspective view which showed an example of the structure of the gripper part of the movement manipulator concerning Embodiment 3. FIG. FIG. 10 is a side view showing an example of a method for calculating a reflecting surface of a plane mirror according to a third embodiment. It is the perspective view which showed typically the structure of the shelf concerning Embodiment 3. FIG. 10 is a flowchart showing a part of an example of an operation procedure of the moving manipulator according to the fourth embodiment; It is the side view which showed an example of the method of calculating the reflective surface of the plane mirror concerning Embodiment 4.

Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In all the drawings, even if the embodiments are different, the same or corresponding members are denoted by the same reference numerals, and the common description will not be repeated. The present invention is not construed as being limited to the description of the embodiments below. Those skilled in the art will readily understand that the specific configuration can be changed without departing from the spirit or the spirit of the present invention.

In this specification and the like, notations such as “first”, “second”, and “third” are attached to identify the constituent elements, and do not necessarily limit the number or order. In addition, a number for identifying a component is used for each context, and a number used in one context does not necessarily indicate the same configuration in another context. Further, it does not preclude that a component identified by a certain number also functions as a component identified by another number.

The position, size, shape, range, etc. of each component shown in the drawings and the like may not represent the actual position, size, shape, range, etc. in order to facilitate understanding of the invention. For this reason, the present invention is not necessarily limited to the position, size, shape, range, and the like disclosed in the drawings and the like.

FIG. 1 is a diagram showing a picking operation of an object by a moving manipulator. In Embodiment 1 of the present invention, as shown in FIG. 1, an operation of grasping and lifting an object 3 placed on a shelf 2 by a moving manipulator 1 will be described. Hereinafter, a series of operations for gripping and lifting the object 3 by the moving manipulator 1 is referred to as a picking operation of the object 3.

(Constitution)
FIG. 2 is a schematic perspective view illustrating a schematic configuration of the moving manipulator 1 according to the first embodiment. In FIG. 2, the movable manipulator 1 is provided at a predetermined portion of the control unit 10, the movable carriage unit 11, the arm unit 12, the gripper unit 13 fixed to the tip of the arm unit 12, and the gripper unit 13. Camera 14.

The control unit 10 receives an instruction from the host control unit 6 (described later) by wire or wireless, and controls the operation of the mobile manipulator 1. Here, the operation control means not only controlling the operation of each part of the moving manipulator 1 based on the instruction, but also planning the operation of the moving manipulator 1 based on information such as the camera 14 of the moving manipulator 1. Including intelligent processing.

The moving carriage unit 11 includes one or more moving mechanisms such as wheels 110, and moves the moving manipulator 1 to an arbitrary place on a flat ground by moving forward / backward / turning based on an operation command of the moving manipulator control unit 10. . Here, the flat ground may include not only a simple plane but also a slope or a small step. Hereinafter, the flat ground movement of the moving manipulator 1 by the moving carriage unit 11 is referred to as traveling of the moving manipulator 1.

The arm unit 12 is fixed to a predetermined portion of the movable carriage unit 11. The arm unit 12 has one or more joint mechanisms, and moves the gripper unit 13 to a predetermined position and direction in a predetermined three-dimensional space based on an operation command of the moving manipulator control unit 10. Hereinafter, the position and direction of the gripper portion 13 are referred to as the posture of the gripper portion 13. In FIG. 2, the arm unit 12 has a configuration having a vertical articulated mechanism in which a plurality of joint mechanisms are connected by the arm unit. However, the arm unit 12 may be configured to move the gripper unit 13 to a predetermined posture. The configuration is not limited to this. For example, a horizontal multi-joint mechanism, an orthogonal multi-joint mechanism, a parallel link mechanism, or a combination thereof may be used.

The gripper unit 13 has a function of gripping a predetermined object. In FIG. 2, the gripper unit 13 shows a configuration in which a predetermined object is gripped by two finger mechanisms 130, but is not limited thereto. For example, the structure which has three or more finger mechanisms, or the structure which hold | grips a target object with a vacuum suction mechanism or a magnet without using a finger mechanism may be sufficient.

FIG. 3 is a schematic side view of the distal end portion of the arm portion 12, the gripper portion 13, and the camera 14 in FIG. In FIG. 3, the camera 14 is offset and fixed at a predetermined angle with respect to the extending direction of the finger mechanism 130 of the gripper unit 13. That is, the lower end direction of the visual field V of the camera 14 is fixed so as to be parallel to the extending direction of the finger. Thereby, it is possible to avoid the finger mechanism 130 from being reflected in the visual field V.

The arm portion 12 is provided with a joint mechanism 121, and the gripper portion 13 can be directed at an arbitrary angle along the arrow P in FIG. Similarly, the arm portion 12 is provided with a joint mechanism 122, and the gripper portion 13 can be directed at an arbitrary angle along the arrow R in FIG. Moreover, the arm part 12 can orient | assign the gripper part 13 to the arbitrary directions according to the arrow Y in FIG. 3 with the other joint mechanism of the arm part 12 which is not shown in figure. Thus, the arm unit 12 can direct the gripper unit 13 in an arbitrary direction. However, the configuration shown in FIG. 3 is an example, and can be configured by combining various known joint mechanisms.

FIG. 4 is a schematic perspective view showing a schematic configuration of the shelf 2 according to the first embodiment of the present invention. The shelf 2 is composed of one shelf board 21 and one shelf top board 22 supported by four columns 20. A plurality of picking objects 3 are placed on the shelf board 21.

Here, a work space is defined for the mobile manipulator 1 according to the first embodiment. The work space is a predetermined space in which one or more objects 3 to be picked by the moving manipulator 1 are placed. In FIG. 4, the work space of the moving manipulator 1 is defined as a three-dimensional space having a rectangular parallelepiped shape that is surrounded by columns 20 from the upper surface of the shelf plate 21 to the lower surface of the shelf top plate 22.

In the work space according to the first embodiment, the plane mirror 4 is arranged at a predetermined position. In FIG. 4, the plane mirror 4 is fixed to a predetermined position of the shelf top plate 22 at a predetermined angle with its reflection surface directed toward the shelf plate 21. However, the existence of the object 3 and the plane mirror 4 is not strictly limited in the work space. A part of the object 3 or the plane mirror 4 protrudes from the work space, or the entire object 3 or the plane mirror 4 works. It does not prevent it from being placed in the vicinity of the space.

FIG. 5 is a drawing schematically showing a configuration in which the reflecting surface of the plane mirror 4 is viewed from the front. The plane mirror 4 can be made of glass, metal, plastic, or the like, and reflects at least the wavelength of the region that can be imaged by the camera 14. In FIG. 5, markers 5 having a predetermined shape are arranged at the four corners of the reflecting surface of the plane mirror 4. In FIG. 5, all the markers 5 have the same shape. The shape of the marker 5 may be any shape that can detect the presence / absence and the number of captured images from an image captured by the camera 14 by predetermined image processing. Furthermore, in the said image process, what is necessary is just the shape which can calculate the center coordinate of each marker 5 in the said image. The marker 5 can be formed by adding protrusions on the surface or inside of the plane mirror, printing, marking, applying a seal, or the like. The positional relationship between the markers 5 arranged on the plane mirror 4 is stored in advance in the control unit 10 or the upper control unit 6 (described later).

FIG. 6 is a diagram showing an example of the configuration of the control system for the mobile manipulator in the first embodiment. The control unit 10 of the mobile manipulator 1 includes an overall control unit 100 that controls the entire manipulator 1, a data storage unit 101 that stores data necessary for control, a mobile cart control unit 102 that controls the mobile cart unit 11, an arm An arm control unit 103 that controls the unit 12, a gripper control unit 104 that controls the gripper unit 13, and an image processing unit 105 that processes an image obtained by the camera 14 are provided. The control unit 10 of the mobile manipulator 1 is configured to be able to communicate with the host control unit 6 by wire or wireless.

The host control unit 6 includes an object data storage unit 60 that stores information such as map data 601, shelf shape data 602, and object data 603. The host control unit 6 may be built in the housing of the mobile manipulator 1 or may be outside the mobile manipulator 1. The upper control unit 6 and the control unit 10 need only be able to perform information processing in cooperation and control the mobile manipulator 1, and may be configured by a single computer, or may include an input device, an output device, a processing device, Any part of the storage device may be composed of a plurality of computers connected by a wired or wireless network.

Here, the map data 601 is data describing a point where the shelf 2 is arranged in a space where the mobile carriage unit 11 of the mobile manipulator 1 can travel (hereinafter referred to as travel space). When a plurality of shelves 2 are arranged in the traveling space, it is assumed that an individual identification code is assigned to each shelf 2 and the identification codes and points are stored in pairs. The shelf shape data 602 is data in which information such as the width, depth, and height of the shelf board 22 is described with reference to a predetermined part of the shelf 2. The object data 603 stores information such as the type or identification number of the object 3 to be picked and the identification code of the shelf 22 on which the object 3 is arranged. The object data 603 may further include characteristic data such as the shape, weight, and material of the object 3.

That is, by combining the map data 601, the shelf shape data 602, and the object data 603, the upper control unit 6 allows the work space where the object 3 to be picked is displayed to be present in the traveling space. The picking command can be transmitted to the control unit 10 of the movement manipulator 1 based on this.

On the other hand, in the work space, the posture (position and orientation) of the target object 3 that is the target of the picking operation. For example, the position is the position of the center of gravity of the bin that is the target. Etc.) cannot be known by the host controller 6. This is because the object 3 to be displayed is picked by the moving manipulator 1 or picked by a human worker in the traveling space, so that the posture may be changed at any time.

Therefore, the moving manipulator 1 needs to pick the object 3 after confirming the posture of the object 3 in the work space.

(Operation)
Next, the procedure of the picking operation of the object 3 by the moving manipulator in the first embodiment will be described with reference to FIGS.

FIG. 7 is a diagram showing an overall outline of the picking operation procedure in the first embodiment. Each step in FIG. 7 may be executed by a predetermined program built in the control unit 10 of the movement manipulator 1 or may be executed by the host control unit 6 to control the control unit 10 of the movement manipulator 1 one by one or collectively. You may make it transmit a command to.

7 is started when, for example, the operator inputs an instruction to identify and pick the target object 3 to the host control unit 6 (START).

In step S701, the moving manipulator 1 travels to the front of the shelf 2 on which the object 3 to be picked is displayed. As described above, the upper control unit 6 stores the point of the shelf 2 where the object 3 to be picked is displayed, that is, the point of the work space, in the traveling space of the mobile manipulator 1. The host control unit 6 transmits a picking command including information indicating the point of the work space to the control unit 10 of the mobile manipulator 1. Then, the mobile manipulator 1 travels by controlling the mobile carriage unit 11 to the point of the work space specified by the picking command. For this traveling, for example, an existing technique such as embedding a magnetic cable indicating a traveling route in the traveling space in advance and moving along the magnetic cable may be used. Alternatively, an existing technology that moves autonomously using the map data 601 may be used.

In step S702, the moving manipulator 1 travels to the front of the shelf 2, and then changes the posture of the gripper unit 13 to move the camera 14 from the front (opening) of the shelf so as to face the work space, Take a picture. The center of the moving manipulator 1 (the center can be arbitrarily determined by the point information of the work space and the shelf shape data 602 stored in advance in the upper control unit 6. In order to simplify the description, the moving manipulator 1 will be described below. The position of the shelf 21 in the work space with respect to the origin) is known. Therefore, the moving manipulator 1 can control the arm unit 12 and the gripper unit 13 so that the camera 14 is positioned in a predetermined posture (position and orientation) relatively determined from the position of the shelf board 21. For example, the center of the moving manipulator 1 is positioned at a position away from the shelf 2 by a predetermined distance, and the arm 12 and the gripper 13 are positioned in a posture set according to the posture of the shelf board 21.

FIG. 8 is a diagram schematically showing a state viewed from the side of the work space in step S702.

In FIG. 8, on the shelf board 21, in addition to the object 3a that is the object of the picking operation, the object 3b that is not the object of the picking operation is positioned behind the object 3a as viewed from the gripper unit 13. is doing. Therefore, when picking the object 3a by the gripper part 13, it is necessary to control the finger mechanism 130 of the gripper part 13 so as not to interfere with the object 3b.

FIG. 9 is a diagram showing an example of a photographed image 141 photographed by the camera 14 in step S702.

As shown in FIG. 9, in the photographed image 141, the image of the object 3b is photographed so as to be behind the object 3a. It is generally difficult to accurately specify the posture of the object 3b from such an image that is behind the object and partially missing. Further, when the object 3b is completely hidden by the object 3a as viewed from the gripper unit 13, the presence of the object 3b cannot be detected from the state of FIG.

Returning to FIG. 7, the description will be continued.

In step S703, detection processing of the markers 5 at the four corners of the plane mirror 4 is performed on the captured image 141 (FIG. 9). Here, the shape and number of the markers 5 are stored in advance as a part of the shelf shape data 602, for example. Since the shape and number of the markers 5 are known, the number of markers 5 photographed in the photographed image 141 and the pixel coordinate values in the photographed image 141 can be specified by a known template matching process. Although the marker 5 may be omitted and the corners of the plane mirror 4 may be extracted instead of the marker by image processing or the like, the burden of image processing can be reduced by preparing and using the marker. The image processing unit 105 performs image processing and calculation after step S703. Further, the image processing unit 105, the overall control unit 100, and the host control unit 6 may share the processing.

In step S703, when the four markers 5 are not detected from the image, the posture of the gripper unit 13 is changed (step S704). For example, the control operation such as changing the posture of the gripper unit 13 by rotating the joint mechanism 121 of the arm unit 12 in the direction of the arrow P in FIG. Then, after changing the posture, the work space is imaged again by the camera 14 (step S702), and the marker 5 detection process (step S703) is performed.

The processing from step S702 to step S704 is repeated until a known number (for example, four) of markers are detected from the captured image of the camera 14. In step S701, it is desirable that the positional relationship between the shelf 2 and the moving manipulator 1 is set to an appropriate positional relationship so that all the markers are within the angle of view of the camera 14. In this case, the marker should be detected once or a predetermined number of trials. However, if the marker cannot be detected within a predetermined number of times, the operator can be notified by issuing an error signal or a warning.

FIG. 10 is a diagram schematically showing the working space viewed from the side when four markers 5 are detected from the image captured by the camera 14. The mirror 4 is within the angle of view of the camera 14.

FIG. 11 is a drawing showing an example of a photographed image of the camera 14 in FIG.

In step S705, the attitude of the plane mirror 4 in the work space is calculated based on the image captured by the camera 14 (FIG. 11). An example of the calculation method is shown in FIG.

12, the origin of the moving manipulator 1 is O, the viewpoint of the camera 14 is C, and the center point of the plane mirror 4 is M. In order to simplify the explanation, in FIG. 12, the plane mirror 4 is illustrated as a rectangle having no thickness, and the position of the marker 5 is the position of each vertex of the rectangle.

In FIG. 12, the posture of the viewpoint C of the camera 14 with respect to the origin O of the moving manipulator 1 (information indicating the position and tilt of the camera. For example, the position of the center point of the camera lens and the direction in which the optical axis of the camera faces Can be uniquely calculated by the angle of each joint of the arm unit 12 and the length of the arm connecting the joints (not shown in FIG. 12). Information relating to the moving manipulator 1 such as the angle of each joint of the arm unit 12 and the length of the arm connecting the joints is stored in advance in, for example, the data storage unit 101 of the control unit 10. The homogeneous transformation matrix indicating the posture of the viewpoint C of the camera 14 with the origin O of the moving manipulator 1 as a reference is calculated as ^O T _C. In FIG. 12, the captured image (FIG. 11) captured at the viewpoint C of the camera 14 is shown as a captured image plane P.

On the other hand, the mutual positional relationship between the markers 5 of the plane mirror 4, that is, the lengths H and W of each side of the plane mirror 4, for example, is stored as a part of the shelf shape data 602 and is known. At this time, the intersections of the four straight lines connecting the marker 5 and the viewpoint C of the camera 14 and the captured image plane P of the camera 14 become the coordinates of the marker 5 detected in step S703. By estimating the homography matrix from the coordinates of the marker 5 on the captured image plane P of the camera 14 and the lengths W and H of each side of the mirror 4, the center point of the plane mirror 4 with respect to the viewpoint C of the camera 14 is estimated. The posture of M (information indicating the position and tilt of the mirror. For example, the position of the center point of the mirror and the direction in which the mirror surface faces) can be calculated. The calculated homogeneous transformation matrix indicating the attitude of the center point M of the plane mirror 4 with respect to the viewpoint C of the camera 14 is defined as ^C T _M.

Then, by integrating the calculated two homogeneous transformation matrices ^O T _C and ^C T _m , the homogeneous transformation matrix ^O indicating the attitude of the center point of the plane mirror 4 with respect to the origin O of the moving manipulator 1 is used. T _M can be calculated. The homogeneous transformation matrix ^O T _M corresponds to the attitude of the plane mirror 4 in the work space (real space).

In this manner, the attitude of the plane mirror 4 in the work space was derived from the image of the plane mirror 4 captured by the camera 14. At this time, if the posture of the plane mirror 4 in the work space is calculated by the markers 5 arranged at the four corners of the plane mirror 4, an accurate posture can be calculated. The calculated attitude of the plane mirror 4 is stored in the control unit 10 or the host control unit 6. (Step S706).

In step S707, the work space is imaged again using the camera 14. Here, the posture is the same as that in step S702, but may be any other posture as long as the work space is photographed. The photographed image is assumed to be an image 141 similar to that in FIG. The image 141 is roughly divided into two image processes, a process in step S708 and a process from step 709 to step 721, in parallel or in series.

In step S708, the object 3a which is the picking target is detected from the image 141 photographed in step S707, and the posture thereof is calculated. For the detection of the object 3a, a known image recognition algorithm such as pattern matching with an image of the object 3 stored in advance (for example, stored as part of the object data 603) may be applied. . The detected posture of the object may be calculated by storing the size of the object in advance and comparing it with the size reflected in the captured image 141 or calculating the homography. . These image processes can be calculated, for example, with coordinates with the viewpoint C of the camera 14 as the origin, but eventually the coordinates are converted to coordinates based on the origin O of the moving manipulator 1 to unify the coordinate system. Also good. At the same time, the postures of the other objects 3b and the gripper unit 13 may be calculated together.

In step S709, for the viewpoint C of the camera 14, a point C ′ that is a surface object with respect to the reflecting surface M of the plane mirror 4 is calculated. In the following description, the viewpoint C ′ is referred to as a mirror image viewpoint C ′ of the camera 14.

FIG. 13 is a side view illustrating a mirror image viewpoint C ′ with respect to the plane mirror of the camera. As can be understood from FIG. 13, with respect to the viewpoint C of the camera 14, the point orientation C ′ that is a surface object with respect to the reflection surface M of the plane mirror 4 becomes the mirror image viewpoint C ′ of the camera 14. That is, if the posture of the camera 14 and the posture of the mirror 4 are known, the posture of the mirror image viewpoint C ′ of the camera 14 can be calculated. The object reflected on the mirror 4 captured by the camera 14 corresponds to the image of the object viewed from the mirror image viewpoint C ′ of the camera 14. Here, the posture of the mirror 4 is stored in step 706, and can be expressed by, for example, coordinates based on the origin O of the moving manipulator 1. Further, the posture of the camera 14 can be calculated based on the same coordinates from the data regarding the moving manipulator 1 stored in the data storage unit 101 as described above.

In step S710, a region of the plane mirror 4 in the photographed image 141, that is, a mirror region image is cut out. An example of the mirror region image cutout process will be described with reference to FIG. 12 again.

First, the homogeneous transformation matrix ^O T _M indicating the attitude of the plane mirror 4 with respect to the origin O of the moving manipulator 1 is stored in step S706 (here, the moving manipulator 1 does not move after step S701). To do). Further, the homogeneous transformation matrix ^O T _C indicating the posture of the viewpoint C of the camera 14 with respect to the origin O of the moving manipulator 1 can be calculated from the angle of each joint of the arm unit 12 and the length of the arm connecting the joints. . From these homogeneous transformation matrices ^O T _M and ^O T _C , a homogeneous transformation matrix ^C T _M indicating the attitude of the plane mirror 4 with reference to the viewpoint C of the camera 14 in step 710 (FIG. 13) can be calculated.

Here, when the length of each side of the plane mirror 4, that is, H and W are known, a homography matrix is calculated from ^C T _M , and the image m of the plane mirror 4 on the imaging plane p of the camera 14 can be calculated. An image m of the plane mirror 4 on the shooting plane p of the camera 14 becomes a mirror region image. Therefore, by cutting out only the area of the image m from the captured image 141, the mirror area image of the plane mirror 4 can be cut out. Note that the method of cutting out the image m is not limited to the above, and for example, markers at four corners of the mirror 4 may be detected, and a rectangular region surrounded by the markers may be used as the image m.

The mirror region image of the plane mirror 4 cut out in this way is an image of the visual field indicated by V ′ from the mirror image viewpoint C ′ of the camera 14 in FIG. As is apparent from FIG. 13, the object 3b is captured in the mirror region image without being shaded by the object 3a. That is, an object that has not been detected as a blind spot in step S708 is captured in the mirror region image.

In step S711, the object 3b is detected by image processing similar to that in step S708 on the mirror region image cut out in step S710. That is, the object 3b that cannot be seen from the viewpoint C of the camera 14 is detected from the mirror region image, and its posture is calculated by pattern matching or homography calculation. As shown in FIG. 13, the image in the mirror region image is an image of the object 3 b viewed from the mirror image viewpoint C ′ of the camera 14.

When image processing similar to that in step S708 is performed using the mirror image viewpoint C 'as a reference and coordinates with this as the origin, the object 3b is calculated as the posture of the camera 14 viewed from the mirror image viewpoint C'. The calculation result can be indicated by coordinates with the mirror image viewpoint C ′ as the origin. At the same time, the postures of the other objects 3a and the gripper unit 13 may be calculated together. Finally, the coordinate system may be unified by converting the coordinates into the coordinates based on the origin O of the moving manipulator 1.

In step S713, the posture information of the target object 3a calculated in step S708 and the posture information of the target object 3b obtained in step S711 are integrated. Thereby, the control part 10 becomes possible [grasping | ascertaining the relative positional relationship of the target object 3a in the work space, the target object 3b, and the gripper 13 etc. as needed.

As described above, if each posture is converted into coordinates based on the origin O of the moving manipulator 1 and the coordinate system is unified, the positional relationship of objects in the work space can be grasped in a unified manner. In addition, when there are feature points that are visible from both the viewpoint C and the mirror image viewpoint C ′ of the camera 14, the three-dimensional posture of the object can be calculated by stereo processing.

Based on the integrated posture information of each object, control is performed so that the gripper unit 13 is moved to a posture in which the object 3a can be gripped (step S714), and the object is gripped by the gripper unit 13 (step S714). S715).

(effect)
According to the present embodiment described above, the moving manipulator 1 becomes a blind spot in the work space, in addition to the posture of the target object 3a that is the target of the picking operation, as a shadow of the target object 3a from the shelf opening. It is possible to detect the posture of the object 3b located at the position. Therefore, it is possible to pick only the object 3a without the object 3b and the gripper 13 colliding with each other.

Further, in the case of the method of storing the posture of the plane mirror 4 arranged in the work space in advance, if the moving manipulator 1 stops at a slight inclination with respect to the shelf 2, it is detected in step S708. The posture of the object 3b viewed from the mirror image viewpoint C ′ is shifted by twice the inclination angle of the mobile manipulator 1 with respect to the shelf 2. Further, the mirror region in step S710 is also cut out from the region shifted by the tilt angle. Due to these effects, the posture of the object 3b cannot be calculated correctly. On the other hand, according to the present embodiment, by providing step S705 for calculating the attitude of the plane mirror 4 with the marker 5 or the like, the shift of the stop position accompanying travel such that the moving manipulator 1 is tilted with respect to the shelf 2 and stopped. Regardless, the calculation of the mirror image viewpoint C ′ of the camera 14 by the plane mirror 4 and the process of cutting out the mirror area image can be performed with high accuracy.

(Modification)
Note that the picking of the object 3a by the gripper portion 13 of the moving manipulator 1 according to the present embodiment does not have to be performed according to the procedure shown in FIG. 7, and can be appropriately changed without departing from the scope of the invention. In FIG. 7, the work space is photographed only once in step S <b> 707 after the attitude of the plane mirror 4 is calculated in step S <b> 705 until the picking operation is completed.

For example, a step of photographing again in a state where the gripper unit 13 is sufficiently close to the object 3a may be provided.

As shown in FIG. 14, when the gripper unit 13 is sufficiently close to the object 3a, only a small part of the object 3a can be photographed from the camera 14. On the other hand, from the mirror image viewpoint C ″ of the camera 14 in the state of FIG. 14, it is possible to photograph three objects: the object 3a, the object 3b, and the finger mechanism 130 of the gripper unit 13. Accordingly, the object 3a and Not only the positional relationship between the target 3b but also the positional relationship between the target 3a and the gripper unit 13 and the positional relationship between the target 3b and the gripper unit 13 can be detected from the mirror region image.

In this case, step S708 to step S711 need only be performed without performing step S708. Thereby, each positional relationship of the target object 3a, the target object 3b, and the gripper part 13 can be grasped | ascertained.

That is, in the operation procedure according to the present embodiment, the procedure for calculating the attitude of the plane mirror 4 by the marker 5 in step S705 may be performed before the procedure in steps S709 to S711. Step S714 for controlling the movement of the gripper unit 13 may be performed based on the result of the procedure of S711.

Further, the camera 14 is not necessarily fixed to the gripper unit 13. An example is shown in FIG.

As shown in FIG. 15, a configuration in which a joint mechanism 131 is further provided between the camera 14 and the gripper unit 13 and the angle T of the camera 14 with respect to the gripper unit 13 can be arbitrarily changed may be employed. If the angle T of the camera 14 with respect to the gripper unit 13 can be observed by the control unit 10, the attitude of the plane mirror 4 from the origin O of the moving manipulator 1 shown in FIG. 12 can be calculated. This eliminates the need to control the arm unit 12 in the search for the marker 5 in step S704.

In step S703, it has been described that the markers 5 at the four corners of the plane mirror 4 are detected, but it is not always necessary to detect the four markers 5. An example is shown in FIG.

As shown in FIG. 16, a method using two types of

markers

5 and 51 arranged in different shapes on the diagonal of the plane mirror 4 may be used. In this case, if at least three markers are detected, The attitude of the plane mirror 4 can be calculated. As a result, the number of repeated processes from step S702 to step S704 can be reduced.

As shown in FIG. 17, if only one checker pattern marker 52 composed of a predetermined square having a known size is used, the attitude of the plane mirror 4 can be calculated by predetermined image processing. This is because each square vertex in the checker pattern plays the same role as the marker 5.

In the present embodiment, the procedure using the plane mirror 4 has been described. However, it is also possible to use a mirror having a shape other than the plane mirror.

A convex mirror 41 having a known shape may be used as shown in FIG. The convex mirror 41 in FIG. 18 has a checker pattern marker 52 attached to the center thereof. In the case where the convex boundary 41 is used, the mirror region image of the convex mirror 41 is converted into a planar image by performing projective transformation determined by the shape of the convex boundary 41 after cutting out the mirror region image from the image in step S710 in FIG. Convert it. By using the convex mirror 41, the number of calculation steps for calculation increases compared to the case of using the plane mirror 4, but it is possible to observe a wider space from the mirror viewpoint of the camera 14.

In the above description, an example in which only one plane mirror 4 is arranged in the work space has been described. Of course, a plurality of plane mirrors 4 in the work space may be arranged. For example, as shown in FIG. 19, in addition to the plane mirror 4a in which the markers 5 are arranged at the four corners in the work space, the plane mirror 4b in which the markers 52 are arranged at the four corners, and the plane mirror 4c in which the markers 53 are arranged at the four corners. Further, the configuration may be further arranged. Here, the marker 5, the marker 52, and the marker 53 are different in shape from each other as long as it can be determined and detected by predetermined image processing. It goes without saying that the operation procedure shown in FIG. 7 can be applied to each of the plane mirrors 4a to 4c.

In the work space shown in FIG. 19, the plane mirrors 4a to 4c may be used in combination. For example, when the plane mirror 4a is used in combination with the plane mirror 4a, it may be realized by a recursive process of further cutting out the mirror area image of the plane mirror 4b from the mirror area image of the plane mirror 4a cut out in step S710. In this way, the posture detection of an object with no blind spots can be performed by complex posture calculation using a plurality of plane mirrors.

In Example 1, although the example which controls the arm part 12 and the gripper part 13 of the movement manipulator 1 using the plane mirror 4 which provided the marker 5 in four corners was demonstrated, the plane mirror 4 provided with the marker 5 is used for all working spaces. Need to be placed in. Therefore, in the present embodiment, an example in which the marker 5 is not arranged on the plane mirror 4 and the marker 5 is arranged at a place other than the plane mirror 4 will be described.

FIG. 20 is a diagram showing a configuration of the shelf 2 according to the present embodiment, and is a diagram contrasted with FIG. 4 in the first example. FIG. 20 schematically shows a configuration in which the shelf 2 is viewed from the front in the left direction of FIG.

20 differs from the shelf 2 shown in FIG. 4 in that the marker 5 is provided on a part of the shelf board 21 and the shelf top board 22. Further, the plane mirror 4 is fixed at a predetermined angle at a predetermined position of the shelf top plate 22 of the shelf 2 shown in FIG. 20 with its reflecting surface directed toward the shelf plate 21. The flat mirror 4 according to the present embodiment is different from the flat mirror 4 shown in FIG. 5 in that the markers 5 are not provided at the four corners.

In the present embodiment, the shelf shape data 602 stored in the object data storage unit 60 of the host control unit 6 includes the shape of the shelf 2 and the attitude of the plane mirror 4 based on a predetermined part of the shelf 2. Are stored. That is, in this embodiment, the relative positional relationship among the shelf 2, the marker 5, and the plane mirror 4 is known in advance.

FIG. 21 is a diagram illustrating a picking procedure of the object 3 by the moving manipulator 1 according to the present embodiment, and is compared with FIG. 7 in the first embodiment.

21 is different from the operation procedure shown in FIG. 7 in the operation procedure shown in FIG.

First, after the mobile manipulator 1 travels to the front of the shelf 2 in step S701, the entire shelf opening is photographed by the camera 14 in step S7021. Thereby, it can be searched whether the four markers 5 stuck on the shelf 21 and the shelf 22 shown in FIG. 20 are reflected in the captured image of the camera 14 (step S7031). If four markers 5 are not detected in step S7031, the direction of the camera 14 is changed (step S704), and photographing is performed again (step S7021).

Next, the attitude of the shelf 2 is calculated using the four markers 5 detected from the captured image of the camera 14 and the shelf shape data 602 (step S7051).

Then, the attitude of the plane mirror 4 fixed to the shelf 2 is calculated from the shelf shape data 602 based on the calculated attitude of the shelf 2 (step S7052), and the attitude is transferred to the control unit 10 or the upper control unit 6. Store (step S706).

As described above, in the present embodiment, the shape of the support object of the plane mirror 4, the posture of the plane mirror 4 with respect to the support object, and the position of the marker 5 attached to the support object, The posture is calculated, and the posture of the plane mirror 4 is indirectly calculated from the posture of the supporting object.

In the present embodiment, the fixed position of the plane mirror 4 and the marker 5 is not limited to the shelf 2.

FIG. 22 is a perspective view showing a schematic configuration of the tray 200 for storing the object 3 according to the present embodiment. A two-dimensional code 53 is affixed to the outside of the front surface of the tray 200, and the plane mirror 4 is fixed to the inside of the back surface of the tray 200. Information such as the contents of the tray 200 can be embedded in the two-dimensional code 53 using the pattern shape as an identification code.

As with the checker pattern marker 52 in the first embodiment, if the size of the two-dimensional code 53 is known, the posture can be calculated from the image by one two-dimensional code 53.

The shape data of the tray 200 including the two-dimensional code 53 and the fixed posture of the plane mirror 4 is stored in the object data storage unit 60, thereby calculating the posture of the plane mirror 4 from the posture of the shelf 2 according to the present embodiment. Similar to the procedure, the attitude of the plane mirror 4 fixed to the back inner side of the tray 200 can be calculated from the two-dimensional code 53 attached to the outer front surface of the tray 200. Then, by using the calculated posture of the plane mirror 4 fixed inside the back surface of the tray 200, the object 3 in the tray 200 can be gripped by the gripper unit 13 so as not to collide with the wall surface of the tray 200. Needless to say.

In this manner, in the operation procedure of the moving manipulator 1 according to the present embodiment, the marker 5 can be attached to a place other than the plane mirror 4 and can be easily photographed with respect to the camera 14 fixed to the gripper unit 13. Can be attached to a place.

In the above-described embodiment, the example in which the marker 5 is arranged on the plane mirror 4 and the shelf 2 arranged in the work space has been described. In the third embodiment, a configuration in which the marker 5 is arranged on a part of the moving manipulator 1 will be described.

FIG. 23 is a perspective view schematically showing the configuration of the gripper portion 13 of the moving manipulator 1 according to the present embodiment. The gripper portion 13 according to the present embodiment is different from the first embodiment in that a checker pattern marker 52 is further provided at the center of the two finger mechanisms 130 of the gripper portion 13.

FIG. 24 is a drawing schematically showing a method for calculating the attitude of the plane mirror 4 in the moving manipulator 1 according to the present embodiment. In FIG. 24, the field of view V of the camera 14 fixed to the gripper unit 13 is a mirror image reflected on the plane mirror 4. Here, a point indicated by the posture of the checker pattern marker 52 detected in the mirror image of the plane mirror 4 is denoted by m1. Then, the plane on which the reflecting surface of the plane mirror 4 exists is a vertical bisector between the posture of the checker pattern marker 52 calculated from the joint angle of the arm portion 12 and the length of the arm between the joints, and m1. It can be calculated. Information on the attachment position of the marker 52 to the manipulator is stored as data in the data storage unit 105 or the like. Therefore, the posture of the marker 52 can be uniquely calculated based on data such as the angle of each joint of the arm unit 12 and the length of the arm connecting the joints, similarly to the posture of the camera 14.

FIG. 25 is a perspective view of the work space in the present embodiment. In the present embodiment, the plane on which the reflecting surface of the plane mirror 4 exists can be calculated, but the position of the mirror area of the plane mirror 4 cannot be calculated. However, by using a plane mirror 4 that is sufficiently larger than the work space as in the shelf 2 shown in FIG. 25, the field of view of the camera 14 that is fixed to the gripper unit 13 can be entirely as long as the gripper unit 13 is in the work space. The plane mirror 4 can be arranged so as to cover it. Such a plane mirror 4 is disposed, and control is performed by regarding all the captured images of the camera 14 as mirror region images.

Thus, in this embodiment, it is possible to control the movement of the gripper unit 13 using the mirror image viewpoint without disposing the marker 5 on the plane mirror 4 or the shelf 2.

Other modifications will be described. This modification is basically a modification of the embodiment described with reference to FIGS. 7 and 21 and is another example of the obstacle posture detection process.

FIG. 26 shows only a portion between steps S707 and S713 in FIGS. 7 and 21, which is different from FIGS. In this example, the mirror image viewpoint is not calculated, and the posture in the work space is derived from the posture in the mirror image space of the obstacle reflected in the mirror region in the image.

Figure 27 shows a conceptual diagram. When viewed from the viewpoint C of the camera 14, the object 3b is reflected in the mirror 4 as 3b '. Therefore, as in the embodiments of FIGS. 7 and 21, a mirror region in the image is cut out (step S710), and the virtual image 3b 'of the object 3b is detected by image processing similar to step S708. That is, the object 3b cannot be seen from the viewpoint C of the camera 14, but the virtual image 3b 'is detected from the mirror region image, and the posture thereof is calculated by pattern matching or homography calculation (step S711). As shown in FIG. 27, the image in the mirror region image is an image obtained by viewing the virtual image 3b 'of the object 3b from the viewpoint C of the camera 14.

The posture of the object 3b with respect to the virtual image 3b 'is obtained by performing mirror image conversion of the posture of the virtual image 3b' with respect to the mirror (step S712). At the same time, the postures of the other objects 3a and the gripper unit 13 may be calculated together. Further, the calculation result may be finally converted into coordinates based on the origin O of the moving manipulator 1 to unify the coordinate system.

Although specific embodiments of the present invention have been described in detail above, these are examples of implementations presented for a clear understanding of the disclosed invention. Accordingly, it goes without saying that various aspects, modifications, and combinations can be further formed in the above-described embodiment without departing from the scope of the claims. For example, a part of the configuration of one embodiment can be replaced with the configuration of another embodiment, and the configuration of another embodiment can be added to the configuration of one embodiment. Further, it is possible to add, delete, and replace the configurations of other embodiments with respect to a part of the configurations of the embodiments. Any component expressed in the singular herein shall include the plural unless the context clearly dictates otherwise.

Although the mobile manipulator 1 having the mobile carriage unit 11 has been described in the above embodiment, the present invention is not limited to this. For example, it goes without saying that the present invention can also be applied to a stationary manipulator in which the arm portion is moved by a linear motion mechanism fixed at a predetermined location.

It can be used to control various mobile manipulators.

DESCRIPTION OF SYMBOLS 1 Mobile manipulator 10 Control part 11 Mobile trolley part 110 Wheel 12 Arm part 121,122 Joint mechanism 13 Gripper part 131 Finger mechanism 14 Camera 2 Shelf 20 Strut 21 Shelf board 22 Shelf top board 200

Tray

3, 3a, 3b Object 4 Plane mirror 41 Convex mirrors 5, 51 Marker 52 Checker pattern marker 53 Two-dimensional code 6 Host control unit 60 Object data storage unit

Claims

A manipulator control method for operating a movable manipulator in a work space where an object is arranged,
The mirror placed in the work space or in the vicinity thereof is imaged with a camera mounted on the manipulator,
Detecting the orientation of the mirror based on the orientation of the camera and the image captured by the camera;
Detecting the mirror area from the image captured by the camera,
Detecting an image of at least a portion of the object from the area of the detected mirror;
Based on the attitude of the mirror and the image of the object, calculate the attitude of the object in the work space;
A manipulator control method for operating the manipulator based on the calculated posture information of the object.
When calculating the orientation of the object in the work space based on the orientation of the mirror and the image of the object,
Based on the camera posture and the mirror posture, the mirror image viewpoint of the camera is calculated,
The manipulator control method according to claim 1, wherein an attitude of the object in the work space is calculated based on a mirror image viewpoint of the camera and an image of the object.
When calculating the orientation of the object in the work space based on the orientation of the mirror and the image of the object,
Based on the posture of the camera and the image of the object, calculate the posture of the virtual image of the object in a mirror image space;
The manipulator control method according to claim 1, wherein the attitude of the object in the work space is calculated based on the attitude of the virtual image of the object and the attitude of the mirror.
The object is placed on a shelf;
Based on the shelf arrangement information, the manipulator is moved to the vicinity of a predetermined shelf,
The manipulator control method according to claim 1, wherein the attitudes of the camera, the mirror, and the object are calculated based on an origin fixed to the manipulator.
Detecting an image of at least a portion of the manipulator from the detected mirror region;
Based on the attitude of the mirror and at least a part of the image of the manipulator, calculating the attitude of at least a part of the manipulator in the work space;
The manipulator control method according to claim 1, wherein the manipulator is operated based on the calculated posture information of at least a part of the manipulator.
Detecting at least a portion of the object from a region other than the detected mirror;
Calculating an attitude of the object in the work space based on an image of at least a part of the object;
The manipulator control method according to claim 1, wherein the manipulator is further operated based on the calculated posture information of the object.
Detecting at least a portion of the manipulator from a region other than the detected mirror;
Calculating an attitude of at least a part of the manipulator in the work space based on an image of at least a part of the manipulator;
The manipulator control method according to claim 1, wherein the manipulator is operated based on the calculated posture information of at least a part of the manipulator.
Place a marker in the mirror or workspace,
The manipulator control method according to claim 1, wherein an attitude of the mirror is detected based on information on the marker in an image captured by the camera.
A manipulator control system for operating a movable manipulator in a work space where an object is arranged,
The manipulator includes a moving mechanism for moving the manipulator, an operating mechanism for operating the object, a camera for imaging the work space, a camera control mechanism for changing the posture of the camera, and a control unit. And
The controller is
A function of detecting the posture of the camera;
A function of detecting the orientation of the mirror in the captured image based on the orientation of the camera and the image captured by the camera;
A function of detecting an area of the mirror from an image captured by the camera and detecting an image of at least a part of the object from the detected area of the mirror;
A function of calculating the posture of the object in the work space based on the posture of the mirror and the image of the object;
A manipulator control system for operating the operation mechanism based on the calculated posture information of the object.
The operation mechanism includes an arm unit connected by at least one joint mechanism, and a gripper unit held by the arm unit and capable of grasping and moving the object.
The camera is disposed in the gripper unit,
The control unit controls the operation of the operation mechanism, stores geometric data of the shape of the operation mechanism, and detects the posture of the camera using the geometric data. The manipulator control system described.
The work space in which the object is arranged is a space defined with reference to a shelf on which the object is arranged,
The manipulator control system according to claim 9, wherein the mirror is disposed on the shelf.
The manipulator is configured to be movable in the vicinity of an arbitrary shelf among the plurality of shelves,
The control unit stores shelf arrangement information relating to the arrangement of the plurality of shelves and object information relating to objects arranged on each shelf,
12. The manipulator control system according to claim 11, wherein the control unit moves the manipulator to a vicinity of a predetermined shelf based on the shelf arrangement information and the object information, and images the work space with the camera.
A marker is arranged in the work space, and a relative positional relationship between the marker and the mirror is stored in the control unit, and the inside of the captured image is based on the marker in the captured image by the camera. The manipulator control system according to claim 12, wherein the control system detects a posture of the mirror.
A manipulator that operates in a work space in which an object is arranged,
A moving mechanism for moving the manipulator;
A gripper for operating the object;
An arm part for moving the gripper part;
A camera for imaging the work space;
A camera control mechanism for changing the posture of the camera;
Having a control unit,
The controller is
A function of detecting the posture of the camera;
A function of detecting the orientation of the mirror in the captured image based on the orientation of the camera and the image captured by the camera;
A function of detecting an image of at least a part of the object from an image captured by the camera;
A function of calculating the orientation of the object in the work space based on the orientation of the mirror and the image of the object;
A manipulator that operates the gripper unit based on the calculated posture information of the object.
Having a marker disposed on the gripper portion;
In the function of detecting the attitude of the mirror,
The manipulator according to claim 14, wherein the posture of the mirror in the captured image is detected using information on the posture of the marker and an image of the marker in the image captured by the camera.