JP2013158846A - Robot device, image generation device, image generation method, and image generation program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To control conveyance of a body considering acceleration/deceleration.SOLUTION: A robot device includes a teaching image generation part (31) that is an image generation device which generates a target image to be a target when conveying a body to a target point by a robot using visual servo control, and includes: a teaching image obtaining part (311) which obtains a teaching image from an image obtained by capturing a range that includes both the target point and the body; an image information obtaining part (312) which obtains information related to the position and posture of the body from the image of the body included in the teaching image; an image information interpolation part (313) which interpolates the information obtained from the image information obtaining part; and a rendering part (314) which renders the target image corresponding to desired conveyance speed using the interpolated information.

Description

  The present invention relates to a robot apparatus, an image generation apparatus, an image generation method, and an image generation program.

  In recent years, SCARA robots, multi-axis robots, and the like have been introduced in electronic equipment production lines for product assembly operations and inspections. For example, in an assembly line, a robot supports an assembly operation of an electronic device by grasping an object such as a single part or an assembly part and transporting it to a predetermined target point.

  As this type of robot control technology, a control technology called visual servo control or visual feedback control is known (see, for example, Patent Document 1). In this visual servo control, a change in relative position between the object to be transported and the target point is measured as visual information using a video camera device, and the measurement result is used as feedback information. It is a kind of servo system that controls the relative position of.

  According to a robot using visual servo control, when an object held by the robot hand is transported to a target point, the target image representing the object at the target point and the object at the local point obtained by the video camera device are displayed. The frame image is compared, and the object is conveyed by controlling the position and orientation of the object so that the frame image matches the target image (see, for example, Patent Document 2). In this case, a plurality of relay points are designated in order to avoid obstacles on the transport route, and a plurality of teaching images for teaching the position and orientation of the object at each relay point are prepared in advance. By using the plurality of teaching images as the target image, the robot conveys the object to the final target point while following the path avoiding the obstacle.

JP 2002-208209 A Japanese Patent Laid-Open No. 2003-256025

  However, according to the conventional visual servo control, as described below, it is difficult to arbitrarily set the conveyance speed when conveying an object, and the object cannot be conveyed in consideration of acceleration / deceleration. There is a problem.

  FIG. 16 is an explanatory diagram for explaining a conventional method for setting the conveyance speed V by visual servo control. The figure schematically shows a change in the conveyance speed V when an object is conveyed from the point A to the point B which is the target point. In the figure, the horizontal axis represents time (t), the vertical axis represents the conveyance speed (V), and the white circles represent speed command values generated from the image information every 33 milliseconds. The white circles are interpolated by a quartic function, for example, and the speed and acceleration are smoothly connected. Actually, since the speed command to the robot is given every 1 millisecond, for example, such interpolation processing is generally performed. According to the conventional visual servo control, the object conveyance speed V is set to a value proportional to the difference between the teaching image and the frame image of the object at the local point. Then, the conveyance speed V is updated every fixed period (for example, 33 milliseconds) corresponding to the frame rate (for example, 30 frames / second) of the frame image obtained by the video camera device. For this reason, as shown in FIG. 16, as the object approaches the point B, the conveyance speed V tends to gradually decrease.

  In the example of FIG. 16, when the object is at the point A, as a difference between the frame image of the object at the point A and the teaching image, for example, the distance from the object to the target point B (in this case, from the point A to the point B ) Is calculated. Then, according to the control rule set in advance, the conveyance of the object is started at the conveyance speed Va corresponding to this distance. When the next frame image is generated, a distance from the object to the target point B is newly calculated from this frame image and the above-described teaching image, and the conveyance speed Vb corresponding to this distance is set. In this case, since the distance between the object and the target point B is shortened by the amount that the object is transported at the transport speed Va, the transport speed Vb is set to a value smaller than the transport speed Va. . In the same manner, the transport speed V is gradually decreased in the process of transporting the object to the target point B, and the transport speed V becomes zero when the object reaches the target point B. In this way, according to the control rule set in advance, as the object is farther from the target point B, the conveyance speed V of the object is set to be larger, and as the object approaches the target point B, the conveyance speed V is set to be smaller. The

  As described above, according to the conventional visual servo control, the conveyance speed V of the object on the conveyance path is controlled based on the difference between the frame image of the object and the teaching image, that is, the distance between the object and the target point. It is uniquely set by the rules. For this reason, for example, the object conveyance speed V cannot be flexibly controlled according to the condition of the conveyance path, and the object conveyance speed V cannot be controlled in consideration of acceleration / deceleration.

  In such visual servo control, as one of methods for controlling the conveyance speed V of an object, for example, a technique for adjusting a control gain (for example, a correspondence relationship between an image difference and a conveyance speed) can be considered. Therefore, the stability of control may be impaired. Accordingly, there is a need for a technique for controlling the conveyance speed V of an object while keeping the control gain constant.

  The present invention has been made to solve the above problems, and provides a robot apparatus, an image generation apparatus, an image generation method, and an image generation program that control the conveyance of an object in consideration of acceleration / deceleration. Objective.

In order to solve the above problems, an aspect of the robot apparatus according to the present invention includes a robot main body (for example, an element corresponding to the robot main body 10) that holds an object so as to be transportable to a target point, and a range including the target point and the object From the imaging unit (e.g., an element corresponding to the imaging unit 20) that captures the image, a teaching image that teaches the conveyance path of the object is acquired from the image captured by the imaging unit, and the teaching image is transferred to the desired conveyance of the object. An information processing unit that generates a target image that is a target when the object is conveyed by interpolation according to speed, and generates control information that controls the conveyance amount of the object based on the target image (for example, An element corresponding to the information processing unit 30) and a robot control unit (for example, a robot) that controls the operation of the robot body based on the control information generated by the information processing unit. Having an element) corresponding to preparative controller 40, the configuration of the robot apparatus having a.
According to the above configuration, the target image is generated by interpolating the teaching image captured by the imaging unit according to the desired conveyance speed. When the object is conveyed, the operation of the robot body is controlled based on the target image. At this time, the larger the difference between the current object image and the target image, the larger the object conveyance speed is set in accordance with a predetermined control rule. Therefore, if the target image is generated according to a desired transport speed, the transport speed of the object held by the robot body can be controlled. Therefore, it is possible to control the conveyance of the object in consideration of acceleration / deceleration.

In the robot apparatus, for example, the information processing unit is generated by an image generation unit (for example, an element corresponding to the image generation unit 31) that generates the target image from an image captured by the imaging unit, and the image generation unit. A difference between a storage unit (for example, an element corresponding to the storage unit 32) that stores the target image that has been recorded, a target image stored in the storage unit, and an image captured by the imaging unit when the object is transported A control information generation unit (for example, an element corresponding to the control information generation unit 33) that generates the control information such that the control information is small.
According to the above configuration, the control information is generated so that the difference between the current image captured by the imaging unit and the target image is small. Therefore, it is possible to control the conveyance amount of the object using the control information.

In the robot apparatus, for example, the image generation unit includes a teaching image acquisition unit (for example, an element corresponding to the teaching image acquisition unit 311) that acquires the teaching image from an image captured by the imaging unit, and the teaching image acquisition. An image information acquisition unit (for example, an element corresponding to the image information acquisition unit 312) that acquires information on the position and orientation of the object in a three-dimensional space from the image of the object included in the teaching image acquired by the unit; An image information interpolation unit (for example, an element corresponding to the image information interpolation unit 313) that interpolates the information acquired by the image information acquisition unit in the three-dimensional space, and the information interpolated by the image information interpolation unit A rendering unit (for example, an element corresponding to the rendering unit 314) that renders the target image in accordance with the transport speed of Obtain.
According to the above configuration, the image information interpolation unit interpolates information on the position and orientation of the object included in the teaching image. Thereby, it is possible to acquire information related to the posture of the object at an arbitrary position other than the position on the conveyance path from which the teaching image is acquired. Therefore, for example, by appropriately selecting the position on the transport path of the image to be interpolated according to the desired transport speed, it is possible to generate a target image corresponding to the desired transport speed.

In the robot apparatus, for example, the teaching image acquisition unit acquires the teaching image by binarizing an image captured by the imaging unit, and the image information acquisition unit obtains the object by modeling the object. Information on the position and orientation of the object is acquired by pattern matching the obtained image and the teaching image obtained by binarization by the teaching image acquisition unit.
According to the above configuration, by pattern matching between the teaching image obtained by binarization and the image obtained by modeling, for example, the coordinates of the object in the three-dimensional space and the object related to each axis in the three-dimensional space The amount of rotation is specified. Therefore, it is possible to acquire the position and orientation of the object imaged by the imaging unit.

In the robot apparatus, for example, the image information interpolation unit interpolates information on the position and orientation of the object using a spline function.
According to the above configuration, since the information on the position and orientation of the object indicated by the teaching image is interpolated by the spline function, in the process of conveying the object via the conveyance path indicated by the teaching image, the position and orientation of the object are determined. The change can be controlled smoothly.

In the robot apparatus, for example, the imaging unit includes a plurality of camera devices, and the information processing unit provides a teaching image that teaches a conveyance path of the object from the captured image for each of the plurality of camera devices. The acquired target image is interpolated according to a desired conveyance speed to generate a target image that is targeted when the object is conveyed, and the image that is captured by the imaging unit when the object is conveyed And a process of obtaining a difference between the target image and the target image.
According to the above configuration, the conveyance speed of the object held by the robot body can be controlled based on the images of the plurality of camera devices. Therefore, for example, even if an object exists in the blind spot area of some camera devices, if the object exists in an imaging area of another camera device, it is possible to continuously control the conveyance of the object.

In the robot apparatus, for example, the plurality of camera apparatuses are installed in association with a plurality of sections set in the conveyance path of the object.
According to the above configuration, it is possible to control the conveyance speed of the object held by the robot body based on the image of the camera device corresponding to each section. Therefore, if a camera apparatus is appropriately installed for each section, an image with a constant quality can be obtained over the entire section, and the control accuracy can be maintained constant.

In order to solve the above-described problem, one aspect of an image generation apparatus according to the present invention is an image generation apparatus that generates a target image that is a target when an object is transported to a target point by a robot using visual servo control. Included in the teaching image acquisition unit that acquires the teaching image that teaches the conveyance path of the object from the image obtained by imaging the range including the target point and the object, and the teaching image acquired by the teaching image acquisition unit An image information acquisition unit that acquires information on the position and orientation of the object in the three-dimensional space from the image of the object, and an image information interpolation unit that interpolates the information acquired by the image information acquisition unit in the three-dimensional space; A rendering unit that renders the target image according to a desired conveyance speed using the information interpolated by the image information interpolation unit; Having the configuration of an image generating apparatus comprising a.
According to the said structure, the effect similar to the image generation part with which the said robot apparatus is provided can be acquired.

In order to solve the above problem, one aspect of an image generation method according to the present invention is an image generation method for generating a target image that is a target when an object is conveyed to a target point by a robot using visual servo control. A teaching image acquisition unit acquires a teaching image that teaches a conveyance path of the object from an image obtained by imaging the range including the target point and the object, and an image information acquisition unit acquires the teaching image. A step of acquiring information on the position and orientation of the object in a three-dimensional space from the image of the object included in the teaching image acquired by the unit, and an image information interpolating unit that acquires the information acquired by the image information acquiring unit. The step of interpolating in the three-dimensional space, and the rendering unit using the information interpolated by the image information interpolation unit, Depending having the configuration of an image generation method comprising the steps of rendering the target image.
According to the said structure, the effect similar to the image generation part with which the said robot apparatus is provided can be acquired as an effect by an image generation method.

In order to solve the above problems, one aspect of the image generation program according to the present invention is an image generation apparatus that generates a target image that is a target when a computer transports an object to a target point by a robot using visual servo control. An image generation program for causing the computer to acquire a teaching image that teaches a conveyance path of the object from an image obtained by imaging the range including the target point and the object; An image information acquisition unit that acquires information on the position and orientation of the object in a three-dimensional space from the image of the object included in the teaching image acquired by the teaching image acquisition unit; and the information acquired by the image information acquisition unit Is interpolated in the three-dimensional space, and the information interpolated by the image information interpolation unit. With, it has a configuration of an image generating program to function as a rendering unit for rendering the target image in accordance with the desired conveying speed.
According to the above configuration, the same function and effect as the image generation unit provided in the robot apparatus can be obtained as the function and effect of the image generation program.

  According to the aspect of the present invention, the target image corresponding to the desired speed is generated from the teaching image, and the conveyance of the object is controlled based on the target image. Therefore, the object can be conveyed in consideration of acceleration / deceleration. .

It is a block diagram showing an example of a functional structure of the robot apparatus by 1st Embodiment of this invention. It is a figure showing an example of the appearance of the robot main body with which the robot apparatus by the embodiment is provided. It is a block diagram showing an example of functional composition of an information processor with which a robot apparatus by the embodiment is provided. It is a block diagram showing an example of a functional composition of an image generation part with which an information processor by the embodiment is provided. It is a flowchart showing an example of the process sequence of the image generation part by the embodiment. It is explanatory drawing for demonstrating the method of acquiring the teaching image by the embodiment. It is explanatory drawing for demonstrating the method to produce | generate the teaching image by the embodiment. It is explanatory drawing for demonstrating the method to acquire the image information by the embodiment. It is explanatory drawing for demonstrating the definition of the position and attitude | position of the object by the embodiment. It is explanatory drawing for demonstrating the information regarding the position and attitude | position of an object by the embodiment. It is explanatory drawing for demonstrating the interpolation process of the information regarding the position and attitude | position of an object by the embodiment. It is explanatory drawing for demonstrating the rendering method of the target image by the embodiment. It is explanatory drawing for demonstrating the relationship between the conveyance speed and position of an object by the embodiment. It is a block diagram showing an example of a functional structure of the robot apparatus by 2nd Embodiment of this invention. It is a block diagram showing an example of functional composition of an information processor with which a robot apparatus by the embodiment is provided. It is a figure showing the modification of the robot main body by 1st Embodiment and 2nd Embodiment. It is explanatory drawing for demonstrating the setting method of the conveyance speed of the object by the conventional visual servo control.

Hereinafter, embodiments for carrying out the present invention will be described in detail with reference to the drawings.
In addition, the same code | symbol represents the same element over all embodiment and all drawings in this specification.
[First Embodiment]
FIG. 1 is a block diagram showing an example of a functional configuration of the robot apparatus 1 according to the first embodiment of the present invention. The robot apparatus 1 according to the present embodiment is an apparatus for assembling an assembly part by transporting an object such as a part to a predetermined target point by visual servo control in an electronic equipment manufacturing line. In the present embodiment, an object to be transported by the robot apparatus 1 is a precision part that constitutes an assembly part incorporated in, for example, a small electronic device or a precision device. As shown in FIG. 1, the robot apparatus 1 includes a robot body 10, an imaging unit 20, an information processing unit 30, and a robot control unit 40.

  FIG. 2 is a diagram schematically illustrating an example of the appearance of the robot body 10. The robot body 10 holds an object so as to be able to be transported to a target point, and includes a support base 10a fixed to the floor surface, an arm part 10b connected to the support base 10a so as to be able to turn and bend, and a rotation. The hand part 10c is connected to the arm part 10b so as to be movable and swingable, and the grip part 10d is attached to the hand part 10c. The robot body 10 holds the object OBJ to be transported by the grip portion 10d in a state where it is gripped.

In the present embodiment, the robot body 10 is, for example, a 6-axis vertical articulated robot, and has a 6-axis degree of freedom by a coordinated operation of the support base 10a, the arm unit 10b, and the hand unit 10c, and the gripping unit 10d. It is possible to freely change the position and orientation of the parts gripped by. Further, the robot body 10 moves one of the arm unit 10b, the hand unit 10c, and the gripping unit 10d or a combination thereof under the control of the robot control unit 40.
Note that the degree of freedom of the robot body 10 is not limited to six axes. Further, the support base 10a may be installed in a place fixed to the floor surface, such as a wall or a ceiling.

  Returning to FIG. The imaging unit 20 captures a range including both a target point that is a transport destination and an object held by the robot body 10. For example, the image capturing unit 20 moves both the target point of the transport destination and the object to be transported upward. It is fixed to the wall or ceiling so that you can look down. In the present embodiment, the imaging unit 20 is a video camera device that captures an image at a frame rate of 30 frames / second. However, the imaging unit 20 may capture an image at an arbitrary frame rate such as 30 frames / second, 120 frames / second, and 200 frames / second. An image (frame image) captured by the imaging unit 20 is supplied to the information processing unit 30.

  The image processing unit 30 generates control information C necessary for controlling the operation of the robot apparatus 10 that conveys an object by analyzing the image captured by the imaging unit 20, and includes, for example, a computer apparatus. The In the present embodiment, the control information C includes information that defines the operation of each part of the robot body 10 such as, for example, the rotation angle or the rotation angular velocity of the arm unit 10b that constitutes the robot body 10.

  In the present embodiment, the information processing unit 30 acquires, for example, a teaching image for teaching an object conveyance path from an image captured by the imaging unit 20 when the operator teaches the object conveyance path. By interpolating the image according to a desired conveyance speed, a target image to be a target when the object is conveyed is generated. Further, in the stage of conveying an object, the information processing unit 30 generates control information C for controlling the conveyance amount of the object based on the image captured by the imaging unit 20 and the target image. In the present embodiment, the amount of conveyance of an object means the amount of change in the position of the object and the amount of change in the posture of the object. The definition of the position and orientation of the object will be described later. The control information C is supplied to the robot control unit 40 as a robot control command.

  The robot control unit 40 directly controls the operation of the robot body 10 based on the control information C generated by the information processing unit 30. That is, in the present embodiment, the robot control unit 40 is based on the control information C supplied from the information processing unit 30 and is one of the arm unit 10b, the hand unit 10c, and the gripping unit 10d of the robot body 10. Alternatively, the operation of the robot body 10 is controlled so as to transport the object held by the grip portion 10d to the target point by controlling the operation of any combination thereof (such as the amount of rotation of the arm portion 10b).

  FIG. 3 is a block diagram illustrating an example of a functional configuration of the information processing unit 30 included in the robot apparatus 10. The information processing unit 30 includes an image generation unit 31, a storage unit 32, and a control information generation unit 33. Here, the image generation unit 31 generates the target image from the image captured by the imaging unit 20. The storage unit 32 stores the target image generated by the image generation unit 31. The control information generating unit 33 generates control information C based on the target image stored in the storage unit 32. In the present embodiment, the control information generation unit 33 controls the control information C so that the difference between the target image stored in the storage unit 32 and the frame image captured by the imaging unit 20 when the object is conveyed is small. Is generated.

  In other words, the control information generation unit 33 generates the control information C so that the object is conveyed in a direction in which the target image stored in the storage unit 32 and the frame image captured by the imaging unit 20 match. To do. Specifically, as will be described later, control is performed so that the transport amount of the object is calculated in a direction in which the position and orientation of the object coincide with the position and orientation of the object indicated in the target image, and such a transport amount is obtained. Information C is generated. In the present embodiment, since the frame rate of the image captured by the image capturing unit 20 is 30 frames / second, a frame image is supplied from the image capturing unit 20 to the image generating unit 31 every 33 milliseconds. Control information C is newly generated every 33 milliseconds.

  FIG. 4 is a block diagram illustrating an example of a functional configuration of the image generation unit 31 included in the information processing unit 30. The image generation unit 31 includes a teaching image acquisition unit 311, an image information acquisition unit 312, an image information interpolation unit 313, and a rendering unit 314. The teaching image acquisition unit 311 acquires the teaching image from the image captured by the imaging unit 20. In the present embodiment, when the robot apparatus 1 is taught the conveyance path, the imaging unit is in a state where an object is positioned at a relay point (points P1 to P5 shown in FIG. 6 described later) on the conveyance path designated by the operator. An object is imaged by 20 to acquire a teaching image.

  In this embodiment, for convenience of explanation, a two-dimensional image of an object at a relay point designated by an operator is referred to as a teaching image, and a two-dimensional image obtained by interpolation processing described later is referred to as a target image. However, such definitions of terms are for convenience of explanation, and in a broad sense, the target image obtained by the interpolation processing is also a kind of teaching image in visual servo control.

  The image information acquisition unit 312 acquires, as image information, information on the position and orientation of the object represented by the teaching image from the teaching image acquired by the teaching image acquisition unit 311. In the present embodiment, as will be described later, this image information includes a CG (Computer Graphics) image generated using a CAD (Computer Aided Design) model of the object and an image of the object imaged by the imaging unit 20. Acquired by performing pattern matching. Pattern matching here refers to the position in the three-dimensional space of the object imaged in the two images by solving the nonlinear minimization problem so as to minimize the difference between the gradation values of the pixels of the two images. And calculating the difference in posture. In the example of this embodiment, the nonlinear minimization problem is solved so as to minimize the difference between the gradation value of each pixel of the teaching image and the gradation value of each pixel of the image captured by the image capturing unit 20, The difference between the position and orientation of the object in the three-dimensional space in the teaching image and the position and orientation of the object in the three-dimensional space in the image captured by the imaging unit 20 is calculated.

  The image information interpolation unit 313 interpolates the image information of each teaching image acquired by the image information acquisition unit 312 in the three-dimensional space. In this embodiment, as will be described later, by interpolating image information in a three-dimensional space using a spline function, it is possible to acquire information on the position and orientation of an object in a three-dimensional space at an arbitrary point on the transport path. And In other words, the trajectory representing the conveyance path of the object is represented by the spline function from the image information of the finite teaching image, and the position and posture of the object at an arbitrary point in the three-dimensional space can be known from the spline function. . That is, it is possible to generate a target image at an arbitrary point on the transport path.

  The rendering unit 314 uses the image information interpolated by the image information interpolation unit 313 to render (draw) the target image as a two-dimensional image according to a desired conveyance speed of the object. Since the image information interpolation unit 313 can obtain image information related to the position and orientation of an object at an arbitrary point, the rendering unit 314 uses the image information obtained by the image information interpolation unit 313 to perform desired processing. It is possible to render a target image at an arbitrary point according to the conveyance speed.

  In the present embodiment, the conveyance speed in the section between the two target images is controlled by setting the distance between two target images adjacent on the conveyance path in accordance with the desired conveyance speed. That is, when it is desired to obtain a small conveyance speed, the distance between target images is reduced (that is, the difference between the images is reduced). Conversely, when the conveyance speed is increased, the distance between target images is increased. (In other words, the image difference is increased). Further, the transport speed in the section between the two target images may be controlled by setting the time interval between two target images adjacent on the transport path in accordance with the desired transport speed. In this case, when the time interval between the relay points of the two target images is increased, the transport speed between these relay points is decreased. Conversely, when the time interval is decreased, the transport speed between these relay points is increased. Become.

Next, the operation of the robot apparatus 1 according to the present embodiment will be described.
The operation of the robot apparatus 1 according to the present embodiment includes a process for generating a target image in advance (hereinafter referred to as “target image generation process”) and a visual servo control using the target image to convey an object. Processing (hereinafter referred to as “object conveyance processing”). Therefore, in the following description of the operation, first, target image generation processing by the image generation unit 31 will be described, and then object conveyance processing by visual servo control using the target image will be described.

(Target image generation processing)
FIG. 5 is a flowchart illustrating an example of a target image generation processing procedure (image generation method) by the image generation unit 31. In step S <b> 1 of FIG. 5, the teaching image acquisition unit 311 constituting the image generation unit 31 acquires a teaching image that teaches the conveyance path of the object from the image captured by the imaging unit 20.

  FIG. 6 is an explanatory diagram for explaining a technique for acquiring a teaching image. The example shown in the figure shows a case where the operator teaches a transport path for transporting the object OBJ (part) into the housing of the product M mounted on the mounting table T. In the figure, the conveyance path of the object OBJ is indicated by a dotted line. An opening H is formed in the housing of the product M, and the object OBJ is carried into the housing of the product M from the opening H. Further, the point P0 represents the starting point of the transport path, and in the initial state, the object OBJ is located at the starting point P˜. Points P1 to P5 represent relay points on the transport path for acquiring the teaching image of the object OBJ, that is, teaching points.

  In the example shown in FIG. 6, the obstacle B exists in the vicinity of the opening H of the product M. For this reason, it is necessary to set a transport route that bypasses the obstacle B as shown by a dotted line in FIG. In order to set such a transport route, for example, by adjusting the operation of the robot body 10 with the object OBJ held by an operator's operation, a point P1 that is a key point of the transport route that bypasses the obstacle B , P2, P3, P4, P5, the object OBJ is positioned at each relay point, and the imaging unit 20 images the object OBJ at each point. Thereby, the actual image P of the object OBJ at each relay point on the transport route is obtained.

  The teaching image acquisition unit 311 constituting the image generation unit 31 binarizes the actual image P captured by the imaging unit 20, thereby relaying points P1, P2, P3, P4, and P5 on the transport route. A teaching image of the object OBJ at is acquired.

  FIG. 7A is an explanatory diagram for explaining a technique for the teaching image acquisition unit 311 to generate a teaching image from the actual image P. FIG. The teaching image acquisition unit 311 generates the image Pa of only the object OBJ from which the background is removed by calculating the background difference of the actual image P captured by the imaging unit 20. Then, by binarizing this image Pa using an appropriate threshold value, for example, a black and white image Pb is obtained as a teaching image. The shape of the object OBJ represented in the image Pb is uniquely determined according to the position and orientation of the object OBJ. Therefore, this image Pb includes information regarding the position and orientation of the object OBJ in the real space. For example, the position of the object OBJ in the real space is reflected in the coordinates and scale (reduction scale) of the object OBJ in the image Pb, and the posture of the object OBJ is reflected in the contour of the object OBJ in the image Pb.

  Subsequently, in step S <b> 2, the image information acquisition unit 312 uses a CAD model CG image obtained by modeling the object OBJ in the CG space set to match the real space, and the teaching image acquisition unit 311. By performing pattern matching with the teaching image Pb obtained from the actual image P, information regarding the position and orientation of the object OBJ in the three-dimensional space is acquired as image information.

  FIG. 7B is an explanatory diagram for explaining a method for the image information acquisition unit 312 to acquire image information related to the position and orientation of the object OBJ. The image information acquisition unit 312 uses the CAD model CM of the object OBJ to generate a CG image Ga of the object OBJ at the position and orientation given as the initial values. Then, the image information acquisition unit 312 performs pattern matching between the teaching image Pb acquired by the teaching image acquisition unit 311 and the CG image Ga, thereby performing an image related to the position and orientation of the object OBJ in the CG space in the three-dimensional space. Get information.

  For example, the image information acquisition unit 312 calculates the luminance difference between the teaching image Pb and the CG image Ga for each corresponding pixel, and obtains the sum of the differences of each pixel over the entire image. Further, a CG image Ga of the object OBJ at different positions and postures is generated using the CAD model CM, and similarly, the sum of luminance differences from the teaching image Pb is repeatedly obtained. Then, the image information acquisition unit 312 identifies the CG image Gb when the sum of the luminance differences is minimum, and determines the position and orientation in the CG space (three-dimensional) of the CAD model that gives the CG image Gb. The position and orientation of the object OBJ in the teaching image Pb are specified as the position and orientation in the three-dimensional space, and information regarding the position and orientation is acquired as image information.

Here, the definition of the position and orientation of the object OBJ in the present embodiment will be described.
FIG. 8 is an explanatory diagram for explaining the definition of the position and orientation of the object OBJ in the CG space that defines the CAD model described above. In this embodiment, the coordinate system of the CG space that defines the CAD model CM is calibrated in advance so as to match the real space in which the object OBJ exists. In such a CG space, information on the position of the object OBJ is defined as coordinates (x, y, z) of a predetermined part (for example, the position of the center of gravity) of the CAD model CM of the object OBJ. Also, the information on the posture of the object OBJ is rotated on each of the three axes (Jx, Jy, Jz) passing through the coordinate points that are parallel to the three axes of the coordinate system of the CG space and give the position of the object OBJ. It is defined as the rotation angle (θx, θy, θz) with the axis. However, without being limited to this example, the position and orientation of the object OBJ may be defined in any way.

FIG. 9 is an explanatory diagram for explaining image information related to the position and orientation of the object OBJ acquired by the image information acquisition unit 312. The points P1 to P4 on the conveyance path of the object OBJ shown in FIG. FIG. 5 is an explanatory diagram for explaining a relationship with image information related to the position and orientation of an object OBJ.
In FIG. 9, the point P5 shown in FIG. 6 is omitted. In FIG. 9, the horizontal axis represents the points P1 to P4, but this may be replaced with the time (time) at which the object OBJ passes through each point. The same applies to FIGS. 10 and 11 described later.

  Parameters PA1, PB1, PC1, PD1, PE1, and PF1 shown in FIG. 9 schematically represent image information related to the position and orientation of the object OBJ at the point P1 shown in FIG. For example, the parameters PA1, PB1, and PC1 correspond to the components of the coordinates (x, y, x) representing the position of the object OBJ at the point P1, and the parameters PD1, PE1, and PF1 are the parameters of the object OBJ at the point P1. This corresponds to each component of the rotation angle (θx, θy, θz) representing the posture. Similarly, the parameters PA2, PB2, PC2, PD2, PE2, and PF2 represent image information regarding the position and orientation of the object OBJ at the point P2, and the parameters PA3, PB3, PC3, PD3, PE3, and PF3 are at the point P3. The parameters PA4, PB4, PC4, PD4, PE4, and PF4 represent image information related to the position and orientation of the object OBJ at the point P4. As shown in FIG. 9, each parameter representing the position and orientation of the object OBJ changes according to the position and orientation of the object OBJ at each point. The amount of this change is determined by the state in which the object OBJ is positioned at each point by the operation of the operator.

Subsequently, in step S3, the image information interpolation unit 313 interpolates the trajectory between the teaching points in the three-dimensional space from the information on the position and orientation of the teaching point object using a spline function. In the present embodiment, the image information interpolation unit 313 uses a spline function to obtain a trajectory between teaching points for interpolating parameters representing the position and orientation of the object OBJ acquired by the image information acquisition unit 312 in the three-dimensional space. .
FIG. 10 is an explanatory diagram for explaining interpolation processing of image information related to the position and orientation of the object OBJ. As shown in the figure, the image information interpolation unit 313 interpolates each parameter shown in FIG. 9 with a spline function s (t) described later. That is, the image information interpolation unit 313 passes the trajectory PA passing through the parameters PA1 to PA4 representing the x component of the position of the object OBJ, the trajectory PB passing through the parameters PB1 to PB4 representing the y component, and the parameters PC1 to PC4 representing the z component. The trajectory PC is calculated using a spline function. Similarly, the image information interpolation unit 313 includes trajectories PD passing through parameters PD1 to PD4 representing θx components of the posture of the object OBJ, trajectories PD passing through parameters PE1 to PE4 representing θy components, and parameters PE1 to PE4 representing θz components. The passing trajectory PE is calculated using a spline function. The spline function that gives the trajectories PA to PF described above represents the interpolated parameters in addition to the parameters at the points P1 to P5. That is, the spline function that gives the trajectories PA to PF includes image information related to the position and orientation of the object OBJ at any point on the transport path from the points P1 to P4.

Therefore, an image representing the position and orientation of the object OBJ at a desired time can be generated from the spline function that gives the trajectories PA to PF. That is, the target image at an arbitrary point on the transport path between the point P1 and the point P4 is rendered using the image information regarding the position and orientation of the object OBJ obtained from the spline function that gives the trajectories PA to PF. With this target image, it becomes possible to interpolate the teaching images obtained at the points P1 to P4.
In the above description, the parameter of the point P5 is omitted. However, in order to transport the object OBJ to the point P5, the trajectory PA shown in FIG. 10 is considered in consideration of the parameters at the points P1 to P5. PF is calculated using a spline function.

Here, an example of a calculation procedure for interpolating the above-described parameters using the spline function s (t) will be described.
The spline function s (t) is expressed as the following formula (1). In Equation (1), a _i is a coefficient, and B _{i, k} (t) is a B spline.

Interpolating the above parameters with the spline function (t) means specifying the spline function s (t) that expresses the value of each parameter, specifically, the coefficient in the spline function s (t). It means that a _i is specified. Hereinafter, the calculation procedure of the coefficient a _i of the spline function s (t) represented by the equation (1) will be described.

(Procedure 1) (t ₀ , p ₀ ), (t ₁ , p ₁ ), (t ₂ , p ₂ ) as information on the position and orientation of the object OBJ at N (N is an arbitrary natural number) teaching points ),..., (T _N−1 , p _N−1 ) are input values. Here, t _{0 to} t _N−1 represent the time at each teaching point. Further, p _{0 to} p _N−1 represent the above-described parameter values at each teaching point, and are, for example, the coordinates of the position of the object OBJ at each teaching point.

(Procedure 2) The node q of the K-1 dimensional spline is calculated by the following equation.
q ₀ = q ₁ = ... = q _K-1 = t ₀
q _{i + K} = (t _i + t _{i + K} ) / 2 (where i = 0, 1,..., N-1-K)
_{q N = q N + 1 =} ... = q N + K-1 = t N-1

(Procedure 3) B spline B _{i, k} (t) is calculated by the following equation using the above-mentioned node q by the Dova Cox algorithm.
B _{i, k} (t) = {(t−q _i ) / q _{i + K−1} −qi)} B _{i, K−1} (t)
+ {(Q _{i + K} −t) / (q _{i + K} −q _{i + 1} )} B _{i + 1, K−1} (t)

(Procedure 4) The coefficient a _i of the spline function s (t) of the above equation (1) expressed as a polynomial of the above B spline function B _{i, k} (t) is _obtained by the following procedure.
From s (t _i ) = p _i (where i = 0, 1,..., N−1), the following N-element linear equation holds.

The above-described N-element linear equation is expressed as a determinant as follows.
Aa = p (2)
However, the matrix A, the matrix a, and the matrix b in the above equation (2) are expressed as follows.

The coefficient a _i which is each element of the matrix a is obtained from the above determinant (2) by Gaussian elimination or the like. Thus, the coefficient a _i of the spline function s (t) is obtained.
As described above, the coefficient a _i of the spline function s (t) is obtained as each element of the matrix a from the value of the parameter forming the image information at each teaching point, and the spline function s representing the above-described trajectories PA to PF. (T) is obtained. According to such a spline function s (t), it is possible to generate a trajectory for smoothly interpolating parameters, and to smoothly accelerate and decelerate the object OBJ conveyed on the trajectory. .

Subsequently, in step S <b> 4, the rendering unit 314 renders the target image according to the desired conveyance speed V using the information on the position and orientation of the object OBJ interpolated by the image information interpolation unit 313. In the present embodiment, the rendering unit 314 renders the target image according to the desired conveyance speed V specified by the operator, using the parameters interpolated by the image information interpolation unit 313.
FIG. 11 is a diagram for explaining a rendering technique of the rendering unit 314. In the figure, points P1 and P2 are points P1 and P2 shown in FIG. 6, and points P11 and P12 represent points where the target image is interpolated between the points P1 and P2. In this embodiment, for simplification of description, the point P11 corresponds to a third point between the point P1 and the point P2, and the point P12 is two-thirds between the point P1 and the point P2. It shall correspond to the point of. That is, the distance from the point P1 to the point P11 is one third of the distance from the point P1 to the point P2, and the distance from the point P1 to the point P12 is three minutes of the distance from the point P1 to the point P2. 2.

  In FIG. 11, an image G11 represents a target image at a point P11 indicated when the object OBJ is located at the point P1, and an image G12 is a point P12 indicated when the object OBJ is located at the point P11. The image G2 represents the target image at the point P2 shown when the object OBJ is located at the point P12. Dotted lines shown in the images G11, G12, and G2 represent transport routes that use the points P1 to P5 shown in FIG. 6 as relay points.

  Here, the target image G11 is an image generated by the image information interpolation unit 313 from parameters PA11, PB11, PC11, PD11, PE11, and P11, which are image information related to the position and orientation of the object OBJ at the point P11. The position and orientation of the object OBG shown in the target image G11 are defined by the parameters PA11, PB11, PC11, PD11, PE11, and PF11. The target image G12 is an image generated by the image information interpolation unit 313 from parameters PA12, PB12, PC12, PD12, PE12, and PF12, which are image information related to the position and orientation of the object OBJ at the point P12. The position and orientation of the object OBG shown in the target image G22 are defined by the parameters PA12, PB12, PC12, PD12, PE12, and PF12. The target image G2 is an image generated by the image information interpolation unit 313 from parameters PA2, PB2, PC2, PD2, PE2, and PF2, which are image information related to the position and orientation of the object OBJ at the point P2, or a teaching image acquisition unit. 311 is a teaching image of the object OBJ at the point P2 acquired by 311.

As illustrated in FIG. 11, if the number of relay points between the point P1 and the point P2 is increased and the target image by interpolation is increased, the conveyance speed V between the point P1 and the point P2 is reduced as will be described later. Become. The rendering unit 314 sets the number of relay points according to a desired conveyance speed between the points P1 and P2, and the above-described image information interpolation unit 313 interpolates the target image of the object OBJ at the relay points. Rendering is performed using the image information. Similarly, a relay point is set in accordance with a desired conveyance speed in the section between the points P2 to P5, and a target image at the relay point is generated.
As described above, the target image generation process is performed by the image generation unit 31, and the target image generated by the target image generation process is stored in the storage unit 32.

  Here, the relationship between the target image obtained by the above-described target image generation processing and the conveyance speed will be described. In FIG. 11, the conveyance speed V when the object is conveyed from the point P1 to the point P11 is parameters PA1, PB1, PC1, PD1, PE1 which are image information relating to the position and orientation of the object OBJ at the point P1 shown in FIG. , PF1 and a parameter PA11, PB11, PC11, PD11, PE11, PF11 which is image information related to the position and orientation of the object OBJ at the point P11. The parameter difference in this case is smaller than in the case where only the target image G2 is generated without setting the relay points P11 and P12 in FIG.

  For example, the difference in the position of the object OBJ when the relay point P11 is set (for example, the distance from the point P1 to the point P11) is the position of the object OBJ when only the target image G2 is generated without setting the relay point P11. (For example, the distance from the point P1 to the point P2). Therefore, by generating the target image G11 at the relay point P11 by interpolation, the conveyance speed V between the point P1 and the point P11 is also reduced. Conversely, if the target image G11 at the relay point P11 is not generated, the conveyance speed V can be increased as compared with the case where the target image G11 is generated. The same applies to the conveyance speed V when the object is conveyed from the point P11 to the point P12 and the conveyance speed V when the object is conveyed from the point P12 to the point P2.

(Object conveyance processing)
Next, the object conveyance process by visual servo control using the target image generated by the image generation unit 31 as described above will be described.
The visual servo control by the robot apparatus 1 of the present embodiment is the same as the conventional visual servo control except that the target image generated by the image generation unit 31 is used as a teaching image. Here, in addition to the target images G11, G12, and G2 at the points P11, P12, and P2 of FIG. 11 generated by the image generation unit 31 described above, the target image G1 (not shown) at the point P1 is used to A process of transporting the object OBJ from the start point P0 shown in FIG. 6 to the point P2 via the points P1, P11, and P12 will be described.

  It is assumed that no relay point is set between the start point P0 and the point P1, and there is no interpolated target image. Further, the target images G1, G11, G12, and G2 of the object OBJ at each of the points P1, P11, P12, and P2 are generated by the image generation unit 31 by the above-described target image generation process and stored in the storage unit 32 in advance. It shall be.

  Schematically, the visual servo control is based on the information processing shown in FIG. 3 from the frame images obtained every moment by the imaging unit 20 and the above-described target images G1, G11, G12, G2 previously stored in the storage unit 32. The control information generation unit 33 constituting the unit 30 generates the control information C at a constant cycle corresponding to the frame rate of the frame image, and the robot control unit 40 feedback-controls the operation of the robot body 10 based on the control information C. Is implemented.

  More specifically, the imaging unit 20 images a predetermined range including the start point P0 to the point P5 where the object OBJ is located at a frame rate of 30 frames / second. This frame image is supplied from the imaging unit 20 to the control information generation unit 33 of the information processing unit 30 every 33 milliseconds corresponding to the frame rate.

  Since the first relay point of the object OBJ located at the start point P0 is the point P1, the control information generation unit 33 reads the target image G1 (not shown) at the point P1 from the storage unit 32. Then, the control information generation unit 33 calculates the difference between the target image G1 and the current frame image. In other words, the control information generation unit 33 calculates the difference of each parameter described above from the target image G1 and the current frame image. Then, the control information generation unit 33 calculates the transport amount of the object OBJ so that the difference becomes small, and instructs the operation amount of the robot body 10 corresponding to the transport amount, for example, the rotation amount of the hand unit 10b. Control information C is generated and output. Here, in the present embodiment, the transport amount of the object OBJ includes both a change amount of the position of the object OBJ and a change amount of the posture of the object OBJ. That is, the control information generating unit 33 changes the position change amount of the object OBJ and the change amount of the posture of the object OBJ so that the object OBJ at the point P0 takes the position and posture shown in the target image G11 at the point P1. And the control information C specifying the movement amount of the robot body 10 that gives such an amount of change to the object OBJ is supplied to the robot control unit 40.

  Each time a new frame image is supplied from the imaging unit 20, the control information generation unit 33 recalculates the difference between the parameters to generate control information C, and supplies the control information C to the robot control unit 40. In the present embodiment, the imaging unit 20 supplies a frame image with a frame rate of 30 frames / second to the control information generation unit 33, so that the control information generation unit 33 calculates the difference of each parameter about every 33 milliseconds. Then, control information C is generated. The robot control unit 40 controls each unit of the robot body 10 according to the operation amount of the robot body 10 specified by the control information C sequentially supplied from the control information generation unit 33. Thereby, the object OBJ held by the robot body 10 is transported from the start point P0 toward the point P1.

  FIG. 12 is a diagram illustrating the relationship between the conveyance speed V of the object OBJ by the robot body 10 and the position on the conveyance path. The robot control unit 40 controls the operation of the robot body 10 based on the control information C supplied from the control information generation unit 33 when the object OBJ is located at the start point P0, and the object OBJ moves from the start point P0 toward the point P1. To start moving. In this case, the difference between the parameters forming the image information related to the position of the object OBJ increases in accordance with the distance from the start point P0 to the point P1, which is the first relay point. For this reason, the control information generation unit 33 generates the control information C by setting a relatively large conveyance speed VA according to this difference as the conveyance speed V of the object OBJ. Based on the control information C generated by setting the transport speed VA, the robot control unit 40 causes the robot body 10 to perform an operation corresponding to the transport speed VA.

  Then, when a new frame image is supplied from the imaging unit 20 after about 33 milliseconds, the control information generation unit 33 acquires parameters regarding the position and orientation of the object OBJ of the new frame image, The control information C is generated by newly recalculating the difference from the parameter representing the position and orientation of the object OBJ in the target image G1 at the point P1. At this time, since the position of the object OBJ is closer to the point P1 than the starting point P0, the distance between the object OBJ and the point P1 is shortened. Therefore, the conveyance speed VB of the object OBJ set by the control information C at this time is smaller than the above-described conveyance speed VA. In this way, as the object OBJ approaches the point P1, as shown in FIG. 12, the conveyance speed V of the object OBJ tends to decrease.

  In FIG. 12, the conveyance speed V tends to decrease substantially linearly between the start point P0 and the point P1, but microscopically, the conveyance speed V gradually decreases every 33 milliseconds. . However, in reality, since the mass of the object OBJ and the inertial mass of the robot body 10 exist, the conveyance speed V of the object OBJ is proportional to the distance between the object OBJ and the point P1, as illustrated in FIG. Then it falls gently. Therefore, in the example shown in FIG. 12, the average value of the conveyance speed V of the object OBJ from the start point P0 to the point P1 is approximately VA / 2.

  Subsequently, when the object OBJ reaches the point P1, the control information generation unit 33 reads the target image G11 at the point P11 from the storage unit 32 because the next relay point is the point P11. Then, the control information generation unit 33 calculates the difference between the parameters described above from the frame image of the object OBJ located at the point P1 and the target image G11, calculates the transport amount of the object OBJ so that the difference becomes small, Control information C for instructing the amount of movement of the robot body 10 corresponding to the transport amount is generated and output. Here, the control information generation unit 33 responds to the difference between each parameter calculated from the frame image of the object OBJ located at the point P1 and the target image G11 as the transport speed V when transporting from the point P1 toward the point P11. Control information C is generated by setting the transport speed VB. The transport speed V in this case also decreases gently in proportion to the distance between the object OBJ and the point P11, as described above.

  The transportation of the object OBJ by visual servo control in the next section P11 to P12 and the section P12 to P2 is the same as the transportation in the section from the start point P0 to the point P1 or the section P1 to P11. Explained. Although not specifically described, the conveyance between the points P2 to P5 is also described in the same manner. In this way, the object OBJ is conveyed by the object conveyance process based on the visual feedback control.

  Here, it is assumed that the distance between the sections of the start points P0 to P1 is equal to the distance between the sections of the points P1 and P2, and the distances of the sections P1 to P11, P11 to P12, and P12 to P2 are the sections of the start points P0 to P1. 12, the transport speed VB is about one third of the transport speed VA in FIG. 12, and the average value VB / 2 of the transport speed V in the section between the points P1 and P2 is the start point. This is about one third of the average value VA / 2 of the conveyance speed V in the section from P0 to P1.

Therefore, according to the present embodiment, it is possible to control the conveyance speed V of each section based on the target image. That is, by generating the target image by adjusting the interval between the relay points according to the desired transport speed, the transport speed V of the object OBJ can be intentionally controlled, and the object is considered in consideration of acceleration / deceleration. The OBJ can be transported.
Further, according to the present embodiment, the position and orientation of the object OBJ at an arbitrary point can be grasped by the spline function s (t), so that the desired transport speed V can be achieved without being restricted by the teaching point. A target image can be prepared accordingly.
In addition, according to the present embodiment, a target image at an arbitrary point can be generated from a limited teaching image, so that the object OBJ is transported to the final target point with a small number of teaching images in consideration of acceleration / deceleration. Can be controlled.

  Further, according to the present embodiment, in the process of transporting the object OBJ to the target point, it is possible to set the transport speed according to the surrounding situation. For example, if the obstacle B does not exist around the object OBJ, the object OBJ can be transported without decreasing the transport speed, and the transport speed can be decreased around the obstacle B. In addition, since the target image is generated by interpolating the teaching image using the spline function, the object OBJ can be smoothly conveyed. In addition, acceleration and deceleration can be performed smoothly.

In the above-described embodiment, the interpolation processing is performed using the spline function. However, any interpolation method such as linear interpolation can be used as long as the conveyance speed can be controlled.
In the above-described embodiment, the image generation unit 31 is a constituent element of the robot apparatus 1, but it may be configured separately as an image generation apparatus. The image generation apparatus in this case is an image generation apparatus that generates a target image to be a target when an object is transported to a target point by a robot using visual servo control, and includes the above-described teaching image acquisition unit 311, The image generation apparatus includes an image information acquisition unit 312, an image information interpolation unit 313, and a rendering unit 314.

  The processing procedure of the image generation unit 31 according to the above-described embodiment can also be expressed as an image generation method. The image generation method in this case is an image generation method for generating a target image that should be a target when an object is transported to a target point by a robot using visual servo control, and includes the above-described steps S1 to S4. Configured as a generation method.

  Furthermore, the processing procedure of the image generation unit 31 can be expressed as an image generation program for causing a computer to execute the processing procedure. The image generation program in this case is an image generation program for causing a computer to function as an image generation device that generates a target image to be targeted when an object is transported to a target point by a robot using visual servo control. Thus, the computer is configured as an image generation program for causing the teaching image acquisition unit 311, the image information acquisition unit 312, the image information interpolation unit 313, and the rendering unit 314 to function.

[Second Embodiment]
Next, a second embodiment of the present invention will be described.
Schematically, in the present embodiment, a plurality of imaging units are prepared, a certain imaging unit images a part of the section on the conveyance path, and another section is captured by another imaging unit, The imaging unit controls the conveyance of the object OBJ over a plurality of sections for imaging. The number of imaging units is arbitrary.

  FIG. 13 is a block diagram illustrating an example of a functional configuration of the robot apparatus 2 according to the present embodiment. The robot apparatus 2 includes two imaging units 21 and 22 and an information processing unit 50 in place of the imaging unit 20 and the information processing unit 30 in the configuration of the first embodiment shown in FIG. Other configurations are the same as those of the first embodiment.

Each of the two imaging units 21 and 22 is the same as the imaging unit 20 in the first embodiment, and is, for example, a video camera device. However, in the present embodiment, the two imaging units 21 and 22 are installed in association with a plurality of sections set in the conveyance path of the object OBJ. In the present embodiment, the imaging unit 21 is installed in the section from the start point P0 to the point P5 shown in FIG. 6 described above, and the imaging unit 21 has a transport path from the object OBJ to the point P5 to the point P5. It is set to image a range that includes it. Further, in the present embodiment, the imaging unit 22 is installed at a desired position inside the housing of the product M shown in FIG. 6 from above, and a final target point (not shown) from the point P5 to the inside of the housing of the product M. It is set to image the range including the transport route up to.
Note that the number of imaging units is arbitrary, and an arbitrary number of imaging units may be installed in accordance with the number of sections set on the conveyance path.

  The information processing unit 50 acquires a teaching image for teaching the conveyance path of the object OBJ from the captured images for each of the two imaging units 21 and 22, and interpolates the teaching image according to a desired conveyance speed. As a result, a target image to be targeted when the object OBJ is conveyed is generated. And when this object OBJ is conveyed, the process which acquires the difference of the image imaged by the imaging parts 21 and 22 and the said target image is performed.

  FIG. 14 is a block diagram illustrating an example of a functional configuration of the information processing unit 50. The information processing unit 50 includes a first information processing unit 51, a second information processing unit 52, and a selection unit 53. Among these, each of the 1st information processing part 51 and the 2nd information processing part 52 is comprised similarly to the information processing part 30 by the above-mentioned 1st Embodiment. The selection unit 53 is for selecting and outputting either the control information C1 output from the first information processing unit 51 or the control information C2 output from the second information processing unit 52 as the control information C. .

  In the present embodiment, the selection operation by the selection unit 53 is such that the object OBJ includes the section of the transport path from the start point P0 to the point P5 included in the imaging range of the imaging unit 21 and the point P5 included in the imaging range of the imaging unit 22. Is controlled based on which section of the transport route from the first to the final target point exists. That is, when the object OBJ exists on the transport path from the start point P0 to the point P5 included in the imaging range of the imaging unit 21, the selection unit 53 uses the control information C1 output by the first information processing unit 51 as control information. Select as C. In addition, the selection unit 53 controls the control information C2 output by the second information processing unit 52 when the object OBJ exists on the transport path from the point P5 included in the imaging range of the imaging unit 22 to the final target point. Is selected as control information C.

Next, the operation of the robot apparatus 2 according to the present embodiment will be described.
In the present embodiment, when the object OBJ exists in the section of the transport path from the start point P0 to the point P5 shown in FIG. 6, the selection unit 53 uses the control information C1 output from the first information processing unit 51 as the control information C. Choose as. And the imaging part 21 and the 1st information processing part 5 function similarly to the imaging part 20 and the information processing part 30 by the above-mentioned 1st Embodiment. Therefore, in this case, the robot apparatus 2 functions in the same manner as the robot apparatus 1 according to the first embodiment.

  In the present embodiment, when the object OBJ is on the transport path inside the housing of the product M that is a blind spot area from the imaging unit 21, the imaging unit 22 and the first information processing unit 51 are replaced with the imaging unit 22 and the first information processing unit 51. 2 The information processing unit 52 functions to generate control information C2, and the selection unit 53 selects the control information C2 as control information C. Therefore, in this case, the image generating unit 31 shown in FIG. 4 constituting the second information processing unit 52 generates each target image from the teaching image, and each relay of the object OBJ on the transport path inside the housing of the product M is performed. A teaching image at a point is acquired, and a target image corresponding to a desired conveyance speed is generated by interpolating the teaching image as in the first embodiment. This target image is stored in the storage unit 32 shown in FIG.

  Further, when the object OBJ is transported from the point P5 to the final target point, the second information processing unit 52 refers to the target image stored in the storage unit 32 and starts from the frame image captured by the imaging unit 22. Control information C2 is generated. The control information C2 is supplied as control information C to the robot control unit 40 via the selection unit 53. The robot control unit 40 controls the operation of the robot body 10 based on the control information C2 supplied as the control information C, thereby causing the object OBJ to be transported from the point P5 to the final target point.

  Here, in the present embodiment, selection switching by the selection unit 53 is performed when the object OBJ reaches the point P5. That is, the information processing unit 50 causes the selection unit 53 to select the control information C1 as the control information C until the object P5 reaches the point P5 corresponding to the end point of the transport path included in the imaging range of the imaging unit 21. . When the object OBJ reaches the point P5, the selection unit 53 is made to select the control information C2 as the control information C. Thus, by switching the content of the control information C from the control information C1 to the control information C2, it is possible to continuously control the conveyance of the object OBJ even when the object OBJ enters the blind spot area of the imaging unit 21. Become.

  In the above example, the two imaging units 21 and 22 are installed in association with the sections on the transport path. However, a plurality of imaging units are installed in the same section, and control is performed from images captured by the imaging units. Information C may be generated. In this case, for example, even if an obstacle dynamically enters the imaging range of a certain imaging unit and a situation in which the object OBJ cannot be imaged by this imaging unit suddenly occurs, the imaging is performed by another imaging unit. It becomes possible to generate the control information C from the image. Accordingly, visual servo control can be stably continued.

Although the embodiment of the present invention has been described above, the present invention is not limited to the above-described embodiment, and various modifications can be made without departing from the gist of the present invention.
FIG. 15 is a diagram schematically illustrating a modification of the robot body 10 according to the first embodiment and the second embodiment described above. The robot body 11 according to this modification includes a pedestal 11a, a support base 11b fixed to the base 11a so as to be able to turn, and two arm portions 11c connected to the support base 11b so as to be turnable and extendable. The hand portion 11d is connected to each arm portion 11c so as to be swingable, and the grip portion 11e is attached to each hand portion 11d. A plurality of wheels for moving the robot body 11 are attached to the lower part of the base 11a. The robot body 11 holds the object OBJ in a state of being gripped by the gripping part 11e. For example, the robot main unit 11 and the robot control unit 40 may be integrally configured by housing the robot control unit 40 shown in FIG. 1 in the pedestal 11a, or the robot control unit 40 and the robot main unit 11 may be integrated. And may be configured separately. Also in this modified example, the robot body 11 is a vertical articulated robot, and the number of axes of the joint is arbitrary.

  DESCRIPTION OF SYMBOLS 1, 2 ... Robot apparatus 10, 11 ... Robot main body 20, 21, 22 ... Imaging part, 30 ... Information processing part, 31 ... Image generation part, 32 ... Storage part, 33 ... Control information generation part, 40 ... Robot Control unit, 50 ... information processing unit, 51 ... first information processing unit, 52 ... second information processing unit, 53 ... selection unit, 311 ... teaching image acquisition unit, 312 ... image information acquisition unit, 313 image information interpolation unit, 314: rendering unit, C: control information, G1, G11, G12, G2 ... image, OBJ ... object, P0 ... start point (point), P1-P5 ... point, S1-S4 ... processing steps, V, VA, VB ... Transport speed.

Claims (10)

A robot body that holds an object to a target point so that it can be transported;
An imaging unit for imaging a range including the target point and the object;
A target image is acquired by acquiring a teaching image that teaches a conveyance path of the object from an image captured by the imaging unit, and interpolating the teaching image according to a desired conveyance speed of the object, thereby conveying the object. And an information processing unit that generates control information for controlling the transport amount of the object based on the target image,
A robot control unit that controls the operation of the robot body based on the control information generated by the information processing unit;
Robot device with The information processing unit
An image generation unit that generates the target image from an image captured by the imaging unit;
A storage unit for storing the target image generated by the image generation unit;
A control information generation unit that generates the control information so that a difference between a target image stored in the storage unit and an image captured by the imaging unit when the object is conveyed;
The robot apparatus according to claim 1, comprising: The image generation unit
A teaching image acquisition unit that acquires the teaching image from an image captured by the imaging unit;
An image information acquisition unit that acquires information on the position and orientation of the object in a three-dimensional space from the image of the object included in the teaching image acquired by the teaching image acquisition unit;
An image information interpolation unit for interpolating the information acquired by the image information acquisition unit in the three-dimensional space;
A rendering unit that renders the target image according to a desired conveyance speed using information interpolated by the image information interpolation unit;
The robot apparatus according to claim 2, further comprising: The teaching image acquisition unit acquires the teaching image by binarizing the image captured by the imaging unit,
The image information acquisition unit performs pattern matching between an image obtained by modeling the object and a teaching image obtained by binarization by the teaching image acquisition unit, thereby obtaining information on the position and orientation of the object. The robot apparatus according to claim 3, which is acquired.   The robot apparatus according to claim 3, wherein the image information interpolation unit interpolates information regarding the position and orientation of the object using a spline function. A plurality of camera devices are provided as the imaging unit,
The information processing unit acquires, for each of the plurality of camera devices, a teaching image that teaches the conveyance path of the object from the captured image, and interpolates the teaching image according to a desired conveyance speed, The target image that is targeted when the object is transported is generated, and the process of acquiring the difference between the image captured by the imaging unit and the target image is performed when the object is transported. 6. The robot apparatus according to any one of 5 above.   The robot apparatus according to claim 6, wherein the plurality of camera apparatuses are installed in association with a plurality of sections set in a conveyance path of the object. An image generation device that generates a target image that is a target when an object is transported to a target point by a robot using visual servo control,
A teaching image acquisition unit that acquires a teaching image that teaches a conveyance path of the object from an image obtained by imaging the target point and a range including the object;
An image information acquisition unit that acquires information on the position and orientation of the object in a three-dimensional space from the image of the object included in the teaching image acquired by the teaching image acquisition unit;
An image information interpolation unit for interpolating the information acquired by the image information acquisition unit in the three-dimensional space;
A rendering unit that renders the target image according to a desired conveyance speed using information interpolated by the image information interpolation unit;
An image generation apparatus comprising: An image generation method for generating a target image when a target is transported to a target point by a robot using visual servo control,
A teaching image acquisition unit acquiring a teaching image that teaches a conveyance path of the object from an image obtained by imaging the target point and a range including the object;
An image information acquisition unit acquiring information on the position and orientation of the object in a three-dimensional space from the image of the object included in the teaching image acquired by the teaching image acquisition unit;
An image information interpolation unit interpolating the information acquired by the image information acquisition unit in the three-dimensional space;
A rendering unit rendering the target image according to a desired transport speed using the information interpolated by the image information interpolation unit;
An image generation method including: An image generation program that causes a computer to function as an image generation device that generates a target image when a target is transported to a target point by a robot using visual servo control,
The computer,
A teaching image acquisition unit that acquires a teaching image that teaches a conveyance path of the object from an image obtained by imaging the target point and a range including the object;
An image information acquisition unit that acquires information on the position and orientation of the object in a three-dimensional space from the image of the object included in the teaching image acquired by the teaching image acquisition unit;
An image information interpolation unit for interpolating the information acquired by the image information acquisition unit in the three-dimensional space;
A rendering unit that renders the target image according to a desired conveyance speed using information interpolated by the image information interpolation unit;
An image generation program that functions as