JP2020107142A - Recognition method, recognition system, robot control method, robot control system, robot system, recognition program, and robot control program - Google Patents

Abstract

To bring less occurrence in fluctuating time required until a subject 51 is recognized among a plurality of types of articles 5.SOLUTION: A recognition method includes first recognition processing and second recognition processing. The first recognition processing recognizes a first characteristic of a subject 51 from image data including the subject 51. In the second recognition processing, a plurality of recognition means 121 corresponding to characteristics of a plurality of types of articles 5 that may be the subject 51 is prepared. The second recognition processing selects, in accordance with at least the first characteristic recognized in the first recognition processing, one piece of recognition means 121 from the plurality of recognition means 121, for recognizing a second characteristic of the subject 51 by the one recognition means 121.SELECTED DRAWING: Figure 1

Description

The present disclosure generally relates to a recognition method, a recognition system, a robot control method, a robot control system, a robot system, a recognition program, and a robot control program. More specifically, the present disclosure relates to a recognition method, a recognition system, a robot control method, a robot control system, a robot system, a recognition program, and a robot control program for recognizing an object from image data.

The random picking system described in Patent Document 1 is a device that grips and takes out workpieces that have been piled up in bulk by a hand provided at the tip of a robot arm. The random picking system includes an image pickup device for picking up images of randomly stacked works, and a work recognition device for performing pattern matching and selecting a work to be held based on image data of the randomly stacked works.

JP, 2017-170567, A

In the random picking system described in Patent Document 1, when recognizing a specific work (object) from a plurality of types of works (articles), pattern matching must be performed for each of the plurality of types of works. Therefore, in the random picking system, when recognizing a specific work from a plurality of types of work, there is a problem that the time required to recognize the specific work tends to vary.

The present disclosure provides a recognition method, a recognition system, a robot control method, a robot control system, a robot system, a recognition program, and a robot control program in which variations in time required to recognize an object from a plurality of types of articles are unlikely to occur. The purpose is to

A recognition method according to an aspect of the present disclosure includes a first recognition process and a second recognition process. The first recognition processing recognizes the first feature of the object from the image data including the object. In the second recognition processing, a plurality of recognition means corresponding to the characteristics of the plurality of types of articles that can be the target object are prepared. The second recognition processing selects one recognition means from the plurality of recognition means according to at least the first feature recognized in the first recognition processing, and the one recognition means selects one of 2 Recognize features.

A recognition system according to one aspect of the present disclosure includes a first recognition unit and a second recognition unit. The first recognition unit recognizes the first feature of the target object from the image data including the target object. The second recognition unit is provided with a plurality of recognition means corresponding to the characteristics of a plurality of types of articles that can be the target. The second recognizing unit selects one recognizing unit from the plurality of recognizing units in accordance with at least the first feature recognized by the first recognizing unit, and the one recognizing unit identifies the first object 2 Recognize features.

A robot control method according to an aspect of the present disclosure controls a robot that picks an object based on the first feature and the second feature of the object recognized by the recognition method.

A robot control system according to an aspect of the present disclosure includes a control unit that controls a robot that picks the object. The control unit controls the robot based on the first feature and the second feature of the target object recognized by the recognition method.

A robot system according to an aspect of the present disclosure includes a robot that picks the above-described object, and a control unit that controls the robot. The control unit controls the robot based on the first feature and the second feature of the target object recognized by the recognition method.

A recognition program according to an aspect of the present disclosure is a program for causing one or more processors to execute the above recognition method.

A robot control program according to an aspect of the present disclosure is a program for causing one or more processors to execute the above robot control method.

The present disclosure has an advantage that variations in the time required to recognize an object from a plurality of types of articles are unlikely to occur.

FIG. 1 is a schematic diagram showing a configuration of a recognition system used when executing a reading method according to an embodiment of the present disclosure, a robot control system used when executing a robot control method, and a robot system. FIG. 2 is a schematic diagram showing an example of image data obtained by imaging a box containing a plurality of types of parts to which the above recognition system is applied. FIG. 3 is an explanatory diagram of an example of a first recognition process of the above recognition system. FIG. 4 is an explanatory diagram of an example of a second recognition process of the above recognition system. FIG. 5: is a flowchart which shows operation|movement in the recognition system and robot control system same as the above.

(1) Outline The recognition system 10 used when executing the recognition method of the present embodiment is applied to, for example, a robot control system 100 as shown in FIG. The robot control system 100 includes a control unit 13 that controls a robot (manipulator) 3 that picks an object 51. The robot control system 100 is installed in, for example, a factory or a distribution center, and is used when the robot 3 replaces human work. The “object” in the present disclosure refers to an article 5 handled by the robot 3 among a plurality of types of articles 5. The target object 51 changes every moment as the work performed by the robot 3 progresses. For example, in the process of assembling the board 7, since the robot 3 performs a work of picking up components according to the command sequentially given from the control unit 13 and incorporating the parts into the board 7, the target object 51 is different depending on the command. Become. “Incorporating into a board” in the present disclosure means holding a part on the board 7 such as inserting a terminal of a lead part into a through hole of the board 7 or connecting a connector of a flat cable to a connector on the board 7. Or, it means temporarily holding.

The “type” of the plurality of types of articles 5 in the present disclosure refers to, for example, the type of the article 5, the shape of the article 5, the size of the article 5, the pattern of the article 5, the color distribution of the article 5, or the brightness of the article 5. Distribution etc. As an example, if the article 5 is a component to be incorporated in the substrate 7, the type of the article 5 is classified according to the unique function of the article 5, such as a capacitor, a resistor, an inductor, a switch, or an IC (Integrated Circuit) chip. It is a division. That is, the “plurality of types of articles” may include, in addition to the plurality of articles 5 of different types, a plurality of articles 5 of the same type but different in shape, size, or pattern. In the present embodiment, for example, even if the two articles 5 are the same capacitor, if the capacitance values are different, the types of the two articles 5 are different from each other.

“Picking” as referred to in the present disclosure means an operation of selecting and taking out the target object 51 from a plurality of types of articles 5. As an example, when the robot control system 100 is installed in a distribution center, picking refers to a work of selecting and picking up an object 51 to be conveyed from a plurality of types of articles placed in the distribution center. In addition, as an example, when the robot control system 100 is installed in a factory for assembling the board 7, the picking is performed from the box 6 in which a plurality of types of parts (articles 5) in the factory are accommodated to the board 7. This is a work of selecting and taking out a part (object 51) to be incorporated.

The recognition method (recognition system 10) of the present embodiment is used to recognize an object 51 from a plurality of types of articles 5. The recognition method includes a first recognition process and a second recognition process. As shown in FIG. 1, the recognition system 10 includes a first recognition unit 11 and a second recognition unit 12.

The first recognition process (first recognition unit 11) recognizes the first feature of the object 51 from the image data D1 including the object 51. In the present embodiment, the image data D1 is a two-dimensional image of the box 6 obtained by the imaging unit 4 (described later) imaging the box 6 from above the box 6. The box 6 stores a plurality of types of articles 5 including the target object 51. Further, in the present embodiment, the first feature of the target object 51 is the type of the target object 51 (that is, the type, shape, size, pattern of the target object 51, the distribution of colors in the target object 51, or the brightness of the target object 51). And the position of the object 51 in the image data D1.

In the second recognition processing (second recognition unit 12), a plurality of recognition means 121 corresponding to the characteristics of the plurality of types of articles 5 that can be the target object 51 are prepared. The “recognition unit” in the present disclosure is a recognition algorithm for recognizing the feature (here, the second feature) of the object 51 such as pattern matching, shape detection (corner detection or edge detection as an example), and the like. May be included. In addition, the “recognition unit” in the present disclosure may include a template T1 used for one recognition algorithm such as pattern matching, for recognizing a feature (here, second feature) of the target object 51.

Then, the second recognition process (second recognition unit 12) selects one recognition unit 121 from the plurality of recognition units 121 according to at least the first feature recognized in the first recognition process (first recognition unit 11). Then, the second recognizing means 121 recognizes the second feature of the object 51. In the present embodiment, the second feature of the target object 51 is the posture (including the orientation) of the target object 51 in the image data D1.

That is, the second recognition process (second recognition unit 12) selects one recognition algorithm from a plurality of recognition algorithms according to the type of the target object 51, for example. Then, the second recognition process (second recognition unit 12) recognizes the second feature of the target object 51 by the selected recognition algorithm. Further, the second recognition processing (second recognition unit 12) selects one template T1 from the plurality of templates T1 according to the type of the target object 51, for example. Then, the second recognition process (second recognition unit 12) recognizes the second feature of the target object 51 by a recognition algorithm (for example, pattern matching) using the selected template T1. Then, the control unit 13 of the robot control system 100 controls the robot 3 based on the first feature and the second feature of the target object 51 recognized by the above-described recognition method.

As described above, in the present embodiment, the first recognition process (first recognition unit 11) recognizes the first feature of the target object 51, so that the second recognition process (second recognition unit 12) performs the first feature. Only the second feature needs to be recognized based on the features. Therefore, in the present embodiment, compared with the case of recognizing the first feature and the second feature of the target object 51 only by the second recognition process, the time required for the second recognition process for each target object 51 is less likely to vary. As a result, in the present embodiment, there is an advantage that variations in the time required to recognize the target object 51 from the plurality of types of articles 5 are unlikely to occur.

(2) Details Hereinafter, the recognition system 10 and the robot control system 100 of the present embodiment will be described in detail with reference to FIGS. 1 and 2. In the present embodiment, the recognition system 10 is described as being incorporated in the robot control system 100, but may be configured as a system different from the robot control system 100. In addition, in the present embodiment, it is assumed that the robot control system 100 is installed in a board 7 assembly factory. Then, in the present embodiment, it is assumed that the robot 3 picks a component to be incorporated in the substrate 7.

Further, in the present embodiment, it is assumed that the robot control system 100 controls one robot 3, but the robot control system 100 may control a plurality of robots 3. Then, in the present embodiment, it is assumed that the recognition system 10 recognizes the target object 51 from the single box 6 accommodating the plurality of types of articles 5, but the recognition system 10 recognizes the target object 51 from the multiple boxes 6. You may recognize.

(2.1) Robot First, the robot 3 will be described. As shown in FIG. 1, the robot 3 includes an arm 31, a hand 32, and an imaging unit 4. In the present embodiment, the robot 3 is fixed at a predetermined place in the factory, but may be configured to be movable on a rail or to be autonomously driven by wheels. The robot 3 further includes a communication interface for communicating with the communication unit 2 (described later) of the robot control system 100.

The arm 31 is a multi-joint robot arm and is configured to be movable within a predetermined range. In the present embodiment, at least a region where the target object 51 can be taken out of the box 6, a region where the target object 51 can be incorporated in the substrate 7, and a plurality of types of hands 32 are accommodated in the predetermined range. And a region where the hand 32 can be taken out from the box for the other hand.

The hand 32 is attached to the tip of the arm 31, and is configured to take out the target object 51 from the box 6 by sandwiching the target object 51. The hand 32 only needs to be a mode capable of holding the target object 51, and may be, for example, a mode in which the target object 51 is sucked by a nozzle in addition to the mode in which the target object 51 is sandwiched. In this embodiment, one hand 32 selected from a plurality of types of hands 32 can be attached to the tip of the arm 31. The robot 3 executes the work of selecting and attaching one hand 32 from the hand box in accordance with a command from the control unit 13 (described later). When the hand 32 is already attached to the tip of the arm 31, the robot 3 executes the work of exchanging the attached hand 32 and the hand 32 according to the command.

The imaging unit 4 has a solid-state imaging device such as a CCD (Charge-Coupled Device) image sensor. In the present embodiment, the imaging unit 4 is attached to the tip of the arm 31 of the robot 3. The image capturing unit 4 captures an image of the box 6 from above the box 6 according to a command from the control unit 13, thereby capturing a plurality of types of articles 5 contained in the box 6. That is, the imaging unit 4 sets at least the entire opening of the box 6 as the imaging range. The image data D1 obtained by being imaged by the imaging unit 4 is transmitted to the robot control system 100 (and the recognition system 10).

An example of the image data D1 is shown in FIG. As shown in FIG. 2, the image data D1 is a two-dimensional image of the box 6 viewed from above. In FIG. 2, the image data D1 includes an image of a wall (side wall) forming the box 6, but the wall may not be included. That is, it is sufficient that the image data D1 includes at least the entire area (or part) inside the box 6. The image data D1 includes images of a plurality of types of articles 5 contained in the box 6. In the example shown in FIG. 2, the plurality of types of articles 5 include capacitors, resistors, IC chips, switches, and the like. Further, in the example shown in FIG. 2, the plurality of types of articles 5 include capacitors that are the same capacitor but have different shapes, sizes, patterns, or the like. Any of the plurality of types of articles 5 can be the target object 51 picked by the robot 3 in response to a command from the control unit 13.

(2.2) Robot Control System and Recognition System Next, the robot control system 100 (and the recognition system 10) will be described. In the present embodiment, the robot control system 100 is realized by a server located at a remote place apart from the installation place of the robot 3. In the present embodiment, the robot control system 100 is inside the factory, but it may be outside the factory. As shown in FIG. 1, the robot control system 100 includes a main computer 1 and a communication unit 2. In the present embodiment, the communication unit 2 is included in the components of the robot control system 100 (and the recognition system 10), but the communication unit 2 is included in the components of the robot control system 100 (and the recognition system 10). It does not have to be.

The main computer 1 is a computer system whose main configuration is one or more processors and memories as hardware. In this main computer 1, various functions are realized by executing a program recorded in the memory by one or more processors. The program may be pre-recorded in the memory of the main computer 1, may be provided through an electric communication line, or may be provided by being recorded in a non-transitory recording medium such as an optical disk or a hard disk drive that can be read by a computer system. May be.

The main computer 1 has a first recognition unit 11, a second recognition unit 12, and a control unit 13. Specifically, the main computer 1 realizes the respective functions of the first recognition unit 11, the second recognition unit 12, and the control unit 13 by executing a predetermined program by one or more processors.

The first recognition unit 11 executes a first recognition process of recognizing the first feature of the target object 51 from the image data D1 including the target object 51. In the present embodiment, the first recognition processing (first recognition unit 11) recognizes the type of the target object 51 and the position of the target object 51 in the image data D1 as the first feature of the target object 51. .. That is, in the present embodiment, the first feature of the target object 51 includes the position of the target object 51. Then, in the present embodiment, the first recognition processing uses the classifier to recognize the first feature of the target object 51. The classifier is obtained by machine learning using image data of a plurality of types of articles 5 that can be the target object 51 as input data. The classifier may include, for example, a linear classifier such as SVM (Support Vector Machine), a classifier using a neural network, or a classifier generated by deep learning using a multilayer neural network.

In this embodiment, the first recognition unit 11 has a classifier that uses a learned neural network. The learned neural network may include, for example, a CNN (Convolutional Neural Network) or a BNN (Bayesian Neural Network). The first recognition unit 11 is realized by mounting a learned neural network on an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field-Programmable Gate Array).

Here, a learning phase of generating a classifier used in the first recognition processing (first recognition unit 11) by machine learning before using the recognition system 10 will be described. Machine learning in the learning phase is executed, for example, by a learning center. The learning center uses one or more processors to perform machine learning of the neural network used in the first recognition processing. When performing machine learning, the weighting coefficient of the neural network is initialized. The “processor” here may include, for example, a general-purpose processor such as a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit), as well as a dedicated processor specialized for calculation in a neural network.

Machine learning of a neural network is performed using a learning data set. The “learning data set” in the present disclosure is a combination of a learning image input to the input layer of the neural network (hereinafter, simply referred to as “learning image”) and teacher data corresponding to the learning image. It is a set of a plurality of learning data, which is one learning data. The learning image is an image of the article 5. In addition, it is preferable to prepare a large number of learning images in which the position, size, and angle of the article 5 in the image are changed for each article 5. The learning data set includes learning data for all types of articles 5 handled by the recognition system 10 (and the robot control system 100). Further, in the learning data, the teacher data is information on a category into which the article 5 shown in the learning image is classified. In this embodiment, the category information is the type of the article 5.

The one or more processors input the learning image into the input layer of the neural network and execute the calculation for each of the plurality of learning data. Then, the one or more processors execute backpropagation (error backpropagation method) processing using the output values of the plurality of neurons in the output layer of the neural network and the teacher data. Here, the plurality of neurons in the output layer respectively correspond to a plurality of categories. In the backpropagation process, one or more processors update the weighting coefficient of the neural network so as to maximize the output value of the neuron corresponding to the teacher data among the plurality of neurons in the output layer. One or more processors perform backpropagation processing on all learning data to optimize the weighting coefficient of the neural network. This completes the learning of the neural network.

FIG. 3 shows image data D1 that visually expresses an example of the processing result of the first recognition processing (first recognition unit 11). In FIG. 3, among the plurality of types of articles 5 accommodated in the box 6, a part of the articles 5 visible from the opening of the box 6 is illustrated. Therefore, in reality, a large number of articles 5 other than the articles 5 shown in FIG. 3 can be accommodated in the box 6 in a stacked state.

As shown in FIG. 3, among the plurality of articles 5 included in the image data D1, one or more articles 5 recognized by the first recognition process are labeled with a label A1 indicating the type of the article 5. That is, the first recognizing unit 11 recognizes the type of the one or more articles 5 recognized by the first recognizing process, and therefore can perform the process of attaching the label A1. Further, as shown in FIG. 3, the one or more articles 5 recognized by the first recognition processing are surrounded by a frame (bounding box) B1. That is, the first recognizing unit 11 also recognizes the position (coordinates) in the image data D1 for the one or more articles 5 recognized in the first recognizing process, and thus the process of enclosing the frame 5 in the frame B1 is possible. Is.

The second recognizing unit 12 is provided with a plurality of recognizing means 121 corresponding to the characteristics of the plurality of types of articles 5 that can be the object 51. The plurality of recognition units 121 are stored in advance in the memory of the main computer 1, for example. Then, the second recognition unit 12 selects one recognition unit 121 from the plurality of recognition units 121 according to at least the first feature recognized in the first recognition process (first recognition unit 11), and performs one recognition. A second recognition process for recognizing the second feature of the object 51 is executed by the means 121. In the present embodiment, the second recognition process (second recognition unit 12) uses the position of the target object 51 in the image data D1 and the posture (direction of the target object 51 in the image data D1 as the second feature of the target object 51. Including), and is aware. That is, in the present embodiment, the second feature of the target object 51 includes at least one of the position and the posture of the target object 51.

Further, in the present embodiment, the second recognition unit 12 selects and selects the template T1 corresponding to the target object 51 from the plurality of templates T1 according to the type of the target object 51 recognized in the first recognition processing. Pattern matching is performed using the template T1. That is, in the present embodiment, each of the plurality of recognition units 121 is the template T1 used for pattern matching. In the present embodiment, the template T1 is a two-dimensional image of the article 5 when the box 6 is viewed from above, and is used when comparing with the image data D1.

The second recognition process (second recognition unit 12) includes the object 51 corresponding to the command from the control unit 13 among the one or more articles 5 recognized by the first recognition process (first recognition unit 11). Pattern matching is performed by applying the template T1 to the region. Specifically, the second recognition process searches for a set of pixels that match the template T1 by repeatedly moving and rotating the template T1 within a region of the object 51 in the image data D1. That is, in the present embodiment, the second recognition process (second recognition unit 12) uses one recognition unit in the image data D1 based on the first feature recognized in the first recognition process (first recognition unit 11). The range to which 121 (here, the template T1) is applied is limited. The “range” here is, for example, a frame B1 surrounding the object 51.

In the present embodiment, as shown in FIG. 4, the template T1 is composed of a plurality of sub-templates T11. The plurality of sub-templates T11 are images that include mutually different characteristic points of the target object 51. The feature point of the target object 51 is, for example, a corner of the target object 51. In the example shown in FIG. 4, the characteristic points of the capacitor, which is the object 51, are the top of the capacitor and the tips of a pair of terminals of the capacitor. In the second recognition process (second recognition unit 12), the template T1 is moved or rotated without breaking the relative positional relationship of the plurality of sub-templates T11. As described above, in the present embodiment, the template T1 is not a set of pixels corresponding to the entire target object 51, but a set of pixels corresponding to a part of the target object 51. Of course, the template T1 may be composed of a set of pixels corresponding to the entire object 51.

As described above, in the present embodiment, both the first recognition process (first recognition unit 11) and the second recognition process (second recognition unit 12) recognize the position of the target object 51. Then, in the present embodiment, after the rough position of the object 51 in the image data D1 is recognized in the first recognition process, the more accurate position of the object 51 in the image data D1 is recognized in the second recognition process. ing. That is, in the present embodiment, the accuracy of recognizing the position of the target object 51 is higher in the second recognition process than in the first recognition process.

Further, in the present embodiment, the first recognition process (first recognition unit 11) and the second recognition process (second recognition unit 12) are both imaged by the imaging unit 4 before the first recognition process is executed. The image data D1 obtained in step 1 is used. That is, in the present embodiment, the second recognition process uses the same image data D1 as the image data D1 used in the first recognition process to recognize the second feature of the target object 51 by the one recognition unit 121.

The control unit 13 has a function of controlling the robot 3 by giving a command for controlling the robot 3 to the robot 3 via the communication unit 2. As an example, the control unit 13 issues a command to pick the target object 51 designated by the robot 3 and the target object 51 picked by the robot 3 to the substrate 7 according to the schedule of the assembly process of the substrate 7 stored in advance in the memory. And a command to be incorporated into. Note that the control unit 13 communicates with a host system that manages the assembly process of the board 7 to generate a command for controlling the robot 3 in accordance with a command from the host system, and the generated command is used by the robot 3 May be given to. The control unit 13 also has a function of controlling the imaging unit 4 so that the imaging unit 4 images the box 6.

The communication unit 2 includes, for example, a wireless communication module. The communication unit 2 uses a wireless communication module to perform optical wireless communication using light such as infrared rays or visible light as a medium, or wireless communication using radio waves as a medium, via a network such as a wireless LAN (Local Area Network). It communicates with the communication interface of the robot 3. The communication unit 2 may be connected to the network via a communication device such as a router.

(3) Operation Hereinafter, an example of operation of the recognition system 10 and the robot control system 100 of the present embodiment will be described with reference to FIG. In the following description, it is assumed that the recognition system 10 has previously received a command from the control unit 13 of the robot control system 100 to specify the object 51 to be picked by the robot 3. It is assumed that the order of the picking target objects 51 is not determined in advance. That is, all the picking target objects 51 are designated in the command. Then, the robot 3 sequentially picks one target object 51 at a position where it is easy to pick, out of all the designated picking target objects 51.

First, the control unit 13 controls the robot 3 so that the tip of the arm 31 of the robot 3 is located above the box 6. Then, the control unit 13 controls the imaging unit 4 to image the plurality of types of articles 5 contained in the box 6 (S1). The image data D1 obtained by being imaged by the imaging unit 4 is transmitted to the recognition system 10 and the robot control system 100 via the communication interface of the robot 3.

When the image data D1 is acquired, the first recognition unit 11 inputs the image data D1 into the classifier to execute the first recognition process (S2). Accordingly, the first recognition unit 11 recognizes the first feature (here, the type of the article 5) of one or more articles 5 among the plurality of types of articles 5 included in the image data D1. In other words, the first recognition unit 11 recognizes the first feature of the object 51 included in the image data D1 (S3). The processing result of the first recognition unit 11 is given to the second recognition unit 12.

Upon receiving the processing result of the first recognizing unit 11, the second recognizing unit 12 executes the second recognizing process (S4). First, in the second recognition processing, the second recognition unit 12 selects one template T1 from the plurality of templates T1 (recognition means 121) based on the first feature of the target object 51 recognized by the first recognition unit 11. Select (S5). The template T1 selected here is the template T1 corresponding to the type of the target object 51 recognized by the first recognition unit 11. When the plurality of target objects 51 are recognized by the first recognition unit 11, the second recognition unit 12 corresponds to the type of the target objects 51 at the position where the robot 3 easily picks them out of the plurality of target objects 51. The template T1 to be processed is selected.

Next, in the second recognition process, the second recognition unit 12 performs pattern matching by applying the selected template T1 to the image data D1. As described above, in the present embodiment, since the position of the target object 51 in the image data D1 is recognized to some extent in the first recognition processing, the second recognition unit 12 causes the range (including the position of the target object 51 ( For example, pattern matching is performed in the frame B1) shown in FIG. As a result, the second recognition unit 12 recognizes the second feature of the target object 51 (here, the position and orientation of the target object 51) (S6). The processing result of the first recognition unit 11 and the processing result of the second recognition unit 12 are given to the control unit 13.

Then, the control unit 13 controls the robot 3 based on the processing result of the first recognizing unit 11 and the processing result of the second recognizing unit 12, thereby picking the object 51 from the box 6 to the robot 3. (S7). Here, since the type of the target object 51, the position of the target object 51, the posture of the target object 51, and the like are recognized by the first recognition process and the second recognition process, the control unit 13 is based on the recognized information. The robot 31 controls the arm 31 and the hand 32 of the robot 3. As an example, the control unit 13 selects the hand 32 corresponding to the target object 51, or moves and rotates the hand 32 to a position where the target object 51 can be sandwiched.

After that, the control unit 13 controls the robot 3 so that the hand 32 with the object 51 picked is located above the substrate 7. Then, the control unit 13 controls the robot 3 to incorporate the object 51 into the corresponding portion of the substrate 7 (S8). After that, the processes of S1 to S8 described above are repeated for each object 51 to be picked by the robot 3.

As described above, in the present embodiment, the first recognition process (first recognition unit 11) recognizes the first feature of the target object 51, so that the second recognition process (second recognition unit 12) performs the first feature. Only the second feature needs to be recognized based on the features. Therefore, in the present embodiment, compared with the case of recognizing the first feature and the second feature of the target object 51 only by the second recognition process, the time required for the second recognition process for each target object 51 is less likely to vary. As a result, in the present embodiment, there is an advantage that variations in the time required to recognize the target object 51 from the plurality of types of articles 5 are unlikely to occur.

Hereinafter, the above advantages will be described with a comparison with the recognition method of the comparative example. The recognition method of the comparative example is different from the recognition method of the present embodiment in that the first feature and the second feature of the object 51 are recognized only by the second recognition process (here, pattern matching). In the recognition method of the comparative example, one of the plurality of templates is used to try to recognize the position and orientation of the object 51 in the image data D1. In this method, for example, a process of moving and rotating the template (hereinafter, referred to as “search process”) is executed by using one of the four corners of the image data D1 as a starting point until a set of pixels matching the template is found. To do. If no set of pixels that matches the template is found, the template is changed and the search process is repeated. In the recognition method of the comparative example, the time required until a template corresponding to the object 51 is found by repeatedly changing the template and the time required until a set of pixels matching the template are found are different for each object 51. Come on. That is, the recognition method of the comparative example has a problem that the time required to recognize the target object 51 from a plurality of types of articles 5 tends to vary.

On the other hand, in the recognition method of the present embodiment, first, the first recognition process (first recognition unit 11) recognizes the first feature of the object 51 (here, the type of the object 51). Accordingly, the second recognition process (second recognition unit 12) can select the template T1 corresponding to the type of the target object 51 as the template T1 used in the search process. Then, in the second recognition process, it is possible to find the set of pixels that match the template T1 by executing the search process using this template T1. That is, in the present embodiment, it is possible to recognize the second feature of the target object 51 by one search process by passing through the first recognition process. Therefore, in the present embodiment, the time required for the second recognition process corresponds to the time required for finding a set of pixels that match the template T1. Variation in the time required for the second recognition processing for each object 51 is unlikely to occur. Moreover, since the time required for the first recognition processing corresponds to the time required for the image processing for the entire image data D1, there is almost no variation among the objects 51. As a result, in the present embodiment, there is an advantage that variations in the time required to recognize the target object 51 from the plurality of types of articles 5 are unlikely to occur.

Further, in the present embodiment, since the position of the target object 51 is recognized as the first feature of the target object 51 in the first recognition process, one recognition means 121 (here, the target object 51 in the second recognition process). It is possible to limit the range to which the template T1) corresponding to is applied. Therefore, the present embodiment has an advantage that the time required for the second recognition process can be easily shortened as compared with the case where the first feature and the second feature of the object 51 are recognized only by the second recognition process. is there.

In addition, a mode in which the first feature and the second feature of the object 51 are recognized only by the first recognition process (first recognition unit 11) is also conceivable. However, in this mode, although the position of the object 51 in the image data D1 can be recognized, a deviation from the actual position of the object 51 may occur. Therefore, in this aspect, in the work of incorporating the target object 51 into the substrate 7 as in the present embodiment, there is a possibility that sufficient accuracy for picking the target object 51, which is a minute component, cannot be obtained.

Therefore, in the recognition method of the present embodiment, in addition to the first recognition process, the second recognition process having higher recognition accuracy of the position of the target object 51 than the first recognition process is executed. As a result, in the present embodiment, the accuracy of recognizing the position of the target object 51 in the image data D1 is improved as compared with the mode in which the first feature and the second feature of the target object 51 are recognized only by the first recognition processing. You can As a result, the recognition method of the present embodiment has an advantage that it is easy to obtain sufficient accuracy for picking the target object 51 even when the target object 51 is a minute component.

(4) Modified Example The above-described embodiment is only one of the various embodiments of the present disclosure. The above-described embodiment can be variously modified according to the design and the like as long as the object of the present disclosure can be achieved. Further, the same function as that of the recognition system 10 may be embodied by a (computer) program, a non-transitory recording medium recording the program, or the like. Further, the same function as the robot control system 100 may be embodied by a robot control method, a (computer) program, a non-transitory recording medium recording the program, or the like.

The robot control method according to one aspect controls the robot 3 that picks the target object 51 based on the first feature and the second feature of the target object 51 recognized by the recognition method described above.

The recognition program according to one aspect is a program for causing one or more processors to execute the above-described recognition method.

A robot control program according to one aspect is a program for causing one or more processors to execute the robot control method described above.

Hereinafter, modifications of the above-described embodiment will be listed. The modified examples described below can be applied by appropriately combining with the above-described embodiments.

The recognition system 10 (or the robot control system 100) according to the present disclosure includes, for example, a computer system in the main computer 1 or the like. The computer system mainly includes a processor as a hardware and a memory. When the processor executes the program recorded in the memory of the computer system, the function as the recognition system 10 (or the robot control system 100) in the present disclosure is realized. The program may be pre-recorded in the memory of the computer system, may be provided through an electric communication line, or may be recorded in a non-transitory recording medium such as a memory card, an optical disk, a hard disk drive, which can be read by the computer system. May be provided. The processor of the computer system is composed of one or a plurality of electronic circuits including a semiconductor integrated circuit (IC) or a large scale integrated circuit (LSI). The integrated circuit such as IC or LSI referred to here is called differently depending on the degree of integration, and includes an integrated circuit called system LSI, VLSI (Very Large Scale Integration), or ULSI (Ultra Large Scale Integration). Further, an FPGA (Field-Programmable Gate Array), which is programmed after the LSI is manufactured, or a logic device capable of reconfiguring the connection relation inside the LSI or reconfiguring the circuit section inside the LSI, should also be adopted as a processor. You can The plurality of electronic circuits may be integrated in one chip, or may be distributed and provided in the plurality of chips. The plurality of chips may be integrated in one device or may be distributed and provided in the plurality of devices. The computer system referred to here includes a microcontroller having one or more processors and one or more memories. Therefore, the microcontroller is also composed of one or a plurality of electronic circuits including a semiconductor integrated circuit or a large scale integrated circuit.

Further, it is not essential for the recognition system 10 (or the robot control system 100) that the plurality of functions of the recognition system 10 (or the robot control system 100) are integrated in one housing. The components of the recognition system 10 (or the robot control system 100) may be distributed and provided in a plurality of housings. Furthermore, at least a part of the functions of the recognition system 10 (or the robot control system 100) may be realized by, for example, a server device and a cloud (cloud computing).

In the above-described embodiment, the communication unit 2 may communicate with the communication interface of the robot 3 via a network such as a wired LAN, for example, by wired communication. That is, the communication method between the robot control system 100 (and the recognition system 10) and the robot 3 may be a wireless communication method or a wire communication method.

In the above embodiment, the recognition system 10 and the robot control system 100 are realized by the server, but the invention is not limited to this, and the recognition system 10 and the robot control system 100 may be provided in the robot 3. In this case, the communication unit 2 is unnecessary.

In the above-described embodiment, the robot 3 includes the arm 31 and the hand 32, but the robot 3 is not limited to this and may include only the hand 32. That is, the robot control system 100 may be configured to control the hand 32 as the robot 3.

Although the imaging unit 4 is provided in the robot 3 in the above-described embodiment, the present invention is not limited to this. For example, the imaging unit 4 may be installed in a place other than the robot 3 in the factory and at a position where the entire box 6 can be imaged in a plan view.

In the above embodiment, the robot control system 100 may be a robot system further including the robot 3. That is, the robot system includes the robot 3 that picks the object 51 and the control unit 13 that controls the robot 3. Then, the control unit 13 controls the robot 3 based on the first feature and the second feature of the target object 51 recognized by the above-described recognition method.

In the above-described embodiment, each of the plurality of recognition units 121 prepared in the second recognition process (second recognition unit 12) may be a recognition algorithm for recognizing the second feature of the target object 51. For example, in the second recognition processing, a recognition algorithm such as pattern matching and shape detection (corner detection or edge detection as an example) may be prepared as the plurality of recognition means 121. Then, the second recognition processing uses pattern matching as the recognition means 121 or shape detection as the recognition means 121 according to the first feature of the object 51 recognized by the first recognition processing (first recognition unit 11). It may be used as.

In the above embodiment, the image data D1 used in the first recognition process (first recognition unit 11) and the image data D1 used in the second recognition process (second recognition unit 12) may be different from each other. Good. For example, before the second recognition process is performed, the robot 3 may be controlled to cause the image capturing unit 4 to capture an image so that the target object 51 is located at the center of the image capturing range. In this case, in the image data D1 used in the second recognition process, since the target object 51 is not located at the periphery of the imaging range, the image of the target object 51 is less likely to be distorted, and the second feature of the target object 51 can be easily recognized. There is an advantage that.

In the above embodiment, the position of the target object 51 may be recognized by only one of the first recognition process (first recognition unit 11) and the second recognition process (second recognition unit 12). As an example, the type of the target object 51 may be recognized in the first recognition process, and the position of the target object 51 may be recognized in the second recognition process.

In the above-described embodiment, it is assumed that the target object 51 cannot be recognized in the first recognition processing because the target object 51 is buried in the box 6, for example. In such a case, for example, the contents of the box 6 may be stirred by the hand 32 of the robot 3 and the first recognition process may be executed again after the stirring. This mode has an advantage that the accuracy of recognizing the target object 51 in the first recognition process can be improved.

In the above-described embodiment, the first recognition process recognizes the first feature of the target object 51 using the classifier using the learned neural network, but the present invention is not limited to this. For example, the first recognition process may recognize the first feature of the object 51 by comparing the brightness of each pixel of the image data D1 with a threshold value.

In the above-described embodiment, the first recognition process and the second recognition process may be executed in consideration of other information. As another example, if the article 5 is a part produced in a factory, the different information is a production time zone of the part or the like. That is, when the first recognition process and the second recognition process are executed, the type of the article 5 that can be the target object 51 is produced in this production time zone by referring to the production time zone of the component as separate information. It is possible to narrow down to parts.

In the above-described embodiment, the recognition system 10 is used to recognize the object 51 picked by the robot 3 from the plurality of types of articles 5, but the invention is not limited to this application. For example, the recognition system 10 may be used to recognize the target object 51 to which the picked parts of the robot 3 are attached from the plurality of types of articles 5 on the substrate 7. As a specific example, if the component picked up by the robot 3 is a flat cable, the recognition system 10 sets the connector (target 51) to which the connector of the flat cable is connected to a plurality of types of components (articles) on the substrate 7. 5) may be used for recognition. In this case, the control unit 13 may acquire the image data D1 by controlling the imaging unit 4 so that the substrate 7 is imaged from above the substrate 7.

(Summary)
As described above, the recognition method according to the first aspect includes the first recognition process and the second recognition process. The first recognition process recognizes the first feature of the object (51) from the image data (D1) including the object (51). In the second recognition processing, a plurality of recognition means (121) corresponding to the characteristics of the plurality of types of articles (5) that can be the target object (51) are prepared. The second recognition processing selects at least one recognition means (121) from the plurality of recognition means (121) according to at least the first feature recognized in the first recognition processing, and the one recognition means (121) selects the target. Recognize the second feature of the object (51).

According to this aspect, there is an advantage that the time required for recognizing the object (51) from the plurality of types of articles (5) is unlikely to vary.

In the recognition method according to the second aspect, in the first aspect, the second feature includes at least one of the position and the posture of the object (51).

According to this aspect, it is possible to recognize not only what kind of object the object (51) is, but also how the object (51) is arranged. is there.

In the recognition method according to the third aspect, in the first or second aspect, the first feature includes the position of the object (51).

According to this aspect, it is possible to perform the second recognition process by narrowing down to the position of the target object (51) recognized in the first recognition process, so that the time required for the second recognition process can be shortened. There is an advantage that it is easy.

In the recognition method according to the fourth aspect, in any one of the first to third aspects, the first recognition process and the second recognition process both recognize the position of the target object (51). The accuracy of recognizing the position of the object (51) is higher in the second recognition processing than in the first recognition processing.

According to this aspect, there is an advantage that the accuracy of recognizing the position of the target object (51) is easily improved as compared with the case where the position of the target object (51) is recognized only by the first recognition processing. Further, according to this aspect, there is an advantage that the time required for the second recognition processing can be easily shortened as compared with the case where the position of the object (51) is recognized only by the second recognition processing.

In the recognition method according to the fifth aspect, in any one of the first to fourth aspects, the second recognition process uses the same image data (D1) as the image data (D1) used in the first recognition process. , The second recognition means (121) recognizes the second feature of the object (51).

According to this aspect, when the image data (D1) is acquired by one image capturing, as compared with the case where the image data (D1) different from each other in the first recognition process and the second recognition process is captured and acquired. I'm done. Therefore, according to this aspect, there is an advantage that it is easy to shorten the time required to acquire the image data (D1).

In the recognition method according to the sixth aspect, in the first recognition process according to any one of the first to fifth aspects, the first recognition processing is performed by using the image data of a plurality of types of articles (5) that can be the target object (51) as input data. The first feature of the object (51) is recognized using the classifier obtained by learning.

According to this aspect, there is an advantage that the first feature of the object (51) can be easily recognized, as compared with the case where the classifier is not used.

In the recognition method according to the seventh aspect, in any one of the first to sixth aspects, each of the plurality of recognition means (121) is a template (T1) used for pattern matching.

According to this aspect, the second feature of the target object (51) can be recognized by using the template (T1) appropriate for the target object (51), and thus the second feature of the target object (51) can be recognized. It has the advantage of being easier to recognize.

In the recognition method according to an eighth aspect, in any one of the first to sixth aspects, each of the plurality of recognition means (121) is a recognition algorithm for recognizing the second feature of the object (51). is there.

According to this aspect, the second feature of the target object (51) can be recognized by using an appropriate recognition algorithm corresponding to the target object (51), and thus the second feature of the target object (51) can be recognized. There is an advantage that it becomes easier.

In the recognition method according to the ninth aspect, in any one of the first to eighth aspects, the second recognition process is performed on the image data (D1) based on the first feature recognized in the first recognition process. The range to which the recognition means (121) is applied is limited.

According to this aspect, there is an advantage that the time required for the second recognition processing can be easily shortened as compared with the case where one recognition means (121) is applied to the entire image data (D1).

The recognition system (10) according to the tenth aspect includes a first recognition unit (11) and a second recognition unit (12). The first recognition unit (11) recognizes the first feature of the target object (51) from the image data (D1) including the target object (51). The second recognition unit (12) is provided with a plurality of recognition means (121) corresponding to the characteristics of the plurality of types of articles (5) that can be the target object (51). The second recognition unit (12) selects one recognition unit (121) from the plurality of recognition units (121) according to at least the first feature recognized by the first recognition unit (11), and performs one recognition. The second feature of the object (51) is recognized by the means (121).

According to this aspect, there is an advantage that the time required for recognizing the object (51) from the plurality of types of articles (5) is unlikely to vary.

A robot control method according to an eleventh aspect is based on the first feature and the second feature of the object (51) recognized by the recognition method according to any one of the first to ninth aspects. Control a robot (3) for picking.

According to this aspect, there is an advantage that the time required for recognizing the object (51) from the plurality of articles (5) hardly varies.

The robot control system (100) according to the twelfth aspect includes a control unit (13) that controls a robot (3) that picks an object (51). The control unit (13) controls the robot (3) based on the first feature and the second feature of the object (51) recognized by any of the first to ninth recognition methods.

According to this aspect, there is an advantage that the time required for recognizing the object (51) from the plurality of types of articles (5) is unlikely to vary.

The robot system according to the thirteenth aspect includes a robot (3) that picks an object (51) and a control unit (13) that controls the robot (3). The control unit (13) controls the robot (3) based on the first feature and the second feature of the object (51) recognized by the recognition method according to any one of the first to ninth aspects.

According to this aspect, there is an advantage that the time required to recognize the target object (51) from the plurality of types of articles (5) hardly varies.

The recognition program according to the fourteenth aspect is a program for causing one or more processors to execute the recognition method according to any one of the first to ninth aspects.

According to this aspect, there is an advantage that the time required for recognizing the object (51) from the plurality of types of articles (5) is unlikely to vary.

The robot control program according to the fifteenth aspect is a program for causing one or more processors to execute the robot control method according to the eleventh aspect.

According to this aspect, there is an advantage that the time required for recognizing the object (51) from the plurality of types of articles (5) is unlikely to vary.

The methods according to the second to ninth aspects are not essential for the recognition method and can be omitted as appropriate.

10 recognition system 11 1st recognition part 12 2nd recognition part 121 recognition means 3 robot 5 article 51 object 100 robot control system D1 image data T1 template

Claims (15)

A first recognition process for recognizing a first feature of the object from image data including the object;
A plurality of recognition means corresponding to the characteristics of a plurality of types of articles that can be the object are prepared, and one recognition means from the plurality of recognition means is at least according to the first characteristics recognized in the first recognition processing. And a second recognition process for recognizing the second feature of the object by the one recognition means.
Recognition method. The second feature includes at least one of a position and a posture of the object,
The recognition method according to claim 1. The first feature includes a position of the object,
The recognition method according to claim 1 or 2. The first recognition process and the second recognition process both recognize the position of the object,
The accuracy of recognizing the position of the object is higher in the second recognition process than in the first recognition process,
The recognition method according to claim 1. In the second recognition process, the second feature of the object is recognized by the one recognition unit using the same image data as the image data used in the first recognition process,
The recognition method according to any one of claims 1 to 4. The first recognition processing recognizes the first feature of the target object using a classifier obtained by machine learning using image data of a plurality of types of articles that can be the target object as input data.
The recognition method according to claim 1. Each of the plurality of recognition means is a template used for pattern matching,
The recognition method according to claim 1. Each of the plurality of recognition means is a recognition algorithm for recognizing the second feature of the object,
The recognition method according to claim 1. The second recognition process limits a range to which the one recognition unit is applied in the image data, based on the first feature recognized in the first recognition process,
The recognition method according to claim 1. A first recognition unit for recognizing a first feature of the object from image data including the object;
A plurality of recognizing means corresponding to the characteristics of a plurality of types of articles that can be the object are prepared, and one recognizing means from the plurality of recognizing means according to at least the first feature recognized by the first recognizing unit. And a second recognition unit that recognizes the second feature of the object by the one recognition unit.
Recognition system. Controlling a robot that picks the object based on the first feature and the second feature of the object recognized by the recognition method according to any one of claims 1 to 9;
Robot control method. A control unit for controlling a robot for picking the object,
The control unit controls the robot based on the first feature and the second feature of the object recognized by the recognition method according to claim 1.
Robot control system. A robot for picking the object,
A control unit for controlling the robot,
The control unit controls the robot based on the first feature and the second feature of the object recognized by the recognition method according to claim 1.
Robot system. On one or more processors,
For executing the recognition method according to any one of claims 1 to 9,
Recognition program. On one or more processors,
For executing the robot control method according to claim 11,
Robot control program.

Images