WO2024024090A1

WO2024024090A1 - Component count device and robot system

Info

Publication number: WO2024024090A1
Application number: PCT/JP2022/029305
Authority: WO
Inventors: 航有田
Original assignee: ヤマハ発動機株式会社
Priority date: 2022-07-29
Filing date: 2022-07-29
Publication date: 2024-02-01

Abstract

A component count device (100) comprises: an oscillation unit (20) which oscillates a housing unit (2) which houses a plurality of components (1); a photographing unit (30) which continuously photographs the components in the housing unit being oscillated by the oscillation unit; and an information processing unit (41) which, on the basis of photographic images (50) sequentially photographed by the photographing unit, tracks the components being moved by oscillation and which counts the number of the components in the housing unit.

Description

Parts counting equipment and robot systems

The present invention relates to a parts counting device and a robot system, and particularly to a parts counting device and a robot system that count parts.

Conventionally, a parts counting device that counts parts is known (for example, see Patent Document 1).

The above Patent Document 1 discloses a parts counting device that counts parts. This parts counting device is configured to count parts based on a photographed image of parts arranged on a sheet taken by a photographing device. Further, this parts counting device is configured to display, when parts on a sheet overlap, an image processed to enable identification of the portion where overlapping parts are detected. If parts overlap, it is not possible to accurately count the parts. Therefore, the user visually recognizes the overlap of parts on the displayed image, manually eliminates the overlap of parts on the sheet, and causes the parts counting device to perform the counting process again.

JP2019-57211A

However, in the parts counting device described in Patent Document 1, the user manually eliminates overlapping parts and re-executes the counting process, which is disadvantageous in that it requires time and effort on the part of the user. Therefore, there is a problem in that it is difficult to accurately count parts while saving the user's effort.

This invention was made to solve the above problems, and one object of the invention is to provide a parts counting device and a robot that can count parts with high accuracy while saving the user's effort. The goal is to provide a system.

A parts counting device according to a first aspect of the present invention includes: a vibrating part that vibrates a housing part that accommodates a plurality of parts; a photographing part that continuously photographs parts in the housing part vibrated by the vibrating part; The apparatus includes an information processing unit that tracks parts that move due to vibration based on a photographed image and counts the number of parts in the storage section. Note that in this specification, vibration is a broad concept that includes shaking.

As described above, the parts counting device according to the first aspect of the present invention is provided with a vibrating part that vibrates the housing part that accommodates a plurality of parts. Thereby, the parts can be moved by causing the vibrating part to vibrate, so there is no need for the user to manually eliminate overlapping parts. As a result, the user's effort can be saved. In addition, by tracking parts that move due to vibration based on images taken sequentially by the imaging unit, parts can be counted while creating a situation with high counting accuracy (a situation where parts are not placed close to each other). Therefore, parts can be counted with high accuracy. As a result, parts can be counted accurately while saving the user's effort.

In the parts counting device according to the first aspect, preferably, the information processing section creates a likelihood map of the parts included in the photographed images based on the photographed images sequentially photographed by the photographing section, and calculates the likelihood map of the parts included in the photographed images. Configured to track parts based on the map. With this configuration, parts can be recognized with high accuracy based on the likelihood map, and therefore parts can be tracked with high accuracy.

In this case, preferably, the information processing unit estimates the position and orientation of the part based on the created likelihood map, and extracts feature points of the part based on the estimated position and orientation of the part. The part is configured to be tracked by tracking feature points. With this configuration, the position and orientation of the component can be estimated with high accuracy based on the likelihood map, so the feature points of the component can be extracted with high accuracy. As a result, by tracking the accurately extracted feature points, the parts can be accurately tracked.

In the configuration for extracting feature points, preferably, the information processing unit sets an extraction region for extracting feature points based on the estimated position and orientation of the part, and extracts the feature points from within the extraction region. It is configured. With this configuration, feature points can be extracted from within the extraction region where there is a high possibility that feature points exist, so feature points of parts can be extracted with higher accuracy. As a result, the parts can be tracked more accurately by tracking the extracted feature points with more accuracy.

In this case, preferably, the information processing unit obtains the degree of coincidence between the feature points extracted from within the extraction region and the feature points in the reference image, and selects the degree of coincidence among the feature points extracted from within the extraction region. It is configured to determine a feature point that is equal to or greater than a threshold value as a feature point to be tracked. With this configuration, by tracking feature points with a high degree of coincidence, parts can be tracked with even higher accuracy.

In the configuration for extracting feature points, preferably, the information processing unit is configured to count the parts in the storage unit by counting the parts whose feature points are tracked. With this configuration, by counting the parts whose feature points are tracked, it is possible to avoid counting the same part twice, and therefore it is possible to count the parts with higher accuracy.

In this case, preferably, the information processing unit counts the parts for each type of parts by extracting the feature points of the parts for each type of parts and counting the number of parts whose feature points are tracked for each type. It is configured. With this configuration, even when a plurality of types of parts are present in the storage section, the parts can be accurately counted for each type of part.

In the configuration for extracting the feature points, preferably, the information processing unit compares the likelihood of the currently created likelihood map with the likelihood of the likelihood map created a predetermined number of times ago for each part region, Among the compared component regions, for the component region for which the likelihood map created this time has the maximum likelihood, the likelihood map created this time is used to extract feature points, and the likelihood map created this time is used to extract the feature points. For component regions for which the likelihood map does not have the maximum likelihood, the likelihood map that has a higher likelihood than the currently created likelihood map among the likelihood maps created a predetermined number of times ago is used to extract feature points. It is composed of With this configuration, it is possible to always track extracted feature points with a high likelihood.

In the parts counting device according to the first aspect, preferably, the information processing section is configured to track parts using a machine learning model generated by machine learning. With this configuration, parts can be easily and accurately tracked using a machine learning model.

In the parts counting device according to the first aspect, the information processing unit is preferably configured to notify the user when the number of counted parts does not satisfy a predetermined number. With this configuration, if the number of counted parts does not meet the predetermined number (if the number of parts is greater or less than the predetermined number), the user can easily confirm that the number of parts does not satisfy the predetermined number. Since it is possible to know the number of parts, it is possible to easily take measures so that the number of parts satisfies the predetermined number.

In the parts counting device according to the first aspect, preferably, the vibrating section includes a vibrator that vibrates the housing section in the horizontal direction. With this configuration, the parts can be easily moved by vibrating them horizontally, so it is easy to create a situation with high counting accuracy (a situation where parts are not placed close to each other). .

A robot system according to a second aspect of the present invention includes a robot that transfers and loads parts in a storage unit or a robot that transfers and loads parts into the storage unit, and a parts counting device that counts the parts in the storage unit, The parts counting device includes a vibrating part that vibrates the housing part, a photographing part that continuously photographs the parts inside the housing part that are vibrated by the vibrating part, and a part counting apparatus that detects the parts that are moved by vibration based on the photographed images that are sequentially photographed by the photographing part. and an information processing unit that tracks the parts and counts the parts in the storage section.

In the robot system according to the second aspect of the present invention, as described above, a vibrating section is provided that vibrates the housing section that accommodates a plurality of parts. Thereby, the parts can be moved by causing the vibrating part to vibrate, so there is no need for the user to manually eliminate overlapping parts. As a result, the user's effort can be saved. In addition, by tracking parts that move due to vibration based on images taken sequentially by the imaging unit, parts can be counted while creating a situation with high counting accuracy (a situation where parts are not placed close to each other). Therefore, parts can be counted with high accuracy. As a result, parts can be counted accurately while saving the user's effort.

According to the present invention, as described above, parts can be counted with high accuracy while saving the user's effort.

FIG. 1 is a schematic diagram showing a parts counting device according to one embodiment. FIG. 3 is a diagram for explaining a parts counting process of a parts counting device according to an embodiment. FIG. 3 is a diagram for explaining feature point extraction processing in parts counting processing according to an embodiment. FIG. 3 is a diagram for explaining a machine learning model used in feature point extraction processing according to an embodiment. 5 is a flowchart for explaining control processing related to parts counting according to an embodiment. 5 is a flowchart for explaining image processing of captured images according to an embodiment. FIG. 1 is a schematic diagram showing a configuration example of a robot system including a parts counting device according to an embodiment. 12 is a flowchart for explaining control processing on the imaging unit side regarding component counting according to a modification of the embodiment. 12 is a flowchart for explaining control processing on the information processing unit side regarding parts counting according to a modified example of one embodiment.

Hereinafter, embodiments embodying the present invention will be described based on the drawings.

First, with reference to FIG. 1, the configuration of a parts counting device 100 according to an embodiment will be described.

(Configuration of parts counting device)
As shown in FIG. 1, the parts counting device 100 is a device that counts a plurality of parts 1 housed in a housing section 2. As shown in FIG. The accommodating part 2 is, for example, a tray. The component 1 is, for example, a screw or a washer. In an assembly line for industrial products, storage units 2 containing parts 1 are sometimes supplied for assembly of industrial products. For example, the parts counting device 100 counts the parts 1 stored in the storage part 2 before being supplied to an assembly line of industrial products, thereby determining whether the correct number of parts 1 is stored in the storage part 2. It is used to test whether The parts counting device 100 can also be said to be an inspection device.

As shown in FIG. 1, the parts counting device 100 includes a mounting section 10, a vibration section 20, an imaging section 30, and a control section 40.

The mounting section 10 is configured such that the accommodating section 2 can be mounted thereon. The mounting section 10 is a table on which the accommodating section 2 can be placed. Further, the mounting section 10 is provided with a vibrating section 20.

The vibrating section 20 is configured to vibrate the housing section 2. Specifically, the vibrating section 20 includes a first vibrator 21, a second vibrator 22, and a diaphragm 23. The first vibrator 21 is configured to horizontally vibrate the housing section 2 mounted on the mounting section 10. Specifically, the first vibrator 21 is configured to vibrate the housing section 2 mounted on the mounting section 10 in the front-rear direction and the left-right direction (two directions substantially orthogonal to each other in a horizontal plane). The second vibrator 22 is configured to vibrate the housing section 2 mounted on the mounting section 10 in the vertical direction. The diaphragm 23 is configured to be vibrated when each of the first vibrator 21 and the second vibrator 22 vibrates. When the diaphragm 23 is vibrated, the component 1 inside the housing section 2 is vibrated. Note that the first vibrator 21 is an example of a "vibrator" in the claims.

The photographing section 30 is a camera that continuously photographs the component 1 inside the housing section 2 that is vibrated by the vibrating section 20. The photographing unit 30 is configured to perform continuous photographing of moving images or still images similar to moving images by photographing at a predetermined frame rate. The photographing section 30 is arranged above the mounting section 10 (that is, above the component 1 that is the subject). Further, the photographing section 30 is provided with an illumination section 31 that irradiates illumination light toward the component 1 that is a subject during photographing.

The control unit 40 is configured to control the operation of the parts counting device 100. Specifically, the control section 40 includes an information processing section 41, a device control section 42, and a storage section 43. The information processing unit 41 includes a processor such as a CPU (central processing unit), and processes various information. In the present embodiment, the information processing unit 41 acquires a captured image 50 (see FIG. 2) of the component 1 captured by the imaging unit 30, and counts the components 1 in the storage unit 2 based on the acquired captured image 50. The system is configured to perform the processing that follows. The device control unit 42 includes a processor such as a CPU (central processing unit), and controls the operations of the vibration unit 20 and the imaging unit 30. When counting the parts 1 in the storage section 2, under the control of the device control section 42, the vibration section 20 performs a vibration operation, and the photographing section 30 performs a photographing operation. The storage unit 43 includes a rewritable nonvolatile memory such as a flash memory, and is configured to be able to store various types of information.

(Configuration of parts counting process)
As shown in FIGS. 2a and 2b, the device control section 42 moves the component 1 inside the storage section 2 by vibrating the storage section 2 with the vibration section 20, and moves the component 1 inside the storage section 2 using the imaging section 30. It is configured to shoot continuously. The information processing section 41 is configured to acquire a photographed image 50 by the photographing section 30 and to count the parts 1 in the storage section 2 based on the acquired photographed image 50.

Here, in this embodiment, as shown in FIGS. 2b to 2e, the information processing unit 41 tracks the component 1 that moves due to vibration based on the photographed images 50 sequentially photographed by the photographing unit 30, and also tracks the component 1 that moves due to vibration and It is configured to count parts 1 in section 2. Specifically, the information processing section 41 is configured to create a likelihood map 60 of the component 1 included in the photographed image 50 based on the photographed images 50 sequentially photographed by the photographing section 30. The likelihood map 60 is a map representing the likelihood distribution of the component 1 in the captured image 50. The information processing unit 41 uses a method using template matching or a method of extracting feature amounts to determine the degree to which the region including each pixel of the photographed image 50 corresponds to the part showing the part 1. The configuration is configured to create a likelihood map 60 by calculating the likelihood expressed. Usually, when the parts 1 are adjacent to each other, the likelihood is calculated to be low, and when the parts 1 are not adjacent to each other, the likelihood is calculated to be high. Furthermore, the information processing unit 41 is configured to track the component 1 based on the created likelihood map 60.

Further, in the present embodiment, as shown in FIGS. 2 and 3, the information processing unit 41 estimates the position and orientation (orientation) of the component 1 based on the created likelihood map 60, and estimates the position and orientation of the component 1. The component 1 is tracked by extracting feature points 1a of the component 1 based on the position and orientation of the component 1, and tracking the extracted feature points 1a. Specifically, as shown in FIGS. 2c, 2d, and 3a, the information processing unit 41 selects a high likelihood whose likelihood is equal to or higher than a threshold value (for example, 90% or higher) based on the created likelihood map 60. The high likelihood area 61 is acquired, and the center position 62 of the acquired high likelihood area 61 is estimated as the center position of the component 1. Further, as shown in FIG. 3A, the information processing unit 41 is configured to estimate the orientation of the component 1 based on the shape of the acquired high likelihood region 61. For example, when the component 1 has an elongated shape such as a screw, the information processing unit 41 is configured to estimate the orientation of the component 1 on the assumption that the longitudinal direction of the high likelihood region 61 is the longitudinal direction of the component 1. .

Further, in this embodiment, as shown in FIGS. 3b and 3c, the information processing unit 41 sets an extraction region 70 for extracting the feature points 1a based on the estimated position and orientation of the component 1, and It is configured to extract feature points 1a from within. The extraction area 70 is an area where there is a high possibility that the feature point 1a of the component 1 exists. Specifically, the extraction area 70 is an area representing the outer shape (contour) of the component 1. Information representing the external shape of the component 1 is stored in the storage section 43 in advance. The information processing unit 41 arranges the component 1 at the estimated position and in the estimated orientation based on the information representing the external shape of the component 1 stored in the storage unit 43 and the estimated position and orientation of the component 1. The extraction area 70 is set based on this assumption.

Further, in the present embodiment, as shown in FIG. 3d, the information processing unit 41 obtains the degree of coincidence between the feature point 1a extracted from within the extraction region 70 and the feature point 1a within the reference image 80, and Among the feature points 1a extracted from within 70, the feature point 1a whose matching degree is equal to or higher than a threshold value is determined as the feature point 1a to be tracked (final feature point 1a). The reference image 80 is an image representing the component 1, and the feature point 1a is set in advance in the reference image 80. As the feature points 1a of the reference image 80, for example, feature points 1a near the outer shape such as corners and edges are set. Thereby, it is possible to set a feature point 1a that holds information such as thickness and length that is highly related to the type of component 1. One or more reference images 80 are stored in the storage section 43 in advance.

Further, in this embodiment, as shown in FIG. 2e, the information processing unit 41 is configured to count the parts 1 in the storage unit 2 by counting the parts 1 tracking the feature point 1a. . In the example shown in FIG. 2e, the number of parts 1 that tracks the feature point 1a (that is, the detected part 1) is one, so the number of parts 1 counted is one. While moving the component 1 inside the storage section 2 by vibrating the storage section 2 with the vibration section 20, the photographing section 30 continuously photographs the component 1 inside the storage section 2, and the photographed images 50 are sequentially acquired. By performing image processing using the image processing method, the number of parts 1 whose feature points 1a are tracked gradually increases. The information processing section 41 is configured to continue the above-described component counting process until the counted number of components 1 reaches a predetermined number (the correct number to be accommodated in the storage section 2).

Furthermore, in this embodiment, the information processing unit 41 is configured to notify the user when the counted number of parts 1 does not satisfy a predetermined number. Specifically, if the number of counted parts 1 is less than a predetermined number even after a predetermined time has elapsed, or if the number of counted parts 1 is greater than a predetermined number, Configured to notify the user. The information processing unit 41 is configured to notify the user that the counted number of parts 1 does not satisfy a predetermined number by displaying the information on the display unit. For example, when the parts counting device 100 includes a display section, the information processing section 41 displays a notification on the display section of the parts counting device 100; for example, when the parts counting device 100 does not include a display section, the information processing section 41 displays a notification on the external display. Display a notification in the section.

Further, in the present embodiment, the information processing unit 41 calculates the likelihood (accuracy) of the likelihood map 60 created this time (the latest likelihood map 60) and the likelihood of the likelihood map 60 created a predetermined number of times ago. (accuracy) for each part area (area with high likelihood such as high likelihood area 61), and among the compared part areas, the part area where the likelihood is maximum in the likelihood map 60 created this time. The likelihood map 60 created this time is used to extract the feature points 1a, and among the compared component regions, for the component regions for which the likelihood does not have the maximum in the likelihood map 60 created this time, the likelihood map 60 created this time is used to extract the feature points 1a. Among the likelihood maps 60 thus created, the likelihood map 60 having a higher likelihood than the likelihood map 60 created this time is used for extracting the feature point 1a. For example, for a component region that does not have the maximum likelihood in the likelihood map 60 created this time among the compared component regions, the information processing unit 41 selects the component region that has the highest likelihood among the likelihood maps 60 created a predetermined number of times before. The configuration is such that the likelihood map 60 having a high degree of accuracy is used for extracting the feature points 1a.

Here, for convenience, FIGS. 2 and 3 illustrate an example in which one type of component 1 is housed in the housing section 2, but a plurality of types of components 1 are housed in the housing section 2. In some cases. Therefore, in the present embodiment, when there are multiple types of parts 1 in the storage unit 2, the information processing unit 41 extracts the feature points 1a of the parts 1 for each type of parts 1, and extracts the feature points 1a from the feature points 1a. By counting the parts 1 to be tracked for each type, the parts 1 are counted for each type of parts 1. In this case, the information processing unit 41 creates a separate likelihood map 60 for each type of component 1, or creates one likelihood map 60 that includes likelihood information for each type of component 1. It is composed of Although a detailed explanation will be omitted, the information processing unit 41 is configured to extract the feature points 1a of the component 1 for each type of component 1, in the same manner as described above, based on the created likelihood map 60. ing.

(Machine learning model configuration)
Further, in this embodiment, as shown in FIG. 4, the information processing unit 41 is configured to track the component 1 using a machine learning model 90 generated by machine learning. Specifically, as shown in FIG. 4a, the information processing unit 41 is configured to extract feature points 1a using a trained machine learning model 90. More specifically, the machine learning model 90 inputs an image within the extraction region 70 of the photographed image 50 and outputs feature point data 91, which is two-dimensional data representing the feature point 1a within the extraction region 70. It is composed of The machine learning model 90 is configured by, for example, a convolution network. After setting the extraction area 70, the information processing unit 41 cuts out an image within the extraction area 70 of the photographed image 50 and inputs it to the machine learning model 90. Furthermore, the information processing unit 41 extracts the feature points 1a within the extraction region 70 by acquiring feature point data 91 from the machine learning model 90 into which the image has been input.

FIG. 4b shows the learning procedure of the machine learning model 90. As shown in FIG. 4b, first, a learning image 92 is input to the machine learning model 90, and the machine learning model 90 is caused to output feature point data 91. Then, feature points are extracted based on the feature point data 91, and tracking is performed using the extracted feature points. Then, a loss value representing tracking accuracy is calculated based on the tracking execution result. Then, the parameters of the machine learning model 90 are updated by reversely assigning the loss values to the parameters of the machine learning model 90. By repeating the updating of the parameters of the machine learning model 90 using the above procedure, it is possible to generate the machine learning model 90 that can accurately extract the feature points 1a. Further, the trained machine learning model 90 is stored in advance in the storage unit 43 and used.

(Control processing related to parts counting)
With reference to FIGS. 5 and 6, control processing regarding component counting by the component counting device 100 of this embodiment will be described based on a flowchart.

As shown in FIG. 5, first, in step S1, the first vibrator 21 and the second vibrator 22 of the vibrating section 20 are activated.

Then, in step S2, the component 1 inside the housing section 2 is photographed by the photographing section 30 while the housing section 2 is vibrated by the vibrating section 20. As a result, a photographed image 50 in which the component 1 inside the storage section 2 is captured is obtained.

Then, in step S3, image processing of the photographed image 50 is performed. Note that details of image processing of the photographed image 50 will be described later.

Then, in step S4, it is determined whether a predetermined number of parts have been detected (counted). If it is determined that a predetermined number of parts have been detected, the process advances to step S5.

Then, in step S5, the first vibrator 21 and the second vibrator 22 of the vibrating section 20 are stopped. Then, the control process is ended.

Furthermore, in step S4, if it is determined that the predetermined number of parts have not been detected, the process proceeds to step S6.

Then, in step S6, it is determined whether the error condition is satisfied. The error conditions include a condition that the number of parts 1 counted is less than a predetermined number even after a predetermined time has elapsed, and a condition that the number of parts 1 counted is greater than the predetermined number. If it is determined that the error condition is satisfied, the process advances to step S7.

Then, in step S7, the user is notified of an error that the counted number of parts 1 does not meet the predetermined number. After that, the first vibrator 21 and the second vibrator 22 of the vibrating section 20 are stopped, and the control process is ended.

Furthermore, if it is determined in step S6 that the error condition is not satisfied, the process proceeds to step S2. Then, the processes of steps S2 to S6 are repeated until a predetermined number of parts are detected or an error condition is satisfied.

With reference to FIG. 6, details of the image processing of the photographed image 50 in step S3 will be described. In addition, in the image processing of the photographed image 50, in order to improve the recognition accuracy of the component 1 and the tracking accuracy of the component 1, processing for emphasizing the component 1, such as binarization processing and edge enhancement processing, may be performed as appropriate.

As shown in FIG. 6, first, in step S11, a likelihood map 60 of the component 1 is created based on the photographed image 50.

Then, in step S12, the position of component 1 and the orientation of component 1 are estimated based on the likelihood map 60. Furthermore, when multiple types of parts 1 are stored in the housing section 2, a separate likelihood map 60 is created for each type of part 1, or a likelihood map 60 for each type of part 1 is created. By creating one likelihood map 60 including information, the type of part 1 is estimated.

Then, in step S13, the likelihood of the likelihood map 60 created this time and the likelihood of the likelihood map 60 created a predetermined number of times ago are compared for each part area, and the likelihood of the likelihood map 60 created this time is compared with the likelihood of the likelihood map 60 created a predetermined number of times ago. It is determined for each part area whether the degree is the maximum. The process of step S14 is performed for the component region for which the likelihood is determined to be the maximum in the likelihood map 60 created this time.

Then, in step S14, the feature points 1a of the component 1 are extracted using the likelihood map 60 created this time. Further, the feature point 1a is updated to the feature point 1a of the extracted part 1 and is stored in the storage unit 43.

Then, in step S15, the extracted feature point 1a of the component 1 is tracked, thereby tracking the component 1. After that, the process advances to step S4.

Further, in step S13, for the component region for which it is determined that the likelihood does not reach the maximum in the likelihood map 60 created this time, the process of step S16 is performed.

Then, in step S16, the feature point 1a is extracted using a likelihood map 60 that has a higher likelihood than the likelihood map 60 created this time among the likelihood maps 60 created a predetermined number of times ago. In addition, the extracted feature point 1a of the component 1 is tracked, so that the component 1 is tracked. After that, the process advances to step S4.

(Configuration of robot system equipped with parts counting device)
Next, with reference to FIG. 7, a robot system 200 including the parts counting device 100 will be described as an application example of the parts counting device 100 of this embodiment.

The robot system 200 includes a parts counting device 100, a robot 110, and a robot control section 120. The robot 110 is a picking robot that transfers the component 1 into the storage section 2 or transfers the component 1 into the storage section 2 . The robot 110 is, for example, a vertically articulated robot. The robot 110 includes an articulated arm 111 and a hand 112 that is attached to the tip of the arm 111 and holds the component 1 . The robot control unit 120 is configured to control the operation of the robot 110. In the robot system 200, the parts counting device 100 counts the parts 1 in the storage unit 2, for example, in order to check whether the number of parts 1 transferred into the storage unit 2 by the robot 110 is correct. . Alternatively, the parts counting device 100 counts the parts 1 in the storage unit 2 in order to check whether the number of parts 1 in the storage unit 2 before being transferred by the robot 110 is correct. Note that the parts counting process by the parts counting device 100 is as described above, so detailed explanation will not be repeated.

In addition, in order to check whether the number of parts 1 in the storage unit 2 before being transferred by the robot 110 is correct, the parts 1 in the storage unit 2 are counted by the parts counting device 100, and the number of parts 1 in the storage unit 2 is correct. When the number of parts is counted, the photographing unit 30 and the information processing unit 41 of the parts counting device 100 operate as follows. That is, the photographing section 30 is configured to continue photographing the component 1 even after the vibrating section 20 stops vibrating the housing section 2 . The information processing unit 41 also recognizes the posture of the component 1 that is stationary within the storage unit 2 based on the image taken by the imaging unit 30 after the vibration unit 20 stops vibrating the storage unit 2. It is composed of The robot control unit 120 is configured to determine, based on the orientation of the component 1 recognized by the information processing unit 41, the portion of the robot 110 where the hand 112 holds the component 1. Thereby, it is possible to effectively utilize the photographing unit 30 and the information processing unit 41 of the parts counting device 100 to determine the part where the part 1 is held by the hand 112 of the robot 110.

In addition, when a plurality of types of parts 1 are stored in the housing section 2, the information processing section 41 displays an image taken by the photographing section 30 after the vibrating section 20 stops vibrating the housing section 2. Based on this, the posture of the component 1 that is stationary within the storage section 2 is recognized for each type of component 1. Then, the robot control unit 120 determines the part where the hand 112 of the robot 110 should hold the part 1 according to the type of the part 1, based on the posture of the part 1 for each type recognized by the information processing unit 41. It is composed of As a result, it is possible to effectively utilize the photographing section 30 and the information processing section 41 of the parts counting device 100 to appropriately determine the part where the part 1 is held by the hand 112 of the robot 110 according to the type of the part 1. It is.

(Effects of this embodiment)
In this embodiment, the following effects can be obtained.

In this embodiment, as described above, the vibrating section 20 that vibrates the accommodating section 2 that accommodates the plurality of components 1 is provided. Thereby, the component 1 can be moved by causing the vibrating section 20 to vibrate the component 1, so there is no need for the user to manually eliminate the overlap of the components 1. As a result, the user's effort can be saved. In addition, by tracking the parts 1 that move due to vibration based on the photographed images 50 sequentially photographed by the photographing unit 30, while creating a situation with high counting accuracy (a situation in which the parts 1 are not placed in close positions), Since the parts 1 can be counted, the parts 1 can be counted with high accuracy. As a result, the parts 1 can be counted with high accuracy while saving the user's effort.

Furthermore, in the present embodiment, as described above, the information processing unit 41 creates the likelihood map 60 of the component 1 included in the photographed image 50 based on the photographed image 50, and also creates the likelihood map 60 of the component 1 included in the photographed image 50. The device is configured to track the part 1. Thereby, the component 1 can be recognized with high precision based on the likelihood map 60, so the component 1 can be tracked with high precision.

Further, in the present embodiment, as described above, the information processing unit 41 estimates the position and orientation of the component 1 based on the created likelihood map 60, and based on the estimated position and orientation of the component 1, The component 1 is configured to be tracked by extracting feature points 1a of the component 1 and tracking the extracted feature points 1a. Thereby, the position and orientation of the component 1 can be estimated with high precision based on the likelihood map 60, so the feature points 1a of the component 1 can be extracted with high precision. As a result, by tracking the accurately extracted feature points 1a, the component 1 can be accurately tracked.

Furthermore, in the present embodiment, as described above, the information processing unit 41 sets the extraction region 70 for extracting the feature points 1a based on the estimated position and orientation of the component 1, and extracts the feature points from within the extraction region 70. 1a. Thereby, the feature point 1a can be extracted from within the extraction region 70 where there is a high possibility that the feature point 1a exists, so the feature point 1a of the component 1 can be extracted with higher accuracy. As a result, the component 1 can be tracked more accurately by tracking the extracted feature points 1a with more accuracy.

Further, in the present embodiment, as described above, the information processing unit 41 obtains the degree of coincidence between the feature point 1a extracted from within the extraction region 70 and the feature point 1a within the reference image 80, and Among the feature points 1a extracted from , the feature point 1a whose matching degree is equal to or higher than a threshold value is determined as the feature point 1a to be tracked. Thereby, by tracking the feature points 1a with a high degree of coincidence, the component 1 can be tracked with even higher accuracy.

Furthermore, in this embodiment, as described above, the information processing unit 41 is configured to count the parts 1 in the storage unit 2 by counting the parts 1 that track the feature point 1a. Thereby, by counting the parts 1 that track the feature point 1a, it is possible to avoid counting the same part 1 twice, and therefore it is possible to count the parts 1 with higher accuracy.

Furthermore, in the present embodiment, as described above, the information processing unit 41 extracts the feature points 1a of the component 1 for each type of the component 1, and counts the number of parts 1 that track the feature points 1a for each type. , is configured to count parts 1 for each type of parts 1. Thereby, even when a plurality of types of parts 1 exist in the storage section 2, the parts 1 can be accurately counted for each type of parts 1.

Further, in the present embodiment, as described above, the information processing unit 41 calculates the likelihood of the likelihood map 60 created this time and the likelihood of the likelihood map 60 created a predetermined number of times ago for each part area. Among the compared component regions, for the component region that has the maximum likelihood in the likelihood map 60 created this time, the likelihood map 60 created this time is used to extract the feature points 1a, and the Among these, for component regions where the likelihood does not reach the maximum in the likelihood map 60 created this time, among the likelihood maps 60 created a predetermined number of times ago, the likelihood is larger than the likelihood map 60 created this time. The map 60 is configured to be used for extracting feature points 1a. Thereby, it is possible to always track the extracted feature point 1a in a state where the likelihood is large.

Furthermore, in this embodiment, as described above, the information processing unit 41 is configured to track the component 1 using the machine learning model 90 generated by machine learning. Thereby, the component 1 can be easily and accurately tracked using the machine learning model 90.

Furthermore, in this embodiment, as described above, the information processing unit 41 is configured to notify the user when the counted number of parts 1 does not satisfy a predetermined number. As a result, if the number of counted parts 1 does not satisfy the predetermined number (if the number of parts 1 is greater or less than the predetermined number), the user can easily confirm that the number of parts 1 does not satisfy the predetermined number. Since this can be known, measures can be easily taken so that the number of parts 1 satisfies the predetermined number.

Furthermore, in this embodiment, as described above, the vibrating section 20 includes the first vibrator 21 that vibrates the housing section 2 in the horizontal direction. Thereby, the component 1 can be easily moved by vibrating the component 1 in the horizontal direction, so a situation with high counting accuracy (a situation in which the components 1 are not placed close to each other) can be easily created.

[Modified example]
Note that the embodiments disclosed this time should be considered to be illustrative in all respects and not restrictive. The scope of the present invention is indicated by the claims rather than the description of the embodiments described above, and further includes all changes (modifications) within the meaning and range equivalent to the claims.

For example, in the above embodiment, an example was shown in which the parts counting device is applied to a robot system, but the present invention is not limited to this. In the present invention, the parts counting device may be applied to systems other than robot systems.

Further, in the above embodiment, an example was shown in which the parts counting device is an inspection device that inspects the number of parts by counting the parts, but the present invention is not limited to this. In the present invention, the parts counting device may be an inspection device further provided with an inspection function other than the function of inspecting the number of parts.

Further, in the above embodiment, an example was shown in which feature points extracted based on a likelihood map are tracked, but the present invention is not limited to this. In the present invention, feature points may be extracted by image processing other than the likelihood map, and the extracted feature points may be tracked.

Furthermore, in the above embodiment, an example was shown in which an extraction region for extracting feature points is set, but the present invention is not limited to this. In the present invention, it is not necessary to set an extraction area for extracting feature points.

Further, in the above embodiment, an example was shown in which the degree of coincidence between feature points extracted from within the extraction region and feature points in the reference image is obtained, but the present invention is not limited to this. In the present invention, it is not necessary to obtain the degree of coincidence between the feature points extracted from within the extraction region and the feature points within the reference image. That is, feature points extracted from within the extraction area may be determined as feature points to be tracked.

Furthermore, in the above embodiment, an example was shown in which the likelihood of the likelihood map created this time and the likelihood of the likelihood map created a predetermined number of times ago are compared for each part area. Not limited. In the present invention, it is not necessary to compare the likelihood of the likelihood map created this time and the likelihood of the likelihood map created a predetermined number of times ago for each part area. That is, the likelihood map created this time may always be used to extract feature points.

Further, in the above embodiment, an example was shown in which feature points are extracted using a machine learning model, but the present invention is not limited to this. In the present invention, feature points may be extracted using rule-based image processing other than machine learning models.

Further, in the above embodiment, an example was shown in which a machine learning model trained to extract feature points is used to extract feature points and track parts, but the present invention is not limited to this. In the present invention, a machine learning model trained to create a likelihood map may be used to create a likelihood map and track parts.

Further, in the above embodiment, an example was shown in which the vibrating section includes both the first vibrator that vibrates the housing section in the horizontal direction and the second vibrator that vibrates the housing section in the vertical direction. Not limited. In the present invention, the vibrating section may include only one of the first vibrator that vibrates the housing section in the horizontal direction and the second vibrator that vibrates the housing section in the vertical direction.

Further, in the above embodiment, for convenience of explanation, the processing operation of the control unit is explained using a flow-driven flowchart in which the processing is performed in order along the processing flow, but the present invention is not limited to this. In the present invention, the processing operation of the control unit may be performed by event-driven processing in which processing is executed on an event-by-event basis. In this case, it may be completely event-driven, or it may be a combination of event-driven and flow-driven.

Furthermore, in the present invention, when the photographing speed is higher than the image processing speed, the photographing process and the image processing may be executed in parallel in separate flows, as shown in the modified examples of FIGS. 8 and 9. Note that in FIGS. 8 and 9, the same processes as those in the flowchart of FIG. 5 are denoted by the same reference numerals, and detailed explanation thereof will be omitted.

As shown in FIG. 8, on the imaging unit 30 side, in step S101, it is determined whether all of the captured images 50 stored in the storage unit 43 have been subjected to image processing. If it is determined that all of the captured images 50 stored in the storage unit 43 have not been subjected to image processing, the process of step S101 is repeated. Further, if it is determined that all of the captured images 50 stored in the storage unit 43 have been subjected to image processing, the process advances to step S102. Then, in step S102, the photographing section 30 photographs the component 1 inside the storage section 2. Further, the photographed image 50 is stored in the storage unit 43. Then, in step S103, it is determined whether or not photography has been performed a predetermined number of times. If it is determined that photography has not been performed a predetermined number of times, the process of step S102 is repeated. Furthermore, if it is determined that photography has been performed a predetermined number of times, the process advances to step S4.

Further, as shown in FIG. 9, on the information processing unit 41 side, in step S111, it is determined whether or not there are any captured images 50 stored in the storage unit 43 for which image processing has not been performed. If it is determined that there is no unprocessed image among the captured images 50 stored in the storage unit 43, the process of step S111 is repeated. If it is determined that some of the photographed images 50 stored in the storage unit 43 have not been subjected to image processing, the process proceeds to step S3, and image processing is performed on the photographed images 50 for which image processing has not been performed. Then, the process advances to step S111. By executing the flowcharts shown in FIGS. 8 and 9 in parallel, the image capturing section 30 can perform imaging while waiting for the information processing section 41 to complete the image processing.

1 Parts 1a Features 2 Housing section 20 Vibration section 21 First vibrator (vibrator)
30 Photographing unit 41 Information processing unit 50 Photographed image 60 Likelihood map 70 Extraction area 80 Reference image 90 Machine learning model 100 Parts counting device 110 Robot 200 Robot system

Claims

a vibrating unit that vibrates a housing unit that accommodates a plurality of components;
an imaging unit that continuously photographs the parts in the housing unit that are vibrated by the vibration unit;
A parts counting device, comprising: an information processing unit that tracks the parts that move due to vibration based on photographed images sequentially taken by the photographing unit, and counts the parts in the storage unit.
The information processing unit creates a likelihood map of the component included in the captured image based on the captured images sequentially captured by the imaging unit, and tracks the component based on the likelihood map. The parts counting device according to claim 1, wherein the parts counting device is configured to.
The information processing unit estimates the position and orientation of the part based on the created likelihood map, extracts feature points of the part based on the estimated position and orientation of the part, and extracts the extracted feature points of the part. 3. The parts counting device according to claim 2, configured to track the parts by tracking feature points.
The information processing unit is configured to set an extraction area for extracting the feature points based on the estimated position and orientation of the part, and to extract the feature points from within the extraction area. 3. The parts counting device according to 3.
The information processing unit obtains a degree of coincidence between the feature points extracted from within the extraction region and the feature points in the reference image, and determines whether the degree of coincidence is among the feature points extracted from within the extraction region. The parts counting device according to claim 4, wherein the component counting device is configured to determine the feature point that is equal to or greater than a threshold value as the feature point to be tracked.
The parts counting device according to claim 3, wherein the information processing unit is configured to count the parts in the storage unit by counting the parts that track the feature points.
The information processing unit counts the parts for each type of parts by extracting feature points of the parts for each type of parts and counting the parts whose feature points are tracked for each type. The parts counting device according to claim 6, wherein the parts counting device is configured to.
The information processing unit compares the likelihood of the likelihood map created this time with the likelihood of the likelihood map created a predetermined number of times ago for each part area, and of the compared part areas, For the component region where the likelihood is maximum in the created likelihood map, the newly created likelihood map is used to extract the feature points, and among the compared component regions, the newly created likelihood map is used to extract the feature points. For the component region where the likelihood does not reach the maximum, the feature points are extracted from the likelihood map that has a greater likelihood than the likelihood map created this time among the likelihood maps created a predetermined number of times ago. The parts counting device according to claim 3, configured to be used for.
The parts counting device according to claim 1, wherein the information processing unit is configured to track the parts using a machine learning model generated by machine learning.
The parts counting device according to claim 1, wherein the information processing unit is configured to notify a user if the number of the counted parts does not satisfy a predetermined number.
The parts counting device according to claim 1, wherein the vibrating section includes a vibrator that vibrates the housing section in a horizontal direction.
a robot that transfers a component within a storage section or transfers the component into the storage section;
a parts counting device that counts the parts in the storage section,
The parts counting device includes:
a vibrating section that vibrates the housing section;
an imaging unit that continuously photographs the parts in the housing unit that are vibrated by the vibration unit;
an information processing unit that tracks the parts moving due to vibration and counts the parts in the storage unit based on photographed images sequentially taken by the photographing unit;
including robotic systems.