WO2023233992A1 - Procédé de traitement d'images pour objet de forme irrégulière - Google Patents

Procédé de traitement d'images pour objet de forme irrégulière Download PDF

Info

Publication number
WO2023233992A1
WO2023233992A1 PCT/JP2023/018115 JP2023018115W WO2023233992A1 WO 2023233992 A1 WO2023233992 A1 WO 2023233992A1 JP 2023018115 W JP2023018115 W JP 2023018115W WO 2023233992 A1 WO2023233992 A1 WO 2023233992A1
Authority
WO
WIPO (PCT)
Prior art keywords
image data
learning
image
data
image processing
Prior art date
Application number
PCT/JP2023/018115
Other languages
English (en)
Japanese (ja)
Inventor
健人 箕浦
直也 宮地
Original Assignee
パナソニックＩｐマネジメント株式会社
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by パナソニックＩｐマネジメント株式会社 filed Critical パナソニックＩｐマネジメント株式会社
Publication of WO2023233992A1 publication Critical patent/WO2023233992A1/fr

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
    • G06V10/774—Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks

Definitions

  • the present invention relates to an image processing method for an amorphous object that uses deep learning to recognize individual objects from a sorted object consisting of a plurality of amorphous objects.
  • picking devices that use robots to pick and sort specific parts from an object to be sorted consisting of multiple types of parts have been widely used in various industries around the world.
  • specific parts are recognized from dismantled or shredded products that are no longer needed and have been dismantled or roughly crushed into parts, and are then sorted and collected by picking and recycled.
  • products with different numbers of parts are randomly disassembled and placed on a conveyor. Therefore, it is difficult to align and transport the dismantled items, and it is necessary to identify the gripping position of randomly placed objects to be sorted and determine whether they can be picked.
  • Patent Document 1 proposes a method of recognizing the gripping position of randomly placed parts by comparing a master model consisting of a simple shape of the part with an object to be sorted.
  • FIG. 12 shows the external configuration of the robot 501 described in Patent Document 1.
  • 501 is a robot
  • 502 is a workbench
  • 503 is a robot main body
  • 504 is a robot controller
  • 505 is a hand
  • 506 is a tray
  • 507 is a component
  • 508 is a component supply unit
  • 509 is a gripping position detection device
  • 510 is a A camera 511 is an image recognition device.
  • the flowchart in FIG. 13 shows the processing procedure executed by the grip position detection device 509 and the image recognition device 511.
  • FIG. 14 shows a schematic diagram for explaining the relationship between the degree of coincidence and the overlapping state of parts.
  • P1 to P3 are images of grippable candidate parts.
  • step S1 the component supply unit 508 is photographed by the camera 510, and the two-dimensional image is input to the image recognition device 511.
  • step S2 a process is performed in which the entire image data is compared with the first master model to determine the degree of matching with the first master model and extracted as a grippable candidate part.
  • step S3 it is determined whether a grippable candidate part has been extracted, and if it has been extracted, in step S4, the grippable candidate part is compared with the second master model, and The degree of matching in the second master model is determined.
  • the grippable candidate parts extracted in the first matching stage of step S2 include those that partially overlap with other parts 507.
  • the grippable candidate parts extracted in the first matching stage of step S2 include those that partially overlap with other parts 507.
  • P1 in FIG. This includes items that have a slight overlap, and items that overlap with other parts 507 at the gripping site, as shown by P3.
  • step S4 which is the comparison of gripped parts
  • state P3 the degree of matching with the second master model becomes small.
  • state P2 although the degree of coincidence in the second master model is relatively large, it is lower than the degree of coincidence in the first master model.
  • state P1 the degree of coincidence in the second master model is relatively large and greater than the degree of coincidence in the first master model, and the area shown with diagonal lines in FIG. If the degree of coincidence is greater than the degree of coincidence in the first master model, it can be determined that there is no overlap with other parts 507 at the gripping site and that gripping is possible.
  • An image processing method for an amorphous object includes: An image processing unit acquires a plurality of learning image data taken from above a conveyance section in which a plurality of amorphous sorting objects are conveyed, An image obtained by cutting out only the sorted object in an area smaller than the area of the learning image data from the plurality of acquired learning image data is used as learning image data of the object, and a background is extracted from the plurality of acquired learning image data. An image obtained by cutting out only a region smaller than the area of the training image data is used as the training image data of the background, and from the plurality of acquired training image data, a boundary part where the plurality of objects to be sorted are in contact with each other is determined.
  • An image obtained by cutting out an image in an area smaller than the area of the learning image data is used as the learning image data of the boundary part, and the learning image data of the object, the learning image data of the background, and the learning image data of the boundary part are used.
  • the image processing unit Based on a trained model generated by learning by deep learning, the image processing unit generates new image data captured from above the transport unit where the plurality of objects to be sorted are transported. Classifying each pixel of the new image data into an object, a background, and a boundary, The image processing unit derives the position coordinates, angle, gripping position, or type of the object to be sorted from the classified image data.
  • FIG. 1 Schematic configuration diagram of a picking device in Embodiment 1 of the present invention
  • FIG. 1 of the present invention A schematic configuration diagram of an image processing unit, etc. of a picking device in Embodiment 1 of the present invention
  • FIG. 1 of the present invention A schematic configuration diagram of another example of the image processing unit of the picking device in Embodiment 1 of the present invention
  • Schematic diagram of captured image data in Embodiment 1 of the present invention Schematic diagram of post-inference image data in Embodiment 1 of the present invention
  • Schematic diagram of binarized image data in Embodiment 1 of the present invention Schematic diagram of a trained model generation procedure in Embodiment 1 of the present invention
  • Schematic diagram of an inference processing method for a trained model in Embodiment 1 of the present invention Flowchart showing image processing procedure in Embodiment 1 of the present invention
  • Schematic diagram of binarized image data in Embodiment 1 of the present invention Schematic diagram of captured image data in Embodiment 1 of the present
  • Patent Document 1 it is possible to determine the gripping position of a fixed-sized object for which a master model can be prepared in advance, but it is not possible to determine the gripping position of an amorphous object for which it is difficult to prepare a master model in advance. have challenges.
  • dismantled used home appliances at a home appliance recycling factory come in a wide variety of manufacturers, model numbers, and years, and may also be dirty or deformed, making them highly variable.
  • the dismantled products may be conveyed on the belt conveyor in a state in which they are in contact with each other or in a state in which they overlap.
  • robot picking disassembled products transported on these belt conveyors it is necessary to determine whether the dismantled products overlap each other in order to specify the gripping position.
  • Patent Document 1 it is extremely difficult to compare a variety of external shapes, such as dismantled used home appliances, with a master model as in Patent Document 1.
  • the present invention solves the above-mentioned conventional problems, and provides an image processing method for an amorphous object, which can derive the position coordinates, angle, gripping position, or type of an amorphous object such as a household electrical appliance.
  • the purpose is to
  • FIG. 1A, FIG. 1B, and FIG. 1C are configuration diagrams of a picking device in Embodiment 1 of the present invention.
  • 301 is a transport unit such as a conveyor
  • 302 is a group of objects
  • 303 is an imaging device such as a camera
  • 304 is a field of view of the imaging device
  • 305 is a robot
  • 306 is a collection box
  • 307 is a specific object
  • 400 is a control unit
  • 401 is an image processing unit
  • 200 is a trained model.
  • the control unit 400 includes an image processing unit 401 and controls the operation of the robot 305 based on processing by the image processing unit 401.
  • the image processing unit 401 includes an image data acquisition unit 401a, an inference processing unit 401b, a binarization processing unit 401c, a circumscribing rectangle generation unit 401d, and an image data cutting unit. It includes a section 401e, an information extraction section 401f, and a transfer section 401h.
  • the image data acquisition unit 401a acquires image data to be processed.
  • the inference processing unit 401b performs inference processing on the acquired data.
  • the binarization processing unit 401c performs binarization processing on the inference result.
  • the circumscribed rectangle generation unit 401d generates a circumscribed rectangle as a result of the binarization process.
  • the image data cutting unit 401e cuts data from within the circumscribed rectangle.
  • the information extraction unit 401f extracts information from the cut data.
  • the transfer unit 401h transfers the extracted information to the control unit 400.
  • the control unit 400 includes a grippability determining unit 401g, which determines whether or not it is grippable based on the extracted information, and controls the operation of the robot 305 based on the result.
  • a conveyance unit 301 moves objects 302 including various types of disassembled products placed on a conveyor belt 301, which is an example of a conveyance unit, in one direction, for example, to the right in FIG. 1A. transport in the direction.
  • the object group 302 may not be arranged at equal intervals on the belt, but may be randomly supplied to the conveyance unit 301 and conveyed.
  • the imaging device 303 is mounted above the upstream side of the transport section 301A toward the transport section 301. Control can be performed by detecting the presence or absence of an image of the object group 302 by the image data acquisition unit 401a of the image processing unit 401 so that the imaging device 303 starts imaging at the timing when the object group 302 is within its field of view.
  • FIG. 2A shows a schematic diagram of captured image data 100 obtained by the image data acquisition unit 401a from the imaging device 303.
  • objects 1a and 1b are amorphous objects such as recycled disassembled home appliances that have different external shapes.
  • Background 2 is the background portion of the image data excluding object 1a and object 1b.
  • the boundary portion 3 is a boundary portion where the object 1a and the object 1b overlap, or a boundary portion where the object 1a and the object 1b are in contact with each other.
  • the inference processing unit 401b of the image processing unit 401 performs image recognition processing on the captured image data 100 obtained from the above-mentioned imaging device 303.
  • the learned model used here is a learning model based on machine learning that is constructed in advance using image data as learning image data.
  • Machine learning includes deep learning, supervised learning, and reinforcement learning.
  • learning by deep learning is adopted.
  • the trained model has two phases. One is a phase in which a trained model is generated, and the other is a phase in which inference processing is performed using the generated trained model.
  • FIGS. 3A and 3B schematically show the generation procedure of the trained model 200 and the inference processing method.
  • Step S11 in FIG. 3A will be explained.
  • step S11 a plurality of pieces of captured image data 100 of the object 1a and the object 1b are collected.
  • each of the plurality of pieces of captured image data 100 may include not only the object 1a and the object 1b but also a plurality of types of objects.
  • step S12 image processing is performed on the plurality of captured image data 100 to create a database of learning images.
  • step S13 a trained model 200 is generated by performing deep learning on the database of training images, and is stored in the inference processing unit 401b.
  • FIG. 3B an inference processing method for the learned model 200 will be described.
  • Step S14 in FIG. 3B will be explained.
  • the captured image data 100 is input to the image data acquisition unit 401a of the image processing unit 401.
  • step S15 in FIG. 3B the learned model 200 of the inference processing unit 401b performs inference processing on the captured image data 100 of the image data acquisition unit 401a.
  • step S16 in FIG. 3B will be explained.
  • the inference processing unit 401b by performing inference processing using the trained model 200, the inference processing unit 401b generates image information of the object to be sorted necessary for picking as an inference result, and outputs it to the binarization processing unit 401c.
  • FIG. 2A is a schematic diagram of captured image data 100.
  • the captured image data 100 is image data in which an object 1a and an object 1b are overlapped.
  • FIG. 2B is a schematic diagram of post-inference image data 101 that is obtained by performing inference processing using the learned model 200 on the captured image data 100 of FIG. 2A.
  • FIG. 2C is a schematic diagram of the binarized image data 102 that is obtained by performing the binarization process on the post-inference image data 101 of FIG. 2B.
  • the flowchart in FIG. 4 shows the processing procedure in the image processing unit 401.
  • step S21 in FIG. 4 will be explained.
  • captured image data 100 obtained by capturing an image of the object group 302 transported on the transport unit 301 with the imaging device 303 is input to the image data acquisition unit 401a of the image processing unit 401.
  • step S22 in FIG. 4 will be explained.
  • the captured image data 100 of FIG. 2A input to the image data acquisition unit 401a is subjected to inference processing by the learned model 200 of the inference processing unit 401b, thereby generating the post-inference image data 101 of FIG. 2B. Then, it is output to the binarization processing section 401c.
  • object data 41 is image data that displays the inference results between object 1a and object 1b.
  • the background data 42 is image data that displays the inference results of the background 2.
  • the boundary data 43 is image data that displays the inference results for the boundary 3.
  • step S23 in FIG. 4 the binarization processing unit 401c of the image processing unit 401 performs binarization processing on the post-inference image data 101 to generate the binarized image data 102 shown in FIG. It is output to the generation unit 401d.
  • the binarized object data 51a and the binarized object data 51b are image data obtained by performing binarization processing on the object data 41 by the binarization processing unit 401c.
  • the post-binarization background data 51c is image data obtained by performing a binarization process on an image data area that is the sum of the background data 42 and the boundary data 43 by the binarization processing unit 401c.
  • step S23 the binarized object data 51a and the binarized object data 51b are generated as two independent object image data by the binarization processing unit 401c.
  • the object 1a and the object 1b partially overlap in the captured image data 100, by generating the binarized image data 102, it is possible to display the overlapping objects individually as image data. Become.
  • FIG. 5A is a schematic diagram showing a circumscribed rectangle 61c and a circumscribed rectangle 61d on the binarized image data 102.
  • the circumscribed rectangle generation unit 401d of the image processing unit 401 generates a circumscribed rectangle 61c and a circumscribed rectangle 61d by surrounding the binarized object data 51a and the binarized object data 51b with circumscribed rectangles, respectively.
  • the image data is then output to the image data cutting section 401e.
  • a circumscribing rectangle 61c is a circumscribing rectangle surrounding the binarized object data 51a.
  • the circumscribed rectangle 61d is a circumscribed rectangle surrounding the binarized object data 51b.
  • step S25 in FIG. 4 will be explained.
  • FIG. 5B is a schematic diagram showing a circumscribed rectangle 61c and a circumscribed rectangle 61d on the captured image data 100.
  • FIG. 5C is a schematic diagram of the transfer image data 61e and the transfer image data 61f.
  • a circumscribed rectangle 61c and a circumscribed rectangle 61d are displayed on the captured image data 100 at the same coordinates as the binarized image data 102.
  • the image data cutting unit 401e cuts out the image data within the circumscribed rectangle 61c and the circumscribed rectangle 61d in the captured image data 100, thereby creating the image data for transfer 61e and the image data for transfer 61f. and outputs it to the information extraction unit 401f.
  • FIG. 5B by displaying the circumscribed rectangle 61c and the circumscribed rectangle 61d in the captured image data 100, the object 1a and the object 1b are surrounded by the circumscribed rectangle 61c and the circumscribed rectangle 61d. This makes it possible to recognize the plurality of different objects 1a and 1b, which were conveyed in an overlapping state, as individual objects.
  • step S26 the information extraction unit extracts information necessary for picking, such as object position coordinates, object angle, object gripping position, and/or object type data, from the transfer image data 61e and the transfer image data 61f. 401f image processing.
  • information necessary for picking such as object position coordinates, object angle, object gripping position, and/or object type data
  • the binarized image data 102 is cut out with the circumscribed rectangle 61c and the circumscribed rectangle 61d instead of the transfer image data 61e and the transfer image data 61f extracted from the captured image data 100 in step S25. You may also use the image extracted in . Furthermore, if the object group 302 flowing to the picking device is composed of multiple types of objects, that is, in the case of multi-item sorting with multiple collection boxes 306, the type is also determined in this step S26, and type information is also obtained. It's fine.
  • an image of the object cut out from the captured image data 100 is good, but if it is possible to discriminate the type from the shape of the object, binary image data An image of the object cut out from 102 may also be used.
  • step S27 the information acquired in step S26 is transferred to the control unit 400 of the picking device by the transfer unit 401h, and based on the transferred information, the grippability determining unit 403 of the control unit 400 determines whether picking, that is, gripping is possible. By doing so, a desired picking operation can be performed, and the specific object 307 can be sorted and collected into the collection box 306 by the robot 305.
  • the grippability determination unit is provided in the image processing unit 401 as shown in FIG. 1C instead of in the control unit 400, and as step S27A, the grippability determination unit of the image processing unit 401 is installed based on the information acquired in step S26. At 401g, it is determined whether picking, that is, gripping is possible. Next, in step S27, the transfer unit 401h transfers the determination result to the control unit 400 of the picking device to perform a desired picking operation, and the robot 305 can also sort and collect the specific object 307 into the collection box 306.
  • FIG. 6A shows an example of captured image data 100 used to create learning image data.
  • the captured image data 100 image data in which various external shapes and materials are imaged can be used.
  • the captured image data 100 is an example of two-dimensional image data captured of a recycled and disassembled home appliance.
  • objects 1a and 1b are amorphous objects such as recycled disassembled home appliances that have different external shapes.
  • Background 2 is the background portion of the image data excluding object 1a and object 1b.
  • the boundary portion 3 is a boundary portion where the object 1a and the object 1b are in contact with each other.
  • each of the plurality of pieces of captured image data 100 may include not only the object 1a and the object 1b but also a plurality of types of objects.
  • FIG. 6B shows a schematic diagram of the post-inference image data 101.
  • the post-inference image data 101 is image data obtained by classifying the captured image data 100 into object data 41, background data 42, and boundary data 43.
  • object data 41 is image data of object 1a and object 1b.
  • Background data 42 is image data of background 2.
  • the boundary data 43 is image data of the boundary 3.
  • FIG. 7A is a schematic diagram showing a rectangular area 20 displayed on the captured image data 100.
  • a rectangular area 20 is image data that displays an area to be cut out from the captured image data 100 in order to create the learning image database 201.
  • FIG. 7B shows a schematic diagram of the learning image database 201.
  • the learning image database 201 is a learning image data group that stores an object data group 31, a background data group 32, and a boundary data group 33.
  • the object data group 31 is an image data group in which a plurality of object learning image data 21 are stored.
  • the background data group 32 is an image data group in which a plurality of background learning image data 22 are stored.
  • the boundary data group 33 is an image data group in which a plurality of boundary part learning image data 23 are stored.
  • object learning image data 21 is image data obtained by cutting out a rectangular region 20 from objects 1a and 1b.
  • the background learning image data 22 is image data obtained by cutting out a rectangular area 20 from the background 2.
  • the boundary part learning image data 23 is image data obtained by cutting out a rectangular area 20 from the boundary part 3.
  • the object 1a and object 1b divide the boundary part learning image data 23 into two rectangles, but even if the boundary surface is horizontal as in the boundary part learning image data 23, it is vertically It does not matter whether it is in a direction or in an oblique direction.
  • the rectangular areas 20 have the same size. However, the rectangular area 20 may be cut out into areas having different sizes.
  • the created learning image database 201 is trained by deep learning to create a trained model 200 and stored in the inference processing unit 401b.
  • post-inference image data 101 can be generated from the captured image data 100 by executing the inference process using the trained model 200 created in this way in the inference processing unit 401b as described in step S22. Then, by generating the post-inference image data 101 in the inference processing unit 401b, the inference processing unit 401b can obtain image data classified into object data 41, background data 42, and boundary data 43 for each pixel. .
  • the inference processing unit 401b of the image processing unit 401 performs binarization processing from an object to be sorted consisting of a plurality of amorphous objects.
  • the objects 1a and 1b are individually image-processed, and the information extraction section 401f extracts the objects 1a and 1b. 1b, that is, the position coordinates, angle, gripping position, or type of amorphous objects such as household electrical appliances can be derived, and sorting operations become possible.
  • each object 1a and 1b can be extracted individually from a sorting target consisting of a plurality of irregular objects.
  • the objects 1a and 1b are image-processed individually, and a sorting operation is possible by deriving the grasping positions of the objects 1a and 1b and identifying the type.
  • the image data of the outer edge of the object 1a which is the boundary between the object 1a and the background 2
  • the image data of the boundary 3, which is the boundary between the objects 1a and 1b are image data that have very similar characteristics. Therefore, in the image processing method of Embodiment 1, if the outer edge of the object 1a is mistakenly recognized as the boundary 3, the binarized object data 51a will be smaller than the actual object 1a by the image data of the boundary 3. It may be extracted smaller than the image data. Therefore, in addition to the object, background, and boundary part trained by the trained model 200, a trained model is created in which the peripheral part, which is the boundary between the object and the background, is learned. An image processing method using a trained model whose outer edges have been trained makes it possible to extract the object 1a as a region having the same size as the actual size, and enables more accurate image processing of the object.
  • FIG. 8 is a configuration diagram of a picking device in Embodiment 2 of the present invention.
  • the same components as those of the picking device in FIG. 1A are denoted by the same reference numerals, and the description thereof will be omitted.
  • 210 is a trained model
  • 402 is a control unit.
  • the control unit 402 is provided separately from the image processing unit 401, the grippability determination unit may be provided in the control unit 402 or in the image processing unit 401, as in the first embodiment.
  • the difference from the first embodiment is that a trained model 210 is stored in advance in the inference processing unit 401b of the image processing unit 401. Therefore, a description of the picking device will be omitted.
  • the trained model 210 is a trained model in which image data is newly trained on the trained model 200. Therefore, the trained model 210 can be generated using a procedure equivalent to the trained model 200 shown in FIG. 3A, separately from the image processing method described below (for example, before implementing the image processing method). , the explanation is omitted.
  • the flowchart in FIG. 9 shows the processing procedure in the image processing unit 401.
  • FIG. 10A is a schematic diagram of captured image data 100.
  • FIG. 10B is a schematic diagram of post-inference image data 111 in which the inference processing unit 401b performs inference processing using the learned model 210 on the captured image data 100 in FIG. 10A.
  • steps S22' and S23' which are different from the processing method in the flowchart of FIG. 4, will be described.
  • Step S22' in FIG. 9 will be explained.
  • the inference processing unit 401b performs inference processing using the learned model 210 on the captured image data 100 input to the image data acquisition unit 401a, thereby generating post-inference image data 111. It is output to the value processing unit 401c. That is, the inference processing unit 401b cuts out the boundary between the object to be sorted and the background from the plurality of acquired learning image data in an area smaller than the area of the learning image data, and uses the image as learning image data of the outer edge.
  • the trained model 210 can be trained to classify each pixel of new image data into an object, a background, a boundary, and an outer edge.
  • object data 71 is image data that displays the inference results between object 1a and object 1b.
  • the background data 72 is image data that displays the inference result of the background 2.
  • the boundary data 73 is image data that displays the inference results for the boundary 3.
  • the outer edge data 74 which is different from the first embodiment, is image data that displays the inference result of the outer edge 4.
  • step S23' in FIG. 9 the binarization processing unit 401c of the image processing unit 401 performs binarization processing on the post-inference image data 111 to generate binarized image data 112, and the circumscribing rectangle generation unit Output to 401d.
  • the binarized object data 81a and the binarized object data 81b are obtained by a binarization processing unit that performs binarization processing on an image data area that is a sum of the object data 71 and the outer edge data 74.
  • This is image data obtained using 401c.
  • the post-binarization background data 81c is image data obtained by performing a binarization process on an image data area that is the sum of the background data 72 and the boundary data 73.
  • step S23' the binarized object data 81a and the binarized object data 81b are generated as two independent object image data and output to the circumscribed rectangle generation unit 401d.
  • the object 1a and the object 1b overlap, but by generating the binarized image data 112, it becomes possible to display the overlapping objects individually as image data.
  • the image data obtained by summing the pixels of the classified objects and the periphery areas is used as the classified image data when deriving the position.
  • the coordinates, angle, gripping position, or type can be derived in step S26.
  • the area area of the binarized object data 81a and the binarized object data 81b is the combined area of the object and the outer edge. Therefore, compared to the area areas of the objects 1a and 1b of the captured image data 100, the area areas of the binarized object data 81a and the binarized object data 81b are extracted as larger areas. Therefore, by shrinking the area area of the extracted binarized object data 81a and the binarized object data 81b in the range of 5.5% to 12.0%, the area equivalent to that of the objects 1a and 1b is obtained. It can be a region.
  • the position coordinates, The angle, gripping position, or type can be derived in step S26.
  • a plurality of captured image data 100 in which objects 1a and 1b are captured in advance are prepared.
  • FIG. 11A is a schematic diagram showing a rectangular area 20 displayed on captured image data 100.
  • a rectangular area 20 is image data that displays an area to be cut out from the captured image data 100 in order to create the learning image database 211.
  • FIG. 11B shows a schematic diagram of the learning image database 211.
  • FIG. 11B the difference from FIG. 7B is that an outer edge data group 34 is added, so a detailed explanation of the other parts will be omitted.
  • the learning image database 211 is a learning image data group that stores an object data group 31, a background data group 32, a boundary data group 33, and an outer edge data group 34.
  • the outer edge data group 34 is an image data group in which a plurality of outer edge learning image data 24 are stored.
  • the outer edge learning image data 24 is image data obtained by cutting out a rectangular area 20 from the outer edge 4.
  • the object 1a or the object 1b and the background 2 divide the inside of the rectangle into two, but as in the image data 24 for learning the outer edge, the boundary surface is horizontal. It does not matter if it is in the direction, vertical direction, or diagonal direction.
  • the created learning image database 211 is trained by deep learning to create a trained model 210 and stored in the inference processing unit 401b.
  • the outer edge By creating a trained model 210 that has trained the image data 24 for learning the image data 24, the captured image data 100 can be classified into object data 71, background data 72, boundary data 73, and outer edge data 74. It becomes possible. As a result, by using the image processing method of the second embodiment, it is possible to extract the object 1a as a region having the same size as the actual size, and it is possible to perform image processing of the object more accurately than in the first embodiment.
  • the first and second embodiments of the present invention have been described above, the first and second embodiments described above are merely examples for implementing the present invention. Therefore, the present invention is not limited to the first and second embodiments described above, but can be implemented by appropriately modifying the first and second embodiments without departing from the spirit thereof. That is, by appropriately combining any of the various embodiments or modifications described above, the respective effects can be achieved. In addition, combinations of embodiments, combinations of examples, or combinations of embodiments and examples are possible, and combinations of features in different embodiments or examples are also possible.
  • the image processing method for an amorphous object by using the image processing unit to individually extract objects from a sorting target consisting of a plurality of amorphous objects, By individually image processing each object, it is possible to derive the position coordinates, angle, gripping position, or type of amorphous objects such as household electrical appliances, making it possible to perform sorting operations. Therefore, it is useful, for example, for gripping irregular objects such as household electrical appliances.
  • each object is individually extracted from a sorting target consisting of a plurality of amorphous objects, and each object is individually image-processed. Sorting operations are possible by deriving the gripping position and identifying the product type.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Evolutionary Computation (AREA)
  • General Physics & Mathematics (AREA)
  • Physics & Mathematics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Databases & Information Systems (AREA)
  • General Health & Medical Sciences (AREA)
  • Medical Informatics (AREA)
  • Software Systems (AREA)
  • Computing Systems (AREA)
  • Artificial Intelligence (AREA)
  • Health & Medical Sciences (AREA)
  • Multimedia (AREA)
  • Image Analysis (AREA)

Abstract

La présente invention provoque l'extraction par une unité de traitement d'images (401) de chaque objet individuel à partir d'objets à trier qui sont formés à partir d'une pluralité d'objets de forme irrégulière (1a, 1b), de sorte qu'il soit possible de soumettre individuellement chaque objet à un traitement d'image et de dériver la position de préhension, le type de produit, etc.
PCT/JP2023/018115 2022-06-01 2023-05-15 Procédé de traitement d'images pour objet de forme irrégulière WO2023233992A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2022089870 2022-06-01
JP2022-089870 2022-06-01

Publications (1)

Publication Number Publication Date
WO2023233992A1 true WO2023233992A1 (fr) 2023-12-07

Family

ID=89026461

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2023/018115 WO2023233992A1 (fr) 2022-06-01 2023-05-15 Procédé de traitement d'images pour objet de forme irrégulière

Country Status (1)

Country Link
WO (1) WO2023233992A1 (fr)

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2019130429A1 (fr) * 2017-12-26 2019-07-04 Ｋｙｏｔｏ Ｒｏｂｏｔｉｃｓ株式会社 Système d'enregistrement d'informations de préhension d'objet pouvant être saisi
JP2021163078A (ja) * 2020-03-31 2021-10-11 Ｊｆｅスチール株式会社 異物検出装置、異物除去装置および異物検出方法
US20220044365A1 (en) * 2020-08-07 2022-02-10 Adobe Inc. Automatically generating a trimap segmentation for a digital image by utilizing a trimap generation neural network

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2019130429A1 (fr) * 2017-12-26 2019-07-04 Ｋｙｏｔｏ Ｒｏｂｏｔｉｃｓ株式会社 Système d'enregistrement d'informations de préhension d'objet pouvant être saisi
JP2021163078A (ja) * 2020-03-31 2021-10-11 Ｊｆｅスチール株式会社 異物検出装置、異物除去装置および異物検出方法
US20220044365A1 (en) * 2020-08-07 2022-02-10 Adobe Inc. Automatically generating a trimap segmentation for a digital image by utilizing a trimap generation neural network

Similar Documents

Publication Publication Date Title
Ramisa et al. Using depth and appearance features for informed robot grasping of highly wrinkled clothes
RU2407633C2 (ru) Способ и устройство для определения местоположения и извлечения предметов из транспортирующего устройства
US20160078414A1 (en) Solid waste identification and segregation system
Koskinopoulou et al. Robotic waste sorting technology: Toward a vision-based categorization system for the industrial robotic separation of recyclable waste
US9483707B2 (en) Method and device for recognizing a known object in a field of view of a three-dimensional machine vision system
Yildiz et al. DCNN-based screw detection for automated disassembly processes
JP2013010155A (ja) 対象物把持装置、対象物把持装置の制御方法、およびプログラム
CN113877836B (zh) 一种基于视觉检测系统的智能识别分拣系统
JP2020197954A (ja) 対象物の画像抽出処理方法、電子・電気機器部品屑の組成解析方法、電子・電気機器部品屑の組成解析装置、及び電子・電気機器部品屑の処理方法
Guo et al. Visual affordance guided tactile material recognition for waste recycling
Li et al. Demonstrating large-scale package manipulation via learned metrics of pick success
WO2023233992A1 (fr) Procédé de traitement d'images pour objet de forme irrégulière
Xie et al. Detection and classification of defect patterns in optical inspection using support vector machines
CN105057899B (zh) 一种应用于智能激光切割的扫描图像识别方法
Kaipa et al. Enhancing robotic unstructured bin-picking performance by enabling remote human interventions in challenging perception scenarios
Han et al. Toward fully automated metal recycling using computer vision and non-prehensile manipulation
CN110689005A (zh) 一种基于深度学习融合位置和形状信息的商品识别方法
CN115591643A (zh) 一种硅棒的料性识别和区分方法、控制装置及控制系统
Khan et al. Interactive perception based on gaussian process classification for house-hold objects recognition & sorting
Koishihara et al. Hanging work of t-shirt in consideration of deformability and stretchability
Dario et al. An investigation on a robot system for disassembly automation
Lin et al. Inference of 6-DOF robot grasps using point cloud data
JP6958663B2 (ja) 解体物の選別装置
JPH0953915A (ja) 重なり状態認識方法
Reddy et al. Bliss bot for pharmaceutical inspection

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 23815752

Country of ref document: EP

Kind code of ref document: A1