WO2023062828A1 - 学習装置 - Google Patents

学習装置 Download PDF

Info

Publication number
WO2023062828A1
WO2023062828A1 PCT/JP2021/038275 JP2021038275W WO2023062828A1 WO 2023062828 A1 WO2023062828 A1 WO 2023062828A1 JP 2021038275 W JP2021038275 W JP 2021038275W WO 2023062828 A1 WO2023062828 A1 WO 2023062828A1
Authority
WO
WIPO (PCT)
Prior art keywords
annotation
image
unit
trained model
feature information
Prior art date
Application number
PCT/JP2021/038275
Other languages
English (en)
French (fr)
Japanese (ja)
Inventor
健 李
雅信 本江
圭祐 大島
Original Assignee
株式会社Pfu
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 株式会社Pfu filed Critical 株式会社Pfu
Priority to PCT/JP2021/038275 priority Critical patent/WO2023062828A1/ja
Priority to JP2023553882A priority patent/JPWO2023062828A1/ja
Publication of WO2023062828A1 publication Critical patent/WO2023062828A1/ja

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N20/00—Machine learning

Definitions

  • This disclosure relates to a learning device.
  • sorting waste is performed manually. While sorting waste is a simple task, it places a heavy burden on workers who sort waste (hereafter sometimes referred to as "sorters"). A device (sometimes referred to hereinafter as a “waste sorter”) has been developed to do so.
  • the waste sorting device When the waste sorting device performs the work that the sorting worker was doing instead of the sorting worker, the waste sorting device recognizes each waste flowing on the belt conveyor, and based on the recognition result, the robot hand It is conceivable to extract the desired waste (hereinafter sometimes referred to as "desired waste") from the waste mass flowing on the belt conveyor using a vacuum cleaner or a suction pad. Therefore, when a plurality of types of waste are mixed and flowed on the belt conveyor, it is necessary for the waste sorting device to identify the type of waste. Recognition using a trained model generated by machine learning is effective for the waste sorting device to recognize various types of waste.
  • this disclosure proposes a technique that can improve the efficiency of annotation work.
  • the learning device of the present disclosure has a first annotation section, a learning section, and a model management section.
  • the first annotation unit uses a second trained model copied from a first trained model used by an object selection device that selects a desired object based on a recognition result of the object image to identify the features of the object image.
  • a first annotation process is performed to add feature information, which is information indicating the object image, to the object image.
  • the learning unit updates the second learned model by performing machine learning using the object image and the feature information assigned by the first annotation process as teacher data.
  • the model management unit updates the first trained model using the updated second trained model.
  • the efficiency of annotation work can be improved.
  • FIG. 1 is a diagram illustrating a configuration example of an object sorting system according to Example 1 of the present disclosure.
  • FIG. 2 is a diagram illustrating an operation example of the learning device according to the first embodiment of the present disclosure
  • FIG. 3 is a diagram illustrating an operation example of the learning device according to the first embodiment of the present disclosure
  • FIG. 4 is a diagram illustrating an operation example of the learning device according to the first embodiment of the present disclosure
  • FIG. 5 is a diagram illustrating an operation example of the learning device according to the first embodiment of the present disclosure
  • FIG. 6 is a diagram illustrating an operation example of the learning device according to the first embodiment of the present disclosure
  • FIG. 7 is a diagram illustrating an operation example of the learning device according to the first embodiment of the present disclosure
  • FIG. 1 is a diagram illustrating a configuration example of an object sorting system according to Example 1 of the present disclosure.
  • FIG. 2 is a diagram illustrating an operation example of the learning device according to the first embodiment of the present disclosure
  • FIG. 3 is
  • FIG. 8 is a diagram illustrating an operation example of the learning device according to the first embodiment of the present disclosure
  • FIG. 9 is a diagram illustrating an operation example of the learning device according to the first embodiment of the present disclosure
  • FIG. 10 is a diagram illustrating an operation example of the learning device according to the first embodiment of the present disclosure
  • FIG. 11 is a diagram illustrating an operation example of the learning device according to the first embodiment of the present disclosure
  • FIG. 12 is a diagram illustrating an operation example of the learning device according to the first embodiment of the present disclosure
  • FIG. 1 is a diagram illustrating a configuration example of an object sorting system according to Example 1 of the present disclosure.
  • the object sorting system 1 has a learning device 10, a camera 20, an object sorting device 30, a display 40, and an input device 50.
  • a display 40 and an input device 50 are connected to the learning device 10 .
  • Examples of the input device 50 include a pointing device such as a mouse and a keyboard.
  • the learning device 10, camera 20, and object sorting device 30 are connected to each other via a network.
  • the learning device 10 includes a first annotation unit 11, a teacher data storage unit 12, a machine learning unit 13, an annotation learned model storage unit 14, a model management unit 15, a replication unit 16, and a second annotation unit. 17.
  • the object selection device 30 has a recognition trained model storage unit 31 , an image recognition unit 32 , and a desired object extraction unit 33 .
  • the object sorting system 1 shown in FIG. 1 is installed in a waste disposal site where a group of wastes flows on a belt conveyor will be described as an example.
  • the case where the objects to be sorted by the object sorting system 1 are waste will be described below as an example.
  • the object sorting system 1 may be installed in an assembly factory or the like where a group of parts flows on a belt conveyor.
  • objects to be sorted by the object sorting system 1 are not limited to waste, and the object sorting system 1 can be used for various objects.
  • the camera 20 is placed above the belt conveyor on which the waste group is conveyed, and photographs the waste group conveyed by the belt conveyor. Images captured by the camera 20 (hereinafter sometimes referred to as “captured images”) are transmitted from the camera 20 to both the learning device 10 and the object sorting device 30 .
  • the first annotation unit 11 and the image recognition unit 32 receive the captured image transmitted from the camera 20 .
  • the image recognition unit 32 uses the trained model for recognition stored in the trained model storage unit 31 for recognition to identify each waste image (hereinafter referred to as " (sometimes referred to as a "waste image”), and outputs the recognition result to the desired object extraction unit 33.
  • the image recognition unit 32 recognizes the waste image by, for example, performing instance segmentation on the captured image.
  • the desired object extraction unit 33 extracts the desired waste from the waste group conveyed by the belt conveyor according to the recognition result of the image recognition unit 32 .
  • Examples of the desired object extraction unit 33 include a robot hand, a suction pad, and the like.
  • the replication unit 16 operates only when the trained model for annotation is initialized.
  • the duplicating unit 16 duplicates the trained model for recognition stored in the trained model for recognition storage unit 31 when the trained model for annotation is initialized, and uses the trained model for recognition after duplication as an annotation.
  • the trained model for annotation is initialized by the trained model for recognition by storing it in the trained model storage unit 14 for annotation.
  • the first annotation unit 11 uses the learned model for annotation stored in the learned model storage unit 14 for annotation to recognize each waste image existing in the captured image, and determines the characteristics of each waste image. (hereinafter sometimes referred to as "feature information") to each waste image (hereinafter sometimes referred to as “first annotation process”) is performed.
  • the first annotation unit 11 adds feature information to the waste image whose image recognition score is equal to or greater than the threshold TH1.
  • the feature information includes information indicating the outline of the waste image (hereinafter sometimes referred to as "outline information”) and information indicating the color of the waste image (hereinafter sometimes referred to as "color information”). include.
  • the first annotation unit 11 associates the waste image with the feature information already assigned to the waste image, and generates teacher data (hereinafter referred to as "first teacher data") including the waste image and the feature information. ) is generated. Therefore, data in which feature information is added to each waste image in the captured image becomes the first training data.
  • the first annotation unit 11 causes the generated first teacher data to be stored in the teacher data storage unit 12 .
  • the first annotation unit 11 recognizes the waste image by, for example, performing instance segmentation on the captured image.
  • the second annotation unit 17 acquires the first training data stored in the training data storage unit 12, and causes the display 40 to display captured images in which characteristic information is added to each waste image.
  • the input device 50 is operated by an operator, and the operator can use the input device 50 to specify any part of the photographed image displayed on the display 40 .
  • the second annotation unit 17 uses the learned model for annotation stored in the learned model storage unit 14 for annotation to create an image of an arbitrarily specified location in the captured image (hereinafter referred to as a “specified location image”). ) is recognized, and processing (hereinafter sometimes referred to as “second annotation processing”) is performed to add feature information to the designated portion image.
  • the second annotation unit 17 assigns feature information to the designated portion image whose image recognition score is equal to or greater than a threshold TH2 having a value smaller than the threshold TH1 used by the first annotation unit 11 .
  • the second annotation unit 17 associates the designated portion image and the feature information already assigned to the designated portion image with each other, and generates teacher data (hereinafter referred to as “second teacher data”) including the designated portion image and the feature information. ) is generated. Therefore, the second teacher data is data in which the feature information is added to the designated portion image in the captured image.
  • the second annotation unit 17 causes the generated second teacher data to be stored in the teacher data storage unit 12 .
  • the second annotation unit 17 recognizes the waste image by, for example, performing instance segmentation on an arbitrarily specified location in the captured image.
  • the machine learning unit 13 performs machine learning using the first teacher data and the second teacher data stored in the teacher data storage unit 12, and uses the learned model after machine learning to create the learned model storage unit 14 for annotation. Update the trained model for annotation stored in .
  • the model management unit 15 uses the updated trained model for annotation stored in the trained model storage unit 14 for annotation (hereinafter sometimes referred to as “updated trained model for annotation”) to perform recognition
  • the trained model for recognition stored in the trained model storage unit 31 for recognition is updated.
  • the model management unit 15 performs a recognition test similar to the recognition processing performed by the image recognition unit 32 using the updated annotation trained model, and when the recognition accuracy reaches the target accuracy, the updated annotation Update the trained model for recognition with the trained model.
  • ⁇ Operation of learning device> 2 to 12 are diagrams showing operation examples of the learning device according to the first embodiment of the present disclosure.
  • empty bottles are assumed as waste
  • each empty bottle flowing on the belt conveyor is a brown empty bottle (hereinafter sometimes referred to as a “brown bottle”) and a non-brown bottle. (hereinafter sometimes referred to as "other color”) empty bins (hereinafter sometimes referred to as "other color bins").
  • the desired waste is a brown bottle, and the first and second training data for the brown bottle are generated.
  • the captured image I1 includes images of empty bins (hereinafter sometimes referred to as "empty bin images”) B11 and B12 as waste images.
  • the empty bin image B11 is an image of a brown bin (hereinafter sometimes referred to as a “brown bin image”)
  • the empty bin image B12 is an image of a different color bin (hereinafter referred to as a “other color bin image”). is).
  • the first annotation unit 11 adds outline information INb1 to the empty bin image B11 and outline information INb2 to the empty bin image B12 in the captured image I1.
  • the contour information INb1, INb2 is formed by a plurality of coordinate points [x1, y1], [x2, y2], .
  • the first annotation unit 11 adds label information INa1 and image information INc1 to the empty bin image B11, and adds label information INa2 and image information INc2 to the empty bin image B12. do.
  • the label information INa1 includes the color information "brown bottle” of the empty bottle image B11
  • the label information INa2 includes the color information "another color bottle” of the empty bottle image B12.
  • the image information INc1 and INc2 includes an image file name "XXX.bmp" indicating the captured image I1, and height information "1536" and width information "2048” indicating the pixel size of the captured image I1.
  • Label information INa1, outline information INb1, and image information INc1 form annotation data AD1 for the empty bin image B11
  • label information INa2, outline information INb2, and image information INc2 form annotation data AD2 for the empty bin image B12. be done.
  • the first annotation unit 11 associates the captured image I1 and the annotation data AD1 and AD2 with each other to generate first teacher data including the captured image I1 and the annotation data AD1 and AD2.
  • the annotation data AD1 and AD2 are generated by the first annotation unit 11, for example, as files in JSON format.
  • the captured image I2 includes empty bin images B21, B22, B23, B24, and B25 as waste images.
  • the first annotation unit 11 that has executed the first annotation process on the photographed image I2 assigns the label information “brown bin” and the outline information CO1 to the empty bin image B21, and labels the empty bin image B22
  • the information "other color bin” and outline information CO2 are added, the label information "brown bin” and outline information CO4A are added to the empty bin image B24, and the label information "brown bin” is added to the empty bin image B25. and contour information CO5.
  • the correct color of the empty bin image B21 is brown
  • the correct color of the empty bin image B22 is another color
  • the correct color of the empty bin image B24 is brown
  • the correct color of the empty bin image B25 is another color.
  • the contour indicated by the contour information CO4A attached to the empty bin image B24 is deviated from the correct contour of the empty bin image B24. Therefore, the empty bin image B24 is also a failed image like the empty bin image B25.
  • the captured image I2 includes the empty bin image B23, as shown in FIG. 3, label information and outline information are not added to the empty bin image B23. That is, the first annotation unit 11 has not detected the empty bin image B23. Therefore, the empty bin image B23 is also a failed image, like the empty bin images B24 and B25.
  • the label information "brown bin” is applied to the area R6. ” and contour information CO6 are added. That is, the first annotation unit 11 erroneously detected the region R6 as an empty bin image.
  • the empty bin images B23, B24, and B25 are failed images, and the region R6 is erroneously detected as an empty bin image.
  • the operator operates the input device 50 to use the pointer PO displayed on the photographed image I2 to click the empty bin image B23 in the photographed image I2 as shown in FIG. B23 is specified.
  • the operator uses the pointer PO to designate the empty bin image B23 by drawing an area RS surrounding the location where the empty bin image B23 exists in the photographed image I2, as shown in FIG. 5, for example.
  • the second annotation unit 17 uses the learned model for annotation stored in the learned model storage unit 14 for annotation, as shown in FIG. , outline information CO3 is added to the empty bin image B23.
  • the operator uses the pointer PO to specify the region R6 by clicking the region surrounded by the contour indicated by the contour information CO6 in the photographed image I2 as shown in FIG.
  • the second annotation unit 17 deletes the contour information CO6 given to the region R6, as shown in FIG.
  • the operator uses the pointer PO to click the label information "brown bottle” attached to the empty bin image B25 in the photographed image I2 as shown in FIG. Specify information.
  • the second annotation unit 17 enables correction of the specified label information. Therefore, the operator uses, for example, a keyboard to modify the label information given to the empty bin image B25 from "brown bin” to "other color bin” as shown in FIG. According to the correction by the operator, the second annotation unit 17 corrects the label information "brown bottle” attached to the empty bin image B25 to label information "other color bin” indicating the correct color of the empty bin image B25.
  • the operator uses the pointer PO to specify the contour indicated by the contour information CO4A in the captured image I2 as shown in FIG.
  • the second annotation unit 17 enables modification of the designated contour. Therefore, the operator uses the pointer PO to correct the outline of the empty bin image B24 to a correct outline as shown in FIG.
  • the operator uses the pointer PO to modify the contour by moving the vertices of the contour indicated by the contour information CO4A.
  • the second annotation unit 17 corrects the outline information CO4A attached to the empty bin image B24 to outline information CO4B indicating the correct outline of the empty bin image B24.
  • the first embodiment has been described above.
  • the teacher data storage unit 12, the annotation trained model storage unit 14, and the recognition trained model storage unit 31 are implemented as hardware by, for example, memory or storage.
  • the first annotation unit 11, the machine learning unit 13, the model management unit 15, the replication unit 16, the second annotation unit 17, and the image recognition unit 32 include hardware such as a CPU (Central Processing Unit), a DSP (Digital signal processor), FPGA (Field Programmable Gate Array), and ASIC (Application Specific Integrated Circuit).
  • the learning device of the present disclosure includes a first annotation unit (first annotation unit 11 of the embodiment), a learning unit (machine learning unit 13 of the embodiment), a model and a management unit (model management unit 15 in the embodiment).
  • the first annotation unit uses the first trained model (learned model for recognition of the embodiment) used by the object selection device (object selection device 30 of the embodiment) that selects a desired object based on the recognition result of the object image.
  • the first annotation process is performed to add feature information, which is information indicating the features of the object image, to the object image.
  • the learning unit updates the second learned model by performing machine learning using the object image and the feature information added by the first annotation process as teacher data.
  • the model management unit updates the first trained model using the updated second trained model.
  • the first annotation unit can automatically annotate the object image, so the efficiency of the annotation work can be improved.
  • the trained model used for recognizing object images in the object selection device for the first annotation process there is no need to separately prepare a trained model for the first annotation process, making it even more efficient. can be annotated.
  • the second trained model is updated as needed, the accuracy of the first annotation processing improves as the second trained model is updated.
  • the object image recognition accuracy in the object sorting device is improved with the update of the second trained model.
  • the learning device of the present disclosure has a second annotation unit (the second annotation unit 17 of the embodiment).
  • the second annotation unit adds feature information to an arbitrarily designated failed image among failed images, which are object images to which feature information has not been added by the first annotation process, using the second trained model. Perform the second annotation processing.
  • the learning unit updates the second learned model by performing machine learning using the object image, the feature information added by the first annotation process, and the feature information added by the second annotation process as training data. do.
  • the feature information includes contour information indicating the contour of the object image.
  • the score threshold used when feature information is added by the second annotation process is the score threshold used when feature information is added by the first annotation process (example is smaller than the threshold TH1).
  • object sorting system 10 learning device 11
  • teacher data storage unit 13
  • machine learning unit 14 trained model storage unit for annotation
  • model management unit 16 replication unit 17
  • second annotation unit 20
  • camera 30 object sorting device 31 for recognition Trained model storage unit 32

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Software Systems (AREA)
  • Data Mining & Analysis (AREA)
  • Evolutionary Computation (AREA)
  • Medical Informatics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Physics & Mathematics (AREA)
  • Computing Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Mathematical Physics (AREA)
  • Artificial Intelligence (AREA)
  • Image Analysis (AREA)
PCT/JP2021/038275 2021-10-15 2021-10-15 学習装置 WO2023062828A1 (ja)

Priority Applications (2)

Application Number Priority Date Filing Date Title
PCT/JP2021/038275 WO2023062828A1 (ja) 2021-10-15 2021-10-15 学習装置
JP2023553882A JPWO2023062828A1 (enrdf_load_stackoverflow) 2021-10-15 2021-10-15

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/JP2021/038275 WO2023062828A1 (ja) 2021-10-15 2021-10-15 学習装置

Publications (1)

Publication Number Publication Date
WO2023062828A1 true WO2023062828A1 (ja) 2023-04-20

Family

ID=85988214

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2021/038275 WO2023062828A1 (ja) 2021-10-15 2021-10-15 学習装置

Country Status (2)

Country Link
JP (1) JPWO2023062828A1 (enrdf_load_stackoverflow)
WO (1) WO2023062828A1 (enrdf_load_stackoverflow)

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8103650B1 (en) * 2009-06-29 2012-01-24 Adchemy, Inc. Generating targeted paid search campaigns
JP2021010970A (ja) * 2019-07-05 2021-02-04 京セラドキュメントソリューションズ株式会社 ロボットシステム及びロボット制御方法
JP2021043881A (ja) * 2019-09-13 2021-03-18 株式会社クレスコ 情報処理装置、情報処理方法および情報処理プログラム
JP2021099582A (ja) * 2019-12-20 2021-07-01 キヤノン株式会社 情報処理装置、情報処理方法、及びプログラム

Family Cites Families (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP5969685B1 (ja) * 2015-12-15 2016-08-17 ウエノテックス株式会社 廃棄物選別システム及びその選別方法
WO2021131127A1 (ja) * 2019-12-23 2021-07-01 パナソニックＩｐマネジメント株式会社 識別情報付与装置、識別情報付与方法、及びプログラム

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8103650B1 (en) * 2009-06-29 2012-01-24 Adchemy, Inc. Generating targeted paid search campaigns
JP2021010970A (ja) * 2019-07-05 2021-02-04 京セラドキュメントソリューションズ株式会社 ロボットシステム及びロボット制御方法
JP2021043881A (ja) * 2019-09-13 2021-03-18 株式会社クレスコ 情報処理装置、情報処理方法および情報処理プログラム
JP2021099582A (ja) * 2019-12-20 2021-07-01 キヤノン株式会社 情報処理装置、情報処理方法、及びプログラム

Also Published As

Publication number Publication date
JPWO2023062828A1 (enrdf_load_stackoverflow) 2023-04-20

Similar Documents

Publication Publication Date Title
KR102220174B1 (ko) 머신러닝 학습 데이터 증강장치 및 증강방법
CN112837371B (zh) 基于3d匹配的物体抓取方法、装置及计算设备
US8649600B2 (en) System and method for segmenting text lines in documents
CN109685052A (zh) 文本图像处理方法、装置、电子设备及计算机可读介质
JPH0772861B2 (ja) プログラム作成装置
WO2019171628A1 (en) Image processing system and image processing method
US9558467B1 (en) Systems and/or methods for grid-based multi-level digitization of enterprise models
JP2021527256A (ja) ビジョンシステムで画像内のパターンを検出及び分類するためのシステム及び方法
US6574351B1 (en) Destination address area detection apparatus
CN113705673A (zh) 一种文字检测方法、装置、设备及存储介质
CN111144480A (zh) 一种可回收垃圾视觉分类方法、系统、设备
Sree et al. Waste management detection using deep learning
KR20190072074A (ko) 악성 코드 검출 방법 및 시스템
Den Hartog et al. Knowledge-based interpretation of utility maps
CN112215147A (zh) 垃圾投放位置的溯源方法、装置、设备及存储介质
CN113807347A (zh) 一种基于目标检测技术的厨余垃圾杂质识别方法
CN110378337A (zh) 金属切削刀具图纸标识信息视觉输入方法及系统
Breuel An algorithm for finding maximal whitespace rectangles at arbitrary orientations for document layout analysis
Koushik et al. Automated marks entry processing in Handwritten answer scripts using character recognition techniques
WO2023062828A1 (ja) 学習装置
Wang et al. Recyclable waste identification using cnn image recognition and gaussian clustering
CN113971746B (zh) 基于单次人手示教的垃圾分类方法、装置及分拣智能系统
CN115810197A (zh) 一种多模态电力表单识别方法及装置
Yi et al. Detecting retail products in situ using CNN without human effort labeling
Zin et al. A mobile application for offline handwritten character recognition

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21960681

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2023553882

Country of ref document: JP

Kind code of ref document: A

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 21960681

Country of ref document: EP

Kind code of ref document: A1