WO2023188417A1 - 作業分析装置 - Google Patents

作業分析装置 Download PDF

Info

Publication number
WO2023188417A1
WO2023188417A1 PCT/JP2022/016971 JP2022016971W WO2023188417A1 WO 2023188417 A1 WO2023188417 A1 WO 2023188417A1 JP 2022016971 W JP2022016971 W JP 2022016971W WO 2023188417 A1 WO2023188417 A1 WO 2023188417A1
Authority
WO
WIPO (PCT)
Prior art keywords
work
unit
object detection
video data
worker
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/JP2022/016971
Other languages
English (en)
French (fr)
Japanese (ja)
Inventor
智史 上野
一洋 大和
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Fanuc Corp
Original Assignee
Fanuc Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Fanuc Corp filed Critical Fanuc Corp
Priority to US18/841,809 priority Critical patent/US20250166361A1/en
Priority to CN202280093997.2A priority patent/CN118985003A/zh
Priority to JP2024511148A priority patent/JP7794952B2/ja
Priority to PCT/JP2022/016971 priority patent/WO2023188417A1/ja
Priority to DE112022006288.6T priority patent/DE112022006288T5/de
Publication of WO2023188417A1 publication Critical patent/WO2023188417A1/ja
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q50/00—Information and communication technology [ICT] specially adapted for implementation of business processes of specific business sectors, e.g. utilities or tourism
    • G06Q50/04—Manufacturing
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
    • G06V10/776—Validation; Performance evaluation
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q10/00—Administration; Management
    • G06Q10/06—Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling
    • G06Q10/063—Operations research, analysis or management
    • G06Q10/0639—Performance analysis of employees; Performance analysis of enterprise or organisation operations
    • G06Q10/06398—Performance of employee with respect to a job function
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G06V10/62—Extraction of image or video features relating to a temporal dimension, e.g. time-based feature extraction; Pattern tracking
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/40—Scenes; Scene-specific elements in video content
    • G06V20/41—Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/50—Context or environment of the image
    • G06V20/52—Surveillance or monitoring of activities, e.g. for recognising suspicious objects
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/70—Labelling scene content, e.g. deriving syntactic or semantic representations
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/20—Movements or behaviour, e.g. gesture recognition

Definitions

  • the present invention relates to a work analysis device.
  • the position of the worker's hand is identified from the image data with depth captured by the depth sensor
  • the position of the object is identified from the image data captured by the digital camera
  • the details of the actions performed by the worker during the work are determined.
  • a classification model such as the trained model of Patent Document 1 has a problem of being complex and having low interpretability. Furthermore, in order to detect tools (objects) used within an image for task classification as in Patent Document 2, a large amount of calculation is required to scan the entire image. Furthermore, in order to accurately judge the work being performed by workers, it is necessary to adjust the work judgment criteria (parameters) and manually search and annotate images of various work scenes, which is a time-consuming process. It takes. Another problem is that it is unclear whether the accuracy of work judgment will improve even if the search is performed manually.
  • One aspect of the work analysis device of the present disclosure is a work analysis device that analyzes the work of a worker, and adds a work label indicating the work of the worker to video data including the work of the worker.
  • a work labeling unit an object detection annotation unit that annotates objects related to the work of the worker on the video data to which the work labels have been added; and an object detection annotation unit that annotates objects related to the work of the worker; an object detection learning unit that generates an object detection model that detects an object from video data, an object detection unit that detects the object from the video data using the object detection model, and the video to which the work label is assigned.
  • a work determination parameter calculation unit that performs a work determination on the data and calculates a determination criterion that minimizes the error with the assigned work label, and newly input video data using the object detection model and the determination criterion. and a work determination unit that determines the work of the worker.
  • determination criteria can be automatically adjusted and determined in order to accurately determine the work.
  • FIG. 1 is a functional block diagram showing an example of the functional configuration of the work analysis system according to the first embodiment. It is a figure showing an example of a work table.
  • FIG. 3 is a diagram illustrating an example of a user interface for assigning a work label.
  • FIG. 3 is a diagram illustrating an example of video data with different router states. It is a figure which shows an example of the determination result of work determination. It is a figure showing an example of false detection.
  • FIG. 3 is a diagram showing an example of an image area in video data. It is a figure showing an example of operation of a moving body detection part. It is a flow chart explaining parameter calculation processing of a work analysis device. It is a flowchart explaining analysis processing of a work analysis device.
  • FIG. 3 is a diagram illustrating an example of the operation of a joint position work estimation model. It is a flow chart explaining parameter calculation processing of a work analysis device. It is a flowchart explaining analysis processing of a work analysis device.
  • a work label indicating the work of the worker is assigned to video data (video) in which the work of the worker is captured in advance, and a work label indicating the work of the worker is assigned to the video data to which the work label is attached.
  • the common structure is that an object (tool) related to the work is annotated and an object detection model is generated for detecting the object from video data of the annotated object.
  • the generated object detection model is used to determine the work of the worker in the video data to which the work label has been added, and the work of the worker is determined using the generated object detection model.
  • the work of the worker in the newly input video data is determined using the object detection model and the calculated criterion.
  • a joint position work estimation model that estimates joint position information regarding the worker's joints and estimates the work of the worker based on the estimated joint position information and the assigned work label is used.
  • the task label is generated and assigned a work label based on the value related to the accuracy of object detection in work determination using the object detection model and the classification probability of the work estimated from the joint position in work determination using the joint position work estimation model.
  • the points of the first implementation are that the judgment criteria are calculated so that the error is minimized, and the work of the worker in the newly input video data is judged using the object detection model, the joint position work estimation model, and the judgment criteria. It differs from the form.
  • the first embodiment will first be described in detail, and then the second embodiment will be mainly described with a focus on the differences from the first embodiment.
  • FIG. 1 is a functional block diagram showing an example of the functional configuration of the work analysis system according to the first embodiment.
  • the work analysis system 100 includes a work analysis device 1 and a camera 2.
  • the work analysis device 1 and the camera 2 may be connected to each other via a network (not shown) such as a LAN (Local Area Network) or the Internet.
  • a network such as a LAN (Local Area Network) or the Internet.
  • the work analysis device 1 and the camera 2 are equipped with a communication section (not shown) for communicating with each other through such a connection.
  • the work analysis device 1 and the camera 2 may be directly connected to each other by wire or wirelessly via a connection interface (not shown).
  • the work analysis device 1 is connected to one camera 2 in FIG. 1, it may be connected to two or more cameras 2.
  • the camera 2 is a digital camera or the like, and captures two-dimensional frame images of objects such as workers and tools (not shown) projected onto a plane perpendicular to the optical axis of the camera 2 at a predetermined frame rate (for example, 30 fps, etc.). Take an image with The camera 2 outputs the captured frame image to the work analysis device 1 as video data.
  • the video data captured by the camera 2 may be a visible light image such as an RGB color image, a grayscale image, or a depth image.
  • the work analysis device 1 is a computer known to those skilled in the art, and has a control section 10 and a storage section 20, as shown in FIG.
  • the control unit 10 also operates a work registration unit 101, a work labeling unit 102, an object detection annotation unit 103, an object detection learning unit 104, a work determination parameter calculation unit 105, an object detection annotation proposal unit 106, and a work determination unit 107.
  • the work determination unit 107 includes an object detection unit 1071 and a moving body detection unit 1072.
  • the storage unit 20 is a storage device such as a ROM (Read Only Memory) or an HDD (Hard Disk Drive).
  • the storage unit 20 stores an operating system, application programs, etc. executed by the control unit 10, which will be described later.
  • the storage unit 20 includes a video data storage unit 201 , a work registration storage unit 202 , and an input data storage unit 203 .
  • the video data storage unit 201 stores video data of objects such as workers and tools captured by the camera 2.
  • FIG. 2 is a diagram showing an example of a work table. As shown in FIG. 2, the work table has storage areas for "objects" and "works.” In the "object” storage area in the work table, tool names such as "Ryuta (registered trademark)" and “sandpaper” are stored, for example. In the "work” storage area in the work table, tasks such as “gripping” and “filing” are stored, for example.
  • the input data storage unit 203 stores, for example, frame image data in which a tool (object) annotated by the object detection annotation unit 103 (described later) among frame images of video data is associated with an image range in which the tool is shown. is stored as input data when the object detection learning unit 104 (described later) generates an object detection model.
  • the control unit 10 includes a CPU, a ROM, a RAM (Random Access Memory), a CMOS memory, etc., which are configured to be able to communicate with each other via a bus, which are well known to those skilled in the art.
  • the CPU is a processor that controls the work analysis device 1 as a whole.
  • the CPU reads the system program and application program stored in the ROM via the bus, and controls the entire work analysis device 1 according to the system program and application program. As a result, as shown in FIG.
  • the control unit 10 includes a work registration unit 101, a work labeling unit 102, an object detection annotation unit 103, an object detection learning unit 104, a work determination parameter calculation unit 105, an object detection annotation proposal unit 106 and the functions of the work determination unit 107. Further, the work determination unit 107 is configured to realize the functions of the object detection unit 1071 and the moving body detection unit 1072. Various data such as temporary calculation data and display data are stored in the RAM.
  • the CMOS memory is backed up by a battery (not shown) and is configured as a non-volatile memory that maintains its storage state even when the work analysis device 1 is powered off.
  • the work registration unit 101 stores tools to be used (detected objects) in the work table shown in FIG. ) and the work using the tool (object) (the work to be recognized) are registered in association with each other.
  • FIG. 3 is a diagram illustrating an example of a user interface 30 for assigning work labels.
  • the user interface 30 includes an area 301 for playing video data (video) stored in the video data storage unit 201, a play stop button 302, a slide 303, and a work labeling unit 102 for adding video data to the video data.
  • a completion button 330 is provided to complete the annotation of the object.
  • the work labeling unit 102 displays the user interface 30 on a display device (not shown) such as an LCD included in the work analysis device 1, and displays the video data storage unit 201 in the area 301 of the user interface 30, for example. Play back the video data (video data) stored in the .
  • the user operates the playback stop button 302 and the slide 303 via the input device (not shown) of the work analysis device 1 to check the video data, and confirms the video data for the time from 13:10 to 13:13.
  • the user when the user confirms the work of a worker "sanding" in the video data from time 13:18 to time 13:20, the user inputs the work name "sanding" and sends it to the work labeling section 102. assigns a work label of "sanding" to the video data from time 13:18 to time 13:20. Furthermore, when the user confirms the work of a "cleaning" worker in the video data from time 13:20 to time 13:22, the user inputs the work name "cleaning" and the work labeling unit 102 , a work label of "cleaning" is given to the video data from time 13:20 to time 13:22.
  • the work labeling unit 102 may display the results of assigning work labels to the area 310 in chronological order on the display device (not shown) of the work analysis device 1. Then, the work labeling unit 102 outputs the video data to which the work label has been added to the object detection annotation unit 103.
  • the object detection annotation unit 103 annotates, for example, a tool (object) related to a worker's work on video data to which a work label has been added. Specifically, the object detection annotation unit 103 selects, for example, a router among the video data to which a work label of "Ryuta is applied" from time 13:10 to time 13:13 is given in the area 301 of the user interface 30. Displays frame images (still images) in which a tool (object) is depicted, divided at predetermined intervals, or frame images (still images) divided at arbitrary intervals by the user.
  • the frame images (still images) to be displayed be set at predetermined intervals or arbitrary intervals so that, for example, about 20 frame images are displayed for each work label.
  • the object detection annotation unit 103 acquires the image range (bold rectangle) of the tool (object) in each frame image (still image), as shown in FIG.
  • the router button 321 or the like When the router button 321 or the like is pressed, the tool (object) is annotated as a router.
  • the object detection annotation unit 103 also adds frame images (still images) in which tools (objects) are included for each of the video data to which work labels of "micrometering,”"sanding,” and “cleaning” are attached.
  • the image range of the tool (object) in the image is acquired, and the tool (object) is annotated.
  • the object detection annotation unit 103 completes the annotation of the image range in which the tool (object) is shown and the tool (object) for all frame images (still images) of the video data to which work labels have been added, and When the completion button 330 is pressed, the frame in which the tool is shown (with a time stamp) out of the video data (video data) of the time when each work was performed (time from the start of the work to the end of the work)
  • a set of frame image data (hereinafter also referred to as “annotated frame image data”) in which the image range of the image (still image) and the annotated tool (object) are associated is stored in the input data storage unit 203.
  • the object detection learning unit 104 generates an object detection model that performs object detection from video data of an annotated object. Specifically, the object detection learning unit 104 uses, for example, training data in which the annotated frame image data stored in the input data storage unit 203 is input data and the annotated tool (object) is label data. Well-known machine learning is performed to generate an object detection model that is a trained model such as a neural network. The object detection learning unit 104 stores the generated object detection model in the storage unit 20.
  • the work determination parameter calculation unit 105 uses the object detection model generated by the object detection learning unit 104 to perform a work determination on the video data to which a work label has been assigned, and minimizes the error with the assigned work label. Calculate the judgment criteria. Specifically, the work determination parameter calculation unit 105 sets initial values of parameters as determination criteria for each work registered in the work table of FIG. 2, for example. Note that the parameters include, for example, the number of seconds (X) in which the work is performed for X seconds after detecting the object, a threshold value related to the accuracy of object detection for determining that the work is being performed, and This includes a threshold value related to the accuracy of object detection for determining that the work "sanding" is being performed.
  • the parameters include, for example, the number of seconds (X) in which the work is performed for X seconds after detecting the object, a threshold value related to the accuracy of object detection for determining that the work is being performed, and This includes a threshold value related to the accuracy of object detection for determining that the work "sanding" is
  • the work analysis device 1 can detect the most recent If a tool (object) is detected for X seconds, it can be determined that work is being performed using the tool.
  • the work determination parameter calculation unit 105 inputs the annotated frame image data of another video data to which the work label stored in the input data storage unit 203 is attached to the object detection model, and detects a tool (object).
  • the work determination parameter calculation unit 105 determines the work based on the object detection result and the work table of FIG. 2, and calculates the error between the determined work and the correct work label. Then, the work determination parameter calculation unit 105 calculates an evaluation index such as the F1 score of the parameter value for each work based on the error calculated with all the annotated frame image data, and the calculated evaluation index for each work is the maximum.
  • Parameter values for each task are calculated using Bayesian optimization etc. so that
  • the object detection annotation proposal unit 106 uses the parameters (judgment criteria) calculated by the work determination parameter calculation unit 105 to perform a work determination on the video data to which the work label has been added, and creates an annotation based on the determination result of the work determination.
  • the object detection annotation proposal unit 106 proposes frame images (still images) that should be automatically annotated, as described later. Specifically, the object detection annotation proposal unit 106 generates, for example, a tool (object) annotated in another video data to which a work label is stored in the input data storage unit 203 and an image showing the tool. The work is determined using the image data associated with the range.
  • FIG. 5 is a diagram illustrating an example of a determination result of work determination. The upper part of FIG. 5 shows a time series of correct work labels given to the other video data. The middle part of FIG. 5 shows the determination result of the worker's work by the object detection annotation proposal unit 106 for the image data using the object detection model and parameters. The lower part of FIG. 5 shows the object detection results in the image data using the object detection model.
  • the object detection annotation proposal unit 106 displays the extracted frame image (still image) on the user interface 30, and adds a router to the extracted frame image (still image) based on the user's input operation.
  • the image range is acquired and annotated as "Ryuta” by pressing the "Ryuta” button 321.
  • the object detection annotation proposal unit 106 stores image data in which the image range of the frame image (still image) in which the router is shown (to which a time stamp is added) and the annotated router is associated with the input data storage unit 203. Store.
  • the object detection annotation proposal unit 106 creates a frame image (still image) in which sandpaper is captured around time 13:43 in the other video data, and Frame images (still images) showing sandpaper at times when "sanding" was not determined (detected) are extracted.
  • the object detection annotation proposal unit 106 displays each of the extracted frame images (still images) on the user interface 30, acquires the image range of sandpaper in each frame image (still image) based on the user's input operation, When the sandpaper button 323 is pressed, the tool (object) is annotated as sandpaper.
  • the object detection annotation proposal unit 106 inputs image data that associates the image range of the frame image (still image) in which the sandpaper is shown (with a time stamp added) and the annotated sandpaper to the input data storage unit. 203. This allows the accuracy of object detection to be increased without requiring the user to spend time searching for various scenes.
  • the object detection annotation proposal unit 106 may be extracted (still image).
  • the object detection annotation proposal unit 106 displays the extracted frame image (still image) on the user interface 30, and acquires the image range of the tool (object) in the extracted frame image (still image) based on the user's input operation.
  • the tool (object) may be annotated.
  • the object detection learning unit 104 performs machine learning using the image data including the frame image (still image) extracted (proposed) by the object detection annotation proposal unit 106 and annotated with a tool (object), and generates an object detection model.
  • the work determination parameter calculation unit 105 determines the work by inputting the annotated frame image data including the frame image (still image) extracted (proposed) by the object detection annotation proposal unit 106 into the updated object detection model. , calculate the error between the given correct work label and the work judgment result.
  • the work determination parameter calculation unit 105 calculates an evaluation index such as the F1 score of the parameter value for each work based on the calculated error, and performs Bayesian optimization etc. so that the calculated evaluation index for each work is maximized.
  • the object detection learning unit 104 and the work determination parameter calculation unit 105 repeat the process until the number of frame images (still images) extracted (proposed) by the object detection annotation proposal unit 106 runs out or becomes less than a predetermined number. Then, the object detection learning unit 104 outputs the generated object detection model to an object detection unit 1071 described later, and the work determination parameter calculation unit 105 outputs the calculated parameters to the work determination unit 107 described later.
  • the work determination unit 107 determines the work of the worker in the video data newly input from the camera 2 using the object detection model and set parameters (judgment criteria). Specifically, the work determination unit 107 uses, for example, a frame image (still image) of video data newly input from the camera 2 with an object detection model of an object detection unit 1071 (described later) and a moving body detection unit 1072 (described later). Enter. The work determination unit 107 determines the work of the worker based on the tool (object) detection result output from the object detection model, the detection result of the moving object detection unit 1072, the work table shown in FIG. 2, and the parameters. judge.
  • the work determination unit 107 determines whether the tool (object) cannot be detected from the frame image (still image) of the video data and the tool (object) has been detected immediately within X seconds of the frame image.
  • the work of the worker in the frame image may be determined based on a parameter of the number of seconds X, assuming that the worker has been working for X seconds after detecting the object.
  • the work determination unit 107 also uses a threshold value (for example, 70 %, etc.), the worker's work may be determined to be "no work". For example, as shown in FIG.
  • the object detection unit 1071 has an object detection model generated by the object detection learning unit 104, inputs a frame image (still image) of video data newly input from the camera 2 to the object detection model, and detects a tool (object). ) along with the detection results, outputs values related to object detection accuracy such as reliability.
  • the moving object detection unit 1072 detects moving objects such as workers and tools based on changes such as changes in pixel brightness in designated image areas of each frame image (still image) of video data newly input from the camera 2. To detect. Specifically, as shown in FIG. 7, if there is movement such as a change in pixel brightness in the image area indicated by the bold rectangle of the frame image (still image), the moving object detection unit 1072 detects the operation of the video data. Alternatively, it may be determined that the person is working. Furthermore, as shown in the upper part of FIG. 8, when the moving object detection unit 1072 periodically detects movement at intervals of X seconds or less (for example, 5 seconds, etc.) indicated by the dashed rectangle, the worker continuously It may be determined that the work is being performed.
  • X seconds or less for example, 5 seconds, etc.
  • the moving object detecting section 1072 detects a router, etc. from the frame image (still image) at the time indicated by the shaded rectangle by the object detecting section 1071 during the period in which the movement of the moving object is detected. If a tool (object) is detected, it may be determined that work is being performed with the detected tool (object) during the period. On the other hand, the moving object detection unit 1072 may determine that the worker is not working if no movement is detected for more than X seconds.
  • FIG. 9 is a flowchart illustrating the parameter calculation process of the work analysis device 1. The flow shown here is executed when a new tool (object) and work are registered in the work table by a user such as a worker.
  • step S1 the work labeling unit 102 reproduces the video data including the worker's work stored in the video data storage unit 201 on the user interface 30, and assigns the worker's work to the video data based on the input operation by the user. Assign a work label to indicate the work being done.
  • step S2 the object detection annotation unit 103 detects frame images (still images) separated at predetermined intervals for each work label among the video data to which work labels have been added in step S1. Acquire the image range of the tool (object) and annotate the tool (object). The object detection annotation unit 103 extracts a frame image (with a time stamp) in which the tool is shown (with a time stamp) out of the video data (video data) of the time when each work was performed (time from the start of the work to the end of the work). The annotated frame image data in which the image range of the still image (still image) and the annotated tool (object) are associated is stored in the input data storage unit 203.
  • step S3 the object detection learning unit 104 generates an object detection model that performs object detection from the annotated frame image data annotated in step S2.
  • step S4 the work determination parameter calculation unit 105 inputs the annotated frame image data of another video data with the work label stored in the input data storage unit 203 into the object detection model, and detects the tool (object). To detect.
  • step S5 the work determination parameter calculation unit 105 determines the work of the worker based on the object detection result in step S4 and the work table.
  • step S6 the work determination parameter calculation unit 105 calculates the error between the correct work label and the determination result in step S5 for each work.
  • step S7 an evaluation index such as the F1 score of the parameter value is calculated for each task based on the errors calculated for all the video data.
  • step S8 the work determination parameter calculation unit 105 calculates parameters for each work by Bayesian optimization or the like so that the evaluation index for each work is maximized.
  • step S9 the object detection annotation proposal unit 106 uses the parameters (judgment criteria) calculated in step S8 to determine the work of another video data to which a work label has been added.
  • step S10 the object detection annotation proposal unit 106 increases the value related to object detection accuracy in locations where the value related to object detection accuracy is low, such as erroneous detection or non-detection, based on the determination result in step S9. It is determined whether there is a frame image (still image) to be proposed. If there is a proposed frame image (still image), the process returns to step S2, and the processes from step S2 to step S9 are performed again, including the proposed frame image (still image). On the other hand, if there is no frame image (still image) to propose, the work analysis device 1 sets the object detection model generated in step S3 in the object detection unit 1071, and also sets the parameters calculated in step S8 in the work determination unit 107. Set, and end the parameter calculation process.
  • FIG. 10 is a flowchart illustrating the analysis process of the work analysis device 1. The flow shown here is repeatedly executed while video data is input from the camera 2.
  • step S21 the object detection unit 1071 inputs a frame image (still image) of video data newly input from the camera 2 to the object detection model and detects a tool (object).
  • the moving object detection unit 1072 detects a moving object such as a worker or a tool based on changes such as changes in pixel brightness in the designated image area of each frame image (still image) of the video data newly input from the camera 2. Detect.
  • step S23 the work determination unit 107 determines the work of the worker based on the tool (object) detection result in step S21, the moving object detection result in step S22, the set parameters, and the work table. do.
  • the work analysis device 1 can automatically adjust the determination criteria in order to accurately determine the work.
  • the user only needs to label the task and annotate the object, and the optimal parameters will be automatically calculated.
  • the work analysis device 1 can automatically suggest frames in a video that can improve the accuracy of work determination by annotating them.
  • the first embodiment has been described above.
  • the generated object detection model is used to determine the work of a worker in video data to which a work label has been assigned, and a determination criterion that minimizes the error with the assigned work label is calculated. By doing so, the work of the worker in the newly input video data is determined using the object detection model and the calculated determination criteria.
  • a joint position work estimation model that estimates joint position information regarding the worker's joints and estimates the work of the worker based on the estimated joint position information and the assigned work label is used.
  • the task label is generated and assigned a work label based on the value related to the accuracy of object detection in work determination using the object detection model and the classification probability of the work estimated from the joint position in work determination using the joint position work estimation model.
  • the points of the first implementation are that the judgment criteria are calculated so that the error is minimized, and the work of the worker in the newly input video data is judged using the object detection model, the joint position work estimation model, and the judgment criteria. It differs from the form.
  • the work analysis device 1A according to the second embodiment can automatically adjust the determination criteria in order to accurately determine the work. The second embodiment will be described below.
  • FIG. 11 is a functional block diagram showing an example of the functional configuration of the work analysis system according to the second embodiment. Note that elements having the same functions as the elements of the work analysis system 100 in FIG. 1 are given the same reference numerals, and detailed explanations are omitted. As shown in FIG. 11, the work analysis system 100 includes a work analysis device 1A and a camera 2. Camera 2 has the same functions as camera 2 in the first embodiment.
  • the work analysis device 1A includes a control section 10a and a storage section 20.
  • the control unit 10a also includes a work registration unit 101, a work labeling unit 102, an object detection annotation unit 103, an object detection learning unit 104, a work determination parameter calculation unit 105a, a joint position estimation unit 108, a joint position work learning unit 109, and a work determination section 107a.
  • the work determination unit 107a includes an object detection unit 1071, a moving body detection unit 1072, and a joint position work estimation unit 1073.
  • the storage unit 20 also includes a video data storage unit 201, a work registration storage unit 202, and an input data storage unit 203.
  • the storage unit 20, video data storage unit 201, work registration storage unit 202, and input data storage unit 203 are the storage unit 20, video data storage unit 201, work registration storage unit 202, and input data storage unit in the first embodiment. It has the same function as 203.
  • the work registration unit 101, the work labeling unit 102, the object detection annotation unit 103, and the object detection learning unit 104 are the same as the work registration unit 101, the work labeling unit 102, the object detection annotation unit 103, and the object detection annotation unit 103 in the first embodiment. It has the same function as the object detection learning unit 104.
  • the object detection section 1071 and the moving object detection section 1072 have the same functions as the object detection section 1071 and the moving object detection section 1072 in the first embodiment.
  • the joint position estimating unit 108 estimates joint position information regarding the worker's joint positions for each frame image (still image) of the video data stored in the input data storage unit 203 and assigned a work label.
  • frame images may be extracted from video data at appropriate intervals. For example, if the frame rate of video data is 60 fps, the frame image may be extracted at about 24 fps, for example.
  • the joint position estimation unit 108 uses a known method (for example, Kosuke Kanno, Kenta Oku, Kyoji Kawagoe, "Motion detection and classification method from multidimensional time series data", DEIM Forum 2016 G4-5, Alternatively, using Shohei Uezono, Satoshi Ono, "Feature extraction of multimodal sequence data using LSTM Autoencoder", Materials of the Society for Artificial Intelligence Research Group, SIG-KBS-B802-01, 2018), input data storage unit 203 Time series data such as coordinates and angles of joints such as hands and arms of the worker are estimated as joint position information for each frame image (still image) of the video data to which the work label is stored.
  • FIG. 12 is a diagram showing an example of joint position information in a frame image.
  • FIG. 12 shows joint position information when a worker is sanding.
  • the joint position work learning unit 109 performs machine learning using the joint position information estimated by the joint position estimation unit 108 as input data and the work label assigned by the work label assignment unit 102 as label data. Generate a joint position and work estimation model that estimates the work of. For example, when the joint position information of the worker's right hand in FIG. 12 makes one reciprocating motion every 0.3 seconds, as shown in FIG. 13, it is determined that the worker is sanding.
  • the joint position work learning unit 109 generates a joint position work estimation model.
  • the joint position work learning unit 109 may generate a rule base based on the joint position information estimated by the joint position estimation unit 108 and the work label assigned by the work label assignment unit 102. .
  • the work determination parameter calculation unit 105a calculates the work classification probability based on the value related to the accuracy of object detection in work determination using the object detection model and the classification probability of the work estimated from the joint position in work determination using the joint position work estimation model.
  • the judgment criteria are calculated so that the error with the work label is minimized.
  • the work determination parameter calculation unit 105a for example, similarly to the work determination parameter calculation unit 105 of the first embodiment, calculates initial values of parameters as determination criteria for each work registered in the work table of FIG. Set.
  • the work determination parameter calculation unit 105a inputs the annotated frame image data of another video data to which the work label stored in the input data storage unit 203 is attached to the object detection model, and detects the tool (object).
  • the work determination parameter calculation unit 105a determines the work of the worker based on the object detection results and the work table shown in FIG. Further, the work determination parameter calculation unit 105a estimates the joint position information of the worker for each frame image (still image) of the same different video data, and inputs the estimated joint position information into the joint position work estimation model. , to estimate the worker's work and obtain the classification probability estimated from the joint positions. Then, the work determination parameter calculation unit 105a uses the following equation (1), where a is the weighting coefficient of the classification probability of the work estimated from the joint position, and b is the weighting coefficient of the value related to the accuracy of object detection.
  • the values of parameters are calculated by Bayesian optimization or the like so that the error between the calculated classification probability of the work and the correct work label is minimized.
  • Work classification probability a (work classification probability estimated from joint positions) + b (value related to object detection accuracy) ...
  • the parameters include, for example, the number of seconds X in which the work is performed for X seconds after the object is detected, the weight a of the classification probability of the work estimated from the joint position, and the value related to the accuracy of object detection. Weight b is included.
  • the work determination parameter calculation unit 105a outputs and sets the calculated parameters to a work determination unit 107a, which will be described later.
  • the work determination unit 107a determines the work of the worker in the video data newly input from the camera 2 using the object detection model, the joint position work estimation model, and the set parameters (determination criteria). Specifically, the work determination unit 107a inputs, for example, a frame image (still image) of video data newly input from the camera 2 to the object detection model in the object detection unit 1071 and the moving body detection unit 1072. The work determination unit 107a determines the work of the worker based on the detected tool (object), the work table shown in FIG. 2, and the parameters, and acquires a value related to the accuracy of object detection.
  • the work determination unit 107a estimates joint position information of the worker for each frame image (still image) of the same newly inputted video data, and uses the estimated joint position information in a joint position work estimation unit 1073, which will be described later. Input into joint position work estimation model.
  • the work determination unit 107a acquires the estimation result of the worker's work and the classification probability of the work estimated from the joint positions from the joint position work estimation unit 1073, which will be described later. Then, the work determination unit 107a calculates the classification probability of the work from the values related to the classification probability of the work and the accuracy of object detection estimated from the acquired joint positions, the set parameters, and equation (1), and calculates the classification probability of the work.
  • the work of the worker is determined based on the classification probability and the detection result of the moving object detection unit 1072.
  • the joint position work estimation unit 1073 has a joint position work estimation model generated by the joint position work learning unit 109, inputs the joint position information estimated by the work determination unit 107a into the joint position work estimation model, and The estimation result of the work and the classification probability of the work estimated from the joint positions are output to the work determination unit 107a.
  • FIG. 14 is a flowchart illustrating the parameter calculation process of the work analysis device 1A. Note that the processing from step S31 to step S33 is similar to the processing from step S1 to step S3 in FIG. 9, and detailed description thereof will be omitted.
  • step S34 the joint position estimating unit 108 estimates the worker's joint position information for each frame image (still image) of the video data to which the work label stored in the input data storage unit 203 is attached.
  • step S35 the joint position work learning unit 109 performs machine learning using the joint position information estimated in step S34 as input data and the work label assigned in step S31 as label data to estimate the worker's work.
  • a joint position and work estimation model is generated.
  • step S36 the work determination parameter calculation unit 105a inputs the annotated frame image data of another video data to which the work label stored in the input data storage unit 203 is attached to the object detection model, and the detected tool ( object) and a value related to the accuracy of object detection.
  • step S37 the work determination parameter calculation unit 105a determines the work of the worker based on the object detection result in step S36 and the work table.
  • step S38 the work determination parameter calculation unit 105a estimates the worker's joint position information from a frame image (still image) of the same different video data.
  • step S39 the work determination parameter calculation unit 105a inputs the joint position information estimated in step S38 to the joint position work estimation model, and obtains the estimation result of the worker's work and the classification probability estimated from the joint positions.
  • step S40 the work determination parameter calculation unit 105a calculates the parameters (judgment criteria) using Bayesian optimization or the like so that the error between the work classification probability calculated by equation (1) and the correct work label is minimized. Calculate the value.
  • FIG. 15 is a flowchart illustrating analysis processing of the work analysis device 1A. The flow shown here is repeatedly executed while video data is input from the camera 2.
  • step S51 the object detection unit 1071 inputs the frame image (still image) of the video data newly input from the camera 2 to the object detection model, detects the tool (object), and calculates a value related to the accuracy of object detection. get.
  • the moving object detection unit 1072 detects a moving object such as a worker or a tool based on changes such as changes in pixel brightness in the designated image area of each frame image (still image) of the video data newly input from the camera 2. Detect.
  • step S53 the joint position work estimation unit 1073 estimates the worker's joint position information for each frame image (still image) of the newly input video data.
  • step S54 the joint position work estimation unit 1073 inputs the joint position information estimated in step S53 to the joint position work estimation model, estimates the work of the worker, and obtains the classification probability of the work estimated from the joint positions. do.
  • step S55 the work determination unit 107a uses the classification probability and object detection accuracy of the work estimated from the joint positions acquired in step S51 and step S54, the moving object detection result in step S52, and the set parameters.
  • the classification probability of the work is calculated from Equation (1) and the work of the worker is determined based on the calculated classification probability.
  • the work analysis device 1A can automatically adjust the determination criteria in order to accurately determine the work.
  • the user only needs to label the task and annotate the object, and the optimal parameters will be automatically calculated.
  • the second embodiment has been described above.
  • the work analysis devices 1 and 1A are not limited to the above-described embodiments, and include modifications, improvements, etc. within a range that can achieve the purpose.
  • the work analysis devices 1 and 1A are connected to one camera 2, but the invention is not limited to this.
  • the work analysis devices 1 and 1A may be connected to two or more cameras 2.
  • the work analysis devices 1 and 1A have all the functions, but the present invention is not limited to this.
  • the server includes some or all of the unit 105a, the joint position estimation unit 108, the joint position work learning unit 109, the work determination unit 107a, the object detection unit 1071, the moving body detection unit 1072, and the joint position work estimation unit 1073.
  • each function of the work analysis apparatuses 1 and 1A may be realized using a virtual server function or the like on the cloud. Further, the work analysis apparatuses 1 and 1A may be configured as a distributed processing system in which each function of the work analysis apparatuses 1 and 1A is distributed to a plurality of servers as appropriate.
  • the work analysis device 1A did not have the object detection annotation proposal unit 106, but it may have the object detection annotation proposal unit 106. By doing so, when the accuracy of work determination is insufficient, the work analysis device 1A can automatically suggest frames in the video that can improve the accuracy of work determination by annotating them.
  • each function included in the work analysis devices 1 and 1A in the first embodiment and the second embodiment can be realized by hardware, software, or a combination thereof.
  • being realized by software means being realized by a computer reading and executing a program.
  • Non-transitory computer-readable media include various types of tangible storage media. Examples of non-transitory computer-readable media are magnetic recording media (e.g., flexible disks, magnetic tapes, hard disk drives), magneto-optical recording media (e.g., magneto-optical disks), CD-ROMs (Read Only Memory), CD-ROMs, R, CD-R/W, semiconductor memory (for example, mask ROM, PROM (Programmable ROM), EPROM (Erasable PROM), flash ROM, RAM).
  • the program may also be provided to the computer on various types of transitory computer readable media. Examples of transitory computer-readable media include electrical signals, optical signals, and electromagnetic waves.
  • the temporary computer-readable medium can provide the program to the computer via wired communication channels such as electrical wires and optical fibers, or via wireless communication channels.
  • the step of writing a program to be recorded on a recording medium includes not only processes that are performed in chronological order, but also processes that are not necessarily performed in chronological order but are executed in parallel or individually. It also includes.
  • the work analysis device of the present disclosure can take various embodiments having the following configurations.
  • the work analysis device 1 of the present disclosure is a work analysis device that analyzes the work of a worker, and includes a task of assigning a work label indicating the work of the worker to video data including the work of the worker.
  • a labeling unit 102 an object detection annotation unit 103 that annotates objects related to the worker's work on the video data to which a work label has been assigned, and an object detection annotation unit 103 that annotates the video data of the object annotated by the object detection annotation unit 103 .
  • An object detection learning unit 104 that generates an object detection model that performs object detection; an object detection unit 1071 that detects an object from video data using the object detection model; and an object detection learning unit 1071 that uses the object detection model to detect an object from video data; , a work determination parameter calculation unit 105 that calculates a determination criterion that minimizes the error with the assigned work label, and determines the work of the worker in the newly input video data using the object detection model and the determination criterion.
  • a work determination unit 107 is provided. According to this work analysis device 1, the determination criteria can be automatically adjusted in order to accurately determine the work.
  • the object detection annotation proposal unit 106 may be provided to propose a frame image to be annotated based on the object detection annotation proposal unit 106.
  • a joint position estimation unit 108 that estimates joint position information regarding the joint positions of the worker, and joint position information estimated by the joint position estimation unit 108. and the work label information given by the work label giving unit 102, and a joint position work learning unit 109 that creates a joint position work estimation model that estimates the work of the worker based on the work label information given by the work label giving unit 102.
  • a joint position and work estimation unit 1073 that estimates work from joint position information based on the joint position and work estimation model, and a work determination parameter calculation unit 105a that performs object detection in work determination using the object detection model.
  • the unit 107a may determine the work of the worker in the newly input video data using the object detection model, the joint position work estimation model, and the determination criteria. By doing so, the work analysis device 1A can achieve the same effect as (1).
  • the work analysis device 1, 1A according to any one of (1) to (3) further includes a moving object detection section 1072 that detects a moving object in newly input video data, and the work determination section 107, 107a. may determine whether the worker continues to work based on the time interval at which the moving object detection unit 1072 detects a moving object. By doing so, the work analysis devices 1 and 1A can determine the work of the worker with higher accuracy.
  • the criterion is that the work using the tool (object) continues at least after the tool (object) is detected. It may also include a time when the object can be estimated to be present, and a threshold value related to the accuracy of object detection. By doing so, the work analysis devices 1 and 1A can accurately determine the work of the worker even when no tool (object) is detected.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Business, Economics & Management (AREA)
  • Multimedia (AREA)
  • Human Resources & Organizations (AREA)
  • Software Systems (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • General Health & Medical Sciences (AREA)
  • Health & Medical Sciences (AREA)
  • Computational Linguistics (AREA)
  • Economics (AREA)
  • Strategic Management (AREA)
  • Entrepreneurship & Innovation (AREA)
  • Development Economics (AREA)
  • Marketing (AREA)
  • Educational Administration (AREA)
  • Tourism & Hospitality (AREA)
  • General Business, Economics & Management (AREA)
  • Evolutionary Computation (AREA)
  • Social Psychology (AREA)
  • Databases & Information Systems (AREA)
  • Computing Systems (AREA)
  • Medical Informatics (AREA)
  • Artificial Intelligence (AREA)
  • Human Computer Interaction (AREA)
  • Psychiatry (AREA)
  • Game Theory and Decision Science (AREA)
  • Quality & Reliability (AREA)
  • Operations Research (AREA)
  • Manufacturing & Machinery (AREA)
  • Primary Health Care (AREA)
  • Image Analysis (AREA)
  • General Factory Administration (AREA)
PCT/JP2022/016971 2022-03-31 2022-03-31 作業分析装置 Ceased WO2023188417A1 (ja)

Priority Applications (5)

Application Number Priority Date Filing Date Title
US18/841,809 US20250166361A1 (en) 2022-03-31 2022-03-31 Work analysis device
CN202280093997.2A CN118985003A (zh) 2022-03-31 2022-03-31 作业分析装置
JP2024511148A JP7794952B2 (ja) 2022-03-31 2022-03-31 作業分析装置
PCT/JP2022/016971 WO2023188417A1 (ja) 2022-03-31 2022-03-31 作業分析装置
DE112022006288.6T DE112022006288T5 (de) 2022-03-31 2022-03-31 Aufgabenanalysevorrichtung

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/JP2022/016971 WO2023188417A1 (ja) 2022-03-31 2022-03-31 作業分析装置

Publications (1)

Publication Number Publication Date
WO2023188417A1 true WO2023188417A1 (ja) 2023-10-05

Family

ID=88200507

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2022/016971 Ceased WO2023188417A1 (ja) 2022-03-31 2022-03-31 作業分析装置

Country Status (5)

Country Link
US (1) US20250166361A1 (https=)
JP (1) JP7794952B2 (https=)
CN (1) CN118985003A (https=)
DE (1) DE112022006288T5 (https=)
WO (1) WO2023188417A1 (https=)

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2020101036A1 (ja) * 2018-11-16 2020-05-22 株式会社 Ｐｒｅｆｅｒｒｅｄ Ｎｅｔｗｏｒｋｓ 教師信号生成装置、モデル生成装置、物体検出装置、教師信号生成方法、モデル生成方法、およびプログラム
JP2020135417A (ja) * 2019-02-20 2020-08-31 Kddi株式会社 食材又は調味料の使用量を推定する情報装置、プログラム及び方法
WO2021059572A1 (ja) * 2019-09-27 2021-04-01 富士フイルム株式会社 情報処理装置、情報処理装置の作動方法、情報処理装置の作動プログラム
WO2021186592A1 (ja) * 2020-03-17 2021-09-23 株式会社村田製作所 診断支援装置及びモデル生成装置
JP2021157548A (ja) * 2020-03-27 2021-10-07 Ｎｔｔテクノクロス株式会社 発芽判定装置及びプログラム

Family Cites Families (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP6733995B2 (ja) 2016-06-23 2020-08-05 Ｎｅｃソリューションイノベータ株式会社 作業分析装置、作業分析方法、及びプログラム
JP7458623B2 (ja) 2019-10-17 2024-04-01 国立大学法人九州大学 作業分析装置及び作業分析方法

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2020101036A1 (ja) * 2018-11-16 2020-05-22 株式会社 Ｐｒｅｆｅｒｒｅｄ Ｎｅｔｗｏｒｋｓ 教師信号生成装置、モデル生成装置、物体検出装置、教師信号生成方法、モデル生成方法、およびプログラム
JP2020135417A (ja) * 2019-02-20 2020-08-31 Kddi株式会社 食材又は調味料の使用量を推定する情報装置、プログラム及び方法
WO2021059572A1 (ja) * 2019-09-27 2021-04-01 富士フイルム株式会社 情報処理装置、情報処理装置の作動方法、情報処理装置の作動プログラム
WO2021186592A1 (ja) * 2020-03-17 2021-09-23 株式会社村田製作所 診断支援装置及びモデル生成装置
JP2021157548A (ja) * 2020-03-27 2021-10-07 Ｎｔｔテクノクロス株式会社 発芽判定装置及びプログラム

Also Published As

Publication number Publication date
DE112022006288T5 (de) 2024-12-05
CN118985003A (zh) 2024-11-19
JPWO2023188417A1 (https=) 2023-10-05
JP7794952B2 (ja) 2026-01-06
US20250166361A1 (en) 2025-05-22

Similar Documents

Publication Publication Date Title
JP4752721B2 (ja) 移動パターン特定装置、移動パターン特定方法、移動パターン特定プログラム、およびこれを記録した記録媒体
JP7010542B2 (ja) 作業分析装置、作業分析方法、及びプログラム
US20230044842A1 (en) Work analyzing device and work analyzing method
TWI776176B (zh) 手部作業動作評分裝置、方法及電腦可讀取存儲介質
JPWO2020026643A1 (ja) 情報処理装置、情報処理方法及び情報処理プログラム
JP2015095164A (ja) ジェスチャ認識装置およびジェスチャ認識装置の制御方法
JP2019139570A (ja) 判別装置、判別方法およびプログラム
CN113228070B (zh) 用于手动装配线操作的自动重复步骤和循环检测的方法和系统
US20230386049A1 (en) Tracking apparatus, tracking system, tracking method, and recording medium
CN112567401B (zh) 动作分析装置、动作分析方法及其程序的记录媒体
JP7769009B2 (ja) 作業分析装置
JP2014238674A (ja) 情報処理システム、情報処理方法及びプログラム
JP2021047698A (ja) 支援装置、支援装置の制御方法およびプログラム
WO2023188417A1 (ja) 作業分析装置
JP7376446B2 (ja) 作業分析プログラム、および、作業分析装置
US12265930B2 (en) Work estimation apparatus, work estimation method, and non-transitory computer-readable storage medium
JP2024112671A (ja) 作業分析装置、作業分析システム、作業分析方法及びプログラム
CN106169057A (zh) 信息处理装置及方法
JP2020184225A (ja) 情報処理装置、情報処理方法
US20210157805A1 (en) Method of performing a data collection procedure for a process which uses articifical intelligence
JP2024112670A (ja) 作業分析装置、作業分析システム、作業分析方法及びプログラム
CN119501954B (zh) 一种基于视觉的动态人机物体交接方法以及系统
JP7641208B2 (ja) 分析システム、分析方法、プログラム、及び記憶媒体
JP7470069B2 (ja) 指示物体検出装置、指示物体検出方法及び指示物体検出システム
JP2025150871A (ja) 移動体追跡システム

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22935554

Country of ref document: EP

Kind code of ref document: A1

WWE Wipo information: entry into national phase

Ref document number: 18841809

Country of ref document: US

WWE Wipo information: entry into national phase

Ref document number: 112022006288

Country of ref document: DE

ENP Entry into the national phase

Ref document number: 2024511148

Country of ref document: JP

Kind code of ref document: A

WWE Wipo information: entry into national phase

Ref document number: 202280093997.2

Country of ref document: CN

122 Ep: pct application non-entry in european phase

Ref document number: 22935554

Country of ref document: EP

Kind code of ref document: A1

WWP Wipo information: published in national office

Ref document number: 18841809

Country of ref document: US