WO2023017647A1 - 行動分析装置、行動分析方法、行動分析プログラム、撮影装置および行動分析システム - Google Patents

行動分析装置、行動分析方法、行動分析プログラム、撮影装置および行動分析システム Download PDF

Info

Publication number
WO2023017647A1
WO2023017647A1 PCT/JP2022/013286 JP2022013286W WO2023017647A1 WO 2023017647 A1 WO2023017647 A1 WO 2023017647A1 JP 2022013286 W JP2022013286 W JP 2022013286W WO 2023017647 A1 WO2023017647 A1 WO 2023017647A1
Authority
WO
WIPO (PCT)
Prior art keywords
behavior
behavior data
analysis device
data
behavior analysis
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/JP2022/013286
Other languages
English (en)
French (fr)
Japanese (ja)
Inventor
龍 成澤
玄太 松川
賢作 緒方
健 小谷
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Sony Semiconductor Solutions Corp
Original Assignee
Sony Semiconductor Solutions Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Sony Semiconductor Solutions Corp filed Critical Sony Semiconductor Solutions Corp
Priority to EP22855713.8A priority Critical patent/EP4386661A4/en
Priority to US18/681,110 priority patent/US20250124739A1/en
Priority to JP2023541218A priority patent/JP7826321B2/ja
Publication of WO2023017647A1 publication Critical patent/WO2023017647A1/ja
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/20—Movements or behaviour, e.g. gesture recognition
    • G—PHYSICS
    • G05—CONTROLLING; REGULATING
    • G05B—CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS
    • G05B19/00—Program-control systems
    • G05B19/02—Program-control systems electric
    • G05B19/418—Total factory control, i.e. centrally controlling a plurality of machines, e.g. direct or distributed numerical control [DNC], flexible manufacturing systems [FMS], integrated manufacturing systems [IMS] or computer integrated manufacturing [CIM]
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q50/00—Information and communication technology [ICT] specially adapted for implementation of business processes of specific business sectors, e.g. utilities or tourism
    • G06Q50/04—Manufacturing
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/20—Analysis of motion
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/50—Context or environment of the image
    • G06V20/52—Surveillance or monitoring of activities, e.g. for recognising suspicious objects
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/60—Type of objects
    • G06V20/64—Three-dimensional [3D] objects
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/20—Movements or behaviour, e.g. gesture recognition
    • G06V40/28—Recognition of hand or arm movements, e.g. recognition of deaf sign language
    • G—PHYSICS
    • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
    • G16Y—INFORMATION AND COMMUNICATION TECHNOLOGY SPECIALLY ADAPTED FOR THE INTERNET OF THINGS [IoT]
    • G16Y10/00—Economic sectors
    • G16Y10/25—Manufacturing
    • G—PHYSICS
    • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
    • G16Y—INFORMATION AND COMMUNICATION TECHNOLOGY SPECIALLY ADAPTED FOR THE INTERNET OF THINGS [IoT]
    • G16Y20/00—Information sensed or collected by the things
    • G16Y20/20—Information sensed or collected by the things relating to the thing itself
    • G—PHYSICS
    • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
    • G16Y—INFORMATION AND COMMUNICATION TECHNOLOGY SPECIALLY ADAPTED FOR THE INTERNET OF THINGS [IoT]
    • G16Y20/00—Information sensed or collected by the things
    • G16Y20/40—Information sensed or collected by the things relating to personal data, e.g. biometric data, records or preferences
    • G—PHYSICS
    • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
    • G16Y—INFORMATION AND COMMUNICATION TECHNOLOGY SPECIALLY ADAPTED FOR THE INTERNET OF THINGS [IoT]
    • G16Y40/00—IoT characterised by the purpose of the information processing
    • G16Y40/20—Analytics; Diagnosis
    • G—PHYSICS
    • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
    • G16Y—INFORMATION AND COMMUNICATION TECHNOLOGY SPECIALLY ADAPTED FOR THE INTERNET OF THINGS [IoT]
    • G16Y40/00—IoT characterised by the purpose of the information processing
    • G16Y40/30—Control
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02P—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS
    • Y02P90/00—Enabling technologies with a potential contribution to greenhouse gas [GHG] emissions mitigation
    • Y02P90/30—Computing systems specially adapted for manufacturing

Definitions

  • the present disclosure relates to a behavior analysis device, behavior analysis method, behavior analysis program, imaging device, and behavior analysis system that perform behavior analysis using machine learning.
  • Patent Document 1 there is a known method of analyzing the posture of a worker by acquiring skeletal data from features such as the skeletal structure and joint positions of the worker and assigning a posture label to each skeletal data.
  • Patent Document 2 there is known a technique for measuring the degree of progress in each process by recognizing the positions of the worker's wrists and shoulders using a skeletal recognition technique, a sensor, or the like and analyzing the work (for example, Patent Literature 2).
  • this disclosure proposes a behavior analysis device, a behavior analysis method, a behavior analysis program, an imaging device, and a behavior analysis system that can perform appropriate behavior analysis while reducing the load required for learning.
  • a behavior analysis device acquires behavior data representing the behavior of an object during a work process recognized by a model pre-trained to recognize the object. and a determination unit configured to determine, based on the behavior data acquired by the acquisition unit, the time required for the process corresponding to the behavior data.
  • FIG. 1 is a diagram (1) for explaining a user interface according to an embodiment
  • FIG. 2 is a figure (2) for demonstrating the action-analysis process which concerns on embodiment.
  • FIG. 3 is a diagram (3) for explaining the user interface according to the embodiment
  • FIG. 4 is a diagram (4) for explaining the user interface according to the embodiment; It is the figure which compared the conventional process and the action-analysis process which concerns on embodiment.
  • Embodiment 1-1 Overview of Behavior Analysis Processing
  • Embodiment 1-2 Configuration of behavior analysis device according to embodiment 1-3.
  • Configuration of detection device according to embodiment 1-4 Configuration of terminal device according to embodiment 1-5.
  • Procedure of processing according to embodiment 1-6 Modified example according to the embodiment 2.
  • Other embodiments Effects of behavior analysis device, photographing device, and behavior analysis system according to the present disclosure4.
  • Hardware configuration
  • FIG. 1 is a diagram showing an overview of a behavior analysis system 1 according to an embodiment. Specifically, FIG. 1 shows components of a behavior analysis system 1 that executes behavior analysis processing according to the embodiment.
  • the behavior analysis system 1 includes a behavior analysis device 100, an edge 200A, an edge 200B, an edge 200C, and an edge 2004D.
  • the edge 200A, the edge 200B, the edge 200C and the edge 200D have the same configuration, respectively, the detection device 300A, the detection device 300B, the detection device 300C and the detection device 300D, and the terminal device 400A, the terminal device 400B and the terminal device 400C. and a terminal device 400D.
  • edge 200 when there is no need to distinguish between the edge 200A, the detection device 300A, the terminal device 400A, and the like, they are collectively referred to as "edge 200,” “detection device 300,” and "terminal device 400.”
  • the behavior analysis device 100 is an example of an information processing device that executes behavior analysis processing according to the present disclosure.
  • the behavior analysis device 100 is a server or PC (Personal Computer) installed in a factory. Specifically, the behavior analysis device 100 analyzes the behavior of the worker 10 in the factory process.
  • PC Personal Computer
  • the edge 200 is a system installed in a factory for photographing the actions of workers (for example, work processes), and is a terminal device in the action analysis system 1.
  • edge 200A is used to analyze worker 10's behavior.
  • Edge 200B, edge 200C, and edge 200D are used to analyze actions of other workers (not shown).
  • the edge 200 is composed of a detection device 300 and a terminal device 400 .
  • the detection device 300 is a device having a function of photographing the worker 10, such as a digital camera. As will be described later, the detection device 300 includes an image sensor 310 (see FIG. 13) and has a function of recognizing a predetermined object by using a pre-learned model for object recognition.
  • the detection device 300 includes a trained model for recognizing objects that are commonly used in processes at various factories.
  • the objects according to the embodiment are human hands, various tools such as screwdrivers and soldering irons, mice, connectors, bar code readers, and the like.
  • pre-learning process for example, when learning the human hand, object recognition is learned using pre-photographed data that takes into account the different colors of gloves, the size of the hand, the angle of view, etc.
  • object recognition is learned using pre-photographed data that takes into account the different colors of gloves, the size of the hand, the angle of view, etc.
  • the detection device 300 it is possible to generalize the learning process related to machine learning.
  • objects that are commonly used in various factories are subject to pre-learning, thereby reducing the burden associated with machine learning annotations.
  • re-learning suitable for each factory is unnecessary.
  • the weight of inference processing can be reduced, and inference can be made at low cost and at high speed.
  • the terminal device 400 is a device that communicates with or is connected to the detection device 300, and acquires video data captured by the detection device 300 and behavior data indicating the behavior of objects recognized by the detection device 300. In addition, the terminal device 400 communicates with or is connected to the behavior analysis device 100 and transmits video data and behavior data acquired from the detection device 300 to the behavior analysis device 100 .
  • Each device in FIG. 1 conceptually shows the function of the behavior analysis system 1, and can take various aspects depending on the embodiment.
  • the behavior analysis device 100 may be composed of two or more devices having different functions, which will be described later.
  • the number of edges 200 included in the behavior analysis system 1 is not limited to the illustrated number.
  • the behavior analysis system 1 is a system realized by a combination of the behavior analysis device 100 and the edge 200.
  • the behavior analysis system 1 acquires behavior data indicating the behavior of an object during a work process, and determines the required time of the process corresponding to the behavior data based on the acquired behavior data.
  • the behavior analysis device 100 can determine the required time (cycle time) indicating how long it took for one process executed by the worker 10 .
  • the behavior analysis device 100 calculates, for example, the average required time for all workers, and determines whether the worker 10 finished the process faster than other workers. It is possible to analyze the work situation such as
  • the behavior analysis system 1 makes the edge 200 recognize a predetermined object using a pre-learned model, extracts its behavior data, and performs behavior analysis.
  • the object recognition processing is a primary analysis that performs inference processing by lightweight machine learning, and the behavior data is taken out and the behavior analysis is performed according to preset rules. perform analysis and
  • the behavior analysis system 1 can analyze the behavior of the worker 10 without requiring time for annotation or the like and without performing heavy-load machine learning.
  • FIG. 2 is a diagram (1) for explaining behavior analysis processing according to the embodiment.
  • FIG. 2 shows a moving image captured by the edge 200 of the worker 10 proceeding with one process (for example, the process of assembling a part of a product, etc.).
  • the video captured by the edge 200 is transmitted from the edge 200 to the behavior analysis device 100. Via the behavior analysis device 100, the manager can view how the worker 10 proceeds with the work in real time or as a recorded moving image.
  • the behavior analysis device 100 provides a user interface 20 displayed on a display and displays a moving image 25 within the user interface 20 .
  • the moving image 25 includes a tracking indication 22 and a tracking indication 24 indicating that the edge 200 has recognized the object.
  • Tracking display 22 shows that edge 200 recognizes an object, right hand 21 of worker 10 .
  • Tracking display 24 shows edge 200 recognizing an object, left hand 23 of worker 10 .
  • the administrator can watch the work of the worker 10 while operating the operation panel 26 on the user interface 20 .
  • the edge 200 acquires behavior data indicating the behavior of each object while tracking the right hand 21 and the left hand 23 .
  • behavior data is coordinates indicating movement of an object within a photographed screen, which is represented in the form of so-called point cloud data. That is, according to the behavior data acquired by the edge 200, it is possible to recognize behavior such as to which coordinates in the screen the right hand 21 has moved and to which coordinates in the screen the left hand 23 has moved along the time axis. is.
  • the point cloud data may be obtained by tracking an arbitrary point (for example, the center point) of the tracking display 22 or the tracking display 24, or may be obtained by tracking a plurality of points such as four corner points of the tracking display 22 or the tracking display 24. It's okay.
  • the point cloud data is not limited to two-dimensional data, and may be three-dimensional data.
  • the edge 200 has a depth sensor such as a ToF (Time of Flight) sensor, the edge 200 can acquire three-dimensional data of the object.
  • ToF Time of Flight
  • Two videos are displayed side by side on the user interface 30 shown in FIG.
  • a moving image 33 is displayed in which a skilled worker who is skilled in one process is photographed.
  • a moving image 38 of the worker 10 to be compared with the skilled worker is displayed.
  • Each moving image column includes an operation panel 34, a shooting information column 35 indicating which edge 200 was used to shoot, and a date and time information column 27 indicating when the shooting was performed. While visually recognizing these pieces of information, the manager can confirm the actions of the worker 10 when compared with those of the skilled worker.
  • FIG. 4 is a diagram (2) for explaining behavior analysis processing according to the embodiment.
  • FIG. 4 shows a graph display 40 displaying behavior data in which the hand movements of the worker 10 are recorded.
  • the graph display 40 records the movement of the hand performed by the worker 10 until a certain process is completed, and the X-axis and Y-axis movements of the right hand and left hand, respectively.
  • the vertical axis 41 indicates the magnitude of values when the behavior is indicated as coordinates.
  • a horizontal axis 42 indicates time.
  • the first behavior data waveform 43 is a record of hand movements of the worker 10 over time.
  • the first cycle time 44 indicates a group of a series of waveforms from the start to the end of one process. The details of determining the cycle time will be described later.
  • the second behavior data waveform 45 is displayed superimposed on the first behavior data waveform 43 indicating the behavior of the worker 10.
  • a second behavior data waveform 45 indicates behavior data of another worker to be compared with the worker 10, and corresponds to, for example, behavior data of an expert.
  • a second cycle time 46 indicates a cycle time when a skilled worker executes the same process as the process executed by the worker 10 . As shown in FIG. 4 , the second cycle time 46 for the expert is shorter than the first cycle time 44 for the worker 10 .
  • the waveform 47 indicates that both hands of the worker 10 have moved to a position outside the area where the coordinates can be observed (for example, the imaging angle of view of the camera).
  • the behavior analysis device 100 executes a predetermined analysis according to the setting of the administrator or the like.
  • the behavior analysis device 100 determines the similarity between the first behavior data waveform 43 indicating the working situation of the worker 10 and the second behavior data waveform 45 indicating the working situation of the expert. Behavioral analysis is performed by comparison. For example, the behavior analysis device 100 applies similarity analysis called DTW (Dynamic Time Warping). According to such a method, the behavior analysis device 100 can obtain quantitative information on how the work of the worker 10 differs from that of a skilled worker, regardless of the time axis. For example, the behavior analysis device 100 can determine that there is no problem with the work of the worker 10, although it takes longer than the expert. Alternatively, the behavior analysis device 100 detects that there is a difference between the first behavior data waveform 43 and the second behavior data waveform 45, and that the worker 10 may have forgotten one task in the process, It is possible to detect the possibility that an incorrect assembly process has been performed.
  • DTW Dynamic Time Warping
  • FIG. 5 is a diagram (3) for explaining behavior analysis processing according to the embodiment.
  • FIG. 5 shows a graph display 50 displaying the results of a plurality of cycle times when the worker 10 repeats one process for a certain period of time.
  • a result 51 in the graph display 50 indicates the time required for each process when the worker 10 repeats one process.
  • a horizontal axis 52 of the graphical representation 50 indicates the time at which the behavior of the worker 10 was observed.
  • each result 51 is color-coded based on the color-coded display 53 and displayed.
  • the behavior analysis device 100 displays each of the results 51 in different colors based on the similarity between the behavior data of the skilled worker to be compared and the behavior data of the worker 10 .
  • the result 54 indicates a process in which the behavior data of the skilled worker to be compared is similar to the behavior data of the worker 10, and there were no problems.
  • the result 55 indicates a process in which the behavior data of the expert to be compared and the behavior data of the worker 10 are not similar, and there is a problem.
  • result 55 is highlighted in a more prominent way, such as red, compared to result 54 . Thereby, the manager can verify whether or not there was a problem in the behavior of the worker 10 at a certain time.
  • the behavior analysis device 100 may perform processing such as sending an alert to the mobile terminal used by the administrator.
  • behavior analysis device 100 may highlight the result according to the cycle time. For example, behavior analysis device 100 may highlight a process that has a significantly longer cycle time than others. In such a case as well, the behavior analysis device 100 may send an alert to the mobile terminal used by the administrator.
  • FIG. 6 is a diagram (2) for explaining the user interface according to the embodiment.
  • the user interface 60 shown in FIG. 6 displays a list of the moving image column 31, the moving image column 32, the graph display 40, and the graph display 50 shown in FIG.
  • the administrator confirms the graph display 50 and, when finding a result with a problem, selects the result.
  • the behavior analysis device 100 displays the moving image corresponding to the result in the moving image column 31 and the moving image column 32 . Also, the behavior analysis device 100 displays the behavior data corresponding to the result on the graph display 40 .
  • the behavior analysis device 100 receives rules (various setting information) for analysis from an administrator or the like, and determines cycle time or the like according to the rules.
  • FIG. 7 is a diagram (3) for explaining the user interface according to the embodiment.
  • a user interface 70 shown in FIG. 7 includes an image 71 captured by the edge 200 .
  • the image 71 includes area settings 72 .
  • the area setting 72 is an area arbitrarily set by an administrator or the like, and indicates an area within the image 71 in which the worker 10 is assumed to work. For example, when an object is observed in the area setting 72, the behavior analysis device 100 determines that the worker 10 is working. On the other hand, the behavior analysis device 100 determines that one step has been completed when the object being recognized exists in a range beyond the region setting 72 for a predetermined time or longer (for example, several seconds or longer).
  • the behavior analysis device 100 may display a reference region 74, which is the range where objects are frequently located, based on past video history.
  • the reference area 74 indicates that many objects are located at that position in a certain process, and is superimposed on the image 71, for example.
  • the administrator determines the area setting 72 while referring to the reference area 74 .
  • the area setting 72 is indicated by a rectangle in the example of FIG. 7, the area setting 72 may be set in an arbitrary shape instead of a rectangle.
  • the behavior analysis device 100 may set the region setting 72 using three-dimensional information.
  • the setting items 75 include an interval setting 76, a minimum time setting 77, a maximum time setting 78, a flexibility setting 79, and the like.
  • the interval setting 76 is an item for setting the time until it is determined that one process is completed within the range where the object being recognized exceeds the area setting 72 .
  • the minimum time setting 77 is an item that is determined as the minimum time when determining the cycle time of one process. For example, when determining the cycle time, the behavior analysis device 100 avoids determining a cycle time that is shorter than the time input to the minimum time setting 77 .
  • the maximum time setting 78 is an item that is determined as the maximum time when determining the cycle time of one process. For example, when determining the cycle time, the behavior analysis device 100 avoids determining a cycle time that is longer than the time entered in the maximum time setting 78 .
  • Flexibility setting 79 is a numerical value that indicates how strictly the above setting should be applied. In this way, the administrator can efficiently perform secondary analysis of the behavior data acquired by the edge 200 by inputting an approximate cycle time estimated in one process as a rule.
  • FIG. 8 is a diagram (4) for explaining the user interface according to the embodiment.
  • a user interface 80 shown in FIG. 8 includes a list 81 of moving images recorded when one process is repeated.
  • the behavior analysis device 100 detects that the administrator has pressed the re-analysis button 84 after setting the rules, the behavior analysis device 100 analyzes the moving images included in the list 81 based on the rules.
  • the behavior analysis device 100 determines the cycle time of the process by referring to the behavior data corresponding to each moving image based on the information set in the rule. Specifically, the behavior analysis device 100 determines the time during which the object is out of the set area in the behavior data, and determines that the process has ended when the time exceeds the set value. In addition, the behavior analysis device 100 determines the cycle time to be the maximum set value for behavior data in which the behavior of an object can be observed but the time exceeds the set value of the maximum time setting. In this case, the behavior analysis device 100 may send an alert to the manager or the like, noting that there is some problem in the process.
  • the behavior analysis device 100 determines the cycle time 82 of each process listed in the list 81 . In this way, the behavior analysis device 100 performs secondary analysis on behavior data based on the set rules, so that it is possible to determine the cycle time of each step without requiring complex machine learning or other processing. can.
  • the administrator can display the video 88 in the video column 85 by pressing the selection button 83 displayed on the user interface 80 .
  • the manager may select the video with the shortest cycle time and use it as a sample for other workers to view.
  • the manager may select the video with the longest cycle time and view what problems there are in the process.
  • the tracking display 86 and the tracking display 87 are displayed superimposed on the video. As a result, the manager can appropriately confirm how the worker 10 is working.
  • FIG. 9 is a diagram comparing the conventional processing and the behavior analysis processing according to the embodiment.
  • FIG. 9 shows a flow 90 according to the prior art and a flow 96 according to the embodiment.
  • a flow 90 according to the conventional technology includes a first procedure 91 for photographing learning data, a second procedure 92 for annotation, a third procedure 93 for learning processing, a fourth procedure 94 for photographing data to be analyzed, and a fifth procedure 95 for inference processing. Five steps are required.
  • a process is photographed, and labeling (annotation) indicating what kind of work the photographed image corresponds to, and learning processing for recognizing the work. , a lot of time is spent on pre-processing.
  • the flow 96 according to the embodiment it is only necessary to perform the analysis target data photographing as the first procedure 97, the primary machine learning processing as the second procedure 98, and the secondary analysis as the third procedure 99.
  • This is achieved in the flow 96 according to the embodiment by introducing the edge 200 into the factory prior to the first step 97 with a pre-trained model of the object to be recognized.
  • the second procedure 98 executes only relatively lightweight inference processing of object recognition, recognition processing can be executed almost simultaneously with photographing.
  • the third procedure 99 analyzes the behavior data on a rule basis, so that the analysis can be completed without requiring much time unlike the fifth procedure 95 according to the flow 90 according to the conventional technology.
  • by quickly performing the third procedure 99 it is possible to immediately send an alert when an abnormality is detected, which contributes to improving the efficiency of the entire factory.
  • FIG. 10 is a diagram showing a configuration example of the behavior analysis device 100 according to the embodiment.
  • the behavior analysis device 100 has a communication section 110, a storage section 120, and a control section .
  • the behavior analysis device 100 includes an input unit (for example, a keyboard, a touch display, etc.) that receives various operations from an administrator or the like who manages the behavior analysis device 100, and a display unit (for example, a liquid crystal display) for displaying various information. etc.).
  • an input unit for example, a keyboard, a touch display, etc.
  • a display unit for example, a liquid crystal display
  • the communication unit 110 is implemented by, for example, a NIC (Network Interface Card), a network interface controller, or the like.
  • the communication unit 110 is connected to the network N by wire or wirelessly, and transmits/receives information to/from the edge 200 or the like via the network N.
  • the network N is realized by a wireless communication standard or method such as Bluetooth (registered trademark), the Internet, Wi-Fi (registered trademark), UWB (Ultra Wide Band), LPWA (Low Power Wide Area), or the like.
  • the storage unit 120 is implemented by, for example, a semiconductor memory device such as RAM (Random Access Memory) or flash memory, or a storage device such as a hard disk or optical disk.
  • the storage unit 120 has an imaging data storage unit 121 and a rule storage unit 122 .
  • each storage unit will be described in order with reference to FIGS. 11 and 12.
  • FIG. 11 is a diagram showing an example of the imaging data storage unit 121 according to the embodiment.
  • the imaging data storage unit 121 has items such as "imaging data ID”, “imaging date and time”, “image data”, and “point cloud data”.
  • the information stored in the storage unit 120 may be conceptually indicated as "A01", but in reality, each piece of information described later is stored in the storage unit 120.
  • FIG. 11 is a diagram showing an example of the imaging data storage unit 121 according to the embodiment.
  • the imaging data storage unit 121 has items such as "imaging data ID”, “imaging date and time”, “image data”, and “point cloud data”.
  • 11 and 12 the information stored in the storage unit 120 may be conceptually indicated as "A01", but in reality, each piece of information described later is stored in the storage unit 120.
  • Captured data ID is identification information for identifying captured data.
  • Photographing date and time indicates the date and time when the image was captured by the edge 200 .
  • Image data indicates image (moving image) data captured by the edge 200 .
  • Point cloud data indicates data indicating the behavior of the object recognized by the edge 200 .
  • FIG. 12 is a diagram illustrating an example of a rule storage unit according to the embodiment.
  • the rule storage unit 122 has items such as "rule ID”, “set date and time”, and “set information”.
  • Rule ID indicates identification information for identifying a rule.
  • Set date and time indicates the date and time when the rule was set.
  • “Setting information” indicates setting information set as a rule.
  • the setting information is each information included in the setting item 75 shown in FIG.
  • the control unit 130 stores a program (for example, a behavior analysis program according to the present disclosure) stored inside the behavior analysis device 100 by means of, for example, a CPU (Central Processing Unit), MPU (Micro Processing Unit), GPU, etc. Access Memory) etc. is executed as a work area. Also, the control unit 130 is a controller, and may be realized by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).
  • ASIC Application Specific Integrated Circuit
  • FPGA Field Programmable Gate Array
  • control unit 130 has an acquisition unit 131, a determination unit 132, a display control unit 133, and a transmission unit .
  • the acquisition unit 131 acquires various types of information. For example, the acquisition unit 131 acquires behavior data representing the behavior of an object during a work process recognized by a model pre-trained to recognize the object.
  • the acquisition unit 131 acquires behavior data from the edge 200 . More specifically, the acquisition unit 131 acquires the behavior data of the object detected by the image sensor from the image sensor using the model incorporated in the logic chip 312 integrated with the image sensor of the detection device 300. .
  • the acquisition unit 131 may acquire only behavior data (point cloud data), or may acquire image data together with behavior data.
  • the acquisition unit 131 stores the acquired data in the captured data storage unit 121 .
  • the determination unit 132 determines the required time (cycle time) of the process corresponding to the behavior data.
  • the determining unit 132 determines the required time for the process corresponding to the behavior data based on the time information set as the required time for the process.
  • the determination unit 132 determines the delimiter of the process corresponding to the behavior data at the timing when the behavior data indicating the predetermined behavior of the object is observed between the minimum time and the maximum time set as the time information. to decide. For example, the determination unit 132 determines a process break corresponding to behavior data when an object behavior exceeding a predetermined value is observed in the point cloud data, like the waveform 47 shown in FIG. In other words, the determining unit 132 determines the division of the process and determines the required time of the process when the behavior of the object exceeding the predetermined value is observed in the point cloud data.
  • the determination unit 132 may determine whether or not the object has exhibited a predetermined behavior based on the area information set as the work area of the process, and determine the division of the process corresponding to the behavior data. For example, the determination unit 132 may determine the delimiter of the process corresponding to the behavior data at the timing when the behavior data indicating that the object has left the area set as the area information and the predetermined time has passed has been observed. . That is, the determining unit 132 refers to the area setting 72 as shown in FIG. 7, determines the division of the process at the timing indicating that the object has left the area and the predetermined time has passed, and Decide how long it will take.
  • the display control unit 133 displays various user interfaces including various information on the display or the like.
  • the display control unit 133 also receives various information from the administrator via the user interface.
  • the display control unit 133 displays, on the user interface, a list of required times for processes corresponding to behavior data determined by the determination unit 132 over multiple times along the time axis. 5 and 6, the display control unit 133 displays, in graph form, the time required for a plurality of steps performed by the worker 10 over a certain period of time. Thereby, the manager can confirm at a glance how long it takes the worker 10 to complete one process.
  • the display control unit 133 compares the first behavior data arbitrarily selected from among the plurality of behavior data and the second behavior data which is behavior data to be compared, and obtains the second behavior data. Based on the degree of similarity between the behavior data corresponding to one process and the behavior data corresponding to one process in the first behavior data, the portion of the required time corresponding to one process is highlighted. For example, as shown in FIG. 5, in a certain process, when the work situations of the skilled worker and the worker 10 to be compared are not similar, the display control unit 133 displays a portion corresponding to the process with a color that emphasizes the process. to display. This allows the manager to grasp at a glance which process the worker 10 is not performing appropriate work.
  • the display control unit 133 displays the result of comparing the first behavior data arbitrarily selected from among the plurality of behavior data and the second behavior data which is the behavior data to be compared as a graph for the user interface. may be displayed above. For example, as shown in FIG. 4, the display control unit 133 superimposes and displays the waveform of the behavior data of the expert and the waveform of the behavior data of the worker 10 to be compared in a certain process.
  • the display control unit 133 determines whether there is an abnormality in the process corresponding to the second behavior data based on the similarity between the waveform corresponding to the first behavior data in the graph and the waveform corresponding to the second behavior data. It may be determined whether there is For example, the display control unit 133 determines the similarity between the expert and the waveform corresponding to the behavior data of the worker 10 to be compared, based on a technique such as DTW.
  • the display control unit 133 determines the waveform corresponding to the second behavior data based on the similarity between the waveform corresponding to the first behavior data in the graph and the waveform corresponding to the second behavior data. It is determined whether or not the plurality of processes match the plurality of processes corresponding to the first behavior data, and if the plurality of processes do not match, it is determined that there is an abnormality in the process corresponding to the second behavior data. .
  • a plurality of steps may indicate a plurality of small steps performed within a certain cycle time.
  • the display control unit 133 compares the waveform of the skilled worker or the model with the waveform of the worker 10 to be compared, and if a dissimilar portion is detected, there is a possibility that some small process has been omitted. , and an abnormality is detected.
  • the transmission unit 134 transmits a warning to a pre-registered transmission destination. For example, the transmission unit 134 transmits an alert to a mobile terminal held by an administrator. Alternatively, the transmission unit 134 may control display of the alert on the user interface displayed by the display control unit 133 .
  • FIG. 14 is a diagram showing a configuration example of the detection device 300 according to the embodiment.
  • the detection device 300 includes an image sensor 310. Although not shown in FIG. 14, the detection device 300 has an optical system for realizing functions as a digital camera, a communication system for communicating with the terminal device 400, and the like.
  • the image sensor 310 is, for example, a CMOS (Complementary Metal Oxide Semiconductor) image sensor composed of a chip, receives incident light from the optical system, performs photoelectric conversion, and outputs image data corresponding to the incident light. .
  • CMOS Complementary Metal Oxide Semiconductor
  • the image sensor 310 has a configuration in which a pixel chip 311 and a logic chip 312 are integrated via a connection portion 313 .
  • the image sensor 310 also has an image processing block 320 and a signal processing block 330 .
  • the pixel chip 311 has an imaging section 321 .
  • the imaging unit 321 is configured by arranging a plurality of pixels two-dimensionally.
  • the imaging unit 321 is driven by the imaging processing unit 322 to capture an image.
  • the imaging processing unit 322 Under the control of the imaging control unit 325, the imaging processing unit 322 performs driving of the imaging unit 321, AD (Analog to Digital) conversion of analog image signals output by the imaging unit 321, imaging signal processing, and the like. imaging processing related to imaging of the image of .
  • AD Analog to Digital
  • the captured image output by the imaging processing unit 322 is supplied to the output control unit 323 and also supplied to the image compression unit 335 . Also, the imaging processing unit 322 passes the captured image to the output I/F 324 .
  • the output control unit 323 selects the captured image from the imaging processing unit 322 and the signal processing result from the signal processing block 330 from the output I / F 324 to the outside (in the embodiment, the terminal device 400 or the behavior analysis device 100) output control to output That is, the output control unit 323 controls to selectively output at least one of the behavior data indicating the behavior of the detected object and the image to the outside.
  • the output control unit 323 selects the captured image from the imaging processing unit 322 or the signal processing result from the signal processing block 330 and supplies it to the output I/F 324 .
  • the output I/F 324 can output both data. Or output I/F324 can output only behavior data, when behavior analysis device 100 requires only behavior data. That is, the output I/F 324 can output only the signal processing result (behavior data) when the captured image itself is not required in the secondary analysis, so the amount of data to be output to the outside can be reduced. .
  • the signal processing block 330 has a CPU 331, a DSP 332, a memory 333, a communication I/F 334, an image compression section 335, and an input I/F.
  • the CPU 331 and DSP 332 recognize objects from images included in the image compression unit 335 using a pre-learning model incorporated in the memory 333 via the communication I/F 334 or input I/F 336 .
  • the CPU 331 and DSP 332 also acquire behavior data indicating the behavior of the recognized object.
  • the signal processing block 330 detects the behavior of the object contained in the image using the pre-learning model for recognizing the object in cooperation with each functional unit.
  • the detection device 300 can selectively output the image data obtained by the image processing block 320 and the behavior data obtained by the signal processing block 330 to the outside.
  • the detection device 300 may include various sensors in addition to the configuration shown in FIG.
  • the detection device 300 may include a ToF sensor, which is a depth sensor that measures the distance to an object located in space.
  • the detection device 300 can acquire, as behavior data, not only two-dimensional point cloud data shown on the image, but also three-dimensional point cloud data including height information.
  • FIG. 14 is a diagram showing a configuration example of the terminal device 400 according to the embodiment.
  • the terminal device 400 has a communication unit 410, a storage unit 420, and a control unit 430.
  • the communication unit 410 is implemented by, for example, a NIC, a network interface controller, or the like.
  • the communication unit 410 is connected to the network N by wire or wirelessly, and transmits/receives information to/from the behavior analysis device 100 or the like via the network N.
  • the storage unit 420 is implemented, for example, by a semiconductor memory device such as a RAM or flash memory, or a storage device such as a hard disk or optical disk.
  • the storage unit 420 has a photographed data storage unit 421 that stores photographed data including images photographed by the detection device 300 and behavior data.
  • the control unit 430 is realized, for example, by executing a program stored inside the terminal device 400 using a RAM or the like as a work area by a CPU, MPU, GPU, or the like. Also, the control unit 430 is a controller, and may be implemented by an integrated circuit such as an ASIC or FPGA, for example.
  • control unit 430 has a conversion unit 431, an acquisition unit 432, and a transmission/reception unit 433.
  • the conversion unit 431 When an object is detected by the detection device 300, the conversion unit 431 reads the behavior on the image and converts it into behavior data. Note that if the detection device 300 has a function of generating behavior data, the conversion unit 431 does not need to perform conversion processing.
  • the acquisition unit 432 acquires image data and behavior data output from the detection device 300 .
  • the acquisition unit 432 stores the acquired image data and behavior data in the captured data storage unit 421 .
  • the transmission/reception unit 433 receives a request from the behavior analysis device 100 and transmits image data and behavior data to the behavior analysis device 100 according to the request. Further, when receiving a request to photograph the work situation of worker 10 from behavior analysis device 100, transmission/reception unit 433 controls detection device 300 and photographs the work situation of worker 10 according to the request.
  • FIG. 15 is a sequence diagram (1) showing the flow of processing according to the embodiment.
  • the edge 200 acquires a pre-learned model that has learned object recognition such as common tools and human hands used in various factories (step S101). Then, the edge 200 starts photographing the work situation of an arbitrary worker according to the operation of the manager or the like (step S102).
  • the edge 200 recognizes the object during shooting, and acquires the data (behavior data) for recognizing the object (step S103). Note that the edge 200 acquires not only the behavior data but also the image data of the work. The edge 200 then transmits the acquired data to the behavior analysis device 100 (step S104).
  • the behavior analysis device 100 acquires data from the edge 200 (step S105). After that, the behavior analysis device 100 receives rule settings for the data from the administrator in order to determine the time required for the process corresponding to the photographed data (step S106).
  • FIG. 16 is a flowchart showing the flow of processing according to the embodiment.
  • the behavior analysis device 100 determines whether or not a rule setting operation has been received from the administrator (step S201). If the rule setting operation has not been received (step S201; No), the behavior analysis device 100 waits until the operation is received.
  • the behavior analysis device 100 provides data on the user interface (step S202). For example, the behavior analysis device 100 displays the user interface 70 shown in FIG. 7 so that the administrator can input information.
  • the behavior analysis device 100 receives rule settings from the administrator (step S203). After that, the behavior analysis device 100 analyzes the data according to the accepted rule (step S204). For example, the behavior analysis device 100 determines the time required for a certain process.
  • the behavior analysis device 100 determines whether the rule setting is completed (step S205). If the rule setting is not completed (step S205; No), for example, if the administrator continues to operate, the behavior analysis device 100 continues the process of providing data on the user interface. On the other hand, when the rule setting is completed (step S205; Yes), the behavior analysis device 100 ends the rule setting process.
  • FIG. 17 is a flowchart (3) showing the flow of processing according to the embodiment.
  • the edge 200 starts photographing the worker 10 who is working (step S301).
  • the edge 200 recognizes the object while shooting, and acquires the data of the recognition of the object (step S302).
  • the edge 200 transmits the acquired data to the behavior analysis device 100 while continuing to shoot (step S303).
  • the behavior analysis device 100 continuously acquires data from the edge 200 (step S304). Then, the behavior analysis device 100 applies the rule to the acquired data and displays the result on the user interface (step S305).
  • the behavior analysis device 100 determines whether or not an abnormality is detected in the obtained results (step S306). If an abnormality is detected (step S306; Yes), the behavior analysis device 100 transmits an alert to a pre-registered destination or the like (step S307). If no abnormality is detected (step S306; No) or if an alert is sent, the behavior analysis device 100 stores image data of the worker 10 and behavior data in the storage unit 120 (step S308).
  • the edge 200 does not have to be composed of two devices, the detection device 300 and the terminal device 400 .
  • the edge 200 may consist of only one digital camera with camera function, sensor, communication function, and object recognition function.
  • the behavior analysis device 100 has shown an example in which the time required for the process and the like are determined according to the rules set by the administrator and the like.
  • the behavior analysis device 100 may automatically determine the time required for the process and the like through learning processing instead of following the preset rules.
  • the behavior analysis device 100 may determine the required time of the process corresponding to the behavior data by learning the features observed in the behavior data. For example, the behavior analysis device 100 may automatically detect the delimitation of the process by learning the characteristics of the waveform shown in FIG. For example, the behavior analysis device 100 determines that a waveform indicating that an object has entered a certain area and that the movement of the object has exceeded a predetermined threshold value (moved out of a certain area) indicates the start to end of the process. The waveform shape is learned based on the labeled teacher data such as . As a result, the behavior analysis device 100 can automatically determine the time required for the process without receiving a rule from the administrator.
  • the learning process described above may be performed by the edge 200 instead of the behavior analysis device 100 . That is, Edge 200 learns the process start and end characteristics based on object behavior data. Then, the edge 200 acquires the behavior data of the object, determines the delimitation of one process, and passes the data to the behavior analysis device 100 for each delimited process. As a result, the behavior analysis device 100 can omit the process of determining the required time of the process and only perform the behavior analysis, so that the process can be performed more quickly.
  • the edge 200 sends the image data of the worker 10 and the behavior data indicating the behavior of the object to the behavior analysis device 100 .
  • edge 200 may transmit only behavior data to behavior analysis device 100 .
  • the behavior analysis system 1 uses only relatively lightweight data such as point cloud data for processing, so processing can be performed quickly.
  • the behavior analysis system 1 can operate even in a factory that does not have enough line equipment that can withstand the amount of information for handling image data, or that does not have sufficient security to prevent leakage of image data. can be done.
  • the behavior analysis device 100 displays the time required when the worker 10 repeats one process in a graph or the like.
  • the behavior analysis device 100 may display not only the time required for one process but also the time required when a plurality of processes are combined in a graph.
  • the behavior analysis device 100 can detect not only a place where work is stagnant in one process, but also a specific process that is stagnant from the upstream process to the downstream process.
  • each component of each device illustrated is functionally conceptual and does not necessarily need to be physically configured as illustrated.
  • the specific form of distribution and integration of each device is not limited to the one shown in the figure, and all or part of them can be functionally or physically distributed and integrated in arbitrary units according to various loads and usage conditions. Can be integrated and configured.
  • the determination unit 132 and the display control unit 133 may be integrated.
  • the behavior analysis device (the behavior analysis device 100 in the embodiment) according to the present disclosure includes the acquisition unit (the acquisition unit 131 in the embodiment) and the determination unit (the determination unit 132 in the embodiment).
  • the acquisition unit acquires behavior data representing the behavior of the object during the work process recognized by the pre-trained model for recognizing the object.
  • the determination unit determines the time required for the process corresponding to the behavior data.
  • the behavior analysis device acquires the behavior data of the object recognized by the pre-learned model without performing machine learning or inference processing by itself, and based on the acquired data, the required time to decide.
  • the behavior analysis device can perform appropriate behavior analysis without annotation for each step or complicated machine learning processing.
  • the determination unit determines the time required for the process corresponding to the behavior data based on the time information set as the time required for the process.
  • the behavior analysis device can perform behavior analysis according to the intention of the manager by determining the required time for the process based on the time information defined as the rule.
  • the determination unit determines a division of the process corresponding to the behavior data at the timing when the behavior data indicating the predetermined behavior of the object is observed between the minimum time and the maximum time set as the time information. .
  • the behavior analysis device can more accurately determine the required time by accepting the setting of the minimum and maximum time estimated in the process as a rule.
  • the determination unit determines whether or not the object has exhibited a predetermined behavior based on the area information set as the work area of the process, and determines the division of the process corresponding to the behavior data.
  • the behavior analysis device can more accurately detect the delimitation of the process by accepting in advance the work area in which the object behaves from the manager or the like.
  • the behavior analysis device can more accurately detect the delimitation of the process by accepting, as area information, an event such as an object moving out of the angle of view, which is generally presumed to be the end of the work. can.
  • the determination unit determines the required time for the process corresponding to the behavior data by learning the features observed in the behavior data.
  • the behavior analysis device determines the required time based on learning rather than on a rule basis, making it possible to perform an appropriate analysis without the hassle of setting rules.
  • the behavior analysis device includes a display control unit (in the embodiment, a display control A unit 133) is further provided.
  • the behavior analysis device displays a list of the time required for each process, making it easier to grasp the timing of the process in which a problem occurred.
  • the display control unit compares first behavior data arbitrarily selected from a plurality of behavior data with second behavior data that is behavior data to be compared, and compares the second behavior data.
  • the behavior data corresponding to one of the processes is highlighted based on the degree of similarity with the behavior data corresponding to the one process in the first behavior data. do.
  • the behavior analysis device compares, for example, the behavior data of a skilled worker or the like, which serves as a model, with the behavior data of a specific worker, and displays the results in different colors to determine the accuracy and results of the work. It is possible to make it easier for an administrator or the like to understand.
  • the behavior analysis device displays the result of comparing the first behavior data arbitrarily selected from the plurality of behavior data and the second behavior data which is the behavior data to be compared as a graph on the user interface. It further includes a display control unit (in the embodiment, the display control unit 133) that displays the .
  • the behavior analysis device compares the behavior data of a skilled person or the like, which serves as a model, with the behavior data of a specific worker, thereby making it easier for the manager or the like to grasp the accuracy and results of the work. can be done.
  • the display control unit determines whether there is an abnormality in the process corresponding to the second behavior data based on the similarity between the waveform corresponding to the first behavior data in the graph and the waveform corresponding to the second behavior data. Determine whether or not there is
  • the behavior analysis device can appropriately detect that there is some kind of abnormality in the process by determining the similarity of waveforms using a technique such as DTW.
  • the display control unit performs a plurality of steps corresponding to the second behavior data based on the similarity between the waveform corresponding to the first behavior data in the graph and the waveform corresponding to the second behavior data. , whether or not the plurality of processes corresponding to the first behavior data match, and if the plurality of processes do not match, it is determined that there is an abnormality in the process corresponding to the second behavior data.
  • the behavior analysis device can detect the omission of some work by judging the similarity of the waveforms, so it can prevent the manufacturing of defective products and the occurrence of serious accidents. can be suppressed.
  • the behavior analysis device includes a transmission unit (in the embodiment, transmission 134).
  • the behavior analysis device can quickly notify the administrator of the abnormality by sending a warning (alert) regarding some abnormality.
  • the acquisition unit acquires the behavior data of the object detected by the image sensor from the image sensor using the model incorporated in the chip integrated with the image sensor (the logic chip 312 in the embodiment).
  • the behavior analysis device acquires data from an integrated chip (called an AI chip or the like) that can perform object recognition, etc., so it does not need to perform complex inference processing by itself. , a rapid analysis can be performed.
  • an integrated chip called an AI chip or the like
  • the imaging device (the edge 200 in the embodiment) includes an imaging unit (the imaging unit 321 in the embodiment), a detection unit (the signal processing block 330 in the embodiment), an output control unit (an output control unit in the embodiment 323).
  • the imaging unit captures an image including an object.
  • the detection unit uses a pre-learned model for object recognition to detect the behavior of the object included in the image.
  • the output control unit selectively outputs to the outside at least one of the behavior data indicating the behavior of the object detected by the detection unit and the image.
  • the imaging device acquires image data and behavior data at the same time, or selectively outputs one of them to the outside, thereby reducing the amount of data to be handled.
  • the processing load related to the next analysis can be reduced.
  • the behavior analysis system includes a photographing device and a behavior analysis device.
  • the imaging device includes an imaging unit that captures an image including an object, a detection unit that detects the behavior of the object included in the image using a pre-learning model for recognizing the object, and an image of the object detected by the detection unit. Behavior data indicating behavior and an output control unit selectively outputting at least one of the images to the outside are provided.
  • a behavior analysis device includes an acquisition unit that acquires behavior data output from an output unit, and a determination unit that determines the time required for a process corresponding to the behavior data based on the behavior data acquired by the acquisition unit. Prepare.
  • the behavior analysis system transfers data in a state where the shooting side (edge side) has performed light inference processing such as object recognition processing, and performs secondary analysis in the latter stage. Appropriate behavior analysis can be performed while reducing the load.
  • FIG. 18 is a hardware configuration diagram showing an example of a computer 1000 that implements the functions of the behavior analysis device 100.
  • the computer 1000 has a CPU 1100 , a RAM 1200 , a ROM (Read Only Memory) 1300 , a HDD (Hard Disk Drive) 1400 , a communication interface 1500 and an input/output interface 1600 .
  • Each part of computer 1000 is connected by bus 1050 .
  • the CPU 1100 operates based on programs stored in the ROM 1300 or HDD 1400 and controls each section. For example, the CPU 1100 loads programs stored in the ROM 1300 or HDD 1400 into the RAM 1200 and executes processes corresponding to various programs.
  • the ROM 1300 stores a boot program such as BIOS (Basic Input Output System) executed by the CPU 1100 when the computer 1000 is started, and programs dependent on the hardware of the computer 1000.
  • BIOS Basic Input Output System
  • the HDD 1400 is a computer-readable recording medium that non-temporarily records programs executed by the CPU 1100 and data used by such programs.
  • HDD 1400 is a recording medium that records the behavior analysis program according to the present disclosure, which is an example of program data 1450 .
  • a communication interface 1500 is an interface for connecting the computer 1000 to an external network 1550 (for example, the Internet).
  • CPU 1100 receives data from another device via communication interface 1500, and transmits data generated by CPU 1100 to another device.
  • the input/output interface 1600 is an interface for connecting the input/output device 1650 and the computer 1000 .
  • the CPU 1100 receives data from input devices such as a keyboard and mouse via the input/output interface 1600 .
  • the CPU 1100 transmits data to an output device such as a display, an edger, or a printer via the input/output interface 1600 .
  • the input/output interface 1600 may function as a media interface for reading a program or the like recorded on a predetermined recording medium.
  • Media include, for example, optical recording media such as DVD (Digital Versatile Disc) and PD (Phase change rewritable disk), magneto-optical recording media such as MO (Magneto-Optical disk), tape media, magnetic recording media, semiconductor memories, etc. is.
  • optical recording media such as DVD (Digital Versatile Disc) and PD (Phase change rewritable disk)
  • magneto-optical recording media such as MO (Magneto-Optical disk)
  • tape media magnetic recording media
  • magnetic recording media semiconductor memories, etc. is.
  • the CPU 1100 of the computer 1000 implements the functions of the control unit 130 and the like by executing the behavior analysis program loaded on the RAM 1200.
  • the HDD 1400 also stores the behavior analysis program according to the present disclosure and the data in the storage unit 120 .
  • CPU 1100 reads and executes program data 1450 from HDD 1400 , as another example, these programs may be obtained from another device via external network 1550 .
  • the present technology can also take the following configuration.
  • the decision unit determining the time required for the process corresponding to the behavior data based on the time information set as the time required for the process; The behavior analysis device according to (1) above.
  • the decision unit At the timing when behavior data indicating a predetermined behavior of the object is observed between the minimum time and the maximum time set as the time information, a step break corresponding to the behavior data is determined; The behavior analysis device according to (2) above.
  • the decision unit Determining whether or not the object exhibits a predetermined behavior based on area information set as the work area of the process, and determining a division of the process corresponding to the behavior data; The behavior analysis device according to (3) above.
  • the decision unit Determining a step break corresponding to the behavior data at the timing when behavior data indicating that the object has left the area set as the area information and a predetermined time has elapsed is observed; The behavior analysis device according to (4) above.
  • the decision unit determining the duration of a step corresponding to the behavior data by learning features observed in the behavior data; The behavior analysis device according to any one of (1) to (5) above.
  • a display control unit that displays, on a user interface, a list of required times for processes corresponding to the behavior data determined by the determination unit over a plurality of times along the time axis;
  • the behavior analysis device according to any one of (1) to (6), further comprising: (8)
  • the display control unit A first behavior data arbitrarily selected from the plurality of behavior data is compared with a second behavior data as behavior data to be compared, and one step of the second behavior data is performed. highlighting the portion of the required time corresponding to the one step based on the similarity of the corresponding behavior data to the behavior data corresponding to the one step in the first behavior data;
  • the behavior analysis device according to (7) above.
  • a display control unit for displaying, as a graph on a user interface, a result of comparing first behavior data arbitrarily selected from the plurality of behavior data and second behavior data as behavior data to be compared.
  • the behavior analysis device according to any one of (1) to (6), further comprising: (10) The display control unit Whether there is an abnormality in the process corresponding to the second behavior data based on the similarity between the waveform corresponding to the first behavior data and the waveform corresponding to the second behavior data in the graph determine the The behavior analysis device according to (9) above.
  • the display control unit Based on the similarity between the waveform corresponding to the first behavior data in the graph and the waveform corresponding to the second behavior data, a plurality of steps corresponding to the second behavior data It is determined whether or not the plurality of processes corresponding to the behavior data match, and if the plurality of processes do not match, it is determined that there is an abnormality in the process corresponding to the second behavior data.
  • the behavior analysis device according to (10) above.
  • a transmission unit configured to transmit a warning to a pre-registered transmission destination when the display control unit determines that there is an abnormality in the process corresponding to the second behavior data;
  • the computer obtaining behavior data indicative of the behavior of an object during a work process as recognized by a model pre-trained to recognize the object; Based on the acquired behavior data, determining the required time of the process corresponding to the behavior data; behavioral analysis methods, including (15) the computer, an acquisition unit for acquiring behavior data indicative of the behavior of an object during a work process as recognized by a model pre-trained to recognize the object; a determination unit that determines the required time of a process corresponding to the behavior data based on the behavior data acquired by the acquisition unit; A behavior analysis program that functions as a (16) an imaging unit that captures an image including an object; a detection unit that detects the behavior of an object included in the image using a pre-learning model for recognizing the object; behavior data indicating the behavior of the object detected by the detection unit; and an output control unit selectively outputting at least one of the images to the outside; A photographing device comprising a (17) an imaging unit that captures an image including an object; a detection unit that detects the behavior of an object included
  • behavior analysis system 10 worker 100 behavior analysis device 110 communication unit 120 storage unit 121 photographed data storage unit 122 rule storage unit 130 control unit 131 acquisition unit 132 determination unit 133 display control unit 134 transmission unit 200 edge 300 detection device 400 terminal device

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Physics & Mathematics (AREA)
  • Multimedia (AREA)
  • Computing Systems (AREA)
  • General Health & Medical Sciences (AREA)
  • Health & Medical Sciences (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Business, Economics & Management (AREA)
  • Manufacturing & Machinery (AREA)
  • Human Computer Interaction (AREA)
  • Psychiatry (AREA)
  • Social Psychology (AREA)
  • Economics (AREA)
  • General Business, Economics & Management (AREA)
  • Medical Informatics (AREA)
  • Databases & Information Systems (AREA)
  • Software Systems (AREA)
  • Evolutionary Computation (AREA)
  • Artificial Intelligence (AREA)
  • Quality & Reliability (AREA)
  • Development Economics (AREA)
  • Accounting & Taxation (AREA)
  • General Engineering & Computer Science (AREA)
  • Biomedical Technology (AREA)
  • Automation & Control Theory (AREA)
  • Human Resources & Organizations (AREA)
  • Marketing (AREA)
  • Primary Health Care (AREA)
  • Strategic Management (AREA)
  • Tourism & Hospitality (AREA)
  • Image Analysis (AREA)
  • General Factory Administration (AREA)
PCT/JP2022/013286 2021-08-12 2022-03-22 行動分析装置、行動分析方法、行動分析プログラム、撮影装置および行動分析システム Ceased WO2023017647A1 (ja)

Priority Applications (3)

Application Number Priority Date Filing Date Title
EP22855713.8A EP4386661A4 (en) 2021-08-12 2022-03-22 Behavior analysis device, behavior analysis method, behavior analysis program, imaging device and behavior analysis system
US18/681,110 US20250124739A1 (en) 2021-08-12 2022-03-22 Action analysis device, action analysis method, action analysis program, photographing device, and action analysis system
JP2023541218A JP7826321B2 (ja) 2021-08-12 2022-03-22 行動分析装置、行動分析方法、行動分析プログラムおよび行動分析システム

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2021-131776 2021-08-12
JP2021131776 2021-08-12

Publications (1)

Publication Number Publication Date
WO2023017647A1 true WO2023017647A1 (ja) 2023-02-16

Family

ID=85200155

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2022/013286 Ceased WO2023017647A1 (ja) 2021-08-12 2022-03-22 行動分析装置、行動分析方法、行動分析プログラム、撮影装置および行動分析システム

Country Status (4)

Country Link
US (1) US20250124739A1 (https=)
EP (1) EP4386661A4 (https=)
JP (1) JP7826321B2 (https=)
WO (1) WO2023017647A1 (https=)

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2016157357A (ja) * 2015-02-26 2016-09-01 株式会社日立製作所 作業者品質管理方法及び作業者品質管理装置
US20170061326A1 (en) * 2015-08-25 2017-03-02 Qualcomm Incorporated Method for improving performance of a trained machine learning model
JP2020013341A (ja) 2018-07-18 2020-01-23 コニカミノルタ株式会社 作業工程管理システム、作業工程管理方法、および作業工程管理プログラム
JP2020201772A (ja) 2019-06-11 2020-12-17 株式会社 日立産業制御ソリューションズ 姿勢分析プログラム、および、姿勢分析装置
JP2021086218A (ja) * 2019-11-25 2021-06-03 オムロン株式会社 協調作業システム、解析装置および解析プログラム
WO2021131552A1 (ja) * 2019-12-27 2021-07-01 パナソニックＩｐマネジメント株式会社 作業分析装置及び作業分析方法

Family Cites Families (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20090112541A1 (en) * 2007-10-26 2009-04-30 Joel Anderson Virtual reality tools for development of infection control solutions
CN111723668A (zh) * 2013-09-06 2020-09-29 日本电气株式会社 安全系统、安全方法和非暂时性计算机可读介质
US11734958B2 (en) * 2015-06-19 2023-08-22 eConnect, Inc. Predicting behavior from surveillance data
WO2017031175A1 (en) * 2015-08-17 2017-02-23 University Of Maryland, Baltimore Automated surgeon performance evaluation
CA3003058A1 (en) * 2015-10-29 2017-05-04 Sharp Fluidics Llc Systems and methods for data capture in an operating room
JP6977296B2 (ja) * 2017-04-03 2021-12-08 株式会社ジェイテクト 作業時間計測システム及びそれを用いた作業指示システム
JP6817974B2 (ja) * 2018-01-10 2021-01-20 株式会社 日立産業制御ソリューションズ 計算機システム
EP3540872A1 (en) * 2018-03-15 2019-09-18 Wöhner Besitz GmbH A hybrid busbar for a busbar system
JP7390628B2 (ja) * 2019-03-28 2023-12-04 パナソニックＩｐマネジメント株式会社 識別情報付与装置、識別情報付与方法、及びプログラム

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2016157357A (ja) * 2015-02-26 2016-09-01 株式会社日立製作所 作業者品質管理方法及び作業者品質管理装置
US20170061326A1 (en) * 2015-08-25 2017-03-02 Qualcomm Incorporated Method for improving performance of a trained machine learning model
JP2020013341A (ja) 2018-07-18 2020-01-23 コニカミノルタ株式会社 作業工程管理システム、作業工程管理方法、および作業工程管理プログラム
JP2020201772A (ja) 2019-06-11 2020-12-17 株式会社 日立産業制御ソリューションズ 姿勢分析プログラム、および、姿勢分析装置
JP2021086218A (ja) * 2019-11-25 2021-06-03 オムロン株式会社 協調作業システム、解析装置および解析プログラム
WO2021131552A1 (ja) * 2019-12-27 2021-07-01 パナソニックＩｐマネジメント株式会社 作業分析装置及び作業分析方法

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
See also references of EP4386661A4

Also Published As

Publication number Publication date
JPWO2023017647A1 (https=) 2023-02-16
JP7826321B2 (ja) 2026-03-09
EP4386661A4 (en) 2024-10-09
US20250124739A1 (en) 2025-04-17
EP4386661A1 (en) 2024-06-19

Similar Documents

Publication Publication Date Title
JP6590609B2 (ja) 画像解析装置及び画像解析方法
Moutinho et al. Deep learning-based human action recognition to leverage context awareness in collaborative assembly
US20230410321A1 (en) Information processing apparatus, control method, and program
JP4752721B2 (ja) 移動パターン特定装置、移動パターン特定方法、移動パターン特定プログラム、およびこれを記録した記録媒体
US12175566B2 (en) Information processing device, display method, and program storage medium for monitoring object movement
JP2010067079A (ja) 行動解析システムおよび行動解析方法
JP6988790B2 (ja) 群衆種類識別システム、群衆種類識別方法および群衆種類識別プログラム
US9595095B2 (en) Robot system
JP7331509B2 (ja) ボルト検査装置、ボルト検査システム、ボルト検査方法及びプログラム
US20230109640A1 (en) Skeleton detection system and work management device
JP2015225630A (ja) 作業管理装置、作業管理システムおよび作業管理方法
JP6191160B2 (ja) 画像処理プログラムおよび画像処理装置
US12536799B2 (en) Moving image integration device, moving image integration method, and moving image integration program
WO2023017647A1 (ja) 行動分析装置、行動分析方法、行動分析プログラム、撮影装置および行動分析システム
JP2023074948A (ja) 作業認識装置、作業認識方法、及び作業認識プログラム
JP6803431B2 (ja) プログラム、情報処理装置及び情報処理方法
JP7484569B2 (ja) 画像処理装置、画像処理方法およびプログラム
JP7350222B1 (ja) 動作分析装置、動作分析方法、及び動作分析プログラム
JP2009110239A (ja) 作業動作分析システム及び作業動作分析方法
US20240070894A1 (en) Object detection device and method
JP7641208B2 (ja) 分析システム、分析方法、プログラム、及び記憶媒体
Lahya et al. Deep Learning for Physical Load Estimation: Insights from ViLoad Video Dataset
Guo et al. Computer vision–based management of construction workers' unsafe behaviour
WO2025109937A1 (ja) 監視システム及び監視方法
Hemalatha et al. Real-Time Object Detection and Tracking Robot with IoT Integration Using Faster R-CNN

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22855713

Country of ref document: EP

Kind code of ref document: A1

WWE Wipo information: entry into national phase

Ref document number: 2023541218

Country of ref document: JP

WWE Wipo information: entry into national phase

Ref document number: 18681110

Country of ref document: US

NENP Non-entry into the national phase

Ref country code: DE

ENP Entry into the national phase

Ref document number: 2022855713

Country of ref document: EP

Effective date: 20240312

WWP Wipo information: published in national office

Ref document number: 18681110

Country of ref document: US