JP2024025410A - Work classification device, work classification method, program and learning device - Google Patents

Abstract

The present invention provides a work classification device, a work classification method, a program, and a learning device that can appropriately classify actual work into a plurality of work categories. [Solution] A work classification device uses a trained model that has learned to classify teaching work into a plurality of work categories based on a plurality of teaching images in which teaching work is photographed by a plurality of cameras; The present invention includes an actual image acquisition unit that acquires a plurality of real images of actual work, and a classification unit that classifies the actual work into a plurality of work categories based on the learned model and the actual images. [Selection diagram] Figure 1

Description

The present disclosure relates to a work classification device, a work classification method, a program, and a learning device.

Patent Document 1 describes a work data management system that classifies work performed by workers at a work site into a plurality of detailed chronological processes based on work data including video data of the work site. In the work data management system described in Patent Document 1, a detailed process performed by a worker is specified based on a change in the position of a predetermined part of the worker's body, such as the head or hand.

Patent No. 6824838

However, the work data management system described in Patent Document 1 has a problem in that it may be difficult to classify detailed processes for work in which it is difficult to photograph changes in the position of a predetermined body part.

The present disclosure has been made in order to solve the above problems, and aims to provide a work classification device, a work classification method, a program, and a learning device that can appropriately classify actual work into a plurality of work categories. purpose.

In order to solve the above problems, a work classification device according to the present disclosure learns to classify the teaching work into a plurality of work categories based on a plurality of teaching images in which the teaching work is photographed by a plurality of cameras. a trained model; a real image acquisition unit that obtains a plurality of real images of the real work taken by the plurality of cameras; and a real image acquisition unit that divides the real work into the plurality of work categories based on the learned model and the real images. A classification unit that performs classification.

The work classification method according to the present disclosure includes the step of acquiring a plurality of actual images of actual work taken by a plurality of cameras, and the step of acquiring a plurality of actual images of actual work taken by a plurality of cameras, The method includes the step of classifying the actual work into the plurality of work categories based on the learned model and the actual image.

A program according to the present disclosure includes a step of acquiring a plurality of actual images of an actual work taken by a plurality of cameras, and a step of acquiring a plurality of actual images of an actual work taken by a plurality of cameras, and a step of obtaining a plurality of work classifications based on a plurality of teaching images of a teaching work taken by the plurality of cameras. A computer is caused to perform a step of classifying the actual work into the plurality of work categories based on the learned model in which the classification of the taught work has been learned and the actual image.

A learning device according to the present disclosure inputs a plurality of real images of real work taken by a plurality of cameras, and generates a machine learning model that outputs a result of classifying the real work into a plurality of work categories. It includes a machine learning unit that performs machine learning based on a plurality of teaching images in which classifications are labeled.

According to the work classification device, work classification method, program, and learning device of the present disclosure, actual work can be appropriately classified into a plurality of work categories.

FIG. 1 is a block diagram illustrating a configuration example of a work classification system according to an embodiment of the present disclosure. FIG. 2 is a schematic diagram showing a configuration example of a work database according to an embodiment of the present disclosure. FIG. 2 is a schematic diagram illustrating an example of the definition of work divisions according to an embodiment of the present disclosure. FIG. 2 is a schematic diagram showing an example of time classification according to an embodiment of the present disclosure. FIG. 2 is a schematic diagram showing an example of a teaching image according to an embodiment of the present disclosure. FIG. 2 is a schematic diagram showing an example of a teaching image according to an embodiment of the present disclosure. FIG. 2 is a schematic diagram showing an example of a teaching image according to an embodiment of the present disclosure. FIG. 2 is a schematic diagram showing an example of a teaching image according to an embodiment of the present disclosure. FIG. 2 is a schematic diagram showing an example of a teaching image according to an embodiment of the present disclosure. FIG. 2 is a schematic diagram showing an example of a teaching image according to an embodiment of the present disclosure. FIG. 2 is a schematic diagram showing an example of a teaching image according to an embodiment of the present disclosure. FIG. 2 is a schematic diagram showing an example of a teaching image according to an embodiment of the present disclosure. FIG. 2 is a schematic diagram showing an example of a teaching image according to an embodiment of the present disclosure. 1 is a flowchart illustrating an example of the operation of the work classification device according to the embodiment of the present disclosure. FIG. 1 is a schematic diagram illustrating an implementation example of a work classification system according to an embodiment of the present disclosure. FIG. 1 is a schematic diagram illustrating an implementation example of a work classification system according to an embodiment of the present disclosure. FIG. 1 is a schematic block diagram showing the configuration of a computer according to at least one embodiment.

Hereinafter, a work classification device, a work classification method, a program, and a learning device according to embodiments of the present disclosure will be described with reference to FIGS. 1 to 17. FIG. 1 is a block diagram illustrating a configuration example of a work classification system according to an embodiment of the present disclosure. FIG. 2 is a schematic diagram illustrating a configuration example of a work database according to an embodiment of the present disclosure. FIG. 3 is a schematic diagram illustrating an example of the definition of work divisions according to the embodiment of the present disclosure. FIG. 4 is a schematic diagram illustrating an example of time classification according to an embodiment of the present disclosure. 5 to 13 are schematic diagrams showing examples of teaching images according to the embodiment of the present disclosure. FIG. 14 is a flowchart illustrating an example of the operation of the work classification device according to the embodiment of the present disclosure. 15 and 16 are schematic diagrams illustrating an implementation example of a work classification system according to an embodiment of the present disclosure. FIG. 17 is a schematic block diagram showing the configuration of a computer according to at least one embodiment. In addition, in each figure, the same reference numerals are used for the same or corresponding components, and the description thereof will be omitted as appropriate.

(Configuration of work classification system)
FIG. 1 shows a configuration example of a work classification system 100 according to an embodiment of the present disclosure. The work classification system 100 shown in FIG. 1 includes a work classification device 1, a plurality of cameras 2, a plurality of worker terminals 3, and a work database 4. The work classification system 100 is a system that classifies the work included in the photographed images into a plurality of work categories based on the images photographed by the plurality of cameras 2. Note that, in this embodiment, examples regarding the content of work etc. are based on the assembling work of an aircraft or its parts. However, the application of this embodiment is not limited to this example.

The camera 2 is installed at the work location of the work to be classified and photographs the work. The camera 2 is, for example, a web camera, and transmits image data representing captured still images or moving images to the work classification device 1 via a communication line (not shown). The image data representing a moving image includes, for example, video data representing an image and audio data representing sounds surrounding the camera 2. Further, each image data captured by the plurality of cameras 2 includes time information that is synchronized with each other. For example, the camera 2 detects parts and products to be worked on, one or more workers who work on the parts and products, and tools, measuring instruments, jigs, auxiliary materials, etc. used by the workers. (hereinafter collectively referred to as "tools, etc."). In addition, in this embodiment, the work category of the photographed work, the tools used, etc. are determined based on the image photographed by the camera 2 using a trained model 14 learned by supervised machine learning, which will be described later. Identify. An image that becomes teaching data during machine learning of the learned model 14 is called a teaching image. Further, the work included in the teaching image is referred to as a teaching work. Further, an image that serves as a standard for classification (a target for classification) is called a real image, and a work included in the real image is called a real work.

The worker terminal 3 is, for example, a tablet terminal, and is carried by each worker. Each worker inputs into the worker terminal 3 information indicating, for example, the start date and time (start time) and the completion date and time (completion time) of the actual work to be performed. The worker terminal 3 registers information indicating the input start date and time, completion date and time, etc. into the work database 4.

FIG. 2 shows an example of the configuration of the work database 4. As shown in FIG. As shown in FIG. 2, the work database 4 stores a plurality of records 41. In this embodiment, the work database 4 includes a collection of files that store information to be managed, and a system that manages the information in the files. In the example shown in FIG. 2, each record 41 includes work content (work location, work item, information indicating whether it is a rework work), start time (year, month, day, hour, and minute), completion time (year, month, day, hour, and minute), and work. Contains multiple fields that store data representing the person's name. The work database 4 responds to an inquiry specifying, for example, a worker's name, date and time, etc. with information indicating a start time and a completion time. Alternatively, the work database 4 responds to an inquiry specifying the work location, date and time, etc. with information indicating the work content, worker name, start time, and completion time. In this embodiment, the work content is, for example, information for distinguishing whether the work is a net work or a work to correct a defect that occurred in the net work (this is referred to as rework work). Contains information. Information indicating that the work is a rework work is registered by the worker from the worker terminal 3, for example.

(Configuration of work classification device)
The work classification device 1 shown in FIG. 1 can be configured using, for example, computers such as one or more servers, personal computers, and tablet terminals, and includes hardware such as computers, peripheral devices, and The following parts are provided as a functional configuration consisting of a combination of software such as programs. That is, the work classification device 1 includes a taught image acquisition section 11 , a teaching section 12 , a machine learning section 13 , a learned model 14 , an actual image acquisition section 15 , a classification section 16 , an input/output section 17 , and a storage section 18 . Further, the learned model 14 includes a learned model 141 for process analysis and a learned model 142 for tool analysis. Note that part or all of the functional configuration within the work classification device 1 may be configured on a network in a cloud environment, an on-premises environment, or the like. Further, the work classification device 1 is also an example of a configuration of a learning device according to the present disclosure. Note that the input/output unit 17 has a functional configuration that inputs and outputs information to and from the operator using, for example, a keyboard, a mouse, a display, a touch display, and the like.

Here, with reference to FIGS. 3 and 4, an example of work classification and an example of classification of time related to work in this embodiment will be described. FIG. 3 shows an example of work divisions in this embodiment. In this embodiment, work is classified into four work categories: main work, incidental work, preparatory work, and spare work. The definition of each work is as follows. The main work is a work in which the worker is holding a tool or is making work sounds. Main work is also called net work. Further, the incidental work is work other than net work, preparatory work, and surplus work. Preparation work is work in which the worker goes to pick up or put things down. Slack work is work in which the worker is talking with someone else.

FIG. 4 shows an example of time classification in this embodiment. In this embodiment, time related to work (for example, working time) is classified into five types of time: main time (net time), incidental time, preparation time, slack time, and withdrawal time. Subject time (net time) is the time for subject work (net work), and is the net time directly required to change the shape of parts, assembly, etc., and is unavoidable time in the work process. be. The main time (net time) is, for example, the time for drilling work, the time for cleaning before sealing, etc. Further, the incidental time is the time for incidental work, and is the time that occurs regularly and is indirectly consumed in order to change the shape of the parts, the state of assembly, etc. The accompanying time is, for example, time for general cleaning, which is cleaning of products and parts, and time for inspection. Further, the preparation time is the time for preparation work, and is the time for preparation and cleanup that occurs when switching tasks. Preparation time is, for example, time for preparing tools or parts and time for cleaning up. In addition, slack time is time for slack work, time to recover from fatigue, time for rework (rework time) for defects, and unavoidable time due to discipline and factory management methods. The spare time is, for example, time for breaks, morning meetings, lunch meetings, education, cleaning of the work place, etc. The separation time is the time required for preparations, setup, and cleanup to be performed while being separated from the work area (or work target) by a predetermined distance or more. This is the time to leave the work area within the above incidental time, preparation time, or slack time. The leaving time includes, for example, the time to go to the tool room to borrow something, the time to mix paint, the time to throw away trash, etc.

The teaching image acquisition unit 11 included in the work classification device 1 acquires a plurality of teaching images obtained by photographing the teaching work by the plurality of cameras 2, and stores them in the storage unit 18. The teaching unit 12 displays the teaching image on the input/output unit 17, and labels the teaching image with the content of the work classification and the type of work tool, etc., according to the operator's input operation to the input/output unit 17. The teaching image thus obtained is stored in the storage unit 18.

In this embodiment, the types of tools are classified into main work tools, preparatory work tools, and auxiliary work tools. The main body work tool is a tool mainly used for the main body work, and includes, for example, a drill tool, a caulking tool, a sealing tool, and the like. The preparatory work tools are tools mainly used in the preparatory work, and include, for example, jigs such as measuring instruments, rulers, and templates, and jigs such as cutting dies. Ancillary work tools are tools mainly used in ancillary work, such as tools used in general cleaning. The machine learning unit 13 performs machine learning (or relearning) on the learned model 14 (or the unlearned model of the learned model 14) using the labeled teaching image as teacher data. The machine learning unit 13 inputs a plurality of real images of the actual work taken by a plurality of cameras, and generates a machine learning model that outputs the results of classifying the actual work into a plurality of work categories. Machine learning is performed based on multiple teaching images labeled by type. In this embodiment, the trained model 14 includes a process analysis trained model 141 and a tool analysis trained model 142, but the tool analysis trained model 142 may be omitted. In this case, the learned model 14 is synonymous with the learned model 141 for process analysis.

The trained model for process analysis 141 is a trained model that has learned classification of teaching work into a plurality of work categories based on a plurality of teaching images in which teaching work is photographed by a plurality of cameras 2. The trained model for process analysis 141 is a trained model that includes, for example, a neural network as an element, and uses machine learning to train the neurons in each layer of the neural network so that a solution to be obtained for a large amount of input data is output. Weighting factors have been optimized. The trained model for process analysis 141 is configured, for example, by a combination of a program that performs calculations from input to output and weighting coefficients (parameters) used in the calculations. The trained model 141 for process analysis is a trained machine learning model that receives one or more frames of real images as input, outputs the estimated classification of work divisions, and performs machine learning using labeled teaching images as training data. It's a model.

Further, the trained model for tool analysis 142 is a trained model that includes, for example, a neural network as an element, similar to the trained model for process analysis 141 described above. The trained model for tool analysis 142 receives one or more frames of real images as input, outputs the estimated classification of the type of tool, etc., and uses the teaching image labeled with the type of tool, etc. as training data for the machine. It is a trained machine learning model. Note that when the tool analysis trained model 142 cannot classify the type of tool, etc., it outputs data to that effect.

5 to 13 show examples of teaching images. FIG. 5 shows an example of a teaching image when the operator within the broken line block is performing the main work (rivet caulking work in this example). The teaching image shown in FIG. 5 is labeled with "main work". FIG. 6 shows an example of a teaching image when a worker within a block indicated by a broken line is performing associated work (in this example, recording work, recording quality, etc.). The teaching image shown in FIG. 6 is labeled with "auxiliary work". FIG. 7 shows an example of a teaching image when a worker within a block indicated by a broken line is performing a preparation work (in this example, a work of picking up a part to prepare the part). The teaching image shown in FIG. 7 is labeled with "preparation work".

Further, FIG. 8 shows an example of a teaching image including a main work tool (a drill tool in this example). The teaching image shown in FIG. 8 is labeled with "tool for main body work." FIG. 9 shows an example of a teaching image including a main work tool (a caulking tool in this example). The teaching image shown in FIG. 9 is labeled with "main work tool". FIG. 10 shows an example of a teaching image that includes a preparatory work tool (in this example, a scale). The teaching image shown in FIG. 10 is labeled with "tool for preparatory work." FIG. 11 shows an example of a teaching image including a preparatory work tool (in this example, a vise). The teaching image shown in FIG. 11 is labeled with "tool for preparatory work." FIG. 12 shows an example of a teaching image including a preparatory work tool (a jig in this example). The teaching image shown in FIG. 12 is labeled with "tool for preparatory work." FIG. 13 shows an example of a teaching image that includes an accessory work tool (in this example, a vacuum cleaner). The teaching image shown in FIG. 13 is labeled with "ancillary work tool."

The actual image acquisition section 15 acquires a plurality of actual images taken by the plurality of cameras 2 and stores them in the storage section 18 .

The classification unit 16 classifies the actual work into a plurality of work categories based on the trained model 14 (trained model for process analysis 141) and the actual image, and stores the classified results in the storage unit 18. In this embodiment, the plurality of work categories include main work, preparatory work, incidental work, and extra work.

Furthermore, based on the trained model (trained model for tool analysis 142) and the actual image, the classification unit 16 divides the actual work into multiple work categories by reflecting the analysis results of the work tools included in the actual image. Classify. For example, even if the trained model for process analysis 141 identifies the work as the main work, if the trained model for tool analysis 142 does not identify it as the main work tool, the classification unit 16 classifies the work classification as the main work. Identify other than work.

Further, the classification unit 16 classifies the actual work into a plurality of work categories based on the learned model 14 and the actual image, with reference to the work database 4 that records the content of the actual work, the start date and time, and the completion date and time. For example, the classification unit 16 refers to the work database 4, and if the work content in the work database 4 is "rework work", the classification unit 16 classifies the work classification as "repair work" regardless of the work classification specified using the learned model 14. Classified as "extra work".

In addition, the classification unit 16 has a function of detecting the area where the worker included in the actual image is located and the distance from the work target, a function of identifying the worker, etc. by performing predetermined image recognition processing. are doing. In this case, there are no limitations on the method of image recognition processing, and existing processing methods can be used. The classification unit 16 also has a function of determining whether the area where the worker is located is within a predetermined work area (for example, within a predetermined distance from the work target).

The storage unit 18 stores information indicating the teaching image, the labeling result, the actual image, the classification result of the actual image, and the like.

(Example of operation of work classification device)
An example of the operation of the work classification device 1 will be described with reference to FIG. 14. The flowchart shown in FIG. 14 shows the flow of the work category classification process by the work classification device 1 shown in FIG. In the process shown in FIG. 14, with real images to be classified for a predetermined period of time stored in the storage unit 18, for example, an operator specifies the real images to be processed to the input/output unit 17 and processes them. It will start when you instruct it to start. It is assumed that the learned models 14 (the learned model 141 for process analysis and the learned model 142 for tool analysis) have undergone machine learning. Further, a case will be described using an example in which the actual image to be processed is an image of a specific worker.

When the process shown in FIG. 14 is started, first, the classification unit 16 acquires one or more frames worth of real images of one or more cameras 2 stored in the storage unit 18 (step S11), and The area where the worker is located in the image is detected (step S12). Next, the classification unit 16 determines whether the detected area is within the work area (step S13). If the detected area is not within the work area (step S13: No), the classification unit 16 classifies the photographing time of the analyzed real image as the departure time (step S14).

If the detected area is within the work area (step S13: Yes), the classification unit 16 refers to the work database 4 and determines whether the shooting time of the actual image being analyzed is "rework work" or not. (Step S15). If the photographing time of the real image being analyzed is "rework" (step S15: Yes), the classification unit 16 classifies the photographing time of the analyzed real image as a margin time (step S16).

If the shooting time of the actual image being analyzed is not "rework work" (step S15: No), the classification unit 16 detects the work classification using the trained model for process analysis 141 (step S17), The tool type is detected using the learned model for tool analysis 142 (step S18). Next, the classification unit 16 determines whether the detected work is the main work (step S19). If the detected work is the main work (step S19: Yes), the classification unit 16 determines whether the detected tool is for the main work (step S20). If the detected tool is for the main work (step S20: Yes), the classification unit 16 classifies the photographing time of the analyzed real image as the main work time (step S21).

On the other hand, if the detected work is not the main work (step S19: No), or if the detected tool is not for the main work (step S20: No), the classification unit 16 classifies the detected work It is determined whether or not is a preparatory work (step S22). If the detected work is a preparation work (step S22: Yes), the classification unit 16 classifies the photographing time of the analyzed real image as a preparation time (step S23). If the detected work is not a preparatory work (step S22: No), the classification unit 16 classifies the photographing time of the analyzed actual image into the associated time (step S24).

After the processing in step S14, step S16, step S21, step S23, or step S24, the classification unit 16 determines whether all classification processing for the real image to be processed has been completed (step S25). If all the classification processes for the real images to be processed have been completed (step S25: Yes), the classification unit 16 ends the process shown in FIG. 14. If all the classification processes for the real images to be processed have not been completed (step S25: No), the classification unit 16 re-executes the processes from step S11 onwards.

(Example of implementation of work classification system)
An implementation example of the work classification system 100 described with reference to FIG. 1 will be described with reference to FIGS. 15 and 16. FIG. 15 shows an implementation example of a function to create the trained model 14, and FIG. 16 shows an implementation example of a function to classify tasks.

In the implementation example shown in FIG. 15, first, the worker 201 performs a teaching task and takes a picture with the web camera 202 (step S201). The photographed image or motion data is registered in the database 203 (step S202). Next, the IE (Industrial Engineering) observer 204 uses the terminal 205 to teach (label) the work classification to image data representing a still image or a moving image (step S203). Next, machine learning is executed to create a trained machine learning model 206 (step S204). Next, model deployment is performed on the trained machine learning model 206 to construct an executable trained machine learning model 207 (step S205).

In the implementation example shown in FIG. 16, a plurality of workers 301 are photographed using a plurality of time-synchronized Web cameras 303 (step S301). The plurality of web cameras 303 are time-synchronized by receiving time data on the network from the time server 304. Image data representing the photographed still image or moving image is registered in the cloud DB (database) 305 (step S302).

Further, the worker 301 uses the tablet terminal 302 to input arrival completion data into the arrival completion system 310 on the cloud computing 309 with which the time is synchronized (step S307). The work supervisor 311 monitors the data registered in the completion system 310 using the terminal 312.

The cloud computing 306 executes the trained machine learning model 207 on the image data representing the registered still image or moving image to classify the work (step S303). The work classification data (work classification results) are registered in the cloud DB 307 (step S304). Furthermore, the learned machine learning model 206 is retrained by utilizing the classification results (step S304 and step S204 in FIG. 15).

Next, the cloud computing 308 executes the query, and the consistency with the completed completion data registered in the completed completion system 310 is verified (step S305). Thereafter, the IE observer 204 uses the terminal 205 to perform work analysis (step S306).

In the work classification system shown in FIGS. 15 and 16, by photographing the worker using a web camera, it is possible to obtain camera data of the worker without deploying a work observer. Also, using synchronized time data, images can be taken with multiple cameras and combined using the time data. In addition, camera data is automatically analyzed using a machine learning model that has learned the work categories (main work, incidental work, preparatory work, slack work (slack time)) through supervised learning, and is compared with work completion data. A consistency check is performed. The analysis results are automatically stored in the database and do not require input by work observers. Since only cameras and data on completed work clothes are used, observations can be made without putting any physical strain on the workers.

Note that the relationship between the implementation examples shown in FIGS. 15 and 16 and the configuration shown in FIG. 1 is as follows. Web camera 202 and web camera 303 correspond to camera 2 shown in FIG. The database 203 corresponds to the combination of the teaching image acquisition section 11 and the storage section 18 shown in FIG. Cloud DB 305 corresponds to the combination of real image acquisition section 15 and storage section 18 shown in FIG. The terminal 205 corresponds to the combination of the teaching section 12 and the input/output section 17 shown in FIG. A trained machine learning model 206 and a trained machine learning model 207 correspond to the trained model 14 shown in FIG. The tablet terminal 302 corresponds to the worker terminal 3 shown in FIG. The combination of cloud computing 309 and completion system 310 corresponds to work database 4 shown in FIG. Cloud computing 306 corresponds to the classification unit 16 shown in FIG. Cloud DB 307 corresponds to storage unit 18 shown in FIG.

The implementation example of the work classification system shown in FIGS. 15 and 16 utilizes the following functional configuration. That is, the implementation examples shown in FIGS. 15 and 16 include a general-purpose camera (web camera) connected to a network, time data synchronized on a network, and an AI (Artificial Intelligence) built in a cloud environment or an on-premises environment. ), machine learning (ML) tools, and work completion data stored in a database built in a cloud or on-premises environment.
Note that the classification results of the work categories can be utilized, for example, in a productivity improvement system that analyzes production efficiency indicators.
In addition, in this implementation example, the productivity improvement system executes the trained machine learning model 207 on the cloud computing 306, but if the trained machine learning model 207 can be executed, how can the productivity improvement system do it? may be configured. As a variant, in an analysis environment where low latency is required, the productivity improvement system may run the trained machine learning model 207 on an edge computing device implemented in conjunction with a general-purpose camera (web camera). May be executed.

According to the work classification system shown in Figures 15 and 16, even tasks for which there is no previous experience in a specific work area can be analyzed using supervised machine learning, and can be generalized to tasks with long cycle times and low frequency. can be applied to Furthermore, by photographing the worker using a web camera, camera data on the worker can be obtained without deploying a work observer. Also, using synchronized time data, images can be taken with multiple cameras and combined using the time data. Camera data acquired with a web camera is automatically analyzed using a trained model that has learned the work categories (main work, preparatory work, incidental work, slack work (or slack time)) using supervised learning, and It is possible to check consistency with complete data. The analysis results are automatically stored in the database and do not require input by work observers. In addition, since supervised learning is completed only by defining main work, preparatory work, incidental work, and spare work, general-purpose observation is possible. This makes it easy to observe operations with long cycle times. In addition, since only the camera and work clothes data are used, availability for work in narrow spaces, for example, is improved. Furthermore, since there is no need to attach markers, there is no physical burden on the worker.

(effect)
In the work classification device, work classification method, program, and learning device configured as described above, the learning method learns the classification of teaching work into a plurality of work categories based on a plurality of teaching images captured by a plurality of cameras. The actual work is classified into multiple work categories based on the completed model and multiple real images taken of the actual work by multiple cameras. Therefore, according to the work classification device, work classification method, program, and learning device of the embodiment, actual work can be appropriately classified into a plurality of work categories.

Furthermore, in the embodiment described above, since the plurality of work classifications include main work, preparatory work, incidental work, and extra work, actual work can be classified in detail.

Furthermore, in the above embodiment, the work classification device, the work classification method, the program, and the learning device reflect the analysis results of the work tools included in the actual image, so that the actual work can be classified in detail. .

Further, in the above embodiment, the work classification device, the work classification method, the program, and the learning device refer to the database (work database 4) that records the contents of the actual work, the start date and time, and the completion date and time. It can be classified in detail.

(Other embodiments)
Although the embodiment of the present disclosure has been described above in detail with reference to the drawings, the specific configuration is not limited to this embodiment, and includes design changes within the scope of the gist of the present disclosure. . For example, in the above embodiment, there are four work categories: main work, auxiliary work, preparatory work, and spare work, but the present invention is not limited to this. For example, the work divisions may be divided into two, main work and non-main work, or three, main work, incidental or preparatory work, and extra work. Further, in the above embodiment, the input to the trained model 14 is a still image or a moving image, but the invention is not limited to this, and for example, audio data recorded together with the moving image may be input. Alternatively, audio data recognition processing may be performed in parallel with the classification using the learned model 14. In this case, for example, the presence or absence of a predetermined work sound can be added to the factors for determining whether or not the work is the main work.

<Computer configuration>
FIG. 17 shows a schematic block diagram showing the configuration of a computer according to at least one embodiment.
Computer 90 includes a processor 91, main memory 92, storage 93, and interface 94.
The above-described work classification device 1 is implemented in a computer 90. The operations of each processing section described above are stored in the storage 93 in the form of a program. The processor 91 reads the program from the storage 93, expands it into the main memory 92, and executes the above processing according to the program. Further, the processor 91 reserves storage areas corresponding to each of the above-mentioned storage units in the main memory 92 according to the program.

The program may be for realizing part of the functions to be performed by the computer 90. For example, the program may function in combination with other programs already stored in storage or in combination with other programs installed in other devices. Note that in other embodiments, the computer may include a custom LSI (Large Scale Integrated Circuit) such as a PLD (Programmable Logic Device) in addition to or in place of the above configuration. Examples of PLDs include PAL (Programmable Array Logic), GAL (Generic Array Logic), CPLD (Complex Programmable Logic Device), and FPGA (Field Programmable Gate Array). In this case, some or all of the functions implemented by the processor may be implemented by the integrated circuit.

Examples of the storage 93 include HDD (Hard Disk Drive), SSD (Solid State Drive), magnetic disk, magneto-optical disk, CD-ROM (Compact Disc Read Only Memory), and DVD-ROM (Digital Versatile Disc Read Only Memory). , semiconductor memory, etc. Storage 93 may be an internal medium connected directly to the bus of computer 90, or may be an external medium connected to computer 90 via an interface 94 or a communication line. Furthermore, when this program is distributed to the computer 90 via a communication line, the computer 90 that received the distribution may develop the program in the main memory 92 and execute the above processing. In at least one embodiment, storage 93 is a non-transitory tangible storage medium.

<Additional notes>
The work classification device 1 described in each embodiment is understood as follows, for example.

(1) The work classification device 1 according to the first aspect is a learning system in which the classification of the teaching work into a plurality of work categories is learned based on a plurality of teaching images in which the teaching work is photographed by a plurality of cameras 2. a real image acquisition unit 15 that obtains a plurality of real images of the real work taken by the plurality of cameras 2; A classification unit 16 for classifying into work categories is provided. According to this aspect, actual work can be appropriately classified into a plurality of work categories.

(2) The work classification device 1 according to the second aspect is the work classification device 1 of (1), in which the plurality of work classifications include main work, preparatory work, auxiliary work, and extra work. According to this aspect, actual work can be classified in detail.

(3) The work classification device 1 according to the third aspect is the work classification device 1 according to (1) or (2), in which the classification unit 16 uses the learned model 14 and the real image to The actual work is classified into the plurality of work categories by reflecting the analysis results of the working tools included in the actual image. According to this aspect, classification accuracy can be further improved.

(4) The work classification device 1 according to the fourth aspect is the work classification device 1 according to any one of (1) to (3), in which the classification unit 16 is configured to combine the learned model 14 and the real image. Based on this, the actual work is classified into the plurality of work categories by referring to a database (work database 4) that records the contents of the actual work, the start date and time, and the completion date and time. According to this aspect, classification accuracy can be further improved.

(5) The work classification method according to the fifth aspect includes the steps of acquiring a plurality of actual images in which the actual work is photographed by the plurality of cameras 2, and a plurality of teaching images in which the teaching work is photographed by the plurality of cameras 2. The method includes the step of classifying the actual work into the plurality of work categories based on the trained model and the actual image, which have learned the classification of the teaching work into the plurality of work categories based on the image. According to this aspect, actual work can be appropriately classified into a plurality of work categories.

(6) The program according to the sixth aspect includes the step of acquiring a plurality of actual images in which the actual work is photographed by the plurality of cameras 2, and the step of acquiring a plurality of actual images in which the teaching work is photographed by the plurality of cameras 2. and classifying the actual work into the plurality of work categories based on the learned model and the actual image, which have learned the classification of the taught work into the plurality of work categories. According to this aspect, actual work can be appropriately classified into a plurality of work categories.

(7) The learning device 1 according to the seventh aspect is a machine learning device that inputs a plurality of real images of real work taken by a plurality of cameras 2 and outputs the results of classifying the real work into a plurality of work categories. A machine learning unit 13 is provided that performs machine learning on the model based on a plurality of teaching images in which the plurality of work categories are labeled. According to this aspect, actual work can be appropriately classified into a plurality of work categories.

100...Work classification system 1...Work classification device (learning device)
2...Camera 3...Worker terminal 4...Work database 11...Taught image acquisition unit 12...Teaching unit 13...Machine learning unit 14...Learned model 15...Actual image acquisition unit 16...Classification unit 17...Input/output unit 18...Storage Unit 41...Record 90...Computer 91...Processor 92...Main memory 93...Storage 94...Interface 100...Work classification system 141...Learned model for process analysis 142...Learned model for tool analysis 201...Worker 202...Web camera 203 …Database 204…IE observer 205…Terminal 206…Machine learning model 207…Machine learning model 301…Worker 302…Tablet terminal 303…Web camera 304…Time server 305…Cloud DB
307...Cloud DB
306...Cloud computing 308...Cloud computing 309...Cloud computing 310...Completion system 311...Work supervisor 312...Terminal

Claims (7)

a trained model that has learned classification of the teaching work into a plurality of work categories based on a plurality of teaching images taken of the teaching work by a plurality of cameras;
an actual image acquisition unit that acquires a plurality of actual images of actual work taken by the plurality of cameras;
a classification unit that classifies the actual work into the plurality of work categories based on the learned model and the actual image;
A work classification device comprising: The work classification device according to claim 1, wherein the plurality of work classifications include main work, preparatory work, incidental work, and slack work. The classification unit classifies the actual work into the plurality of work categories based on the learned model and the actual image, reflecting an analysis result of a work tool included in the actual image. 2. The work classification device described in 2. The classification unit classifies the actual work into the plurality of work categories based on the learned model and the actual image, with reference to a database that records the content, start date and time, and completion date and time of the actual work. 2. The work classification device according to 1 or 2. acquiring a plurality of real images of the actual work taken by a plurality of cameras;
Based on a plurality of teaching images in which the teaching work is photographed by the plurality of cameras, and based on the trained model that has learned to classify the teaching work into a plurality of work categories and the actual image, the actual work is divided into the plurality of teaching images. a step of classifying into work categories;
Work classification methods, including: acquiring a plurality of real images of the actual work taken by a plurality of cameras;
Based on a plurality of teaching images in which the teaching work is photographed by the plurality of cameras, and based on the trained model that has learned to classify the teaching work into a plurality of work categories and the actual image, the actual work is divided into the plurality of teaching images. a step of classifying into work categories;
A program that causes a computer to execute. A machine learning model that inputs a plurality of real images of the actual work taken by a plurality of cameras and outputs the results of classifying the actual work into a plurality of work categories is applied to a plurality of teachings in which the plurality of work categories are labeled. A learning device equipped with a machine learning section that performs machine learning based on images.

Images