JP7254546B2 - Information processing device, information processing method and program - Google Patents

Description

The present invention relates to an information processing device, an information processing method, and a program.

2. Description of the Related Art Conventionally, a method of detecting what kind of work a person in an image is doing and a method of detecting the timing of the work based on the result are known, mainly at a production site such as a factory. Further, Patent Literature 1 discloses a recognition device that recognizes a situation from time-series data by machine learning using a learning model.

JP 2013-235346 A

However, in image recognition using machine learning as described above, there is a problem that erroneous recognition may occur. On the other hand, when it is decided to perform machine learning considering the time series, it takes a lot of time, and there are cases where learning data cannot be sufficiently collected.

SUMMARY OF THE INVENTION The present invention has been made in view of such problems, and it is an object of the present invention to improve the accuracy of identifying work content.

Therefore, the present invention is an information processing apparatus, and an acquisition unit that acquires a frame that constitutes a video; work identifying means; a first period corresponding to a continuous frame in which the first work content is identified by the work identifying means among the periods included in the image; Based on a time-series relationship with a second period corresponding to consecutive frames in which the second work content is specified in a period different from the first period, and a predetermined work order for a plurality of work contents. and period identifying means for correcting the work contents associated with the frame image and identifying the period of each work corresponding to the work order.

ADVANTAGE OF THE INVENTION According to this invention, the precision which specifies a work content can be improved.

1 is an overall diagram of a work identification system; FIG. 3 is a hardware configuration diagram of a server device; FIG. 3 is a functional configuration diagram of a server device; FIG. 4 is a flowchart showing work detection processing; 6 is a flowchart showing correction processing; FIG. 10 is an explanatory diagram of correction processing; FIG. 10 is an explanatory diagram of correction processing; 6 is a functional configuration diagram of a server device according to the second embodiment; FIG. 9 is a flowchart showing work detection processing according to the second embodiment; It is a figure which shows the example of a preservation|save of a detection result. FIG. 11 is an explanatory diagram of correction processing according to the third embodiment; 6 is a flowchart showing correction processing; 14 is a flowchart showing correction processing according to the fourth embodiment; It is a figure which shows an example of an expected value. It is explanatory drawing of the correction process of 5th Embodiment. FIG. 15 is a flowchart showing correction processing according to the fifth embodiment; FIG. It is a figure which shows a similarity table.

BEST MODE FOR CARRYING OUT THE INVENTION An embodiment of the present invention will be described below with reference to the drawings.
(First embodiment)
FIG. 1 is an overall diagram of a work identification system according to the first embodiment. The work identification system has a server device 100 and an imaging device 110 . The server device 100 and the imaging device 110 communicate using, for example, TCP/IP. Note that the server device 100 and the imaging device 110 may use other communication protocols. The imaging device 110 captures an image of the person A performing a series of tasks on the object 120 and transmits the image to the imaging device 110 . Here, the object 120 is an object to be worked on. The object 120 is not limited to the size that can be placed on the desk 130 as shown in FIG. It may be of large size. Also, it is assumed that the series of work performed by the person A includes a plurality of works, and the work order of each work is determined in advance. The series of work includes, for example, work A, work B, and work C, and each work is executed in this order. The server device 100 of the present embodiment assigns consecutive integers such as 1, 2, and 3 as labels to the work order of each work included in a series of work, and manages the work. The symbol used for the label is not limited to the embodiment, as long as the order of each work can be identified. Other examples of symbols used in labels include odd numbered sequences such as 1, 3, 5, and alphabetical order such as a, b, c. Note that the server device 100 is an example of an information processing device.

FIG. 2 is a hardware configuration diagram of the server apparatus 100. As shown in FIG. The CPU 201 executes processing using computer programs and data stored in the RAM 206 and ROM 207 . As a result, the CPU 201 controls the operation of the entire server apparatus 100 and executes or controls each process described later as what the server apparatus 100 performs.

The input device 202 is a user interface such as a group of buttons, and can perform various inputs to the server device 100 by being operated by the user. The output device 203 has a display unit such as a liquid crystal screen and a speaker, and can display the results of processing by the CPU 201 and output audio. A touch panel screen may be configured by combining the user interface function of the input device 202 and the display function of the output device 203 . The communication device 204 is configured with an interface such as a USB port and a LAN port for receiving input/output of data from outside the server. A camera for acquiring video data, a medium for recording video data, a medium for recording the work time measured by this device, etc. are connected to the communication device 204, and input/output of data to be processed is performed.

The storage device 205 is configured by a hard disk drive device, flash memory, or the like. The storage device 205 stores an OS (operating system), and computer programs and data for causing the CPU 201 to execute or control each process described later as what the server device 100 performs. In the following description, the storage device 205 is the storage device 205 as a registration destination of video data used for processing, detected work execution history, and measured work time data. Computer programs and data stored in the storage device 205 are appropriately loaded into the RAM 206 under the control of the CPU 201 and are processed by the CPU 201 .

RAM 206 has areas for storing computer programs and data loaded from storage device 205 . Furthermore, the RAM 206 has a work area used when the CPU 201 executes various processes. Thus, the RAM 206 can provide various areas as appropriate. The ROM 207 stores computer programs and data that do not need to be rewritten, such as setting data and a boot program for the server device 100 . Functions and processes of the server device 100, which will be described later, are realized by the CPU 201 reading a program stored in the ROM 207 or the storage device 205 and executing the program. A CPU 201 , an input device 202 , an output device 203 , a communication device 204 , a storage device 205 , a RAM 206 and a ROM 207 are all connected to a bus (BUS) 208 .

FIG. 3 is a functional configuration diagram of the server device 100. As shown in FIG. The work identification unit 301 identifies work content by machine learning using a learned learning model based on images of frames forming a moving image input from the imaging device 110 . It is assumed that the work identification unit 301 identifies the work content using only the information of the image of one frame without using information such as time-series information indicating the time-series relationship of frames. The period identification unit 302 identifies a plurality of frames having the same work content identified by the work identification unit 301 as the work period of the identified work content. Further, the period identification unit 302 appropriately corrects the work content and the work period based on the preset work order information. Here, the work order information is information indicating work contents and order of a plurality of works included in a series of works. The display processing unit 303 controls to display various information on the display unit.

FIG. 4 is a flow chart showing work detection processing by the server device 100 . Work detection processing is processing for detecting work content based on a video. In S<b>401 , the work identifying unit 301 acquires the imaging device 110 via the communication device 204 in units of frames. It is assumed that moving images are obtained sequentially in a streaming format. As another example, the moving image obtained by the imaging device 110 may be temporarily recorded in the storage device 205, and the work identifying unit 301 may acquire the moving image stored in the storage device 205 frame by frame. good.

Next, in S402, the work identification unit 301 identifies work content based on the acquired image information of the frame. This processing is an example of work identification processing. Specifically, a learned model that has learned motion characteristics of a plurality of predetermined work contents is created, and the work identifying unit 301 obtains a score for each work content. The work identification unit 301 identifies the work content with the highest score as the work content corresponding to the frame. In this embodiment, it is assumed that a work label for identifying work content is set in advance, and the work identification unit 301 associates the work label corresponding to the identified work content with a frame and stores it in a storage unit such as the RAM 206, for example. to record.

In this embodiment, the work label is an integer from 1 to 8. Among these, 1 to 7 are predetermined work contents, and indicate work contents different from each other. 8 indicates that the predetermined work content was not detected. Furthermore, in the present embodiment, a video in which a plurality of tasks with a predetermined execution order are continuously performed is processed, and the work label is information that allows the execution order to be identified. In this embodiment, work labels 1, 2, and 3 indicate the order of work, and different numbers indicate different work contents.

There are various methods for creating the learning model described above, and the method is not particularly limited. For example, when using a method called deep learning, a large amount of data of correct work content is first prepared, and a model is created by inputting this data into a learning network.

Further, the work identification unit 301 may identify the work content based on the image, and specific processing therefor is not limited to the embodiment. As another example, an image of a representative movement of each work content is prepared, and the work identification unit 301 compares the prepared image with the image to be processed, and identifies the work content according to the comparison result. You may As another example, an area that is always touched during each work may be set, and the content of the work may be specified based on whether or not the area is touched, how much the area is touched, and the like.

Next, in S403, the period specifying unit 302 determines whether the frame image acquired in S401 is the final frame of the video. If the period specifying unit 302 determines that the current frame is the final frame (YES in S403), the process proceeds to S405 because there is no next frame. When the period identification unit 302 determines that the frame is not the final frame (NO in S403), the process proceeds to S404. In S404, the period identifying unit 302 determines whether or not the work content identified in the frame image acquired in S401 is different from the work content of the previous frame. At this time, the period specifying unit 302 refers to the work label. If the work content is different (YES in S404), the period identifying unit 302 advances the process to S405. If the work content is the same (NO in S404), the period specifying unit 302 advances the process to S401.

In S405, the period identification unit 302 counts the continuous period of the work content of the frame acquired in S401 in chronological order. Next, in S406, the period identification unit 302 compares the number of frames (the number of consecutive frames) counted in S405 with the first threshold. Here, the first threshold is a predetermined number of frames. When the period identifying unit 302 determines that the number of consecutive frames is greater than the first threshold (YES in S406), the process proceeds to S407. When the period identification unit 302 determines that the number of consecutive frames is equal to or less than the first threshold (NO in S406), the process proceeds to S408. In S407, the period identifying unit 302 extracts the continuous frames as the work period of the work content identified in S401, records the start frame, end frame, and work content of the work period in a storage unit such as the RAM 206, and then performs processing. Proceed to S409.

On the other hand, in S408, the period identifying unit 302 adds the continuous frames to the immediately preceding work period, and then advances the process to S409. This process is a process corresponding to the fact that there is a high possibility of erroneous recognition when the continuous period is extremely short. In S408, specifically, if the immediately preceding work period is a period identified by work label 2, the period identifying unit 302 replaces the work contents with the period of work label 2 for the continuous frames as well, and sets the period to Integrate.

In S409, the period identification unit 302 determines whether or not there is a work period to be corrected. In this embodiment, when there is a work period in which the immediately preceding work period is specified and the immediately following work period is specified, the period specifying unit 302 corrects the work period that is sandwiched before and after the work period. It shall be specified as the target work period. Hereinafter, the work period to be corrected is referred to as a target period. If the target period exists (YES in S409), the period identifying unit 302 advances the process to S410. If the target period does not exist (NO in S409), the period specifying unit 302 advances the process to S401. Note that there is no next work period in the work period that includes the final frame, but in this case, as a special case, the process proceeds to S410.

In S410, the period specifying unit 302 performs correction processing for the target period. The correction process is a process of correcting the length of the target period, correcting erroneous recognition, and determining the presence or absence of work. The correction processing will be described later with reference to FIG. Next, in S411, the period identifying unit 302 determines whether the frame obtained in S401 is the final frame. When the period identification unit 302 determines that the frame is the last frame (YES in S411), the work detection process ends. If the period specifying unit 302 does not determine that the frame is the final frame (NO in S411), the process proceeds to S401.

FIG. 5 is a flowchart showing detailed processing in the correction processing (S410) described with reference to FIG. 6 and 7 are explanatory diagrams of the correction process. 6A shows the work period column 600, and FIG. 6B shows the work period column 610. FIG. The work period column 600 is the work period column obtained by the processing up to S408 before the correction processing (S410) is performed. is the work period column of . Similarly, FIG. 7(a) shows a work period column 700 before correction processing, and FIG. 7(b) shows a work period column 710 after correction processing. The work period column 710 corresponds to the work period column 700 .

In the work period columns 600 and 700, the number identifying the work period, the start frame, the end frame, and the work label are associated with each other. Here, No is information for identifying the work period. A work label is information for identifying work content. In this embodiment, the work label is an integer from 1 to 8. Among these, 1 to 7 are predetermined work contents, and indicate work contents different from each other. 8 indicates that the predetermined work content was not detected.

In FIG. 5, in S501, the period identification unit 302 determines whether the target period is a non-work period. Specifically, the period identification unit 302 determines that the period is a non-work period when the work label of the target period is "8". If the period specifying unit 302 determines that the current period is the non-work period (YES in S501), the process proceeds to S502. When the period specifying unit 302 determines that the current period is not the non-work period (NO in S501), the process proceeds to S503. In S502, the period identification unit 302 deletes the target period from the period corresponding to the work content corresponding to the target period. The correction process (S410) ends here. In this case, the period specifying unit 302 advances the subsequent process to S411.

On the other hand, in S503, the period identification unit 302 determines whether or not the target period and the work period immediately preceding it have the same work label. When the period identification unit 302 determines that the work labels are the same (YES in S503), the process proceeds to S504. When the period specifying unit 302 determines that the work labels are different (NO in S503), the process proceeds to S506. In S504, the period identification unit 302 identifies the target period as the period corresponding to the assigned work content (work label). Next, in S505, the period identification unit 302 integrates the period of the target period into the immediately preceding work period. With this, the correction process (S410) ends. For example, in the work period column 600 of FIG. 6A, work label 3 is identified as work content for work period No. 5 in S504, and work period No. 5 is integrated with work period No. 4 in S505.

In S506, the period identification unit 302 determines whether the work label of the target period has a smaller numerical value than the work label of the immediately preceding work period. In the following description, the work label for the target period is called the target label, and the work label for the previous work period is called the previous label. If the period identifying unit 302 determines that the target label is smaller than the immediately preceding label (YES in S506), the process proceeds to S507. When the period identification unit 302 determines that the target label is equal to or greater than the immediately preceding label (NO in S506), the process proceeds to S508. In S507, the period identification unit 302 deletes the target period from the period corresponding to the work content corresponding to the target period. The correction process (S410) ends here. Note that the work period referred to in S506 may be a work period before the target period, and is not limited to the immediately preceding work period.

In this embodiment, when the work is performed according to the predetermined execution order, the work labels should be in ascending order. Corresponding to this, the processing of S506 and S507 is to delete it as an erroneous recognition result when the ascending order is not satisfied. For example, in the work period column 600 of FIG. 6A, the work label of No. 7 work period is "2", which is smaller than the work label "4" of the previous No. 6 work period. Therefore, the work period of No. 7 is deleted in the process of S507. Note that the processing of S507 is an example of processing for deleting the target period when the work content of the target period is defined as the work content before the work content of the immediately preceding work period in the order of work.

In S508, the period identification unit 302 determines whether the difference between the target label and the immediately preceding label is one. As described above, the work labels correspond to the order of work, and the process of S508 confirms whether the work is performed in the order that matches the order of work. Therefore, if the difference between the work labels according to the order of work is a value other than 1, the period identifying unit 302 makes a determination based on the other value. If the period specifying unit 302 determines that the difference is 1 (YES in S508), the process proceeds to S509. When the period specifying unit 302 determines that the difference is not 1 (NO in S508), the process proceeds to S510.

In S509, the period identification unit 302 identifies the target period as the period corresponding to the work content (work label) identified in the target period. The correction process (S410) ends here. For example, in the work period column 600 of FIG. 6A, the work period of No. 3 is specified as the work period of work label 2 in the process for which the work period of No. 3 is the target period. Similarly, in the process using the No. 4 work period as the target period, the No. 4 work period is identified as the work period of work label 3 . In the process of S509, when the work content of the target period, the previous work content, and the work content of the target period are defined as continuous work content in this order in the work order, the target period corresponds to the work content. It is an example of the process of specifying as the period of work content.

Also, there are cases where there is no previous work period, such as the work period No. 10 in the work period column 700 in FIG. 7A. In this case, the period identifying unit 302 determines NO in both S503 and S505, and determines YES in S508, thereby determining the work period. As another example, when the work label of the work period to be corrected is "1", the work label "1" may be determined as the work content when there is no previous work period. In this process, it is possible to determine the work contents of the work period to be corrected without comparing with work labels of other work periods.

In S510, the period identification unit 302 determines whether or not the difference between the work label (next label) of the work period next to the target period and the immediately preceding label is one. This process is a process for confirming whether or not the next work and the immediately preceding work are consecutive works. When the period specifying unit 302 determines that the difference between the work labels is 1 (YES in S510), the process proceeds to S511. When the period specifying unit 302 determines that the difference between the work labels is not 1 (NO in S510), the process proceeds to S512. In S511, the period identification unit 302 deletes the target period from the period corresponding to the work content corresponding to the target period. Thus, the correction process (S410) ends.

If the difference between the next label and the immediately preceding label is 1, the previous work period and the next work period are periods corresponding to continuous work order, and therefore the period of interest is considered to be the period of noise. Therefore, in this embodiment, in this case, in S511, the target period is deleted from the period corresponding to the corresponding work content. For example, in the work period column 700 in FIG. 7A, work labels 2, 4, and 3 are assigned to work periods No. 11 to No. 13. FIG. On the other hand, when the processing of S511 is executed, the work period of No. 12 is deleted. The process of S510 is a process of determining whether or not the work contents of the previous period and the next period are defined as continuous work contents in the work order. Further, the processing of S511 is an example of processing for identifying the target period as the period of the corresponding work content when the work content of the previous period and the next period are defined as continuous work content in the order of work. be.

On the other hand, in S512, the period specifying unit 302 determines that there is work omission, outputs notification information indicating that there is work omission, and the correction process (S410) ends. Specific processing for outputting the notification information includes sending the notification information to the work manager's PC or the like by e-mail, outputting an alarm sound, and displaying the notification information on the display unit.

For example, in the work period column 700 in FIG. 7A, work labels 4, 6, and 7 are assigned to the work periods No. 20 to No. 22. FIG. In this case, a determination of NO is made in S508, a determination of NO is made in S510, and the process of S512 is performed. Then, by the processing of S511, it is determined that work No. 5 has not been performed, and notification information indicating that work No. 5 has been omitted is output. In addition, in S512, predetermined specific processing may be performed, and specific processing is not limited to the embodiment. 6(a) to the work period column 610 of FIG. 6(b), and the work period column 700 of FIG. 7(a) to the work period column of FIG. 7(b). 710 is confirmed. Note that the correction process (S410) is a process of specifying the period of each work corresponding to the order of work based on the temporal relationship between a plurality of periods and a predetermined order of work. An example.

As described above, the server device 100 can improve the accuracy of specifying the work content and the work period by considering the work order.

(Second embodiment)
Next, the work identification system according to the second embodiment will be described mainly with respect to differences from the work identification system according to the first embodiment. FIG. 8 is a functional configuration diagram of the server device 100 of the work identification system according to the second embodiment. A server device 100 according to the second embodiment has a score calculation unit 801 , a period identification unit 802 , and a display processing unit 803 .

The score calculation unit 801 calculates the score of each task by machine learning using a learned learning model, based on the images of the frames forming the moving image input from the imaging device 110 . It is assumed that the score calculation unit 801 identifies the work content using only the information of the image of one frame without using information such as time-series information indicating the time-series relationship of frames. The period identifying unit 802 detects work content and work period based on the score calculated by the score calculating unit 801, the output probability model, and the transition probability model. Here, the output probability model is a model for estimating the content of work from the score of each work. A transition probability model is a model that defines transition probabilities between multiple tasks. The display processing unit 803 controls to display various information on the display unit.

FIG. 9 is a flow chart showing work detection processing by the server device 100 . Work detection processing is processing for detecting work content based on a video. In S901, the score calculation unit 801 determines whether to end the work detection process. There are various methods for determination, and the method is not particularly limited. For example, there are methods of making a judgment based on the time, making a judgment based on whether or not there is a moving object, making a judgment based on whether the power is turned off, or making a judgment by detecting the end of the video. . When the score calculation unit 801 determines to end the work detection process (YES in S901), it ends the work detection process. If the score calculation unit 801 does not determine to end the work detection process (NO in S901), the process proceeds to S902.

In S<b>902 , the score calculation unit 801 acquires a moving image frame by frame from the imaging device 110 via the communication device 204 . It is assumed that moving images are obtained sequentially in a streaming format. As another example, the moving image obtained by the imaging device 110 may be temporarily recorded in the storage device 205, and the work identifying unit 301 may acquire the moving image stored in the storage device 205 frame by frame. good.

Next, in S903, the score calculation unit 801 obtains a score for each work content for the frame acquired in S902. This processing is an example of score identification processing. Specifically, it is assumed that a learned model that has learned motion features of a plurality of predetermined work contents is set in advance, and the score calculation unit 801 obtains a score for each work content based on the learned model. , the score is recorded in a storage unit such as the RAM 206 .

There are various methods for creating the learning model described above, and the method is not particularly limited. For example, when using a method called deep learning, first prepare a large amount of data that captures the movement of the correct work content and data that specifies the work content, and input these data to the learning network. Create a model by doing When a frame of a moving image of the content of work is input to the trained model, the content of the work is specified as an output.

Also, the score calculation unit 801 may calculate the score of each task based on the image, and the specific processing for that purpose is not limited to the embodiment. As another example, an image of representative motion of each work content is prepared, and the score calculation unit 801 compares the prepared image with the image to be processed, and calculates the score of each work according to the comparison result. may be calculated. As another example, an area that is always touched during each task may be set, and the score for each task may be calculated based on whether or not the area is touched and how much it is touched.

Next, in S904, the period identifying unit 802 detects a sufficient period for processing in S905 and subsequent steps from among the periods included in the moving image acquired in S902. For example, the period identifying unit 802 determines that the sufficient period is detected when the period from the start to the end of the series of work periods is set as a sufficient period and the timing of the end of the series of work is detected. Specifically, when targeting a series of tasks in which four tasks, task 1, task 2, task 3, and task 4 are performed in this order, the period identifying unit 802 determines whether the task 4 ends based on the score. is searched for, and it is determined that a sufficient period has been detected when the end of task 4 is detected. Here, the sufficient period is an example of a series period or a predetermined period. Note that the process of detecting a sufficient period is an example of the series period determination process of determining the series period based on the score. If a sufficient period is detected (YES in S904), the period specifying unit 802 advances the process to S905. If a sufficient period is not detected (NO in S904), the period specifying unit 802 advances the process to S901.

In S905, the period identifying unit 802 combines scores corresponding to the sufficient period detected in S904 to generate a score series. Next, in S906, the period identifying unit 802 generates a label sequence in which each frame corresponding to each score included in the score sequence generated in S905 is labeled with a likely work content. The period specifying unit 802 uses an output probability model and a transition probability model to generate a label sequence. Also, the transition probability model is a model that defines the probability of transition from each task to each task. By calculating the likelihood for these models, the period identification unit 802 assigns a likely label to each score in the score series and the corresponding frame.

The period identifying unit 802 further adjusts the likelihood based on the duration of each task. For example, the average and variance of the duration of each task are measured in advance, and the normal distribution of the duration of each task is created from the average and variance. When calculating the likelihood, the period identifying unit 802 lowers the likelihood if the generated duration of each task deviates from the normal distribution of the duration of each task.

Note that there are various methods for adjusting the likelihood based on the duration of each task, and the method executed by the period identification unit 802 is not particularly limited. For example, a method using another distribution instead of the normal distribution can be used. As another example, the period identifying unit 802 may calculate the average duration of each task in advance, and make adjustments based on how different it is from the average. In addition, since the duration of work differs depending on the worker, the period specifying unit 802 creates a distribution corresponding to the difference in duration based on the duration of work of another worker created in advance. good too. Also, the process of changing the likelihood based on the duration of each task described above is not essential.

Next, in S907, the period identification unit 802 counts the consecutive periods of the labels in the label series generated in S906 in chronological order, and summarizes the consecutive periods as work period candidates. Next, in S908, the period identifying unit 802 filters the work period candidates summarized in S907. Specifically, when the difference between the length of a work period candidate and the length of a predetermined work period is shorter than a threshold, the period identification unit 802 identifies this work period candidate as a work period. On the other hand, if the difference is greater than the threshold, the period identification unit 802 does not identify this work period candidate as a work period. Thus, the period identification unit 802 identifies the work period based on the duration of the work. Furthermore, the period identifying unit 802 determines that, among the frames included in the work period candidate, if the number of frames for which the difference between the score of the frame and the preset expected score is equal to or greater than the score threshold is less than the frame threshold, Identify the period candidate as a work period. On the other hand, if the number of frames is equal to or greater than the frame threshold, the period identification unit 802 does not identify this work period candidate as a work period.

Next, in S909, the period identifying unit 802 identifies the work period candidate filtered in S908 as a work period, and stores the start frame, end frame, label, and likelihood calculated in S906 of the work period in the storage device 205, etc. is recorded in the memory of FIG. 10(a) is an example of a format for saving detection results. As shown in FIG. 10A, the period specifying unit 802 describes information about the detected work period for each line.

Next, in S910, the period identification unit 802 determines whether there is any omission of work in the work period summarized in S909. The period identifying unit 802 detects work omissions by comparing the detection results arranged in chronological order as shown in FIG. 10B with a predetermined definition order. For example, in the example of FIG. 10B, the predetermined order of definition of the work labels is 3, 4, 5, and 6, but the work labels of the detection results are in the order of 3, 4, and 6. Therefore, it is determined that the work label 5 is missing. If there is work omission (YES in S910), the period identifying unit 802 advances the process to S911. If there is no work omission (NO in S910), the period specifying unit 802 advances the process to S901.

In S<b>911 , the period identifying unit 802 outputs notification information to the effect that there is work omission. Specific processing for outputting the notification information includes sending the notification information to the work manager's PC or the like by e-mail, outputting an alarm sound, and displaying the notification information on the display unit. After outputting the notification information indicating that there is work omission, the process proceeds to S901.

As described above, the server device 100 can improve the accuracy of specifying the work content and the work period by modeling the score series.

It should be noted that the determination of whether or not there is an omission of work in S910 and the processing of notification information to the effect that there is omission of work in S911 are not essential processes in this embodiment, and it is sufficient to simply record the content of the work. If the work omission determination process is not to be performed, the period identifying unit 802 may advance the process to S901 after the process of S909. Further, in S909, the storage area of the detection result recorded in the storage unit may be released after the omission of work has been determined. Also, in S909, the period identification unit 802 does not necessarily store all of the start frame, end frame, label, and likelihood calculated in S906 of the work period. For example, the period identification unit 802 may store only work labels that are necessary for determining the omission of work.

Also, in the present embodiment, the period identifying unit 802 detects the period of a series of tasks as a sufficient period in S904 of the task detection process shown in FIG. However, in S904, it is sufficient to detect a period of a certain length, and the period to be detected is not limited to the embodiment. As another example, the period identification unit 802 may detect the period of one task as a sufficient period. In this case, from S905 onward, a score series is generated, a work period is determined, and the like is performed for the work period.

As another example, the period identifying unit 802 may detect a certain number of frames, a certain amount of time, or the like as a sufficient period. As another example, it is assumed that an area through which the worker's body passes during work is set, and the sensor notifies the server device 100 when the worker passes through the area. Then, when the server apparatus 100 receives a notification from the sensor, the period identification unit 802 may detect a sufficient period from this notification. Also, as another example, when a program that operates during work is executed, the device on which the program is installed notifies the server device 100, and the period identification unit 802 receives the notification, A sufficient period may be detected. Further, as another example, equipment such as a button may be prepared, and when the equipment is used, the equipment may notify the server device 100 .

(Third Embodiment)
Next, the work identification system according to the third embodiment will be described mainly with respect to the differences from the work identification systems according to the other embodiments. As shown in FIG. 11, it is assumed that work periods No. 1 to No. 6 are obtained. While the correct work labels for each of the work periods No. 1 to No. 6 are 1, 2, 3, 3, 3, and 4, the work labels 1, 2, 3, 4, 3, and 4 are obtained by processing using the learned model. Suppose it is specified. In this case, in the first embodiment, the No. 3 period is specified as the work label 3 period, and the No. 4 to No. 6 periods are specified as the work label 4 period. That is, the result is different from the correct work content. In the work identification system according to the third embodiment, even when such identification results are obtained, the correct work period is identified.

FIG. 12 is a flowchart showing detailed processing in the correction processing (S410) by the server device 100 according to the third embodiment. If the period identification unit 302 determines in S501 that the current period is not the non-work period (NO in S501), the process proceeds to S901. In S901, the period identifying unit 302 compares the number of frames included in the target period (the number of target frames) with a second threshold. Here, the second threshold is a predetermined number of frames. Also, the second threshold is a value larger than the first threshold referred to in S406 (FIG. 4). If the number of target frames is smaller than the second threshold (YES in S901), the period specifying unit 302 advances the process to S902. If the number of target frames is greater than or equal to the second threshold (NO in S901), the period specifying unit 302 advances the process to S503. In S902, the period identification unit 302 deletes the target period from the period corresponding to the work content corresponding to the target period. The correction process (S410) ends here.

The process of S406 is a process for removing erroneous recognition results that appear in several frames, while the process of S901 is a process for removing erroneous recognition results that occur for a relatively long period of time. If the second threshold value is referred to in S406, there is a possibility that correct recognition results that are fragmented due to erroneous recognition may be excluded. In order to avoid this, threshold processing is performed in two steps as described above.

The processing of S901 and S902 will be specifically described with reference to FIG. Here, it is assumed that the first threshold referred to in S406 (FIG. 4) is 8 frames, and the second threshold referred to in S901 is 15 frames. When the period No. 4 is processed as the work period, the number of frames in the work period No. 4 is less than the second threshold value, so the period No. 4 is recognized as an error by the processes of S901 and S902 and is deleted from the processing target. be. Subsequently, when the period No. 5 is processed as the work period, the period No. 5 is integrated with the period No. 3, which is the immediately preceding work period. That is, the period No. 3 to No. 5 is specified as the period for work label 3, and the subsequent period No. 6 is specified as the period for work label 4. FIG. Other configurations and processes of the work identification system according to the third embodiment are the same as those of the work identification systems according to other embodiments.

As described above, in the work identification system according to the third embodiment, it is possible to improve the accuracy of identifying work content and work period by using two thresholds.

As a modification of the third embodiment, the server apparatus 100 may specify each work period by changing the determination condition in S506 and allowing overlap of work. For example, the determination condition of S506 may be "(target label+1)<previous label?" As a result, for example, there is a pattern in which the first work is interrupted due to lack of parts in the middle, and the first work is restarted when the parts have arrived after part of the second work has started. can also handle. Also, it is possible to cope with a pattern in which the first work is finished once, and the following second work is started, but the first work is performed again just in case. In this case, for example, a work period column corresponding to work labels 3, 4, and 3 can be obtained. A period can be specified.

(Fourth embodiment)
Next, the work identification system according to the fourth embodiment will be described mainly with respect to differences from the work identification system according to the third embodiment. In the work identification system according to the fourth embodiment, as in the work identification system according to the third embodiment, even when the identification result described with reference to FIG. 11 is obtained, the correct work period is determined. Identify.

FIG. 13 is a flowchart showing detailed processing in the correction processing (S410) by the server device 100 according to the fourth embodiment. If the period identification unit 302 determines in S501 that the current period is not the non-work period (NO in S501), the process proceeds to S1001. In S1001, the period identification unit 302 obtains the absolute value of the difference between the number of frames included in the target period and the expected value. The period specifying unit 302 then compares the absolute value of the difference with the third threshold. FIG. 14 is a diagram illustrating an example of expected values. Note that the expected value is a value arbitrarily set according to the number of frames expected in the work content identified by the work label. Also, the third threshold is a threshold for the difference between the predetermined number of frames. The expected value and the third threshold are stored in advance in the storage device 205, for example.

If the absolute value of the difference is greater than or equal to the third threshold (YES in S1001), the period identifying unit 302 advances the process to S1002. If the absolute value of the difference is smaller than the third threshold (NO in S1001), the period specifying unit 302 advances the process to S503. In S1002, the period identification unit 302 deletes the target period from the period corresponding to the work content corresponding to the target period. The correction process (S410) ends here.

Next, the processing of S1001 and S1002 will be specifically described with reference to FIG. Here, it is assumed that the frame rate of video is constant and the number of frames corresponds to time. Also, the third threshold is assumed to be 10 frames. The difference between the number of frames (10) in the work period of No. 4 and the expected value (50) of work label 4 is greater than or equal to the third threshold (10). Therefore, when the period No. 4 is processed as the work period, the work period No. 4 is deleted from the processing targets as an erroneous recognition. Subsequently, when the period No. 5 is processed as the work period, the period No. 5 is integrated with the period No. 3, which is the immediately preceding work period. That is, the period No. 3 to No. 5 is specified as the period for work label 3, and the subsequent period No. 6 is specified as the period for work label 4. FIG. Other configurations and processes of the work identification system according to the fourth embodiment are the same as those of the work identification systems according to other embodiments.

As described above, in the work identification system according to the fourth embodiment, by referring to the difference from the expected value and the threshold value, it is possible to improve the accuracy of identifying work content and work period.

As a modification of the fourth embodiment, similar to the modification of the third embodiment, the server device 100 changes the determination conditions in S506 to allow overlapping of tasks, so that each You may also specify a working period.

(Fifth embodiment)
Next, the work identification system according to the fifth embodiment will be described mainly with regard to differences from the work identification system according to the third embodiment. The work identification system according to the fifth embodiment performs processing for appropriately changing the work content according to the order of work. Assume that a plurality of work periods as shown in FIG. 15 are obtained. Suppose that the correct work label for the period No. 6 is 6, and the work label 1 is specified by the process using the learned model. In the work identification system of this embodiment, such erroneous identification results are changed to correct work content.

FIG. 16 is a flowchart showing detailed processing in the correction processing (S410) by the server device 100 according to the fifth embodiment. If the period identification unit 302 determines in S501 that the current period is not the non-work period (NO in S501), the process proceeds to S1301. In S1301, the period identification unit 302 determines whether or not the work content of the target period is the work content expected from the work order. In this embodiment, the period identifying unit 302 determines that the expected work period is when the target label has a value obtained by adding 1 to the immediately preceding label. If the work content is expected (YES in S1301), the period identifying unit 302 advances the process to S503. The period specifying unit 302 advances the process to 1302 if the work content is not expected (NO in S1301).

In S1302, the period identifying unit 302 refers to the similarity table and determines whether or not there is work content similar to the work content of the target period. Here, the similarity table is information defining work content similar to each work content, and is stored in advance in the storage device 205, for example. FIG. 17 is a diagram showing an example of a similarity table. If the period specifying unit 302 determines that there is similar work content (YES in S1302), the process proceeds to S1303. When the period specifying unit 302 determines that there is no similar work content (NO in S1302), the process proceeds to S503. In S1303, the period identification unit 302 changes the work label of the target period to a work label of similar work content, and then advances the process to S503.

For example, if the period No. 6 shown in FIG. 15 is processed as a work period, referring to the similarity table shown in FIG. ing. Therefore, the period identification unit 302 determines in S1302 that there is similar work content, and advances the process to S1303. Then, in S1303, the period identification unit 302 changes the work label of the target period from "1" to "6" following the previous work label "5". As a result, in the example of FIG. 15, the periods Nos. 6 and 7 are integrated and specified as the period corresponding to work label 6 . Other configurations and processes of the work identification system according to the fifth embodiment are the same as those of the work identification systems according to the other embodiments. In the processing of S1301 to S1303, when the work contents of the target period and the work contents of the previous work period do not match the work contents determined in the work order, the work contents of the target period are changed to similar work contents. This is an example of change processing to be performed.

As described above, in the work identification system according to the fifth embodiment, it is possible to improve the accuracy of identifying the work content and the work period by considering the work order and the work content.

In the above, the processing of the server device 100 described in each embodiment may be executed by one server device 100 switching the processing. For example, it is desirable that the server device 100 performs the processing according to the first embodiment when it is expected that misrecognition is less likely to occur during work, such as when simple work is repeated. Then, it is desirable that the server device 100 performs the processes according to the third to fifth embodiments when there is a possibility of misrecognition, such as when a mistake or an unexpected movement is assumed during work. Furthermore, the server device 100 performs the processing of the third embodiment when there is little variation in the required time required for each task, or when it is assumed that the duration of a movement that may cause misrecognition is short. is desirable. Conversely, if there is a large variation in the required time required for each task, or if it is assumed that the duration of a movement that may cause misrecognition is long, the server device 100 performs the processing of the fourth embodiment. It is desirable to

As another example, when similar tasks are included, the server device 100 preferably performs the processing according to the fifth embodiment. Also, as another example, the server device 100 may execute a process combining processes according to a plurality of embodiments. For example, the server apparatus 100 can combine and execute the processing according to the third embodiment and the processing according to the fourth embodiment, thereby further increasing the accuracy of determination as to whether recognition is erroneous.

In the above description of the embodiment, an example in which the threshold used for determination is fixed is given, but the threshold may not be a fixed value, and a threshold other than the number of frames may be used as the threshold. For example, the server device 100 may learn the tendency of the misrecognition period by machine learning, and change the threshold according to the situation. Specifically, when the tendency of working hours differs for each worker, or when working hours change depending on the production plan, the server device 100 performs learning including elements other than working hours, A suitable threshold may be obtained for each situation. As another example, the server device 100 uses the recognition score of the work for each frame, and the difference between the second or third highest score and the highest score in the misrecognized frame is If it is small, the determination may be made in consideration of those recognition results.

Although the preferred embodiments of the present invention have been described in detail above, the present invention is not limited to such specific embodiments, and various modifications can be made within the scope of the gist of the invention described in the claims.・Changes are possible.

The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by processing to It can also be implemented by a circuit (for example, ASIC) that implements one or more functions.

100 server device 110 imaging device

Claims (20)

Acquisition means for acquiring frames constituting video;
Work identifying means for identifying work content by machine learning using a learned learning model based on the frame image;
Among the periods included in the image, a first period corresponding to the continuous frames in which the first work content is identified by the work identifying means, and Among the periods included in the image, a first period different from the first period an image of the frame based on the time-series relationship with the second period corresponding to the continuous frames in which the second work content is specified in the period and the predetermined work order of the plurality of work content; and period identifying means for correcting the work content associated with the work order and identifying the period of each work corresponding to the order of work. 2. The information processing apparatus according to claim 1, wherein said second period is a period before said first period. 3. The information processing apparatus according to claim 2, wherein said second period is a period immediately before said first period. When the second work content and the first work content are defined as continuous work content in the work order, the period identifying means sets the first period to the first work content. 4. The information processing apparatus according to claim 3, wherein the period is specified as a period corresponding to the content of work. When the first work content is defined as a work content that precedes the second work content in the work order, the period identifying means sets the first period to the first work content. 4. The information processing apparatus according to claim 3, wherein the period corresponding to the content is deleted. The period identifying means defines a third work content in a third period, which is the period following the first period, and the second work content as consecutive work content in the order of work. 6. The information processing apparatus according to claim 4 or 5, wherein the first period is deleted from the period corresponding to the first work content when there is one. The period identifying means identifies the first period as a period corresponding to the first work content when the second work content and the first work content are the same, and determines the first period. 7. The information processing apparatus according to claim 3, wherein the second period is integrated with the second period. If the first work content and the second work content do not match the work content determined in the order of work, the first work content is set in advance as work content similar to the first work content. 8. The information processing apparatus according to any one of claims 3 to 7, further comprising changing means for changing the set work content. 2. The period specifying means deletes the first period from the period corresponding to the first work content when the first period is shorter than a preset threshold value. 9. The information processing apparatus according to any one of items 1 to 8. When a difference between an expected value preset in association with the first work content and the length of the first period is greater than a preset threshold value, the period specifying means determines the first period. 10. The information processing apparatus according to claim 1, wherein the period is deleted from the period corresponding to the first work content. Acquisition means for acquiring frames constituting video;
score specifying means for specifying a score for each of a plurality of tasks based on the image of the frame;
a period identifying means for identifying a work period based on a score sequence combining scores of a plurality of frames included in a predetermined period and a predetermined work order of the plurality of works; An information processing device characterized by: Based on the score, further comprising series period determination means for determining a series period,
12. The information processing apparatus according to claim 11, wherein said period identifying means identifies said work period based on said score series obtained by combining scores of said series periods and said work order. 13. The information processing apparatus according to claim 12, wherein said series period determining means determines a period of one task as said series period. 13. The information processing apparatus according to claim 12, wherein said series period determining means determines a period from start to end of a series of tasks as said series period. 15. The information processing apparatus according to any one of claims 11 to 14, wherein said period identifying means further identifies the period of said work based on the duration of said work. The period specifying means specifies a work period candidate based on the score series and the work order, and if a difference between the length of the work period candidate and a predetermined work period is equal to or less than a threshold 16. The information processing apparatus according to any one of claims 11 to 15, wherein the work period candidate is specified as the work period. The period specifying means specifies a work period candidate based on the score series and the work order, and compares the scores of the frames included in the work period candidates with a preset expected score. 17. The information processing according to any one of claims 11 to 16, wherein when the number of frames whose difference is equal to or greater than the score threshold is less than the frame threshold, the work period candidate is specified as the work period. Device. An information processing method executed by an information processing device,
an acquisition step of acquiring frames that make up a video;
a work identification step of identifying work content by machine learning using a learned learning model based on the image of the frame;
A first period, among the periods included in the image, corresponding to the continuous frames in which the first work content is identified in the work identification step, and a period different from the first period, among the periods included in the image. an image of the frame based on the time-series relationship with the second period corresponding to the continuous frames in which the second work content is specified in the period and the predetermined work order of the plurality of work content; and a period specifying step of correcting work contents associated with the work order and specifying a period of each work corresponding to the order of work. An information processing method executed by an information processing device,
an acquisition step of acquiring frames that make up a video;
a score identification step of identifying a score for each of a plurality of tasks based on the image of the frame;
and a period identifying step of identifying a work period based on a score sequence combining scores of a plurality of frames included in a predetermined period and a predetermined work order of the plurality of works. Information processing method characterized by: A program for causing a computer to function as each means of the information processing apparatus according to any one of claims 1 to 17.

Images