JP2023078983A - Work management method or system - Google Patents

Abstract

To provide a method or a system, capable of grasping steps without taking labor.SOLUTION: A method executed by a system including one or more calculation devices, includes processing of reproducing a first moving image recording first work configured of a plurality of steps, processing of causing an explainer to orally explain the first work displayed in the first moving image to acquire a voice of the explainer, and measuring processing of analysing the voice to measure a first step time set indicating a time required for each of the plurality of steps.SELECTED DRAWING: Figure 8

Description

The present invention relates to a work management method or system.

2. Description of the Related Art A device for managing work processes is known as a conventional technology (see, for example, Patent Document 1).

JP-A-2000-24850

However, with the above technology, it is difficult to accurately grasp the time required for each step or the name of the step and to draw up a process chart without much effort.

In view of such problems, as one aspect of the present invention, there is provided a method executed by a system including one or more computing devices, wherein a first moving image recording a first work composed of a plurality of steps is reproduced. processing, causing an explainer to verbally explain the first work displayed in the first moving image, obtaining the voice of the explainer, analyzing the voice, and determining the time required for each of the plurality of steps. and a measurement process for measuring the indicated first step time set.

It is possible to provide a method or system that can grasp the process without labor.

1 is an overall configuration diagram of an information processing system according to an embodiment; FIG. It is a figure which shows the hardware constitutions of the information processing apparatus which concerns on this embodiment. It is a figure which shows the functional structure of (a) distribution server and (b) machine-learning server which concern on this embodiment. It is a figure which shows the structure of the learning device which concerns on this embodiment. It is a figure which shows the process information which concerns on this embodiment. It is a figure which shows the data for learning which concerns on this embodiment. FIG. 4 is a sequence diagram showing process chart creation processing according to the present embodiment; It is a figure which shows the outline|summary of the speech analysis which concerns on this embodiment. It is a figure which shows the process chart produced in the process which concerns on this embodiment. 4 is a flowchart showing learning processing according to the embodiment; 4 is a diagram showing an outline of image processing in this embodiment; FIG. 6 is a flow chart showing estimation processing in the present embodiment.

At least the following matters will become apparent from the description of the present specification and the accompanying drawings. DESCRIPTION OF THE PREFERRED EMBODIMENTS The present invention will now be described in accordance with one embodiment thereof with reference to the accompanying drawings.

FIG. 1 shows the configuration of an information processing system 1 according to one embodiment of the present invention. The information processing system 1 includes computing devices of a distribution server 10 , a machine learning server 20 , a photographing device 40 , and one or more user terminals 30 . The distribution server 10, the machine learning server 20, the imaging device 40, and the user terminal 30 are connected via the communication network 5 so that they can transmit and receive data to each other. The communication network 5 is wireless or wired communication means, such as the Internet, WAN (Wide Area Network), LAN (Local Area Network), public communication network, dedicated line, and the like. Although the information processing system 1 according to this embodiment is composed of the plurality of information management devices, the present invention does not limit the number of these devices. Therefore, the information processing system 1 can be configured with one or more devices as long as they have the following functions.

The user terminal 30 is an information processing device operated by a user (such as an operator or an explainer described later), and is, for example, a smart phone, a tablet, a mobile phone, a personal computer, or the like.

The distribution server 10 is an information processing device that has a function of managing processes and creating a process chart for various works such as product assembly and construction work performed by humans or robots.

The machine learning server 20 has a function of executing machine learning to create a learning model and a function of creating a process chart using the learning model.

The photographing device 40 is a device having a function of photographing work and generating a moving image, and corresponds to, for example, a digital video camera.

FIG. 2 shows an example of hardware used to realize the distribution server 10, the machine learning server 20, and the user terminal 30 (hereinafter referred to as "information processing apparatus 100"). As shown in the figure, the information processing apparatus 100 includes a processor 101 , a main memory device 102 , an auxiliary memory device 103 , an input device 104 , an output device 105 and a communication device 106 . These are communicably connected to each other via communication means such as a bus (not shown).

Note that the information processing apparatus 100 does not necessarily have to be implemented entirely by hardware. may be realized by virtual resources such as

The processor 101 is configured using a CPU (Central Processing Unit), an MPU (Micro Processing Unit), and the like. The functions of the distribution server 10, the machine learning server 20, the user terminal 30, and the imaging device 40 are realized by the processor 101 reading and executing the programs stored in the main storage device 102. FIG.

The main memory device 102 is a device that stores programs and data, and is a ROM (Read Only Memory), a RAM (Random Access Memory), a nonvolatile semiconductor memory (NVRAM (Non Volatile RAM)), or the like.

The auxiliary storage device 103 includes, for example, SSD (Solid State Drive), various non-volatile memories (NVRAM: Non-volatile memory) such as SD memory card, hard disk drive, optical storage device (CD (Compact Disc), DVD (Digital Versatile Disc), etc.), a storage area of a cloud server, and the like. Programs and data stored in the auxiliary storage device 103 are read into the main storage device 102 at any time.

The input device 104 is an interface for receiving input of information, and includes, for example, a keyboard, a mouse, a touch panel, a card reader, a voice input device (such as a microphone), and a voice recognition device. The information processing device 100 may be configured to receive input of information from another device via the communication device 106 .

The output device 105 is an interface for outputting various kinds of information. synthesizer and the like. The information processing device 100 may be configured to output information to another device via the communication device 106 .

The communication device 106 is a wired or wireless communication interface that realizes communication with other devices via the communication network 5. Examples of the communication device 106 include a NIC (Network Interface Card), a wireless communication module, and a USB (Universal Serial Interface Card). interface) module, serial communication module, and the like.

[Functional configuration of distribution server]
FIG. 3A shows main functions (software configuration) of the distribution server 10. As shown in FIG. As shown in the figure, the distribution server 10 has functions of a storage area 110 and a management unit 120 . These functions are realized by processor 101 of distribution server 10 reading and executing programs stored in main memory 102 or auxiliary memory 103 of distribution server 10 . In addition to the functions described above, the distribution server 10 also has functions such as an operating system, a file system, a device driver, and a DBMS (DataBase Management System).

Among the functions described above, the storage area 110 is formed in the main storage device 102 or the auxiliary storage device 103 of the distribution server 10 . The storage area 110 stores data of a work video 151, process information 152, and work videos 153 and 154 obtained by photographing the work situation. The storage area 110 stores these data as, for example, database tables and files managed by a file system.

The process information 152 holds a plurality of combinations of the shooting time of the work moving image 151 (or the reproducing time of the moving image), the time required for the process, and the process name (FIG. 5). Details of the process information 152 and the method of generating the work animations 153 and 154 will be described later.

The management unit 120 has a function of recognizing human voice and converting it into character data, a function of creating a process, and the like.

[Functional configuration of machine learning server]
FIG. 3B shows main functions (software configuration) of the machine learning server 20. As shown in FIG. As shown in the figure, the machine learning server 20 has a storage area 210 and a management unit 220 . These functions are realized by the processor 101 of the machine learning server 20 reading and executing a program stored in the main storage device 102 of the machine learning server 20 . In addition to the functions described above, the machine learning server 20 also has functions such as an operating system, a file system, a device driver, and a DBMS (DataBase Management System).

A storage area 210 is formed in the main storage device 102 or the auxiliary storage device 103 of the machine learning server 20 . As shown in FIG. 3B, the storage area 210 stores learning data 251 used for machine learning by a learning device 221 (described later). The learning data 251 has a plurality of combinations of frames and process name labels (FIG. 6). Details such as a method of generating the learning data 251 will be described later.

The management unit 220 has various functions to be described later, and includes at least a learning device 221 . The learning device 221 has a function of learning the image feature amount of the input image and outputting information indicating the result of estimating the elements in the image with respect to the input image. As the learning device 221 having such a function, models of various types and structures can be adopted. Build and do deep learning.

The learning device 221 has an input layer that receives an input of an image, an output layer that outputs an estimation result of the target element, and an intermediate layer that extracts the feature amount of the input image. Each of the input layer, the output layer, and the intermediate layer has nodes (indicated by white circles in the figure), and the nodes of these layers are connected by edges (indicated by arrows in the figure). Note that the configuration of the learning device 221 shown in FIG. 4 is an example, and the number of nodes and edges, the number of intermediate layers, and the like can be changed as appropriate.

[Work schedule creation process]
Processing executed by the information processing system 1 configured as described above will be described below mainly using the sequence diagram of FIG. As an example, the process of creating a process chart for an article assembly work composed of a plurality of processes will be described below.

The processing performed by the information processing system 1 reads and executes programs stored in the main storage device 102 or the auxiliary storage device 103 in each of the distribution server 10, the machine learning server 20, the user terminal 30, and the imaging device 40. It is realized by In the following processing, it may be assumed that the distribution server 10 and the machine learning server 20 execute the processing of the management units 120 and 220 generated by the program.

In step S1, the assembly work is photographed. Filming is done for work related to at least one step, but it is preferable to film the entire work. The moving image data obtained by shooting is transmitted from the imaging device 40 and stored as the working moving image 151 in the storage area 110 of the distribution server 10 (S2).

In step S3, the user terminal 30 reproduces the moving image and the explainer explains the work. Specifically, the work video 151 is played back via the user terminal 30, and an explainer who is familiar with the assembly work explains each process shown in the video while watching the video. The content to be explained may be the name of the process or the details of the work, but hereinafter, the comment of the explainer in this process will be uniformly treated as the "name of the process".

The distribution server 10 acquires the voice of the presenter via the user terminal 30 (S4). At the time of acquisition, the voice may be saved in the storage area 110, or the next process (step S5) may be performed simultaneously without saving.

The distribution server 10 then creates process information 152 (S5). The process information 152 is created by associating the time required for each process with the name of each process, as shown in FIG.

Specifically, as shown in FIG. 8, the voice of the explainer is analyzed by the distribution server 10 (management unit 120), and the voice of the explainer is converted into characters. In parallel with this, the duration of the process is measured by measuring the duration of the presenter's speech. For example, if the voice "screw tightening" is generated for 2 seconds, the distribution server 10 sets the screw tightening process time to 2 seconds. Process information 152 is created by associating the characters "screw tightening" obtained as a result of this processing with the time required for the process (FIG. 5).

Alternatively, intervals of speech may be measured as the time taken for the process. For example, if the voice "tighten a screw" and the voice "moving a product" are acquired in sequence, the interval from the time when the voice "tightening a screw" occurs to the time when the voice "moving the product" occurs is calculated by the delivery server. 10 is the time for the screw tightening process.

Note that "silence" shown in FIG. 8 and the like does not necessarily mean that there is no sound. For example, if the sound volume in the frequency band of the human voice is below a threshold value, there is no sound, and if the speech analysis yields a speech waveform that cannot be converted into text, there can be no sound.

The distribution server 10 can create a process chart as shown in FIG. 9 based on the created process information 152 (S7). In the process information 152, as shown in FIG. 5, the video recording time (or video playback time, the same shall apply hereinafter), the name of each process, and the time required for each process are associated and stored. The distribution server 10 creates a process chart using these associations.

There are various forms of process charts. For example, a flow chart format as shown in FIG. 9, a Gantt chart, a line diagram, a table format such as FIG. 5, and various other formats may be adopted according to the user's wishes.

〔study〕
The work animation 151 and the process information 152 created by the above process are used for learning by the learning device 221 as described below. Learning by the learner 221 is performed according to the flowchart of FIG.

In step S<b>11 , the management unit 220 of the machine learning server 20 acquires the work video 151 from the distribution server 10 and labels the work video 151 . In this process, each frame, which is a still image that constitutes the work moving image 151, is labeled with the name of the process to indicate which process the work displayed in the frame corresponds to (FIG. 6). Note that this process may be executed by the distribution server 10 instead of the machine learning server 20 .

When labeling, the management unit 220 reads the name of each process and the time associated with each process from the process information 152 . Furthermore, the management unit 220 labels the process names according to the shooting time of the work moving image 151 .

In the next step S<b>13 , the management unit 220 performs image processing including edge processing on the work moving image 151 . By performing the edge processing, the outline of the operator is clearly displayed, which facilitates the learning process. As a result of the processing up to step S13, each frame of the edge-processed working video 151 is associated with the process name and shooting time, and is stored in the storage area 210 as the learning data 251 (FIG. 6). ). At this time, the learning data 251 may be modified by the user.

Note that edge processing generally employs a method of extracting pixels with large luminance changes in a frame and creating a binarized image that is distinguished from pixels in other portions (FIG. 11). Further, in the image processing (S13), not only edge processing but also processing such as expansion processing and contraction processing may be used together for the purpose of removing noise and clarifying contours.

In the next step S<b>15 , the management unit 220 causes the learning device 221 to learn using the learning data 251 . The learning device 221 acquires a feature amount of each frame in which a worker, a tool, a product, or the like is displayed, and a feature amount relating to changes in each frame, and learns them together with the name of the process.

By executing this learning process, the learning device 221 functions as a learned model that estimates which process the work displayed in the moving image corresponds to.

Note that the learning process may be executed a plurality of times based on not only one work moving image 151 but also a plurality of work moving images. By using videos about different work contents, it may be possible to create a trained model corresponding to various work. Alternatively, the accuracy of estimation processing (described later) can be improved.

〔Estimated〕
By using the trained learner 221, it is possible to execute an estimation process for estimating the process from the moving image without using the explanation by the explainer. Details of the estimation process will be described below with reference to the flowchart of FIG. 12 .

In step S<b>21 , the learning device 221 acquires a new work video 153 different from the work video 151 from the imaging device 40 . The work video 153 captures the worker performing work across a plurality of processes. When acquiring the work video 153, the learning device 221 performs image processing including edge processing on the work video 153 as shown in FIG. 154 is generated (S22).

Next, the learning device 221 reads a plurality of frames forming the work video 154, and estimates the process name of the work shown in the work video 154 and the time required for each process (S23). Specifically, the learning device 221 acquires the feature amount of the image of each frame in which the worker, the tool, the product, or the like is displayed, or the feature amount related to the change in each frame, and converts the acquired feature amount into the frame. Estimate the process name of the displayed operation.

At the same time, the learning device 221 can grasp the time required for each process from the shooting time of the frame. Alternatively, the time required for the process can be calculated based on the number of frames with the same process name and the frame rate. In this way, the name of each process and the time required for the process are estimated.

The learning device 221 creates a process chart using the process name estimated for each frame (S25). A process chart is created based on the name of each process estimated in step S23 and the time required for each process. As with the processing of step S5 (FIG. 9), an appropriate format of the process chart is prepared according to the user's request, settings, and the like. Also, a table in the same format as the process information 152 may be created.

<Modification>
In the embodiment, each device is communicably connected by the communication network 5, but the devices do not necessarily have to be connected by communication means. For example, the photographing device 40 may be independent from the communication network 5, and data such as the work moving image 151 may be transferred to each device via a storage medium such as a memory. The same is true for other devices.

Also, the processes and functions of each device shown in the embodiment may be executed by another device. For example, part or all of the image processing (S11, S13, S22) including labeling and edge processing may be performed by the distribution server 10, the imaging device 40, or the user terminal 30. Similarly, part or all of the learning (S15) and process estimation (S23) may be performed by the distribution server 10, the photographing device 40, or the user terminal 30. The same applies to other processing or functions.

<effect>
In the above-described embodiment, the information processing system 1 performs a process (S3) of reproducing a work video 151 (corresponding to a first video) in which work composed of a plurality of steps is recorded, and the work displayed in the work video 151 will be described. A process (S4) of having the person give an oral explanation and acquiring the voice of the explainer, and having the distribution server 10 analyze the voice of the explainer, measuring the time required for each of the plurality of steps, and performing the processing as shown in FIG. A measurement process (S5) for acquiring process information 152 having a process time set is executed.

In the above configuration, the process chart is created only by the explainer verbally stating the name of the process. There is no need for the explainer to enter the name of the process from the keyboard, measure the time required for each process with a stopwatch, and take notes. Conventionally, due to such trouble and labor, much time and manpower were required to grasp the process. In addition, there is a possibility that quality defects may occur due to an error in measuring the time or an error in entering the details of the work. On the other hand, with the above configuration, the process can be grasped quickly or accurately without requiring labor.

In the measurement process (S5), the distribution server 10 measures the time required for each of the multiple steps based on the duration of the voice of the explainer.

By adopting such a method, it is possible to easily or accurately measure the time required to perform each step.

The distribution server 10 analyzes the explainer's voice and acquires the process information 152 that holds a name set indicating the names of each of a plurality of processes as text, that is, characters (S5).

In the above configuration, the explainer's voice is recognized and the text data is automatically created, so the explainer does not need to input the name of the process from the keyboard. Data can be created quickly.

The distribution server 10 creates a process chart showing the name of each process in association with the process time from the process information 152 (S7).

The information processing system 1 executes processing (S11, S13) of creating learning data 251 using the work moving image 151 and the process information 152. FIG. In the machine learning server 20, the learning data 251 is used to cause the learning device 221 to learn, and a process (S15) of creating a trained model is executed.

With the above configuration, it is possible to create a learning model for estimating a process or creating a process chart by using the work video 151 as data for machine learning.

The learned learning device 221 performs a process (S21) of reading a work video 153 recording a work composed of a plurality of processes, a name set indicating the names of each of the processes that constitute the work, and each of the work An estimation process (S23) for estimating a process time set indicating the time required for the process is executed.

By executing such processing, the process name and time can be acquired without the explainer's oral explanation. Process data can be obtained without human intervention other than photographing the work.

The learning device 221 creates a process chart in which each name of the process is associated with the process time (S25).

With the above configuration, it is possible to easily create a process chart without human intervention, and to express the time required for each process and the details of the work in an appropriate format.

In the estimation process, the working moving image 153 is edge-processed (S21). Also, the learning device 221 acquires the feature amount of the image and estimates at least one of the process name and the process time (S23).

By performing edge processing as described above, the learning device 221 can grasp the contour of the operator or changes in the contour, and can accurately acquire the feature amount. Therefore, it becomes easy to estimate the work content of each process or the time required for each process.

Information processing system 1
server 10
machine learning server 20
User terminal 30
imaging device 40

Claims (9)

A method performed in a system comprising one or more computing devices, comprising:
a process of reproducing a first moving image recording a first work composed of a plurality of steps;
a process of causing an explainer to verbally explain the first work displayed in the first moving image and acquiring the voice of the explainer;
and a measurement process of analyzing the audio and measuring a first process time set indicative of the time taken for each of the plurality of processes. In the measurement process,
2. The method of claim 1, wherein the duration of each of the plurality of steps is measured based on the duration of the speech. 3. The method of claim 1 or 2, further comprising parsing the speech to obtain a first set of names indicating names of each of the plurality of steps as characters. 4. The method according to claim 3, further comprising creating a process chart showing each name in said first name set and each time in said first process time set in association with each other. A process of creating learning data using the first moving image, the first name set, and the first process time set;
a process of making the arithmetic device learn using the learning data to create a trained model;
5. The method of claim 3 or 4, further comprising: The trained model is
a process of reading a second video recording a second work composed of a plurality of steps;
an estimation process of estimating a second name set indicating the names of each of the plurality of processes constituting the second work, and a second process time set indicating the time required for each process of the second work;
6. The method of claim 5, performing The trained model is
7. The method according to claim 6, further comprising creating a process chart showing each name of said second name set and each time of said second process time set in association with each other. The trained model, in the estimation process,
a process of acquiring an image obtained by performing edge processing on the second moving image;
a process of acquiring a feature amount of the image;
estimating at least one of the second name set and the second process time set based on the image;
8. A method according to claim 6 or 7, wherein performing A system comprising one or more computing devices,
a process of reproducing a first moving image recording a first work composed of a plurality of steps;
a process of causing an explainer to verbally explain the first work displayed in the first moving image and acquiring the voice of the explainer;
and a measurement process of analyzing the voice and measuring a first process time set indicating the time required for each of the plurality of processes.

Images