WO2017017737A1

WO2017017737A1 - Display method, monitor result output method, information processing device, and display program

Info

Publication number: WO2017017737A1
Application number: PCT/JP2015/071170
Authority: WO
Inventors: 伊藤　史; 威彦西村; 一樹 ▲高▼橋
Original assignee: 富士通株式会社
Priority date: 2015-07-24
Filing date: 2015-07-24
Publication date: 2017-02-02

Abstract

In a method for displaying, in order to improve task-learning effectiveness, a task state imaged using an imaging device, a computer executes processing in which a completed task among tasks included in a prescribed task scenario is specified in accordance with a motion detected on the basis of a captured image, and a stored image associated with a task in the task scenario specified as the task following the completed task, is displayed superimposed on an image captured after the motion was detected.

Description

Display method, monitor result output method, information processing apparatus, and display program

The present invention relates to a display method, a monitor result output method, an information processing apparatus, and a display program.

The assembly of products is partly automated, but it is still often performed by people (operators). From the perspective of maintaining product quality and ensuring worker safety, it is important to monitor that work is being performed accurately and to allow workers to learn work efficiently. Yes.

Conventionally, a technique for recognizing correct operation by displaying a model image and an actual image side by side is known (see, for example, Non-Patent Documents 1 and 2).

In the conventional technology described above, the model image and the actual image are displayed side by side, so it is difficult to recognize subtle differences in hand movements, body postures, etc. It was not enough.

Therefore, in one aspect, the purpose is to enhance the effect of learning the work.

In one embodiment, in a method for displaying a work situation imaged using an imaging device, a completed work is identified and completed among work included in a predetermined work scenario according to a motion detected based on a captured image. The computer executes a process of displaying the image stored in association with the work specified in the work scenario as the next work after the work and the captured image after the detection of the motion.

* The effect of mastering work can be enhanced.

It is a figure which shows the example of the external appearance of the system concerning one Embodiment. It is a figure which shows the example of the external appearance of a pallet. FIG. 3 is a diagram illustrating a software configuration example of an information processing apparatus. It is a figure (the 1) which shows the example of a data structure of various information. It is FIG. (2) which shows the data structure example of various information. It is a figure which shows the hardware structural example of information processing apparatus. It is a flowchart which shows the process example of work management. FIG. 10 is a diagram (part 1) illustrating an example of a display screen; It is a flowchart which shows the process example which determines entering / exiting of the hand to a pallet. It is FIG. (2) which shows the example of a display screen. FIG. 10 is a third diagram illustrating an example of a display screen; It is a flowchart which shows the process example of expert image superimposition display. It is a flowchart which shows the example of a process of adjustment of a trunk axis. It is a figure which shows the example of acquisition of a trunk axis. It is a figure which shows the example of a screen which superimposed the image of the expert on the operator's image. It is a flowchart which shows the process example of a timing adjustment. It is a flowchart which shows the process example of deviation specific evaluation. It is a figure which shows the example of application of an error range. It is a figure which shows the example of a display or reporting of deviation | shift. It is a flowchart which shows the process example of work evaluation. It is a flowchart which shows the process example of determination of the coincidence degree of a locus | trajectory. It is a figure which shows the example of application of a tolerance | permissible_range. It is a flowchart which shows the example of a process of calculation of the mismatch degree of a locus | trajectory.

Hereinafter, preferred embodiments of the present invention will be described.

<Configuration>
FIG. 1 is a diagram illustrating an example of the appearance of a system according to an embodiment. In FIG. 1, a motion sensing device 2 and a display / speaker 3 are connected to an information processing device 1 such as a PC (Personal Computer). The motion sensing device 2 performs imaging with a view of a plurality of pallets P and a worker H arranged on a workbench, and the movement (motion) of the worker's hand and body parts (skeletal elements such as joints). Is detected. The pallet P accommodates parts and tools used for product assembly. In the example shown in the drawing, the motion sensing device 2 is arranged above the front of the worker H, but the place where the motion sensing device 2 can be arranged is not limited as long as the pallet P and the worker H can be viewed. The display / speaker 3 displays work instructions and alerts to the worker H, and outputs sound as necessary. The display / speaker 3 may be provided with a touch panel.

FIG. 2 is a diagram showing an example of the appearance of the pallet P, and markers (reference objects) M including code information are pasted at positions such as four corners that are easily imaged from the motion sensing device 2. The marker M is associated with information for designating an image region (a three-dimensional image region including depth information) used to determine whether or not the worker H has entered the corresponding pallet P. An image area (pallet area) as indicated by a broken line with reference to the marker M is set. When the four corners of the pallet P are used as a reference, at least two markers M may be captured by the motion sensing device 2 with respect to each pallet P. An image area can also be set by the marker M.

Since the pallet area is automatically set based on the marker M, it is not necessary to arrange the pallet P at a strict position, and even if the position is slightly shifted, the detection of the operation is not affected.

FIG. 3 is a diagram illustrating a software configuration example of the information processing apparatus 1. In FIG. 3, the information processing apparatus 1 includes a motion sensing processing unit 11, a work management unit 12, an expert image superimposing display unit 13, and a work evaluation unit 14. The motion sensing processing unit 11 includes, in addition to the captured image, a data stream of the captured image (color moving image) and the depth image (the distance information from the viewpoint associated with each pixel of the captured image) of the motion sensing device 2. It has a function of generating skeleton information (skeleton information, joint information) including the position of a human body part (such as a joint) by calculation. Skeletal information includes "head", "shoulder center" "left shoulder" "right shoulder" "left elbow" "left wrist" "left palm" "right elbow" "right wrist" "right palm" "center of spine" Position information such as the center of the waist, the base of the left foot, the base of the right foot, the left knee, the left ankle, the right knee, the right ankle, the left toe, and the right toe can be obtained in real time. Note that when the function of the motion sensing processing unit 11 is included in the motion sensing device 2, it is not necessary to provide the motion sensing processing unit 11 in the information processing device 1.

The work management unit 12 has a function of managing the work performed by the worker H based on the captured image and the skeleton information output from the motion sensing processing unit 11. Details of the processing contents will be described later. The expert image superimposing display unit 13 superimposes and displays the expert's image on the work performed by the worker H based on the captured image and the skeleton information output from the motion sensing processing unit 11, so that the skill of the skilled worker is displayed on the worker H. Has a function to promote learning. Details of the processing contents will be described later. The work evaluation unit 14 has a function of numerically evaluating the work of the worker H based on the captured image and the skeleton information output from the motion sensing processing unit 11. Details of the processing contents will be described later.

Information used for processing by the work management unit 12 or the like includes work information D1, pallet information D2, marker information D3, worker information D4, delay log information D5, error log information D6, skilled worker work information D7, and error range information D8. , Deviation information D9, model locus information D10, allowable range information D11, and evaluation information D12 are held.

4 and 5 are diagrams showing examples of data structures of various types of information. In FIG. 4, the work information D1 includes “product ID / product name”, “process ID / process name”, “work ID”, “work title”, “instruction screen”, “work instruction text”, “corresponding pallet ID”, “hand to pallet”. “In / out”, “Right / left / both hands”, “Maximum start time / minimum time”, “Standard time / allowable time”, and the like. “Product ID / Product name” is information for identifying a product to be assembled. The “process ID / process name” is information for identifying a process in which product assembly or the like is divided into several stages. “Work ID” is information for identifying a work (procedure) obtained by subdividing a process. The work includes picking up an object, placing an object, assembling an object, and the like.

“Work title” is a title indicating an outline of the work. The “instruction image” is an image showing details of work. The “work instruction text” is a character string indicating details of the work. The “corresponding pallet ID” is information indicating a pallet used for the work. When the process is combined, the pallet required in the process is specified. “Hand entering / exiting the pallet” is information indicating whether or not the operator's hand entering / exiting the pallet is used to determine the operation. “Separate right hand / left hand / both hands” is information indicating a worker's hand (right hand / left hand / both hands) designated to be entered / exited in the work. In addition, it is possible to include information on the position when each pallet is placed on the work table in association with the work or process. The position of the pallet on the work table is associated with the hand that the worker enters and exits, and is a position at which the work can proceed optimally. This position information can be used as a reference for placement by displaying on the screen when placing the pallet on the workbench before assembly of the product is started.

“Start maximum time / minimum time” is information indicating the maximum time and minimum time allowed from the start of the work to the hand when the work involves entering / exiting the pallet. “Standard time / allowable time” is the standard time and allowable time until the work is completed when the work involves entering and exiting the pallet (allowable time for long and allowable time for short) ). Note that a series of work by the worker H is performed in units of processes, and work information regarding one process can be referred to as a “work scenario”.

The pallet information D2 has items such as “pallet ID” and “corresponding marker ID”. “Pallet ID” is information for identifying a pallet. The “corresponding marker ID” is information for identifying a marker attached to the palette.

The marker information D3 has items such as “marker ID” and “pallet area designation”. “Marker ID” is information for identifying a marker. “Palette area designation” is information for designating an image area for detecting the entry / exit of a hand with reference to a marker.

The worker information D4 has items such as “worker ID” and “attribute”. “Worker ID” is information for identifying a worker. “Attribute” is information such as the name and affiliation of the worker.

The delay log information D5 includes items such as “log ID”, “date and time”, “worker ID”, “product ID / process ID / work ID”, and “delay content”. “Log ID” is information for identifying delay log information. “Date / time” is information indicating the occurrence date / time of an event recorded as delay log information. The “worker ID” is information for identifying a worker who has performed work that is the target of the delay log information. “Product ID / Process ID / Work ID” is information for identifying the product / process / work of the work that is the target of the delay log information. The “delay content” is information indicating the content of the delay (such as the delay in the time when the hand has been put into the pallet and the delay time).

The error log information D6 has items such as “log ID”, “date and time”, “worker ID”, “product ID / process ID / work ID”, and “error content”. “Log ID” is information for identifying error log information. “Date and time” is information indicating the date and time of occurrence of an event recorded as error log information. The “worker ID” is information for identifying a worker who has performed a work that is a target of error log information. “Product ID / Process ID / Work ID” is information for identifying the product / process / work of the work that is the target of the error log information. “Error contents” is information indicating the contents of an error (such as putting a hand different from the designated hand in the palette).

5, the expert work information D7 includes items such as “expert work information ID”, “video”, “depth”, “skeleton information”, and “frame / work correspondence information”. The “skilled worker work information ID” is information for identifying skilled worker work information. “Video” is information of a captured image of work of an expert. “Depth” is information on the depth image of the work of an expert. “Skeleton information” is skeleton information of the work of an expert. The “frame / work correspondence information” is information for associating a frame of a captured image of work of an expert with a target product / process / work.

The error range information D8 has items such as “error range information ID” and “error range”. “Error range information ID” is information for identifying error range information. “Error range” is information indicating an error range (for example, a radius centered on a reference point).

The deviation information D9 includes items such as “deviation information ID”, “date and time”, “worker ID”, “product ID / process ID / work ID”, and “deviation content”. The “deviation information ID” is information for identifying deviation information. “Date and time” is information indicating the date and time of occurrence of an event recorded as deviation information. The “worker ID” is information for identifying a worker who has performed a work that is a target of deviation information. “Product ID / Process ID / Work ID” is information for identifying the product / process / work of the work that is the target of the deviation information. The “deviation content” is information indicating the content of the deviation.

The model locus information D10 has items such as “example information ID”, “frame number”, and “coordinate”. The “example information ID” is information for identifying example trajectory information as a reference for evaluation. The model trajectory information is in principle hand trajectory information, but if there is another body part whose motion is to be evaluated, the trajectory information about the part may be added. “Frame number” is information indicating the frame number of an image (moving image) in which a locus is recorded. “Coordinates” is information indicating the coordinates of the locus (two-dimensional coordinates and depth on the screen).

The allowable range information D11 includes items such as “allowable range information ID” and “allowable range”. “Allowable range information ID” is information for identifying allowable range information. “Allowable range” is information indicating an allowable range (for example, a radius centered on a reference point).

The evaluation information D12 includes items such as “evaluation information ID”, “date and time”, “worker ID”, “product ID / process ID / work ID”, and “evaluation value”. “Evaluation information ID” is information for identifying evaluation information. “Date and time” is information indicating the date and time of evaluation. “Worker ID” is information for identifying a worker who has performed a work recorded as evaluation information. “Product ID / Process ID / Work ID” is information for identifying the product / process / work of the work to be evaluated. “Evaluation value” is information indicating the contents of evaluation.

FIG. 6 is a diagram illustrating a hardware configuration example of the information processing apparatus 1. 6, the information processing apparatus 1 includes a CPU (Central Processing Unit) 102, a ROM (Read Only Memory) 103, a RAM (Random Access Memory) 104, and an NVRAM (Non-Volatile Random Access Memory) connected to a system bus 101. 105. In addition, the information processing apparatus 1 includes an I / F (Interface) 106, an I / O (Input / Output Device) 107, an HDD (Hard Disk Drive) / flash memory 108, a NIC (Network) connected to the I / F 106. Interface Card) 109, a monitor 110 connected to the I / O 107, a keyboard 111, a mouse 112, and the like. A CD / DVD (Compact Disk / Digital Versatile Disk) drive or the like can also be connected to the I / O 107. When the display / speaker 3 is also used as a monitor, the monitor 110 is not necessary.

The functions of the information processing apparatus 1 described with reference to FIG. 3 are realized by the CPU 102 executing a predetermined program. The program may be acquired via a recording medium or may be acquired via a network.

<Example of work management processing>
FIG. 7 is a flowchart illustrating an example of work management processing performed by the work management unit 12 of the information processing apparatus 1. In FIG. 7, when the process is started, the work management unit 12 acquires a situation (product ID, process ID, pallet ID, worker ID) set in advance by an administrator or the like (step S101). The situation is set based on the work scenario (work information D1) by the administrator or the worker via the user interface screen provided by the information processing apparatus 1 for each work table. At this time, a product to be assembled (product ID), an assembly process (process ID), and a worker (work ID) are set. By determining the process, the pallet (pallet ID) used in the assembly is uniquely determined. The set situation data is held in a storage area in the information processing apparatus 1.

Next, the work management unit 12 waits for a work start trigger (step S102). As for the work start trigger, the manager sends a start signal based on time, etc., the worker H operates the information processing apparatus 1 to send a start signal, or the worker H stands in front of the work table. Suppose that it is detected that the vehicle has stopped for a predetermined time.

When the work start trigger is detected, the work management unit 12 starts substantial processing (step S103).

The work management unit 12 acquires the next work ID from the work information D1 and displays the screen (step S104). In addition, in the case of work in which the worker H does not enter or leave the pallet P, the process shifts to the next work after a predetermined time considering the standard time of work. Even in the case where the worker H does not enter or leave the pallet P, the work content is monitored by image processing or the like, and when it is determined that the work is completed, the process proceeds to the next work. Also good.

FIG. 8 is a diagram showing an example of the display screen. The product number and the process number are displayed in the display area A1, the work title is displayed in the display area A2, and the work included in the current process is displayed in the display area A3. The title is displayed as a list, and the current work is highlighted as indicated by C1. An instruction image is displayed in the display area A4, and a work instruction text is displayed in the display area A5. The display area A6 is an area where error information is displayed, but no error information is displayed in the illustrated example. B1 and B2 are buttons for moving the work forward and backward.

Returning to FIG. 7, the work management unit 12 starts measurement from the initial value 0 for the first elapsed time (step S105). The first elapsed time is for determining the time to start moving from the instruction to start work on the screen until the worker H starts some operation.

Next, the work management unit 12 determines whether or not the worker H has entered the pallet associated with the work in the work scenario (work information D1) (step S106).

FIG. 9 is a flowchart showing an example of a process for determining entry and exit of the hand from the pallet P, which is a process executed in parallel with each operation. In FIG. 9, the work management unit 12 detects a marker M corresponding to the current work ID from the captured image (step S121), and specifies a pallet area (step S122). The palette area is as shown by the broken line in FIG.

Next, in FIG. 9, the work management unit 12 compares the three-dimensional position (two-dimensional coordinates and depth of the captured image) of the skeletal information of the hand (right hand / left hand) with the three-dimensional position of the pallet area. A detection status (detection → non-detection, non-detection → detection) of whether the hand has entered or not is determined (step S123).

Returning to FIG. 7, when the work management unit 12 determines that the hand of the worker H is not in the predetermined pallet (NO in step S <b> 106), the work management unit 12 determines whether the first elapsed time has passed a predetermined value. (Step S107). If it is determined that the time has not elapsed (NO in step S107), the process returns to the determination of whether or not the hand has entered the pallet P (step S106). It should be noted that hand entry / exit detection is performed for all pallets on the workbench, and when a hand enters a pallet that is not associated with the current work, the screen display and recording are performed as an error. Good.

If the work management unit 12 determines that the hand of the worker H has entered the pallet associated with the work (YES in step S106), the work management unit 12 determines whether or not the hand matches the hand associated with the work. (Step S108). When it is determined that the hand of the worker H has entered the predetermined pallet (YES in step S106), even when the first elapsed time is shorter than the minimum time, an alert display and recording that the work is started too early May be performed.

When the work management unit 12 determines that the hand matches the hand associated with the work (YES in step S108), the work management unit 12 starts measurement of the second elapsed time from an initial value: 0 (step S109). The second elapsed time is for determining whether or not the time for putting the hand in the pallet P is appropriate.

Next, the work management unit 12 determines whether or not the hand of the worker H has come out of the pallet P (step S110). For this determination, the hand detection situation described with reference to FIG. 9 is used.

In FIG. 7, when the work management unit 12 determines that the hand of the worker H has not come out of the pallet P (NO in step S110), the work management unit 12 determines whether or not the second elapsed time is within an allowable range (step S111). If it is determined that the value is within the allowable range (YES in step S111), the process returns to the determination (step S110) as to whether or not the hand of the worker H has come out of the pallet P.

When the work management unit 12 determines that the hand of the worker H has come out of the pallet P (YES in step S110), the work management unit 12 stops counting the second elapsed time (step S112), and displays the screen for the next work (step S112). Return to S104). When it is determined that the hand of the worker H has come out of the pallet P (YES in step S110), the work is too early even when the second elapsed time is shorter than the allowable time with respect to the standard time of the work. The alert display and recording may be performed.

Further, when the work management unit 12 determines that the first elapsed time has passed the predetermined value when the worker H's hand is not in the predetermined pallet (NO in step S106), (YES in step S107). It is determined that the worker H is at a loss. In this case, the work management unit 12 records the delay log information D5 as a delay, displays the delay information on the screen (step S113), resets the first elapsed time (step S114), and whether the pallet P has entered the hand. It returns to judgment (step S106) of no. FIG. 10 is a diagram showing an example of the display screen, and shows a state in which an alert “Error: This operation is not performed” is displayed in the display area A6.

Returning to FIG. 7, when the work management unit 12 determines that the hand of the worker H has entered the predetermined pallet (YES in step S <b> 106), the work management unit 12 determines that the hand does not match the hand associated with the work. If so (NO in step S108), it is determined that a work error (work error) has occurred. In this case, the work management unit 12 records the error log information D6 as an error (step S115), displays the error information on the screen (step S116), resets the first elapsed time (step S117), and stores it in the palette P. The process returns to the determination of whether or not a hand has entered (step S106). FIG. 11 is a diagram showing an example of the display screen, and shows a state in which an alert “Error: using right hand” is displayed in the display area A6.

Returning to FIG. 7, the work management unit 12 determines that the second elapsed time is not within the allowable range when it is determined that the hand of the worker H is not out of the pallet P (NO in step S110) ( The delay log information D5 is recorded as a delay (step S111), the delay information is displayed on the screen (step S119), and the process proceeds to the reset of the first elapsed time (step S117).

<Example of expert image overlay display>
FIG. 12 is a flowchart illustrating a processing example of expert image superimposition display by the expert image superimposition display unit 13 of the information processing apparatus 1. The expert image superimposed display process may be performed independently or in parallel with the work management process illustrated in FIG.

In FIG. 12, when processing is performed independently, the expert image overlay display unit 13 acquires a situation (step S21), detects a work start trigger, and starts substantial processing (step S22). This corresponds to steps S101 to S103 in FIG. 7. However, when there are a plurality of expert work information and error range information to be superimposed, information for specifying what to use for processing (experts) Work information ID, error range information ID) are added to the situation. When performing in parallel with the process of FIG. 7, the process starts from step S23 after step S103 of FIG. In step S103 in FIG. 7, the first work of each process is started, and the next work is started when it is determined that the previous work has been completed by motion detection (such as a hand coming out of the pallet).

Referring to FIG. 12, the expert image superimposing display unit 13 acquires expert work information D7 and work error range information D8 (step S23).

Next, the expert image superimposed display unit 13 adjusts the trunk axis with respect to the expert work information D7 and the worker work information (captured image, skeleton information) (step S24).

FIG. 13 is a flowchart showing a processing example of adjustment of the trunk axis (step S24 in FIG. 12). In FIG. 13, the expert image superimposing display unit 13 acquires the trunk axis of the expert on the two-dimensional coordinates from the expert work information D7 (step S241).

Next, the expert image overlay display unit 13 acquires the operator's trunk axis on the two-dimensional coordinates (step S242).

FIG. 14 is a diagram showing an example of acquiring a trunk axis. For each of the worker H indicated by a solid line and the skilled person SH indicated by a broken line, the center of the head / shoulder, the center of the spine, and the waist The line that passes through the center of is acquired as the trunk axis.

Next, returning to FIG. 13, the expert image superimposing display unit 13 calculates the deviation of the trunk axes of the two (step S243), and rotates the image display of the expert so as to correct the deviation (step S244). .

FIG. 15 is a diagram illustrating a screen example in which the image of the worker SH is superimposed on the image of the worker H. At this time, the image of the expert SH is set to have high transparency so as not to hinder the visual recognition of the image of the worker H. Since the trunk axis is adjusted so as to coincide with each other as much as possible, even if there is a difference in standing position or the like, a difference in a portion to be watched such as a hand movement is easily understood.

Next, returning to FIG. 12, the expert image overlay display unit 13 adjusts the timing with respect to the expert work information D7 and the worker work information (step S25).

FIG. 16 is a flowchart showing a processing example of timing adjustment (step S25 in FIG. 12). In FIG. 16, the expert image overlay display unit 13 compares the current work IDs of the worker and the skilled worker (step S251), and determines whether or not the same work is being performed based on the work ID (step S252). When the expert image superimposing display unit 13 determines that the same operation is being performed (YES in step S252), the timing adjustment ends.

When it is determined that the expert image superimposing display unit 13 is not performing the same operation (NO in step S252), the expert image superimposing display unit 13 determines whether the operation is advanced by the expert from the order of the operation ID (step S253).

If it is determined that the expert is working (YES in step S253), the expert image superimposing display unit 13 pauses the reproduction of the expert work video and delays the timing (step S254). The process returns to ID acquisition (step S251).

Further, when it is determined that the expert is not working (the worker is working) (NO in step S253), the expert image superimposing display unit 13 displays the work image of the expert. (Step S255), the process returns to the acquisition of the work ID (step S251).

Next, returning to FIG. 12, the expert image overlay display unit 13 evaluates the degree of deviation of the expert work information D7 and the worker work information (step S26).

FIG. 17 is a flowchart showing an example of processing for evaluating the deviation (step S26 in FIG. 12). In FIG. 17, the expert image superimposing display unit 13 acquires the joint positions of the expert and the worker (step S261).

Next, the expert image overlay display unit 13 determines whether each joint position of the worker is included in a predefined error range (step S262). FIG. 18 is a diagram showing an example of application of the error range. A sphere whose radius is centered on the position a of the expert's joint is set as an error range, and it is determined whether or not the joint position of the operator is included therein. . Since the position b is inside the sphere, it is determined that it is within the error range, and since the position c is outside the sphere, it is determined that it exceeds the error range.

Next, returning to FIG. 17, when the expert image superimposing display unit 13 is included in the error range (YES in step S <b> 263), the evaluation ends.

When it is not included in the error range (NO in step S263), the expert image superimposing display unit 13 marks the frame as a shifted frame (moving image frame) (step S264). Then, the expert image overlay display unit 13 highlights a range between the displaced joint and the adjacent joint (step S265).

FIG. 19 is a diagram showing an example of misalignment display or reporting. Since the positions of the right hand and the left hand are greatly deviated from the skilled person, the joint of the hand is highlighted (indicated by a bold line in the figure), and information about the misalignment is displayed. it's shown. In addition, you may display in real time and you may preserve | save as a report for confirmation after the fact. Further, when the part where the deviation occurs is a part of the head, shoulders, back muscles or the like other than the hand, it may be mentioned in the report that the posture is not appropriate.

Next, returning to FIG. 12, the expert image overlay display unit 13 stores the information on the deviation (step S27), and when the work is not completed (NO in step S28), the trunk axis is adjusted (step S24). If the work is finished (YES in step S28), the process is finished. Note that when storing the information on the shift (step S27), the moving image superimposed and displayed during the shift period may be cut out and stored separately so that it can be played back later.

In addition, although the case where the process is performed based on the captured image and the skeleton information of the worker acquired in real time has been described, the process may be performed based on the captured image and the skeleton information of the worker already recorded. Good.

In addition, as described above, it is possible to operate in parallel with the basic work management process shown in FIG. 7, but in that case, the superimposing display with the expert image is not always performed, Only when an error or the like occurs in the basic work management process, the superimposed display with the expert image can be performed.

<Work evaluation processing example>
FIG. 20 is a flowchart illustrating an example of work evaluation processing performed by the work evaluation unit 14 of the information processing apparatus 1. The work evaluation process may be performed independently, or may be performed in parallel with the work management process illustrated in FIG. 7 and the expert image superimposed display process illustrated in FIG.

In FIG. 20, when processing is performed independently, the work evaluation unit 14 acquires a situation (step S31), detects a work start trigger, and starts substantial processing (step S32). This corresponds to steps S101 to S103 in FIG. 7, but when there are a plurality of model trajectory information and allowable range information which are the criteria for evaluation, information for identifying the one to be used for processing (example trajectory). Information ID, allowable range information ID) is added to the situation. When performing in parallel with the process of FIG. 7, the process starts from step S33 after step S103 of FIG. In step S103 in FIG. 7, the first work of each process is started, and the next work is started when it is determined that the previous work has been completed by motion detection (such as a hand coming out of the pallet).

In FIG. 20, the work evaluation unit 14 acquires the model trajectory information D10 specified in the situation (step S33), and acquires the trajectory information of the worker (step S34). The model trajectory information is, in principle, hand trajectory information, and the trajectory of the worker to be compared with the hand trajectory information is also hand trajectory information. The trajectory information may be added. For example, for parts such as the head, shoulders, and back muscles, it is possible to evaluate whether or not the posture is appropriate from the deviation from the trace of the model.

Next, the work evaluation unit 14 determines the degree of coincidence between the trace information of the model and the trace information of the worker (Step S35).

FIG. 21 is a flowchart showing a processing example of determination of the degree of coincidence of trajectories (step S35 in FIG. 20). In FIG. 21, the work evaluation unit 14 compares the model trajectory information with the joint position of the worker (step S351).

Next, when the joint position of the worker deviates from the allowable range of the model trajectory information, the work evaluation unit 14 marks the frame as a mismatched frame (step S352). FIG. 22 is a diagram showing an example of application of the allowable range, in which a sphere having a radius around the position d in each frame of the model trajectory information is set, and the trajectory position of the worker is included therein. Determine whether or not.

Next, returning to FIG. 20, when the current work is not completed (NO in step S36), the work evaluation unit 14 determines the degree of coincidence between the trace information of the model and the trace information of the worker (step S35). Return.

When the current work is completed (YES in step S36), the work evaluation unit 14 calculates the degree of trajectory mismatch (step S37).

FIG. 23 is a flowchart showing a processing example of the calculation of the degree of discrepancy (step S37 in FIG. 20). In FIG. 23, the work evaluation unit 14 determines the inconsistency as
The number of unmatched frames / the number of frames of the entire work is calculated (step S371). The number of unmatched frames is the number of frames marked as unmatched frames.

In addition, although the case where the position of the trajectory is compared in consideration of the allowable range in each frame has been described, a tube-shaped area considering the allowable range is set in the model trajectory, and the operator's trajectory is included in the area. The evaluation may be performed without considering the time required for the movement depending on whether it is included. Further, the time axis of either the model trajectory or the worker trajectory may be enlarged or reduced, and evaluation may be performed in consideration of both positional coincidence and temporal coincidence.

Next, returning to FIG. 20, the work evaluation unit 14 stores the calculated degree of mismatch (step S38), and ends the process. The stored inconsistency is used as a report as operator evaluation information. Evaluation of posture is useful for guidance for performing work safely.

<Summary>
As described above, according to the present embodiment, there are the following advantages.
(1) The work can be monitored even when the position serving as a reference for monitoring varies.
(2) The effect of learning the work can be enhanced.
(3) The work actually performed can be appropriately evaluated.

The present invention has been described above by the preferred embodiments of the present invention. While the invention has been described with reference to specific embodiments, various modifications and changes may be made to the embodiments without departing from the broad spirit and scope of the invention as defined in the claims. Obviously you can. In other words, the present invention should not be construed as being limited by the details of the specific examples and the accompanying drawings.

DESCRIPTION OF SYMBOLS 1 Information processing apparatus 11 Motion sensing process part 12 Work management part 13 Expert image superimposition display part 14 Work evaluation part D1 Work information D2 Pallet information D3 Marker information D4 Worker information D5 Delay log information D6 Error log information D7 Expert work information D8 Error range information D9 Deviation information D10 Model trajectory information D11 Tolerable range information D12 Evaluation information 101 Bus 102 CPU
103 ROM
104 RAM
105 NVRAM
106 I / F
107 I / O
108 HDD / flash memory 109 NIC
110 Monitor 111 Keyboard 112 Mouse 2 Motion sensing device 3 Display / speaker H Worker P Palette M Marker SH Expert

Claims

In a method for displaying a work situation imaged using an imaging device,
According to the motion detected based on the captured image, identify the completed work among the work included in the predetermined work scenario,
The image stored in association with the work specified in the work scenario as the next work of the completed work and the captured image after the detection of the motion are displayed in an overlapping manner.
A display method characterized in that a computer executes processing.
The image and the captured image are moving images.
The display method according to claim 1, wherein:
The image is displayed with a higher transparency than the captured image.
The display method according to claim 1, wherein:
The display controls a display position of an image stored in association with the work according to a position of the motion detection target.
The display method according to claim 1, wherein:
The position of the motion detection target is the position of the human trunk axis where the motion is detected.
The display method according to claim 4, wherein:
The image is a moving image, and a work completion timing or a new work start timing is specified according to the motion detected based on the captured image, and the image that is a moving image is determined at the completion timing or the start timing. Reproduce,
The display method according to claim 1, wherein:
Detecting a shift between the contour or skeleton or trunk axis of a person to be detected in motion and the contour or skeleton or trunk axis of the person included in the image;
Display the occurrence of the deviation according to the detected deviation,
The display method according to claim 1, wherein:
In a method for displaying a work situation imaged using an imaging device,
Identify the target work among the work included in a given work scenario,
An image stored in association with the target work and a captured image are displayed in an overlapping manner.
A display method characterized in that a computer executes processing.
According to the motion detected based on the captured image, identify the completed work among the work included in the predetermined work scenario,
Based on the image stored in association with the work scenario defined in the work scenario as the next work of the completed work, and the captured image after the detection of the motion, Detecting a deviation from the position of the person included in the image,
Output timing information when the deviation is detected, or output a captured image after detection of the motion when the deviation is detected,
A work monitoring result output method, characterized in that a computer executes processing.
According to the motion detected based on the captured image, identify the completed work among the work included in the predetermined work scenario,
Based on the image stored in association with the work scenario specified in the work scenario as the next work of the completed work and the captured image after the detection of the motion, Detecting a shift between the skeleton or the trunk axis and the outline or skeleton or trunk axis of the person included in the image;
Output timing information when the deviation is detected, or output a captured image after detection of the motion when the deviation is detected,
A work monitoring result output method, characterized in that a computer executes processing.
In an apparatus for displaying a work situation imaged using an imaging apparatus,
Means for identifying a completed work among the work included in a predetermined work scenario according to the motion detected based on the captured image;
Means for displaying the image stored in association with the work defined in the work scenario as the next work of the completed work and the captured image after the detection of the motion,
An information processing apparatus comprising:
In an apparatus for displaying a work situation imaged using an imaging apparatus,
Means for identifying the target work among the work included in the predetermined work scenario;
Means for superimposing and displaying an image stored in association with the target work and a captured image;
An information processing apparatus comprising:
Means for identifying a completed work among the work included in a predetermined work scenario according to the motion detected based on the captured image;
Based on the image stored in association with the work scenario defined in the work scenario as the next work of the completed work, and the captured image after the detection of the motion, Means for detecting a deviation from a position of a person included in the image;
Means for outputting timing information when the shift is detected, or a captured image after detection of the motion when the shift is detected;
An information processing apparatus comprising:
Means for identifying a completed work among the work included in a predetermined work scenario according to the motion detected based on the captured image;
Based on the image stored in association with the work scenario specified in the work scenario as the next work of the completed work and the captured image after the detection of the motion, Means for detecting a deviation between a skeleton or a trunk axis and a human contour or a skeleton or a trunk axis included in the image;
Means for outputting timing information when the shift is detected, or a captured image after detection of the motion when the shift is detected;
An information processing apparatus comprising:
In the display program of the work situation imaged using the imaging device,
According to the motion detected based on the captured image, identify the completed work among the work included in the predetermined work scenario,
The image stored in association with the work specified in the work scenario as the next work of the completed work and the captured image after the detection of the motion are displayed in an overlapping manner.
A display program that causes a computer to execute processing.
In the display program of the work situation imaged using the imaging device,
Identify the target work among the work included in a given work scenario,
An image stored in association with the target work and a captured image are displayed in an overlapping manner.
A display program that causes a computer to execute processing.
According to the motion detected based on the captured image, identify the completed work among the work included in the predetermined work scenario,
Based on the image stored in association with the work scenario defined in the work scenario as the next work of the completed work, and the captured image after the detection of the motion, Detecting a deviation from the position of the person included in the image,
Output timing information when the deviation is detected, or output a captured image after detection of the motion when the deviation is detected,
A display program that causes a computer to execute processing.
According to the motion detected based on the captured image, identify the completed work among the work included in the predetermined work scenario,
Based on the image stored in association with the work scenario specified in the work scenario as the next work of the completed work and the captured image after the detection of the motion, Detecting a shift between the skeleton or the trunk axis and the outline or skeleton or trunk axis of the person included in the image;
Output timing information when the deviation is detected, or output a captured image after detection of the motion when the deviation is detected,
A display program that causes a computer to execute processing.