WO2019087870A1 - Work assistance device, work assistance method, and program - Google Patents

Abstract

The objective of the present invention is to notify a worker of the occurrence of work process errors and the like with high precision, thereby improving the work efficiency of the worker. This work assistance device comprises: a storage unit for storing, in association with each other, work that includes at least a first process and a second process carried out after the first process, and process-specific representative images representing the work carried out respectively in the first process and the second process; an input unit for inputting an image of the worker in the process of carrying out the work; a confirmation unit for comparing the input image and the representative images and confirming whether the work is being carried out in a prescribed sequence; and a reporting unit for reporting the result of the confirmation to the worker.

Description

Work support apparatus, work support method and program

(Description of related application)
The present invention is based on the claim of priority of Japanese Patent Application: Japanese Patent Application No. 2017-209179 (filed on October 30, 2017), and the entire contents of the same application are incorporated and described herein by reference. It shall be.
The present invention relates to a work support device, a work support method and a program.

Patent Document 1 discloses a maintenance support system capable of reducing the burden on workers and reducing labor of maintenance and inspection work in maintenance and inspection work of field equipment such as power stations and substations. ing. The document describes that a server 10 having a plurality of databases holding information on maintenance and inspection operations is provided. On the other hand, the portable terminal 6 carried by the worker is provided with a function of photographing the progress status for each predetermined process of maintenance work by the camera 2 attached to the helmet 1 worn by the worker and transmitting it to the server 10 . Then, the server 10 extracts the data necessary for the work from the database according to the progress taken by the camera 2 and provides the information necessary for the work from the display 5 attached to the helmet 1 worn by the worker It is described.

Patent Document 2 discloses a work management system that can analyze and evaluate a work state, a situation of a work site, and the like without relying on a supervisor. According to this document, the input unit 1 of the work management system captures an image of a worker with a high degree of skill in progress, and patterns the moving image using a known image processing method before the storage unit. Store in 3. In addition, the detection unit 2 of the work management system captures an image of a normal worker performing a work, patterns the moving image using a known image processing method, and then stores the pattern in the storage unit 3. , To the central processing unit 4. Then, the central processing unit 4 of the work management system compares the two moving images to determine the difference in work speed of a normal worker with respect to a highly skilled worker or an error in the work procedure.

JP 2005-216137 A Japanese Patent Application Publication No. 2003-167613

The following analysis is given by the present invention. When dispatching workers to the site for maintenance and maintenance work, work process errors may occur. According to the present inventor's analysis, many of these work errors are caused by worker's misrecognition, forgetting, oversight, assumption, etc., and it is necessary to confirm only a specific scene in the work. Can be determined. In this respect, Patent Document 1 describes that the main control unit 11 of the server 10 determines whether the maintenance work of the process is good or not from the progress of the maintenance work transmitted from the portable terminal 6 at each predetermined maintenance process. It plays a role of judgment means. "

In the method of Patent Document 2, the work video of a highly skilled worker is taken in advance and compared with the work video of a normal worker, and the evaluation of the work of the normal worker is evaluated as bad or bad Will depend on the work image of a highly skilled worker. Moreover, although the method of patent document 2 can be used for the desk work etc. shown by FIG. 2 of the same document, it may not be able to be used for confirmation of the visit type and business trip type of work where the environment of the work site may be different each time. It is believed that there is.

An object of the present invention is to provide a work support apparatus, a work support method, and a program that can notify a worker of occurrence of a work process error or the like with high accuracy and contribute to improvement of the work efficiency.

According to the first aspect, an operation including at least a first step and a second step performed after the first step, an operation performed in each of the first step and the second step A storage unit that associates and holds a representative image for each process representing the image, an input unit that inputs an image during the process in which the worker is working, the input image, and the representative image There is provided a work support apparatus including a check unit that checks whether or not the work is performed in a predetermined order, and a notification unit that notifies the operator of the result of the check. Ru.

According to the second aspect, an operation including at least a first step and a second step performed after the first step, and each step representing the operations performed in the first and second steps A computer accessible to the means for associating and holding the representative image with the step of inputting an image during the process of the worker, collating the input image with the representative image; There is provided a work support method including a step of confirming whether the work is performed in a predetermined order and a step of notifying the worker of the result of the confirmation. The method is tied to a specific machine, a computer that functions as a work support device.

According to the third aspect, an operation including at least a first step and a second step performed after the first step, and each step representing the operations performed in the first and second steps Processing for inputting an image in a process in which the operator is working, collating the input image with the representative image on a computer accessible to the means for associating and holding the representative image There is provided a program for executing a process of confirming whether the work is performed in a predetermined order and a process of notifying the worker of the result of the confirmation. Note that this program can be recorded on a computer readable (non-transient) storage medium. That is, the present invention can also be embodied as a computer program product.

According to the present invention, it is possible to accurately notify the operator of the occurrence of an operation process error or the like and to improve the operation efficiency. That is, according to the present invention, the work support device described in the background art is converted into one whose leap function of checking the presence or absence of a work process error has been dramatically improved.

It is a figure which shows the structure of one Embodiment of this invention. It is a figure which shows another structural example of one Embodiment of this invention. It is a figure for demonstrating the information prepared to the memory | storage part of the work assistance apparatus of one Embodiment of this invention. It is a figure for demonstrating the operation | movement of one Embodiment of this invention. It is a figure which shows the structure of the 1st Embodiment of this invention. It is a figure which shows the structure of the wearable terminal used in the 1st Embodiment of this invention. It is a figure which shows the structure of the server of the 1st Embodiment of this invention. It is a figure which shows an example of the information hold | maintained at the work information database (work information DB) of the server of the 1st Embodiment of this invention. It is a figure which shows an example of the information hold | maintained at process image database (process image DB) of the wearable terminal of the 1st Embodiment of this invention, and a server. It is a flowchart showing operation | movement of the wearable terminal of the 1st Embodiment of this invention. It is a figure which shows the structure of the 2nd Embodiment of this invention.

First, an outline of an embodiment of the present invention will be described with reference to the drawings. The reference numerals of the drawings appended to this summary are added for convenience to the respective elements as an example for aiding understanding, and the present invention is not intended to be limited to the illustrated embodiments. In addition, connection lines between blocks such as drawings referred to in the following description include both bidirectional and unidirectional directions. The unidirectional arrows schematically indicate the flow of main signals (data), and do not exclude bidirectionality. Also, although there are ports or interfaces at the input / output connection points of each block in the drawing, they are not shown.

In one embodiment, the present invention can be realized by a work support apparatus 10a including a storage unit 11a, an input unit 12a, a confirmation unit 13a, and a notification unit 14a, as shown in FIG. The storage unit 11a represents the “work” including at least the first process and the second process performed after the first process, the first process, and the work performed in the second process. The representative image for each process to be performed is associated and held.

The input unit 12a inputs an image in a process in which the worker is performing the selected work.

The confirmation unit 13a collates the input image with the representative image to confirm whether the work is performed in a predetermined order.

The notification unit 14a notifies the worker of the result of the confirmation.

The work support apparatus 10a shown in FIG. 1 can also be realized by a computer 10b as shown in FIG. In this case, the memory 11b functions as the storage unit 11a described above, and the input unit 12b functions as the input unit 12a. The memory 11 b also includes auxiliary storage devices such as various disk drives. In addition, the input unit 12b may be a camera that captures an image in the process of working, an input interface that receives an image from such a camera, or the like. Further, the confirmation unit 13a and the notification unit 14a can be realized by the processor 13b by the above-described confirmation processing of the confirmation unit 13a and a computer program that causes the output unit 14b to output (notify) the result.

Here, the operation of the present embodiment will be described by taking support of the installation work of a certain application program on a computer as an example. In this application program, it is assumed that prior to the installation step (the second step), an uninstall operation (the first step) of the program of the old version is required. FIG. 3 shows an example of a representative image stored in the storage unit 11a in this case. In the example of FIG. 3, an image in which a subwindow displayed on the monitor during uninstallation is displayed is stored as a representative image of the first step. Similarly, as a representative image of the second step, an image in which a subwindow displayed on the monitor during installation is displayed is stored.

Then, when an image is input to the input unit 12a in the process of performing the work selected by the worker via, for example, a camera, the confirmation unit 13a selects the input image and the representative image. The collation is performed to confirm whether the work is performed in a predetermined order. For example, when the input image matches the representative image of the first step, the confirmation unit 13a determines that the work has been correctly started. Thereafter, when an image that matches the representative image of the second step is input, the confirmation unit 13a determines that the work has been correctly performed. Here, the confirmation unit 13a may cause the notification unit 14a to notify the worker that the work is correctly performed. By doing this, the worker can confirm that the application installation work has been correctly performed. After that, even if a problem occurs in the operation of the application program, the worker can recognize that the problem is in another place because the information that the installation work itself was correctly performed is obtained.

On the other hand, when the input image and the representative image of the first step coincide with each other, if the representative image of the second step matches, the confirmation unit 13a performs the second process before the first step. It is determined that the following process has been performed. In this case, it is preferable that the confirmation unit 13a notify the notification unit 14a that the work is not properly performed. By doing this, the worker can recognize that the application installation work was wrong. This allows the program installation to be redone before the next work or process. In addition, at that time, the confirmation unit 13a may instruct the notification unit 14a to notify the content of the specific correction operation. For example, in the case of the example of FIG. 4, after the confirmation unit 13a uninstalls the application of V1.1 and the application of V1.0 for the worker, the installation of V1.1 should be performed. It is conceivable to notify that.

First Embodiment
Subsequently, a first embodiment of the present invention using a wearable terminal as a work support apparatus will be described in detail with reference to the drawings. FIG. 5 is a diagram showing the configuration of the work support apparatus of the first embodiment of the present invention. Referring to FIG. 5, a configuration in which a glasses-type wearable terminal (worker attached terminal) 100 worn by a worker 100 a and the server 200 are connected via the network 101 is shown. In the following embodiments, a wearable terminal is described as a work image input device, but the present invention is not limited to a wearable terminal as a work image input device. For example, it is also possible to use a smartphone or other business terminal with a camera as an input device for work images. In addition, the camera itself may be configured to be independent, and an image transmitted from the camera may be transmitted to the work support apparatus via the wearable terminal, the smartphone, the business terminal, or the like. In addition, as a wearable terminal of a glasses type, what is called a smart glass or a headset type terminal can be used.

In addition, as a connection form (communication system) to the network 101 of the wearable terminal 100 and the server 200, various systems can be used. For example, connection via a mobile communication network, connection by wireless LAN (Local Area Network), etc. can be considered. In addition, the wearable terminal 100 and the server 200 may be connected to the network 101 via near field wireless communication such as Bluetooth (registered trademark) or infrared communication. Of course, it is also possible to adopt a method in which the wearable terminal 100 and the server 200 connect to the network 101 via wired communication such as a wired LAN.

FIG. 6 is a diagram showing a configuration of the wearable terminal of the first embodiment of the present invention. 6, wearable terminal 100 includes control unit 102, communication unit 103, image input unit 104, voice input unit 105, input unit 106, display unit 107, voice output unit 108, clock unit 109, and storage unit 110. It is shown.

The communication unit 103 performs data communication between the wearable terminal 100 and the server 200 via the network 101.

The image input unit 104 can be configured, for example, by an imaging device (camera). Specifically, in response to an instruction from the control unit 102, the image input unit 104 displays an image of an area visually recognized by the worker (a moving image or a predetermined interval (the imaging time interval may be wider than the moving image)) It may be a still image captured every time). The image input unit 104 stores the captured image in the process image DB 111 of the storage unit 110. The image input unit 104 does not necessarily have to be an imaging device (camera), and may be an interface to which an imaging device (camera) or the like can be connected.

The voice input unit 105 can be configured by a microphone (microphone). A voice instruction may be received from the operator via the voice input unit 105 to control the operation of the wearable terminal 100. A sound emitted from the work target such as a specific device or a sound around the worker may be taken from the voice input unit 105 to confirm whether the work is performed correctly.

The input unit 106 includes an information input unit such as a touch panel, a slide pad, a mouse, and a keyboard. An instruction may be received from the worker via the input unit 106, and the operation of the wearable terminal 100 may be controlled. In addition, information input by the worker via the input unit 106 (for example, correction data of the process image DB) may be stored in the process image DB 111 or the like of the storage unit 110. Note that the voice input unit 105 and the input unit 106 described above can be omitted as appropriate. For example, when the instruction from the worker is received by the image input unit 104 by recognizing the gesture of the worker, the voice input unit 105 and the input unit 106 can be omitted.

Note that the input of information such as an image or voice in the image input unit 104, the voice input unit 105, and the input unit 106 described above is performed at the timing instructed by the control unit 102.

The display unit 107 is configured by a liquid crystal display or an organic EL (Electro Luminescence) display that displays an output image (or a moving image) according to an instruction from the control unit 102.

The voice output unit 108 outputs voice according to an instruction from the control unit 102. For example, when it is necessary to obtain an operation sound at the time of an operation performed on a device to be worked, the control unit 102 requires the display unit 107 or the voice output unit 108 to the operator. You may make it output the message (speech guidance etc.) which urges execution of operation. Also, instead of the sound output unit 108, a vibration motor for alerting the operator, a device that generates vibration, or a lamp can be used as the output means.

The clock unit 109 measures the required time (elapsed time) from the start of work to each process. By comparing the measured required time with the standard time or the maximum allowable time, it is possible to grasp the presence or absence of a delay in work, the presence or absence of a trouble, and the like.

The control unit 102 controls the respective units of the wearable terminal 100 described above, collates the image input from the image input unit 104 with the representative image or the error image, and performs work progress management (see the flowchart of FIG. 10). Further, the control unit 102 according to the present embodiment manages the progress of the work by comparing the required time (elapsed time) of each process with a separately determined standard time or a maximum allowable time. Further, the control unit 102 of the present embodiment has a function of outputting a warning to the worker when a predetermined warning activation condition is satisfied. Such a control unit 102 can also be realized by reading a program 112 for controlling each unit of the wearable terminal 100 from the storage unit 110 and causing the processor to execute the program.

The storage unit 110 functions as a process image database (process image DB) 111. The process image information received from the server 200 is registered in the process image DB 111. The details of the process image information will be described in detail later.

FIG. 7 is a diagram showing the configuration of the server 200 according to the first embodiment of this invention. Referring to FIG. 7, a server 200 including a control unit 202, a communication unit 203, an image input unit 204, an audio input unit 205, an input unit 206, a display unit 207, an audio output unit 208, and a storage unit 209 is shown.

The communication unit 203 performs data communication between the wearable terminal 100 and the server 200 via the network 101.

The image input unit 204 is an image of a region visually recognized by the worker from the wearable terminal 100 side (a moving image or a still image captured at predetermined intervals (the imaging time interval may be wider than the moving image)). Is an interface that receives

The voice input unit 205 is an interface that receives voice data from the wearable terminal 100 side. A voice instruction may be received from the worker via the voice input unit 205 to control the operation of the server 200. The sound emitted by the work target such as a specific device or the sound around the worker may be taken in from the voice input unit 205 to check whether the work is properly performed.

The input unit 206 is configured by an information input unit such as a touch panel, a slide pad, a mouse, and a keyboard. An instruction may be received from the worker via the input unit 206 to control the operation of the server 200. In addition, information (for example, correction data of the process image DB) input by the worker via the input unit 206 may be stored in the process image DB 211 or the like of the storage unit 209.

The display unit 207 is configured by a liquid crystal display or an organic EL display that outputs a message or the like necessary for the operator on the server 200 side, and displays input / output contents.

In response to an instruction from the control unit 202, the voice output unit 208 outputs a message (voice guidance or the like) prompting the server side operator to perform a necessary operation. Further, instead of the voice output unit 208, a lamp or the like for alerting the operator with light can be used as an output means.

The control unit 202 controls each unit of the server 200 described above. The control unit 202 can also be realized by reading a program (not shown) for controlling each unit of the server 200 from the storage unit 209 and causing the processor to execute the program. Further, when an image input from the image input unit 104 is transmitted from the wearable terminal 100 to the server 200 side, the control unit 202 performs processing for performing work progress management (see the flowchart in FIG. 10). Also good. Specifically, the control unit 202 collates the image received from the wearable terminal 100 with the representative image or the error image, and manages the progress of the work. In addition, the control unit 202 may record the elapsed time since the start of the work, and determine whether there is a delay in the work by comparison with a separately determined standard required time.

Here, the information held in the storage unit 209 of the server 200 will be described. Work information and process image information are registered in the storage unit 209, and function as a work information database (work information DB) 210 and a process image database (process image DB) 211, respectively.

FIG. 8 is a diagram showing an example of information held in the work information DB 210. As shown in FIG. Referring to FIG. 8, an operation No. uniquely assigned to the “operation” specified by the device name, the operation type, and the fault content. Entries are shown. When the work information DB 210 receives a transmission request for a process image from the wearable terminal 100, the work information DB 210 receives the work content input from the work information DB 210. Used to identify the For example, when a certain worker carries out failure repair of a personal computer which does not start up, the work is identified as the work of # 1. The content of the work is not limited to the example of FIG. 8, and information for specifying various “work” to which the present invention can be applied may be registered. For example, in addition to the example of FIG. 8, the device removal operation and the periodic inspection operation may be registered in the operation information DB 210. Further, for example, in the case of oil change of an automobile, information indicating the model name and model number (model) of the automobile is set in the device field, and oil change is set in the operation type field (note that the fault content field). May be blank as in the case of the installation of the server 4 of # 4 in FIG. 8, or the necessity or the like of oil element replacement may be set).

FIG. 9 is a diagram showing an example of information held in the process image DB 211. As shown in FIG. Referring to FIG. Information for performing process progress management is registered in association with a representative image of each process in “Work” in 2 and this representative image and the like. The example of FIG. 9 corresponds to the task No. in the task information DB of FIG. The information at the time of performing the failure repair (paper jam) of the printer 2 identified as 2 is registered. Also, FIG. Of the six steps constituting step 2, the process proceeds normally to step 3, and a state in which an operation error is detected in step 4 is shown. Hereinafter, each item of FIG. 9 will be described.

The “representative image” field is a field for registering an image for confirming that the work has been correctly performed in each process. This representative image is assumed to be an area (strictly speaking, an input is made to wearable terminal 100) that a worker (for example, a maintenance worker of the apparatus may) visually recognize so that the matching of the image can be easily confirmed in wearable terminal 100 Image taken in the same region as the image to be More preferably, the angle, the size, the number of pixels, and the like of the representative image are set to be the same as the image input to the wearable terminal 100.

Moreover, in order to prevent an erroneous determination, it is preferable that the representative image is not an image that can be photographed in another process, but an image obtained only in that process. Further, this representative image may be an image to be collated with a part of the image assumed to be input to the wearable terminal 100. In addition, the angle, the size, the number of pixels, and the like of the image assumed to be input to the wearable terminal 100 and the representative image do not necessarily have to match. For example, it is also possible to adopt a method of collating both by setting one of the images as an inclined image by coordinate conversion, or converting the size and the number of pixels of the image.

Examples of representative images include an external appearance image of a hard disk drive in hard disk replacement work, and an external appearance image of a circuit board or mechanical component assuming a mounting state on a target device in a replacement operation of a circuit board or mechanical component. Further, as another example of the representative image, there is an image showing the appearance of the connector exposed to the outside of the work target device in the device connection work and the connection state thereof.

The “captured image” field in FIG. 9 is a field in which, of the images input from the image input unit 104 of the wearable terminal 100, an image obtained by matching with a representative image or an error image (described later) is registered. In the example of FIG. 9, although the “captured image” field is provided in the process image DB 211 on the server 200 side, when the comparison with the representative image is performed only on the wearable terminal 100 side, “ The "captured image" field can be omitted. As described above, in the example of FIG. 9, since steps 1 to 3 have been confirmed, an image determined to match the representative image is registered in the “captured image” field. Since a match with the error image is detected in step 4 of FIG. 9, an image determined to match the error image is registered in the “captured image” field.

The “execution status” field in FIG. 9 is a field for recording the progress status of the process confirmed by the comparison with the representative image by the control unit 102 of the wearable terminal 100 or the control unit 202 of the server 200. In the example of FIG. 9, the “implementation status” field is provided in the process image DB 211 on the server 200 side, but when the comparison with the representative image is performed only on the wearable terminal 100 side, “Step image DB 211 of the server 200 side The Implementation Status field can be omitted. As described above, in the example of FIG. 9, since steps 1 to 3 have been confirmed, information such as “operation start” and “step confirmation” are registered in the “execution status” field. In step 4 of FIG. 9, since a match with an error image is detected, information “error detection” is registered in the “execution status” field. Further, since the process 5 and the subsequent steps are not performed, the information of the initial state of "not performed" is registered.

The “procedure information” field in FIG. 9 is a field in which the work content and the like of the process next to each process are registered. The content of the “procedure information” field is notified to the worker via the display unit 107 or the voice output unit 108 as necessary.

The “notification information” field in FIG. 9 is a field in which information to be notified to the worker is registered when each process is reached. For example, when the captured image and the representative image match, and it is determined that the work is started, the worker is notified of the content registered in the notification information field. While the difference between work information and work information is created from the perspective that work information conveys the work content, notification information is used to prevent work errors when performing notes, contact matters, and the next process work. This is the point where alert information is set. In addition, forms, such as a character, a voice, an operation | movement, a figure, can be taken as said work information and notification information. In addition, characters and figures indicating the work information and the notification information may be superimposed and displayed on the glasses-type screen of the wearable terminal 100 to support the work of the worker.

The "error image" field is a field for registering an image for detecting that an operation error has occurred in each process. As in the case of the representative image, this error image is also a region (strictly speaking, wearable) in which the worker (for example, a maintenance worker of the apparatus) may visually check so that the matching of the image can be easily confirmed in the wearable terminal 100 It is preferable that the same image as the image assumed to be input to the terminal 100 be captured. More preferably, the angle, the size, the number of pixels, and the like of the error image are set to be the same as the image input to the wearable terminal 100.

Further, in order to prevent an erroneous determination, it is preferable that the error image is not an image that can be photographed in other steps, but an image obtained only in that step. Also, this error image may be an image to be collated with a part of the image assumed to be input to the wearable terminal 100. In addition, the angle, the size, the number of pixels, and the like of the image assumed to be input to the wearable terminal 100 and the error image do not necessarily have to match. For example, it is also possible to adopt a method of collating both by setting one of the images as an inclined image by coordinate conversion, or converting the size and the number of pixels of the image.

The “standard time” field in FIG. 9 is a field in which a standard working time between each process is registered. By comparing the standard time with the actual work time, it is possible to grasp the progress (the degree of delay) of the work. Also, the "maximum allowable time" field in FIG. 9 is a field in which an allowable time required for each process is registered (that is, maximum allowable time> standard time). The "elapsed time" field of FIG. 9 is a field in which the actual required time between each process is registered. In the example of FIG. 9, the standard time and the maximum allowable time of the final process, step 6, are blank. However, the operation ends with the agreement with the representative image of this process, and there is no need for determination. It is for.

The “delay determination” field in FIG. 9 is a field in which the determination result of whether the progress of each process is advanced or delayed compared to the standard required time is stored. In the example of FIG. 9, when the elapsed time is shorter than the standard time, "early determination" is made. For example, since the elapsed time is 8 minutes with respect to the standard time of 15 minutes, the process 3 is “early determination”. In the example of FIG. 9, when the elapsed time is longer than the maximum allowable time, “delay determination” is set. For example, in steps 2 and 4, since the elapsed time is longer than the maximum allowable time, it is “delay determination”. In the example of FIG. 9, when the elapsed time is shorter than the maximum allowable time but longer than the standard time, “delay determination” is made. For example, in step 1, the elapsed time is 8 minutes and the maximum allowable time is 10 minutes with respect to the standard time of 5 minutes, so “delay determination” is made. In addition, these determination criteria are an example to the last, and can be suitably changed with the classification and property of operation | work which perform determination. For example, in addition to the determination in process units as shown in FIG. 9, the cumulative standard time or the cumulative maximum allowable time may be set, and the determination result may be obtained by comparing with the cumulative elapsed time.

Also, the relationship between the standard time and the maximum allowable time can be set appropriately according to the contents of the process. Quite simply, a standard time plus a predetermined time may be set as the maximum allowable time, or a standard time multiplied by a predetermined rate may be set as the maximum allowable time.

At the timing when the “delay determination” field in FIG. 9 is updated, the determination result or the delay amount may be notified to the operator by voice or image. As a result, the worker can grasp the delay of work and the degree thereof.

The “warning information” field in FIG. 9 is a field in which the warning content notified to the worker is stored when the condition set in the “warning activation condition” field is satisfied. For example, in the example of FIG. 9, the warning information and the warning activation condition are set in the steps 2 and 4. This makes it possible to warn (notify) the worker when an error or a delay occurs with respect to a particularly important process in operation.

Subsequently, the operation of the present embodiment will be described in detail with reference to the drawings. FIG. 10 is a flowchart showing the operation of the wearable terminal according to the first embodiment of this invention. Referring to FIG. 10, first, the worker inputs information (device name, work type, fault content, etc.) for specifying the work shown in FIG. 8 from the input unit 106 of the wearable terminal 100 (step S301). .

The control unit 102 of the wearable terminal 100 transmits the input information to the server 200, and requests specification of a work No. and transmission of a process image corresponding to the specified work No. The server 200 specifies, from the work information DB 210, a work No. corresponding to the information (apparatus name, work type, fault content, etc.) received from the wearable terminal 100. Next, the server 200 specifies data corresponding to the work No. from the process image DB 211, and transmits the data together with the work No. to the wearable terminal 100. The wearable terminal 100 stores the information received from the server 200 in the process image DB 111 (step S302).

The control unit 102 of the wearable terminal 100 sets “1” to the variable indicating the number of steps, and starts the matching operation with the input image (step S303).

The control unit 102 of the wearable terminal 100 captures an image of a region visually recognized by the worker from the image input unit 104 (step S304). The captured image is temporarily stored in the captured image field of the process image DB 111. The control unit 102 of the wearable terminal 100 compares the captured image with the representative image of the process number = 1 of the corresponding operation No. of the process image DB 111 (step S305). Here, when the captured image does not match the representative image of the process number = 1 of the corresponding work No. of the process image DB 111 ("do not match" in step S305), the control unit 102 of the wearable terminal 100 performs the work Determine that it has not been started. Therefore, the control unit 102 of the wearable terminal 100 continues to capture an image of the area visually recognized by the worker from the image input unit 104 (to step S304).

In addition, various methods can be used for collation of the images in step S305 or subsequent ones. The specific example will be described in detail later.

On the other hand, when the captured image and the representative image of the process number of the corresponding operation No. = 1 of the process image DB 111 match (“coincidence” in step S305), the control unit 102 of the wearable terminal 100 starts the operation. (Step S306). At this stage, for example, process No. 1 in FIG. The implementation status of 1 becomes “work start”, and the counting unit 109 starts counting the elapsed time.

The control unit 102 of the wearable terminal 100 adds 1 to the variable indicating the number of steps (increment), and starts the matching operation between the input image and the next step (step S307).

The control unit 102 of the wearable terminal 100 captures an image of a region visually recognized by the worker from the image input unit 104 (step S308). The captured image is temporarily stored in the captured image field of the process image DB 111. The control unit 102 of the wearable terminal 100 compares the captured image with the representative image of the corresponding number of steps of the step image DB 111 (step S309). Here, when the captured image and the representative image of the corresponding number of steps of the process image DB 111 do not match (“do not match” in step S309), the control unit 102 of the wearable terminal 100 acquires the captured image and the current number of steps. Matching with representative images of all earlier steps is performed (step S316). Here, if the captured image does not match the representative images of all the processes before the current number of processes (“do not match” in step S316), the control unit 102 of the wearable terminal 100 returns to step S309 and captures The collation of the base image with the representative image of the corresponding number of steps of the step image DB 111 is continued.

On the other hand, when any one of the representative images compared in step S316 matches the captured image, the control unit 102 of the wearable terminal 100 recognizes that the work is not performed in the correct work order (step S317). Then, the control unit 102 of the wearable terminal 100 notifies the worker of a work order error via the display unit 107 or the voice output unit 108 (step S318). At that time, the control unit 102 of the wearable terminal 100 may output work information and notification information of the corresponding process, and may instruct the worker on the correct work content and the content of the return work.

On the other hand, if the captured image and the representative image of the corresponding number of steps in the process image DB 111 match in step S309 ("match" in step S309), the control unit 102 of the wearable terminal 100 performs the work in the correct work order It recognizes that it is being performed (step S310). In this case, the control unit 102 of the wearable terminal 100 notifies the worker that the work is being performed in the correct order via the display unit 107 or the audio output unit 108 (step S311).

Next, the control unit 102 of the wearable terminal 100 confirms whether or not the variable indicating the number of steps matches the final number of steps (step S312). Here, when the variable indicating the number of steps is less than the final number of steps, the work is not finished, so the control unit 102 of the wearable terminal 100 returns to step S308 and corresponds to the captured image and the step image DB 111. The comparison of the number of steps with the representative image is continued ("not the last step" in step S312).

On the other hand, when the variable indicating the number of steps matches the final number of steps, the control unit 102 of the wearable terminal 100 recognizes that all steps of the corresponding work No. have been correctly performed (step S313). The control unit 102 of the wearable terminal 100 notifies the worker that the work has been completed in the correct order via the display unit 107 or the audio output unit 108 (step S314). This completes the progress management of the series of tasks.

Although omitted in the flowchart of FIG. 10, the control unit 102 of the wearable terminal 100 determines the elapsed time at the time when the match between the new image in step S309 and the representative image in the next step is confirmed. Compare the standard time of the previous process with the maximum allowable time. According to the result of the comparison, the control unit 102 of the wearable terminal 100 confirms the delay or the like of the previous process, and updates the content of the delay determination field of the process image DB. In addition, when a delay is detected, the control unit 102 of the wearable terminal 100 notifies the worker of a delay in operation and the like. Although omitted in the flowchart of FIG. 10, the control unit 102 of the wearable terminal 100 confirms whether or not the condition defined in the warning activation condition field of the process image DB 111 of FIG. 9 is satisfied at a predetermined cycle. . Here, when it is determined that the condition defined in the warning activation condition field is satisfied, the control unit 102 of the wearable terminal 100 gives a warning to the worker using the content set in the warning information field.

According to the wearable terminal 100 and the server 200 of the present embodiment described above, it is possible to observe in real time the work performed by the worker and to confirm whether or not the work is performed in the correct order. Further, in the present embodiment, it is possible to provide the worker with necessary information in each process, and to issue a warning about whether or not the operation is delayed. For this reason, it is possible to save labor or automate management of the progress of the work process.

An example of progress management of a process to which this embodiment is applied will be described.
(Step 1) Step of Removing the Cover As a representative image, an image of the entire cover is used. For this image, an image of a cover to which a marker is added in advance may be used. Also, instead of this marker, an image may be prepared which can recognize the shape and size of the outer shape of the cover (in consideration of the imaging magnification) and the cover. This detects the start of work.

(Step 2) Fixing of the cover Fixing of the cover shall be screw fixing by means of a plurality of removable bolts (meaning male screws). In this case, by using the cover image after screw fixation as a representative image, it is possible to detect that the cover is correctly fixed. Alternatively, instead of collation with this representative image, the positions of these bolts are stored in advance, the cover portion is detected from the acquired captured image, and the relative position portion of each bolt is detected from the position of the entire cover. The image is cut out, and it is detected and judged whether the bolt-specific color (for example, the color of zinc plating) is in the area of the predetermined range in which the size of the bolt head is included in the cut-out image. Thus, a method of determining whether there is a bolt at that position can also be employed. If there are parts to attach the cover with bolts at four corners and four places of the cover, it is detected that there are bolts in that area, and if it is judged that there are bolts at all places, it is correct. Judge that the cover is attached and fixed.

Furthermore, if there is a mounting order of the bolts (for example, if there is an order of tightening in the order of upper right, lower left (upper right diagonal), upper left, lower right (upper left diagonal) of the cover) May subdivide step 2. For example, every time the bolt is tightened, a representative image in which an image of the bolt is increased is prepared in advance at the position of the bolt attached in that order, and the captured image, the representative image according to the order of tightening the bolt, and the order Matching can be done as it is. Then, if the images match according to the order, it can be determined that the bolts are tightened according to the defined order.

(Step 3) Withdrawal of the cable The end of the cable is connected to a predetermined place of the device, a terminal, that is, a representative image obtained by imaging the inserted state and a representative image obtained by imaging the absence of the cable are prepared. Do. Then, when the image obtained from the image input unit 104 changes from the image into which the end of the cable is inserted to an image without a cable, it can be determined that the cable has been pulled out in accordance with the work procedure.

Alternatively, the image on the device side of the removed connector may be used as a representative image, and it may be determined that the cable has been correctly removed from the match with the image acquired from the image input unit 104.

(Step 4) Operation of the switch When the operation part of the switch, toggle switch, etc. changes according to the on / off of the switch, or when there is a change in the color of part of the switch when the switch is turned on / off, the switch An image in the on state and an image in the off state are prepared as representative images. If the image acquired from the image input unit 104 matches the image in the on state after the switch matches the image in the on state, it can be determined that the switch has been correctly operated. Then, when a series of work is completed by turning off the switch, it is possible to confirm that the work is completed according to this process.

Second Embodiment
Subsequently, a second embodiment of the present invention will be described. FIG. 11 is a diagram showing the configuration of the wearable terminal 100b according to the second embodiment of this invention. The difference in configuration from the wearable terminal 100 of the first embodiment shown in FIG. 6 is that the work information DB 113 is arranged in the storage unit 110a.

Specifically, since the wearable terminal 100b of the present embodiment includes the work information DB 113, it is possible to identify the work No. by the wearable terminal 100b alone. In addition, data necessary for the work performed by the worker is stored in advance in the process image DB 111 of the wearable terminal 100 b. Then, the wearable terminal 100b according to the present embodiment reads and sets data corresponding to the specified work No. from the process image DB 111. The subsequent operation is the same as that of step S303 and subsequent steps of the first embodiment shown in FIG.

As described above, according to the present embodiment, the work of the worker can be checked by the wearable terminal 100b alone, and in particular, it is suitable for the case where the work is performed in a place where the communication infrastructure is not ready It can be applied to

Finally, several alternative embodiments of the invention will be described.
[Wearable terminal]
As the wearable terminal described above, it is possible to use a head-mounted terminal called a smart glass or a headset terminal, which can display information in the field of view of the person who wears it. Further, in such a wearable terminal, a light flux formed on the retina of the worker is directly irradiated to the eye of the subject to form an image on the retina, and the image of the visual field on the subject (a virtual image It is also possible to use a device of the type recognized as).

More desirably, a sensor that senses the movement of the wearer's eye can be attached to the inside of the smart glass or the headset terminal. For example, the wearable terminal provided with these sensors detects an operation to narrow the eye of the operator, which is performed when the image displayed on the display unit 107 is difficult to see. When such an operation is performed, these wearable terminals zoom the input image obtained from the image input unit 104 and display the display unit 107 in an enlarged manner. Whether or not the operation of narrowing the eyes is being performed can be determined based on whether or not the amount of reflected light obtained from the sensor has continued within a predetermined range and for a predetermined time or more. Further, in order to improve the accuracy of the sensor, it is desirable to irradiate auxiliary light in the direction of the eye in order to input the reflected light, and the irradiated light is invisible light (preferably near infrared light) and The light intensity is preferably in a range that does not affect the eye and retina.

In the above-described terminal such as the wearable terminal, it is also preferable to incorporate a 3-axis acceleration sensor for detecting the attitude of the device and to detect the direction of gravity. This makes it possible to obtain information such as the imaging direction, the size of the object in the image range, and the tilt of the target device. Thereby, in the following matching, it becomes possible to appropriately perform pre-processing such as correction of the inclination of the image and change of the size.

[Matching process with representative image]
A modification of the process of determining whether the image acquired by the image input unit 104 and the representative image is the same or not, which is performed in steps S305, S309, and S316 of FIG. 10, will be described. The matching determination of the image fetched from the image input unit 104 and the representative image can be performed using a method called image matching processing in which feature points in the image are extracted and matching between the feature points is performed.

In order to easily match the representative image with the image fetched from the image input unit 104 in the confirmation of the process of the specific work, the image fetched from the image input unit 104 and the representative image have predetermined angles and sizes. Preferably, it is adjusted to be imaged. For example, as a representative image, it is preferable to use an image captured by the camera of the same wearable terminal as the wearable terminal 100 worn by the worker. Moreover, it is preferable that a representative image is not an image that can be detected in another process, but an image that is obtained only in that process.

In addition, if at least one of the image or the representative image fetched from the image input unit 104 is inclined or the size is different, the inclination is corrected by coordinate conversion, or the size of the image is changed before matching. Is preferred. In confirmation of the process of the specific operation, the captured image of the circuit board can be used as a representative image. In the matching of the circuit board, the outer shape may be recognized to perform match detection, or match detection may be performed with a predetermined shape or color (a color different from the background color in order to make the background float up). In addition, a plurality of markers (at least three points, three points can detect the position, size, and angle of the image) in advance for both the target object and the representative image to be subjected to the work. Therefore, matching may be performed using these markers as feature points. When markers are used, binarization may be performed based on the difference in color from the background of the comparison target image, and the positions of the markers may be extracted to perform matching. Furthermore, with these markers, the position (an axis perpendicular to each other in the imaging plane in the X and Y directions and in the plane perpendicular to the imaging direction), posture (angle θz, center in the direction perpendicular to the imaging plane) The matching may be performed by extracting the rotation direction and the size.

Further, in addition to the coincidence determination by the marker, the coincidence degree of the pattern of each part from the reference position may be calculated, and the final identity judgment may be performed using this coincidence degree. More specifically, the difference between the image and the comparison image after matching the angle and the size, for example, the luminance for each pixel, is normalized after the entire image so that the average of the luminances agrees, and then the predetermined The method of judging by the difference of the area of the part which has the above luminance differences can be used.

In the matching process, in order to reduce the amount of calculation, matching may be performed between images binarized with a predetermined threshold (threshold). In the matching, not a whole image but a part of the image may be extracted and matched. For example, it is possible to perform a matching process to determine whether a part of the images in a predetermined ratio of the images captured from the image input unit 104 matches the representative image by a predetermined degree or more. In this case, a partial area of the image captured from the image input unit 104 is moved at a predetermined ratio, and matching processing is performed each time the movement is performed, and the image input is performed depending on whether a predetermined matching degree or more is obtained. It may be detected whether there is a representative image in the image captured by the unit 104.

Further, when cutting out from the image taken in from the image input unit 104, the predetermined ratio is changed at a predetermined interval from equal magnification to a predetermined magnification, and there is a region in the acquired image that matches the representative image. You may repeat the judgment. Thereby, even if the size of the image changes, it is possible to detect the coincidence of the images.

Further, it is also possible to adopt a method of rotating the predetermined angle image each time the area of the image is moved, and performing matching each time. By checking whether or not a predetermined degree of coincidence can be obtained before the 360 ° rotation, it is possible to make a correct determination even if the target image is rotated.

[Guidance to workers when replacing parts]
When the work is part replacement work, when replacing a part storing an image of part replacement, image matching of the part to be replaced may be performed using the image of the part before replacement as a representative image. At that time, in the process of photographing an operation of replacing a part, it is preferable to define an operation procedure that can obtain an image that does not overlap with other articles or the like. Such an operation procedure can be guided to the worker by notification information or the like.

Further, in the work procedure, it is also preferable to instruct gripping of the part to be replaced such that the inclination about an axis (two axes) perpendicular to the imaging direction central axis does not become a predetermined angle or more. By doing this, new and old parts can be detected individually without overlapping in the image. Furthermore, by doing this, it is possible to track the movement of each part and to judge from the image that no erroneous attachment has been made. By doing as described above, when comparing two images, it becomes possible to omit pre-processing to match the posture and size of the image to be matched. Further, by doing as described above, since the postures of parts appearing in two images are similar and the positions are not largely different, the determination of matching becomes easy, and it becomes possible to improve the accuracy.

Furthermore, by arranging such a work procedure, imaging an image including the removed part at a relatively short time interval (imaging the image at a time interval at which the detection target image does not move significantly) Is possible. Then, by calculating the change in the position of the target part, it is possible to detect that the removed part and the replacement part are mistaken for reattaching the removed part.

It is also preferable to notify a warning when the removed part is remounted (it may be determined by tracking the image of the removed part in an image and approaching within a predetermined distance from the mounting position). By giving such a notice, it is possible to prevent an exchange mistake that the parts and the like are not exchanged.

As mentioned above, although each embodiment of the present invention was described, the present invention is not limited to the above-mentioned embodiment, and further modification, substitution, adjustment in the range which does not deviate from the basic technical idea of the present invention Can be added. For example, the network configuration shown in each drawing, the configuration of each element, and the form of expressing messages are merely examples for helping to understand the present invention, and the present invention is not limited to the configuration shown in these drawings.

Finally, the preferred form of the invention is summarized.
[First embodiment]
(Refer to the work support device from the above first viewpoint)
[Second form]
The storage unit of the work support apparatus described above holds the representative image in association with each of a plurality of types of work for which the worker is in charge;
The confirmation unit
Furthermore, from the worker, the input of information on the work to be started is received,
Based on the information input from the worker, the work to be started can be specified, and it can be configured to confirm whether the specified work is performed in a predetermined order.
[Third form]
The storage unit of the work support apparatus described above is
In addition to the representative image, an error image representing a situation when an erroneous work is performed is stored,
further,
The confirmation unit
Collating the input image with the error image to check whether an erroneous work has been performed;
The notification unit
If it is determined in the confirmation unit that an erroneous work has been performed, a configuration can be employed to notify the worker that an error has occurred.
[Fourth embodiment]
The storage unit of the work support apparatus described above is
The required time from the first step to the second step is held in association with the representative images of the first and second steps,
further,
The confirmation unit
Check whether or not a delay has occurred in the operations from the first step to the second step,
The notification unit may be configured to notify the worker of the presence or absence of the delay of the work.
[Fifth embodiment]
The storage unit of the work support apparatus described above is
Furthermore, a representative image of a third step performed after the second step is held,
The confirmation unit
If the input image matches the representative image of the third step after the first step, it is determined that an order error has occurred in the work,
The notification unit may be configured to notify the worker that an order error has occurred in the work.
Sixth Embodiment
The storage unit of the work support apparatus described above is
Furthermore, a representative image of a process performed after the second process is held,
The confirmation unit
After the first step, if the input image matches any of the representative images of the steps to be performed after the second step, it is determined that an order error has occurred in the operation;
The notification unit may be configured to notify the worker that an order error has occurred in the work.
[Seventh embodiment]
In the work support apparatus described above,
An image captured by a camera of a wearable terminal worn by the worker can be used as the image in the process in which the worker is performing work.
[Eighth embodiment]
In the work support apparatus described above,
As the representative image, an image captured by a camera of the same wearable terminal as the wearable terminal worn by the worker can be used.
[Ninth form]
The work support device mentioned above is
A work support apparatus (wearable terminal) that receives an input of information on a work to be started from the worker;
According to a configuration including a server that specifies the work to be started based on the information input from the worker and provides data corresponding to the specified work to the work support apparatus (wearable terminal) it can.
[Tenth embodiment]
(Refer to the work support method from the second viewpoint above)
[Eleventh embodiment]
(Refer to the program from the above third viewpoint)
The above tenth to eleventh embodiments can be expanded to the second to ninth embodiments as in the first embodiment.

The disclosures of the above-mentioned patent documents are incorporated herein by reference. Within the scope of the entire disclosure of the present invention (including the scope of the claims), modifications and adjustments of the embodiments or examples are possible based on the basic technical concept of the invention. In addition, various combinations or selections of various disclosed elements (including each element of each claim, each element of each embodiment or example, each element of each drawing, and the like) within the scope of the disclosure of the present invention Selective deletion is possible). That is, the present invention of course includes the entire disclosure including the scope of the claims, and various modifications and alterations that can be made by those skilled in the art according to the technical concept. In particular, with regard to the numerical ranges described herein, it should be understood that any numerical value or small range falling within the relevant range is specifically described even if it is not otherwise described.

10a Work support device 11a Storage unit 12a Input unit 13a Confirmation unit 14a Notification unit 10b Computer 11b Memory 12b Input unit 13b Processor 14b Output unit 100, 100b Wearable terminal 100a Worker
101 Network 102, 202 Control Unit 103, 203 Communication Unit 104, 204 Image Input Unit 105, 205 Voice Input Unit 106, 206 Input Unit 107, 207 Display Unit 108, 208 Voice Output Unit 109 Timekeeping Unit 110, 110a, 209 Storage Unit 111, 211 process image database (process image DB)
112 programs 113, 210 work information database (work information DB)
200 server

Claims (10)

1. A representative image for each process representing an operation including at least a first process and a second process performed after the first process, an operation performed in the first process, and an operation performed in the second process And a storage unit that associates and holds the
   An input unit for inputting an image in a process in which the worker is working;
   A confirmation unit that collates the input image with the representative image to confirm whether the work is performed in a predetermined order;
   A notification unit that notifies the worker of the result of the confirmation;
   Work support device equipped with
2. The storage unit associates and holds the representative image for each of a plurality of types of work performed by the worker.
   The confirmation unit
   Furthermore, from the worker, the input of information on the work to be started is received,
   The work support apparatus according to claim 1, wherein the work to be started is specified based on the information input from the worker, and whether or not the specified work is performed in a predetermined order.
3. The storage unit is
   In addition to the representative image, an error image representing a situation when an erroneous work is performed is stored,
   further,
   The confirmation unit
   Collating the input image with the error image to check whether an erroneous work has been performed;
   The notification unit
   The work support apparatus according to claim 1 or 2, wherein when it is determined in the confirmation unit that an erroneous work has been performed, the worker is notified that an error has occurred.
4. The storage unit is
   The required time from the first step to the second step is held in association with the representative images of the first and second steps,
   further,
   The confirmation unit
   Check whether or not a delay has occurred in the operations from the first step to the second step,
   The notification unit notifies the worker of the presence or absence of the delay of the work.
   The work support apparatus according to any one of claims 1 to 3.
5. The storage unit further holds a representative image of a third step performed after the second step,
   The confirmation unit
   If the input image matches the representative image of the third step after the first step, it is determined that an order error has occurred in the work,
   The notification unit notifies the worker that an order error has occurred in the work.
   The work support apparatus according to any one of claims 1 to 4.
6. The storage unit further holds a representative image of a process performed after the second process,
   The confirmation unit
   After the first step, if the input image matches any of the representative images of the steps to be performed after the second step, it is determined that an order error has occurred in the operation;
   The notification unit notifies the worker that an order error has occurred in the work.
   The work support apparatus according to any one of claims 1 to 4.
7. The work support apparatus according to any one of claims 1 to 6, wherein the image in the process in which the worker is working is an image captured by a camera of a wearable terminal worn by the worker.
8. The work support apparatus according to claim 7, wherein an image captured by a camera of a wearable terminal identical to the wearable terminal worn by the worker is used as the representative image.
9. An operation including at least a first step and a second step performed after the first step is associated with a representative image for each step representing the operation performed in the first and second steps. A computer with access to the means to hold
   Inputting an image in a process in which the worker is working;
   Checking the input image and the representative image to confirm whether the work is performed in a predetermined order;
   Notifying the worker of the result of the confirmation;
   Work support method including.
10. An operation including at least a first step and a second step performed after the first step is associated with a representative image for each step representing the operation performed in the first and second steps. On a computer accessible to the means for holding
    A process of inputting an image in a process in which an operator is working;
    A process of comparing the input image with the representative image to confirm whether the work is performed in a predetermined order;
    A process of notifying the worker of the result of the confirmation;
    A program that runs

Images