JP2020071864A - Display system, machine learning device, and display device - Google Patents

Abstract

To control a display timing of normal work.SOLUTION: A display system 1 includes: a state obtaining portion 22 that obtains sate data; a label obtaining portion 23 that obtains label data indicating whether work when obtaining the state data is abnormal or not; a threshold value calculating portion 24 that calculates a threshold value for determining whether work to be performed is likely to be abnormal or not on the basis of the state data and the label data; a display control portion 32 that controls a display timing of the normal work on the basis of the threshold value; and a display portion 33 that displays the normal work according to the display timing.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a display system, a machine learning device, and a display device, and more particularly to a display system, a machine learning device, and a display device that support work.

  During human work (for example, assembly work, inspection work, cleaning work, packing work, office work, etc.), due to oversight of workers, assembly mistakes, inspection mistakes, insufficient cleaning, forgetting to pack, mistakes in work etc. occur. I have something to do. Therefore, there is known a technique of displaying a work procedure to reduce work mistakes.

  Patent Document 1 discloses a head-mounted display device that estimates a current procedure by recognizing an outside scene image and displays a character string indicating the instruction content or an image indicating the content to be performed by the user. ing.

  Patent Document 2 discloses an information processing system that superimposes and displays an instruction image related to work correction on a real space when a difference between standard data in which work is registered and acquired work data is a threshold value or more. ing.

  Patent Document 3 discloses a work recognition device that identifies the work of a user using the output of a speed sensor and the output of an imaging unit.

  Patent Document 4 discloses an endoscope cleaning management system that determines work procedure information to be provided according to the skill level of a worker.

JP, 2014-155207, A Japanese Patent No. 6224873 International Publication No. 2010/143361 JP, 2017-131335, A

  By displaying the normal work such as the work procedure and the work result before the work, the work mistakes of especially inexperienced workers are reduced. However, displaying the normal work is bothersome and disturbing for a skilled worker, and reduces the work efficiency. On the other hand, even a skilled worker may make a work error depending on the complicated work, the work having a long work time, the work time or the degree of working days.

  Therefore, there is a demand for a technique for controlling the display timing of normal work.

One aspect of the present disclosure is a status acquisition unit that acquires status data including at least work content, a label acquisition unit that acquires label data indicating whether or not work at the time of acquiring status data is abnormal, and a status. Based on the data and the label data, a threshold calculation unit that calculates a threshold for determining whether the work to be performed is likely to be abnormal, and a display control unit that controls the display timing of the normal work based on the threshold, A display system including one or a plurality of display units that display normal work according to display timing.
Another aspect of the present disclosure is a status acquisition unit that acquires status data including at least work content, a label acquisition unit that acquires label data indicating whether or not the work when the status data is acquired is abnormal, A threshold value calculation unit that calculates a threshold value for determining whether or not the work to be performed is likely to be abnormal, based on the state data and the label data, and the threshold value learns the state data and the label data as teacher data. Provide a machine learning device that is updated based on a model.
Another aspect of the present disclosure is a status acquisition unit that acquires status data including at least work content, a label acquisition unit that acquires label data indicating whether or not the work when the status data is acquired is abnormal, Based on the state data and label data, a threshold value calculation unit that calculates a threshold value for determining whether the work to be performed is likely to be abnormal, and a display control unit that controls the display timing of the normal work based on the threshold value. And a display unit that displays normal work according to display timing.

  According to the present disclosure, the threshold value is updated every time the state data and the label data are acquired, and thus the prediction accuracy of whether or not the work to be performed is likely to be abnormal is improved over time. Since the display timing of the normal work is controlled by such a threshold value, it is possible to reduce the work mistake without the display of the normal work hindering the operator.

It is a block diagram of the display system in one embodiment. FIG. 11 is a block diagram of a display device according to another embodiment. It is a block diagram of the display system in another embodiment. It is a flow chart which shows an example of operation of a display system or a display.

  Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In each drawing, the same or similar reference numerals are given to the same or similar components. Further, the embodiments described below do not limit the technical scope of the invention described in the claims and the meaning of terms.

  FIG. 1 is a block diagram of a display system 1 according to this embodiment. FIG. 1 shows an environment 10 in which a work is performed, a work target 11 placed in the environment 10, and a worker 12 who performs a predetermined work on the work target 11. The work includes, for example, assembling work, inspection work, cleaning work, packing work, office work, and the like. The display system 1 has a function of displaying information that supports such work.

  The display system 1 includes a machine learning device 20 and a display device 30. The machine learning device 20 and the display device 30 are communicably connected via a wired or wireless network or the like. The machine learning device 20 is configured as a PC (personal computer), a server, or the like. On the other hand, the display device 30 is configured as a laptop, a PDA (personal digital assistant), a smartphone, a wearable device, or the like arranged in the environment 10. The display system 1 may include a plurality of display devices 30, and in this case, each of the display devices 30 is arranged in the same environment or different environments.

  The display system 1 may include a first processor 21, a second processor 31, and a display unit 33. The first processor 21 and the second processor 31 are configured as known processors such as ASIC (application specific integrated circuit), FPGA (field programmable gate array), and CPU (central processing unit). The first processor 21 has a function of calculating a threshold value for determining whether or not work is likely to be abnormal, based on various data acquired from the environment 10. The second processor 31 has a function of controlling the display timing of normal work based on the threshold value. In addition, the second processor 31 has a function of determining whether the work target 11 is a normal product, a function of displaying a defective portion of the work target 11 and instructing replacement, and a function of checking whether the work is normal or not. You may have the function to judge.

  The machine learning device 20 may include a first processor 21. The first processor 21 may include a state acquisition unit 22, a label acquisition unit 23, and a threshold value calculation unit 24. The state acquisition unit 22, the label acquisition unit 23, and the threshold value calculation unit 24 can be configured as a program executable by the CPU. The status acquisition unit 22 acquires status data from the environment 10. The state data is data representing a work state and includes at least work content. Further, the state data may include at least one of worker information, a work time length, and a work date and time. The definition and acquisition method of each information will be described below.

  “Work content” is defined as information indicating what kind of work is being performed. For example, the work content includes a work type code, work difficulty level, and the like. The work content is specified using a device such as a camera or a bar code reader. The device may be provided in the display device 30 or may be provided separately (the same applies to the “device” described later). The work content may be specified based on the ID or image of the work target 11, for example. For example, in the case of an assembly work for assembling the parts A and B into the part C, the work contents are specified by collating the three IDs of the parts A, B, and C with the reference data. In addition, the work content may be used to determine whether or not the work was normal.

  The “worker information” is defined as information indicating the skill level of the worker 12 who performs the work. For example, the worker information includes a worker ID, age, years of work, years of work, and the like. The worker information is specified based on the ID of the worker 12 input when verifying the authority to enter and leave the environment 10 and the authority to access the display device 30, for example.

  The “work time length” is defined as information indicating a time interval from the start of work to the end of work. The length of working time is specified by using a device such as a camera, a bar code reader, an internal clock, or the like. The work start time may be the time when the ID of the work target 11 is acquired, the time when the power of the display device 30 is turned on, or the time when the worker 12 inputs the work start to the display device 30. The work end time may be the time when the image of the work end mark given to the work target 11 is acquired, the time when the power of the display device 30 is turned off, or the worker 12 indicates the work end on the display device 30. It may be the time you entered.

  The “work date and time” is defined as information indicating the date, time, and day of the week when the work was performed. For example, the work date and time may be the work start date and time, the work end date and time, or an intermediate date and time between the work start date and time and the work end date and time. The work date and time is specified using a device such as a camera, a barcode reader, an internal clock, or the like. It is desirable to acquire the work date and time as status data because the accuracy of the work may be impaired if the work time exceeds a certain value or the number of work days exceeds a certain value.

  The label acquisition unit 23 acquires label data from the environment 10. The label data is data indicating whether or not the work when the state data is acquired is abnormal. Label data is data that shows the correct answer.If the label data represents normal work, it is true that the work was normal, and if the label data represents abnormal work, it is true that the work was abnormal. is there. The label data is specified using a device such as a camera or a bar code reader. The label data is specified based on, for example, the ID of the work target 11 and the image. For example, in the case of an assembly work in which the parts A and B are connected by the wiring D, the label data is obtained by collating the three IDs of the parts A, B, and the wiring D and the image of the work target 11 after assembly with the reference data. Is specified.

  The threshold calculation unit 24 calculates a threshold for determining whether or not the work is likely to be abnormal, based on the state data and the label data. The threshold is learned using machine learning, especially supervised learning. That is, the threshold is updated based on the learning model 25 using the state data and the label data as teacher data. The learning model 25 may be shared among the plurality of display units 33. Since the threshold value is increased / decreased each time the state data and the label data are acquired, the prediction accuracy of whether or not the work to be performed is likely to be abnormal is improved over time. However, in other embodiments, the threshold value may be calculated without learning using a known method such as regression analysis.

  The display device 30 may include a second processor 31 and a display unit 33. The second processor 31 may include the display control unit 32. The display control unit 32 may be configured as a program executable by the CPU. The display control unit 32 has a function of controlling the display timing of normal work based on the threshold value described above. The “normal work” may be a normal work procedure or a normal work result. The display unit 33 is configured as a known display display such as a liquid crystal display or a touch panel display. The display unit 33 displays information for supporting the work according to the instruction of the display control unit 32, and particularly displays the normal work according to the display timing of the display control unit 32. The display unit 33 may be configured as an augmented reality device, and in this case, the normal work is displayed by being superimposed on the image in the physical space.

  FIG. 2 is a block diagram of a display device 40 according to another embodiment. The display device 40 also has the function of the machine learning device 20 described above, and operates as a stand-alone device. The display device 40 includes a processor 41 and a display unit 33. The processor 41 is configured as a well-known processor such as ASIC, FPGA, and CPU. The processor 41 has a function of calculating a threshold value for determining whether the work is likely to be abnormal based on various data acquired from the environment 10, and a function of controlling the display timing of the normal work based on the threshold value. It has both. The processor 41 includes the above-described state acquisition unit 22, label acquisition unit 23, threshold value calculation unit 24, and display control unit 32. The state acquisition unit 22, the label acquisition unit 23, the threshold value calculation unit 24, and the display control unit 32 may be configured as a program executable by the CPU.

  FIG. 3 is a configuration diagram of a display system 1 according to another embodiment. For example, like the environments 10a-10c, there may be a plurality of environments instead of one. Further, the above-mentioned machine learning device may exist individually for each environment 10a-10c, or one machine learning device may exist on the network. That is, the machine learning device may be configured as a local server 20a-20c arranged in or near the environment 10a-10c, or a cloud server arranged in a remote place apart from the environment 10a-10c. It may be configured as 20d. Furthermore, the state acquisition unit, the label acquisition unit, and the threshold value calculation unit described above may be arranged on the local servers 20a-20c, or may be arranged on the cloud server 20d on the network. In addition, the above learning model may be modeled for each work and shared on the network.

  Here, an example of the learning model 25 will be described. Formula 1 is an example of a function that models the relationship between the abnormal work flag F and the state data A, B, C, and D. The abnormal work flag F indicates whether the work to be performed is likely to be abnormal. The state data A, B, C, and D respectively represent the work content, the worker information, the length of work time, and the work date and time described above. a0, b0, c0, and d0 respectively represent work content weight, worker information weight, work time length weight, and work date / time weight. The initial values of these weights are obtained by regression analysis or the like after the state data and the like have been accumulated for a certain amount or more. Equation 1 is a linear function that predicts an abnormal work flag from state data, but may be modeled as a non-linear function. Further, the constant e0 may be added to the right side of Expression 1.

  Formula 2 is an example of a function that models the relationship between the label data weights 11 and 12 and the state data weights a0, b0, c0, and d0. The label data weight l1 represents the weight of the normal work label data L1, and the label data weight l2 represents the weight of the abnormal work label data L2. a1, b1, c1, and d1 respectively represent weights of the state data at the time of L1 acquisition, and a2, b2, c2, and d2 respectively represent weights of the state data at the time of L2 acquisition.

  Formula 3 is a function that models the relationship between the abnormal work flag F and the threshold value X at the time of L1 acquisition and L2 acquisition. Y represents a threshold value when the label data was acquired last time, and the initial value of Y may be 0. As shown in Expression 3, the threshold value X is increased when L1 is acquired, and the threshold value X is decreased when L2 is acquired. As a result, the threshold value X is updated every time the state data and the label data are acquired. Expressions 1 to 3 are implemented in the above-described threshold value calculation unit 24.

  Expression 4 is a function that determines whether the work to be performed is likely to be abnormal. As shown in Expression 4, when the abnormal work flag F is equal to or more than the threshold value X, the work is likely to be abnormal, and thus the normal work is displayed before the work. On the other hand, when the abnormal work flag F is less than the threshold value X, the work is unlikely to be abnormal, and therefore the normal work is displayed only when the performed work is not normal. Expression 4 is implemented in the display control unit 32 described above.

  FIG. 4 is a flowchart showing an example of the operation of the display system 1 or the display device 40. In step S10, state data is acquired. The state data includes worker information, the length of work time, and work date and time in addition to the work content. When the state data is acquired, the above threshold value is also updated.

  In step S11, it is determined whether the work target 11 is a normal product. Whether or not the work target 11 is a normal product is specified by image recognition or the like. For example, in the case of assembling work in which the parts A and B are assembled to the part C, the three images of the parts A, B, and C are compared with the reference data, respectively, so that the parts A, B, and C are normal products. Whether or not there is specified. If the work target 11 is not a normal product in step S11 (NO in step S11), the defective portion is displayed and replacement is instructed in step S12. Then, the operation is returned to the start. When the work target 11 is a normal product in step S11 (YES in step S11), the process proceeds to step S13.

  In step S13, it is determined whether the work to be performed is likely to be abnormal. That is, it is determined whether the abnormal work flag is equal to or greater than the threshold value. If the work is unlikely to become abnormal in step S13 (NO in step S13), it is determined whether or not the work is normal in step S14, and only when the work is not normal (NO in step S14), the step In S15, normal work is displayed, a defective portion is displayed, and replacement is instructed. As a result, even a skilled worker can confirm normal work without feeling bothersome. Next, in step S16, the abnormal work flag is increased, and the process returns to the start. If the work is normal in step S14 (YES in step S14), the abnormal work flag is decreased in step S17, and it is determined in step S18 whether the work is completed.

  If the work to be performed is likely to be abnormal in step S13 (YES in step S13), the normal work is displayed before the work in step S19. As a result, work errors can be reduced even if the operator is immature, a complicated work, a work with a long working time, or a large number of working hours or working days. In step S20, it is determined whether or not the work is normal, and only when the work is not normal (NO in step S20), the normal work is re-displayed after the work in step S21, and the defective portion is displayed and replaced. Instruct. As a result, even when the normal work is displayed before the work, but the work is abnormal, the work error can be reduced. Next, in step S22, the abnormal work flag is increased, and the process returns to the start of work. If the work is normal in step S20 (YES in step S20), the abnormal work flag is decreased in step S23, and it is determined in step S18 whether the work is completed.

  In step S18, if the work is not finished (NO in step S18), the process returns to the start of work, and if the work is finished (YES in step S18), the work is finished.

  According to the above embodiment, the threshold value is updated every time the state data and the label data are acquired, so that the prediction accuracy of whether or not the work to be performed is likely to be abnormal is improved over time. Since the display timing of the normal work is controlled by such a threshold value, it is possible to reduce the work mistake without the display of the normal work hindering the operator.

  The program for executing the above-described flowchart may be provided by being recorded in a computer-readable non-transitory recording medium, for example, a CD-ROM.

  Although various embodiments have been described in the present specification, it is recognized that the present invention is not limited to the above-described embodiments and various modifications can be made within the scope of the claims below. I want to be done.

1 Display System 10 Environment 10a-10c Environment 11 Work Target 12 Worker 20 Machine Learning Device 20a-20c Local Server 20d Cloud Server 21 First Processor 22 State Acquisition Unit 23 Label Acquisition Unit 24 Threshold Calculation Unit 25 Learning Model 30 Display Device 31 Second processor 32 Display control unit 33 Display unit 40 Display device 41 Processor

Claims (11)

A status acquisition unit that acquires status data including at least the work content,
A label acquisition unit that acquires label data indicating whether or not the work when the state data is acquired is abnormal,
Based on the state data and the label data, a threshold value calculation unit that calculates a threshold value for determining whether the work to be performed is likely to be abnormal,
A display control unit that controls the display timing of normal work based on the threshold value,
One or more display units for displaying the normal work according to the display timing;
A display system comprising:   The display system according to claim 1, wherein the threshold value is updated based on a learning model using the state data and the label data as teacher data.   The display system according to claim 1, wherein the status data further includes at least one of worker information, a length of work time, and a work date and time.   The display system according to claim 2, wherein the learning model is shared among the plurality of display units.   The display system according to claim 1, wherein the display unit is configured as an augmented reality device.   The display system according to claim 1, wherein the display unit displays information supporting the assembly work.   The display system according to claim 1, wherein the state acquisition unit, the label acquisition unit, and the threshold value calculation unit are arranged on a local server or a cloud server.   The display system according to claim 2, wherein the learning model is modeled for each work and shared on a network. A status acquisition unit that acquires status data including at least the work content,
A label acquisition unit that acquires label data indicating whether or not the work when the state data is acquired is abnormal,
Based on the state data and the label data, a threshold value calculation unit that calculates a threshold value for determining whether the work to be performed is likely to be abnormal,
Equipped with
The machine learning device, wherein the threshold value is updated based on a learning model using the state data and the label data as teacher data.   The machine learning device according to claim 9, wherein the learning model is modeled for each work and shared on a network. A status acquisition unit that acquires status data including at least the work content,
A label acquisition unit that acquires label data indicating whether or not the work when the state data is acquired is abnormal,
Based on the state data and the label data, a threshold value calculation unit that calculates a threshold value for determining whether the work to be performed is likely to be abnormal,
A display control unit that controls the display timing of normal work based on the threshold value,
A display unit for displaying the normal work according to the display timing,
A display device comprising:

Images