WO2024157705A1 - Presentation system, presentation method, and program - Google Patents

Abstract

Provided is a presentation system which makes it easy for a worker to recognize a contact position of a robot. The presentation system (10) comprises a contact position acquisition unit (11), a state acquisition unit (12), a recognition determination unit (13), a presentation mode selection unit (14), and a presentation unit (15). The contact position acquisition unit (11) acquires contact position information that indicates a contact position of the robot, which performs an operation on an object-to-be-operated, with the object-to-be-operated. The state acquisition unit (12) acquires the state of a worker who performs an operation on the object-to-be-operated. The recognition determination unit (13) determines, on the basis of the acquired state of a worker and the contact position information, whether the worker can recognize the contact position. The presentation mode selection unit (14) selects, on the basis of the contact position information and the determination result of the recognition determination unit (13), a presentation mode for presenting the contact position. The presentation unit (15) presents the contact position according to the selected presentation mode.

Description

Presentation system, presentation method, and program

This disclosure relates to a presentation system, a presentation method, and a program that presents the contact position of a robot performing work on a work object.

Robots are used to perform tasks on work objects. For example, there are cases where a person works in collaboration with a robot. In addition, there are cases where a person operates a robot and teaches it how to perform a specific task in order to have the robot perform a specific task.

Patent document 1 describes a device that intuitively shows a robot's movements and work content to people in a work environment, enabling them to share work information with the robot and work in cooperation without fear of the robot's movements.

Patent document 2 describes a robot that allows a person to visually recognize a danger zone that changes as the robot's movements change.

As a result, when a person works collaboratively with a robot, the actual position where the robot plans to make contact (hereinafter referred to as the contact position) is displayed, allowing the person to recognize the contact position in advance. This makes it possible to reduce the accumulation of anxiety and stress in people during collaborative work. Also, when a person is teaching a robot, the robot's contact position is displayed, allowing the person to move the robot while referring to the displayed contact position. This simplifies the teaching work and judgment, and makes it possible to create teaching data that is not dependent on the worker. In this way, by displaying the robot's contact position, it is possible to support people.

JP 2004-122313 A JP 2009-123045 A

However, even if the contact position of a robot is presented using the technology described in Patent Documents 1 or 2, there are cases where the contact position is presented outside the person's line of sight and the person is unable to recognize it, or where the contact position is within the person's line of sight but is not recognized. In such cases, there is a problem in that it is difficult to support the person by presenting the contact position.

The present disclosure provides a presentation system, presentation method, and program that make it easier for workers to recognize the robot's contact position.

A presentation system according to one embodiment of the present disclosure includes a contact position acquisition unit that acquires contact position information indicating the contact position between a robot working on a work object and the work object, a status acquisition unit that acquires the status of a worker working on the work object, a recognition determination unit that determines whether the worker can recognize the contact position based on the acquired status of the worker and the contact position information, a presentation mode selection unit that selects a presentation mode for presenting the contact position based on the contact position information and the determination result of the recognition determination unit, and a presentation unit that presents the contact position according to the selected presentation mode.

The presentation method according to one aspect of the present disclosure includes a contact position acquisition step of acquiring contact position information indicating the contact position between a robot performing work on a work object and the work object, a status acquisition step of acquiring the status of a worker performing work on the work object, a recognition determination step of determining whether or not the worker can recognize the contact position based on the acquired status of the worker and the contact position information, a presentation mode selection step of selecting a presentation mode for presenting the contact position based on the contact position information and the determination result in the recognition determination step, and a presentation step of presenting the contact position according to the selected presentation mode.

A program according to one aspect of the present disclosure is a program for causing a computer to execute the above-mentioned presentation method.

These comprehensive or specific aspects may be realized as a system, method, integrated circuit, computer program, or computer-readable recording medium such as a CD-ROM, or may be realized as any combination of a system, method, integrated circuit, computer program, and recording medium.

The presentation system, presentation method, and program according to one embodiment of the present disclosure make it easier for workers to recognize the contact position of the robot.

1 is a block diagram showing an example of a presentation system according to an embodiment; 1A and 1B are diagrams illustrating application examples of a presentation system according to an embodiment. 10 is a flowchart illustrating an example of an operation of the presentation system according to the embodiment. FIG. 13 is a diagram for explaining a line of sight range of a worker. 11 is a diagram showing an example of a state in which the contact position is not included in the line of sight of the worker; FIG. 11 is a diagram showing an example of a state in which the contact position is not included in the line of sight of the worker; FIG. 11 is a diagram showing an example of a state in which the contact position is not included in the line of sight of the worker; FIG. 10 is a flowchart illustrating another example of the operation of the presentation system according to the embodiment. 11A and 11B are diagrams illustrating another application example of the presentation system according to the embodiment. FIG. 13 is a diagram showing an example of a camera image of a worker captured from above the worker's head. 11A and 11B are diagrams illustrating another application example of the presentation system according to the embodiment. FIG. 1 is a diagram illustrating an example of a camera worn by a worker. FIG. 13 is a diagram showing another example of a camera worn by a worker. FIG. 11 is a diagram showing an example of a camera image of a work target captured by a camera worn by a worker. FIG. 13 is a diagram showing an example of a facial expression of a worker. FIG. 11 is a diagram showing another example of a facial expression of a worker. FIG. 13 is a diagram showing an example of presentation according to a newly selected presentation mode. FIG. 13 is a diagram showing another example of presentation according to a newly selected presentation mode. FIG. 11 is a block diagram showing an example of a presentation system according to a first modified example of the embodiment. FIG. 11 is a block diagram showing an example of a presentation system according to a second modified example of the embodiment. FIG. 11 is a block diagram showing an example of a presentation system according to a third modified example of the embodiment. FIG. 13 is a diagram showing an example of presentation according to a presentation mode selected based on the skill level of a worker.

The following describes the embodiment in detail with reference to the drawings.

The embodiments described below are all comprehensive or specific examples. The numerical values, shapes, materials, components, component placement positions, connection forms, steps, and order of steps shown in the following embodiments are merely examples and are not intended to limit the present disclosure.

(Embodiment)
A presentation system according to an embodiment will be described below.

FIG. 1 is a block diagram showing an example of a presentation system 10 according to an embodiment.

The presentation system 10 is a system for presenting the contact position of a robot performing work on a work object to a worker performing work on the work object. For example, the robot is a manipulator, and the robot comes into contact with the work object to perform work such as grasping. In other words, when the robot performs a grasping task, the contact position becomes the grasping position.

The presentation system 10 includes a contact position acquisition unit 11, a status acquisition unit 12, a recognition determination unit 13, a presentation mode selection unit 14, and a presentation unit 15. The presentation system 10 includes a computer including a processor and a memory. The memory is a ROM (Read Only Memory) and a RAM (Random Access Memory), etc., and can store programs executed by the processor. The contact position acquisition unit 11, the status acquisition unit 12, the recognition determination unit 13, the presentation mode selection unit 14, and the presentation unit 15 are realized by a processor that executes programs stored in the memory, etc.

For example, the presentation system 10 may be a computer in a single housing, a system made up of multiple computers, or a device equipped with one or multiple computers. For example, the presentation system 10 may be mounted on a robot. Also, for example, the presentation system 10 may be a server. Note that the components of the presentation system 10 may be located on one server, or may be distributed across multiple servers.

The contact position acquisition unit 11 acquires contact position information indicating the contact position of the robot with the work object. For example, the contact position acquisition unit 11 acquires information indicating the contact position of the robot with the work object from the robot. In this case, the contact position acquisition unit 11 may also acquire information such as the type of work object, the speed at which the robot approaches the contact position, or the pressure when the robot contacts the contact position. The contact position acquisition unit 11 may be a part with a processor, or a device with a processor. The contact position acquisition unit 11 may also be a computer including a processor and memory, or a device including a processor and memory. The contact position acquisition unit 11 may also be equipped with a camera or a sensor, and the camera or sensor may acquire information such as the contact position of the robot and the pressure when the robot contacts the contact position. The contact position acquisition unit 11 may also be a device with a computer, for example a sensor device with a built-in computer including a processor and memory.

The status acquisition unit 12 acquires the status of a worker performing work on a work object. For example, the status acquisition unit 12 acquires the status of a worker by acquiring video images of the worker from a camera that captures the worker. Details of the operation of the status acquisition unit 12 will be described later. The status acquisition unit 12 may be a component with a processor, or a device with a processor. The status acquisition unit 12 may also be a computer including a processor and memory, or a device including a processor and memory. The status acquisition unit 12 may also be equipped with an imaging device such as a camera that captures an image of the worker, or may also be equipped with a sensor. The status acquisition unit 12 may also be a device equipped with a computer, for example an imaging device with a built-in computer including a processor and memory.

The recognition determination unit 13 determines whether or not the worker can recognize the contact position based on the acquired worker's state and contact position information. Details of the operation of the recognition determination unit 13 will be described later. The recognition determination unit 13 may be a component with a processor, or a device with a processor. The recognition determination unit 13 may also be a computer including a processor and memory, or a device including a processor and memory. The recognition determination unit 13 may also be a device with a computer.

The presentation mode selection unit 14 selects a presentation mode for presenting the contact position based on the contact position information and the determination result of the recognition determination unit 13. Details of the operation of the presentation mode selection unit 14 will be described later. The presentation mode selection unit 14 may be a component having a processor, or a device having a processor. The presentation mode selection unit 14 may also be a computer including a processor and memory, or a device including a processor and memory. The presentation mode selection unit 14 may also be a device including a computer.

The presentation unit 15 presents the contact position according to the selected presentation mode. Details of the operation of the presentation unit 15 will be described later. The presentation unit 15 may be a component with a processor, or a device with a processor. The presentation unit 15 may also be a computer including a processor and memory, or a device including a processor and memory. The presentation unit 15 may also be a device with a presentation function, such as a projector. The presentation unit 15 may also be provided in a device with a presentation function, and may be a device built into a projector or speaker, for example.

FIG. 2 is a diagram showing an application example of the presentation system 10 according to the embodiment.

For example, as shown in FIG. 2, the presentation system 10 is applied when the worker 30 works with the robot 21 to perform work on the work target 40. For example, in the example shown in FIG. 2, the presentation system 10 is a system consisting of the computer 24 and the computer 25. For example, the contact position acquisition unit 11, the presentation mode selection unit 14, and the presentation unit 15 are realized by the computer 24, and the state acquisition unit 12 and the recognition determination unit 13 are realized by the computer 25. The presentation system 10 may include at least one of the robot 21, the projector 22, and the camera 23. In the example shown in FIG. 1, the presentation system 10 includes all of the robot 21, the projector 22, and the camera 23. The computer 24 is connected to the robot 21 and the projector 22 via the cable 34a and the cable 34b, respectively. The computer 25 is connected to the camera 23 via the cable 34c. The computer 24 and the computer 25 are also connected to each other via the cable 34d. The cables 34a-d may be, for example, LAN (Local Area Network) cables or USB (Universal Serial Bus) cables. Also, instead of using cables, the connection between the computer 24 and the computer 25, the connection between the computer 24 and the robot 21 and the projector 22, and the connection between the computer 25 and the camera 23 may be made using wireless technology such as wireless LAN (Local Area Network) or Bluetooth (Bluetooth (registered trademark)).

For example, the presentation unit 15 presents the contact position 50 of the robot 21 with the work target 40 via the projector 22. For example, when the presentation mode is display, the presentation mode selection unit 14 selects a display mode for displaying the contact position 50 as the presentation mode based on the contact position information and the judgment result of the recognition judgment unit 13, and the presentation unit 15 uses the projector 22 to perform a display on the work target 40 according to the selected display mode. In FIG. 2, the line of sight range 31 of the worker 30 is shown, but the details of the line of sight range 31 will be described later. Note that in FIG. 2 and subsequent figures, the contact position 50 is shown in a circular display mode, but this will be described later.

Next, we will explain the details of the operation of the presentation system 10.

FIG. 3 is a flowchart showing an example of the operation of the presentation system 10 according to the embodiment. Note that FIG. 3 illustrates an example in which it is determined whether the worker 30 can recognize the contact position 50 before the contact position 50 is presented.

First, the contact position acquisition unit 11 acquires contact position information indicating the contact position 50 between the robot 21 performing work on the work object 40 and the work object 40 (step S11), and the status acquisition unit 12 acquires the status of the worker 30 performing work on the work object 40 (step S12). Note that the order in which steps S11 and S12 are performed is not particularly limited. For example, the status acquisition unit 12 acquires a camera image of the worker 30 captured by the camera 23 as the status of the worker 30. The camera image of the worker 30 includes line-of-sight information of the worker 30, and the recognition determination unit 13 estimates the line-of-sight range 31 of the worker 30 from the line-of-sight information. Here, the line-of-sight range 31 of the worker 30 will be described with reference to FIG. 4.

FIG. 4 is a diagram for explaining the line of sight 31 of the worker 30. Note that, as will be described in detail later, FIG. 4 is also a diagram showing an example of a state in which the contact position 50 is included in the line of sight 31 of the worker 30.

For example, as shown in FIG. 4, the line of sight range 31 is a range determined by the field of view angle 32 and the viewing distance 33 of the worker 30. The field of view angle 32 and the viewing distance 33 vary from person to person, but for example, the field of view angle 32 and the viewing distance 33 are predetermined to be the field of view angle and the viewing distance of a standard person. The recognition determination unit 13 estimates the line of sight range 31 based on the field of view angle 32 centered on the direction in which the worker 30 is facing in the camera image and the viewing distance 33 in that direction. Note that the line of sight range 31 of the worker 30 may be estimated by using an eye tracker.

Returning to the explanation in FIG. 3, the recognition determination unit 13 determines whether or not the worker 30 can recognize the contact position 50 based on the acquired state of the worker 30 and the contact position information (step S13). Here, the recognition determination unit 13 determines whether or not the worker 30 can recognize the contact position 50 if some kind of display or the like is performed in the future on the contact position 50 indicated by the contact position information. For example, the recognition determination unit 13 determines whether or not the worker 30 can recognize the contact position 50 by determining whether or not the contact position 50 is included in the line of sight range 31. This will be explained using FIG. 4 and FIG. 5A to FIG. 5C.

As described above, FIG. 4 is a diagram showing an example of a state in which the contact position 50 is included in the line of sight 31 of the worker 30, and each of FIGS. 5A to 5C is a diagram showing an example of a state in which the contact position 50 is not included in the line of sight 31 of the worker 30.

As shown in FIG. 4, if the contact position 50 is within the viewing angle 32 and within the viewing distance 33, the recognition determination unit 13 determines that the contact position 50 is within the line of sight 31 and that the worker 30 can recognize the contact position 50.

If the display mode indicating the contact position 50 is small or the contrast between the display mode and the surroundings is low, the angle indicating the field of view angle 32 will be small or the viewing distance 33 will be short. In this case, the angle indicating the field of view angle 32 may be small and the viewing distance 33 may be short. Conversely, if the worker 30 is in a state where he or she can adequately recognize the contact position 50, for example, the display mode indicating the contact position 50 can be made small.

On the other hand, if the display mode indicating the contact position 50 is large or the contrast between the display mode and the surroundings is high, the angle indicating the field of view 32 will be large or the viewing distance 33 will be long. In this case, the angle indicating the field of view 32 may be large and the viewing distance 33 may be long. Conversely, if the worker 30 is in a state where he or she cannot fully recognize the contact position 50, for example, the display mode indicating the contact position 50 may be made large or the contrast between the display mode and the surroundings may be increased, so that the worker 30 can fully recognize the contact position 50.

5A, when the contact position 50 is outside the viewing angle 32, the recognition determination unit 13 determines that the contact position 50 is not included in the line of sight range 31, and determines that the worker 30 cannot recognize the contact position 50. Also, as shown in FIG. 5B, even if the contact position 50 is within the viewing angle 32, when the contact position 50 is outside the viewing distance 33, the recognition determination unit 13 determines that the contact position 50 is not included in the line of sight range 31, and determines that the worker 30 cannot recognize the contact position 50. Also, as shown in FIG. 5C, when the contact position 50 is in the blind spot of the worker 30, the recognition determination unit 13 determines that the contact position 50 is not included in the line of sight range 31, and determines that the worker 30 cannot recognize the contact position 50. For example, whether or not the contact position 50 is in the blind spot of the worker 30 is determined based on the positional relationship between the work object 40, the worker 30, and the contact position 50 shown in the camera image.

Note that the contact position 50 is position information obtained from the robot 21, and has a different coordinate system from the line of sight 31 estimated from the direction in which the worker is facing as seen in the camera image of the camera 23. Therefore, first, the recognition determination unit 13 performs coordinate conversion so as to match the coordinate system of the contact position 50 with the coordinate system of the line of sight 31. For example, the recognition determination unit 13 may perform coordinate conversion from the coordinate system of the contact position 50 to the coordinate system of the line of sight 31, or may perform coordinate conversion from the coordinate system of the line of sight 31 to the coordinate system of the contact position 50. In addition, for example, the recognition determination unit 13 may perform coordinate conversion from the coordinate system of the contact position 50 to the coordinate system of the line of sight 31 to the same coordinate system such as a world coordinate system. By performing the coordinate conversion, it becomes possible to compare the contact position 50 with the line of sight 31, and the recognition determination unit 13 can determine whether or not the contact position 50 is included in the line of sight 31.

Returning to the explanation in FIG. 3, the presentation mode selection unit 14 selects a presentation mode for presenting the contact position 50 based on the contact position information and the determination result of the recognition determination unit 13 (step S14). For example, the presentation mode selection unit 14 selects what type of presentation to perform for the contact position 50 indicated by the contact position information based on the determination result of the recognition determination unit 13. For example, when it is determined that the worker 30 cannot recognize the contact position 50, the presentation mode selection unit 14 selects a presentation mode that makes it easier for the worker 30 to recognize the contact position 50. For example, when the presentation is a display, when it is determined that the worker 30 cannot recognize the contact position 50, the presentation mode selection unit 14 selects a display mode that differs in at least one of the display hue, the display color intensity (saturation), the display blinking frequency, and the display size, compared to the display mode selected when it is determined that the worker 30 can recognize the contact position 50. For example, the presentation mode selection unit 14 selects a display mode in which the color of the display is easily recognized by the worker 30 (e.g., a color different from that of the work object 40), selects a display mode in which the color of the display is highly intense, selects a display mode in which the display blinks frequently, or selects a display mode in which the size of the display is large.

The display form showing the contact position 50 is a circle projected near the worker 30 by the projector 22. The size and color of this circle are set so that the worker 30 can recognize it.

The display form may be a mark other than a circle, such as an oval, triangle, rectangle, polygon with 5 or more sides, a star, a cross or a x (x mark), a mesh, dots or contour lines, letters or arrows, or a heat map. The display form may be in a variety of colors, such as red, yellow, blue, green, white, purple, etc. The display form indicating the contact position 50 may be a mark indicating a danger area.

Furthermore, additional information such as the control error of the robot 21, the time until contact, the contact pressure, the contact speed, the gripping stability, or the contact order when the robot 21 contacts the contact position 50 may also be presented. In other words, the presentation mode selection unit 14 may select a presentation mode for presenting the additional information.

For example, among the additional information, information that can be expressed numerically may be displayed as letters (numbers).

For example, the displayed color may change depending on the pressure of the contact. For example, a contact position 50 that is contacted with a strong force may be displayed in red, and a contact position 50 that is contacted with a weak force may be displayed in blue.

For example, the size of the display may change depending on the time until contact. For example, the size of the display may gradually decrease as the time until contact approaches.

For example, if there is a control error in the robot 21, there is a possibility that the robot 21 may also come into contact with the area around the contact position 50. Therefore, a contour line may be displayed that is highest at the contact position 50 and becomes lower the further away from the contact position 50, or dots or meshes may be displayed that become denser the closer to the contact position 50.

The presentation mode selection unit 14 may also instruct the presentation unit 15 to output a sound such as a warning sound or warning voice while selecting the same display mode as the display mode that would be selected if it was determined that the worker 30 can recognize the contact position 50. This makes it easier for the worker 30 who hears the sound such as the warning sound or warning voice to recognize the contact position 50.

The presentation mode selection unit 14 may also select a display mode that displays a color according to the color of the work object 40. For example, if the work object 40 is a red object, the red color has low visibility even when displayed on the work object 40, and the worker 30 may have difficulty noticing the display. Therefore, for example, a display based on green, which is the contrasting color of the work object 40, may be performed.

Returning to the explanation in FIG. 3, the presentation unit 15 presents the contact position 50 according to the selected presentation mode (step S15). For example, the presentation unit 15 causes the projector 22 to display light on the contact position 50 or to output sound from a speaker (not shown) or the like.

If the recognition determination unit 13 determines that the worker 30 cannot recognize the contact position 50, the presentation mode selection unit 14 selects a presentation mode that makes it easier for the worker 30 to recognize the contact position 50. As a result, the presentation unit 15 presents the contact position 50 according to the selected presentation mode, making it easier for the worker 30 to recognize the contact position 50 of the robot 21. For example, a warning using sound or voice may be considered as a presentation mode.

In addition, if a contact position 50 is determined to be unrecognizable to the worker 30, it may be determined that there is no need to present the contact position 50, and the contact position 50 may not be presented.

Next, an example in which after the contact position 50 is presented, it is determined whether or not the worker 30 can recognize the presented contact position 50 (whether or not the worker recognizes the contact position 50) will be described with reference to FIG. 6.

FIG. 6 is a flowchart showing another example of the operation of the presentation system 10 according to the embodiment.

First, the contact position acquisition unit 11 acquires contact position information indicating the contact position 50 of the work object 40 of the robot 21 performing work on the work object 40 (step S21).

Next, the presentation mode selection unit 14 selects a presentation mode for presenting the contact position 50 based on the contact position information (step S22). In the example of FIG. 6, since it is not determined at step S22 whether the worker 30 can recognize the presented contact position 50, for example, the presentation mode selection unit 14 selects a predetermined presentation mode for the contact position 50 indicated by the contact position information. For example, when the presentation mode is display, the presentation mode selection unit 14 selects a display mode of a predetermined display color, a display color intensity, a display blinking frequency, or a display size. Also, for example, when the presentation is sound output, the presentation mode selection unit 14 selects a presentation mode of a predetermined sound output. The output of the predetermined sound may be the output of a sound indicating the contact position 50, or may be a voice output such as "Grab the sole of the shoe."

Then, the presentation unit 15 presents the contact position 50 according to the selected presentation mode (step S23). For example, the presentation unit 15 causes the projector 22 to display light on the contact position 50 or causes a speaker (not shown) to output sound.

Next, the state acquisition unit 12 acquires the state of the worker 30 performing work on the work object 40 (step S24). Specifically, the state of the worker 30 in a state in which the contact position 50 has been presented is acquired. Since the contact position 50 has been presented, if the worker 30 notices the presentation of the contact position 50, the state of the worker 30 becomes a state corresponding to the presentation of the contact position 50. For example, the state acquisition unit 12 acquires a camera image of the worker 30 obtained by the camera 23 as the state of the worker 30. The camera image of the worker 30 includes line-of-sight information of the worker 30, and the recognition determination unit 13 estimates the line-of-sight range 31 of the worker 30 from the line-of-sight information.

Next, the recognition determination unit 13 determines whether or not the worker 30 can recognize the contact position 50 based on the acquired state of the worker 30 and the contact position information (step S25). Here, as a result of the presentation of the contact position 50, the recognition determination unit 13 determines whether or not the worker 30 recognizes the contact position 50 based on the acquired state of the worker 30 and the contact position information. For example, the recognition determination unit 13 determines whether or not the worker 30 recognizes the contact position 50 by determining whether or not the contact position 50 is included in the line of sight range 31.

3, the presentation mode selection unit 14 selects a presentation mode for presenting the contact position 50 based on the contact position information and the determination result of the recognition determination unit 13 (step S26). As in step S15 of FIG. 3, the presentation unit 15 presents the contact position 50 according to the selected presentation mode (step S27). Note that when the presentation is a sound output, if the presentation mode selection unit 14 determines that the worker 30 cannot recognize the contact position 50, it selects a presentation mode with a different volume or announcement content compared to the presentation mode selected when it is determined that the worker 30 can recognize the contact position 50. For example, the presentation mode selection unit 14 selects a presentation mode with a higher volume or a presentation mode that announces the detailed location of the contact position 50.

Next, we will explain other application examples of the presentation system 10.

FIG. 7 shows another application example of the presentation system 10 according to the embodiment.

As shown in FIG. 7, the camera 23 may be positioned above the worker 30.

Figure 8 shows an example of a camera image of worker 30 captured from above the worker's head.

When the camera 23 is positioned above the head of the worker 30, the state acquisition unit 12 can easily acquire the head angle information of the worker 30 as the line of sight information of the worker 30 from the camera image as shown in FIG. 8. Therefore, the recognition determination unit 13 may estimate the line of sight range 31 of the worker 30 based on the head angle information of the worker 30 as the line of sight information of the worker 30. The recognition determination unit 13 can easily estimate the direction in which the worker 30 is facing from the head angle information of the worker 30, and can easily estimate the line of sight range 31 from the field of view angle 32 centered on the direction in which the worker 30 is facing and the viewing distance 33 to that direction.

FIG. 9 is a diagram showing another application example of the presentation system 10 according to the embodiment.

For example, the projector and camera may be installed in the same location, and as shown in FIG. 9, one device 26 having the functions of a projector and a camera may be used for the presentation system 10. In this case, the contact position acquisition unit 11, state acquisition unit 12, recognition determination unit 13, presentation mode selection unit 14, and presentation unit 15 may be realized by a computer 27 connected to the device 26.

The camera used in the presentation system 10 may also be a camera worn by the worker 30 (a wearable camera).

FIGS. 10A and 10B show an example of a camera worn by a worker 30.

As shown in FIG. 10A, the camera worn by the worker 30 may be an eye gaze camera 28 worn near the eyes of the worker 30. Also, as shown in FIG. 10B, the camera worn by the worker 30 may be AR (Augmented Reality) glasses 29 worn by the worker 30.

FIG. 11 shows an example of a camera image of a work object 40 captured by a camera (eye camera 28 or AR glasses 29) worn by a worker 30.

As shown in FIG. 11, a camera image captured by a camera worn near the eyes of worker 30 is an image close to the field of view of worker 30. Therefore, if contact position 50 is included in the camera image, it can be determined that worker 30 can recognize contact position 50, and if contact position 50 is not included in the camera image, it can be determined that worker 30 cannot recognize contact position. In other words, when a camera worn by worker 30 is used, estimation of face direction and line of sight 31 and coordinate transformation are not required, making it possible to simplify the design of the algorithm.

In this way, the state acquisition unit 12 acquires a camera image captured by a camera worn by the worker 30 as the state of the worker 30 (specifically, acquires where the worker 30 is looking as the state of the worker 30), and the recognition determination unit 13 may determine whether the contact position 50 is included in the camera image, thereby determining whether the worker 30 can recognize the contact position 50.

Note that if the content displayed in the camera image by the camera worn by the worker 30 is small (for example, if the worker 30 is far away from the work object 40), the worker 30 may not be able to recognize the contact position 50. Therefore, whether or not the worker 30 can recognize the contact position 50 may be determined according to the size of the content displayed in the camera image.

Note that the presentation system 10 may use an electromyography sensor attached near the eyes of the worker 30. In this case, the state acquisition unit 12 may acquire information on the electrooculography of the worker 30 as the gaze information of the worker 30, and the recognition determination unit 13 may estimate the gaze range 31 based on the information on the electrooculography of the worker 30. For example, if the range of movement of the eyeballs of the worker 30 is narrow, it is considered that the worker 30 is gazing at a certain range, and it can be determined that the recognition range of the worker 30 is narrow and that the worker 30 recognizes details within that recognition range. Therefore, in this case, the recognition determination unit 13 can estimate a gaze range 31 with a large visual distance 33 and a narrow field of view angle 32. Also, for example, if the range of movement of the eyeballs of the worker 30 is wide, it is considered that the worker 30 is looking at a wide range, and it can be determined that the recognition range of the worker 30 is wide. Therefore, in this case, the recognition determination unit 13 can estimate a gaze range 31 with a wide field of view angle 32.

The state acquisition unit 12 may also acquire a video showing the face of the worker 30 (specifically, acquire the facial expression of the worker 30 from the video showing the face of the worker 30 as the state of the worker 30), and the recognition determination unit 13 may further determine whether the worker 30 can recognize the contact position 50 (specifically, whether the worker 30 recognizes it) based on the facial expression of the worker 30 shown in the video.

12A and 12B are diagrams showing examples of facial expressions of a worker 30. In FIG. 12A, a worker 30 with a frowning expression is shown, and in FIG. 12B, a worker 30 with a squinting expression is shown.

For example, if the facial expression of the worker 30 shown in the acquired video is frowning or squinting as in FIG. 12A or 12B, the recognition determination unit 13 can determine that the worker 30 is unable to recognize the presented contact position 50. Therefore, in this case, the presentation mode selection unit 14 may select a presentation mode that makes it easier for the worker 30 to recognize the contact position 50. Furthermore, if the facial expression of the worker 30 shown in the acquired video is one in which the worker can recognize the contact position 50 by opening the eyes wide, moving the eyebrows, or opening the mouth, the recognition determination unit 13 can determine that the worker 30 can recognize the presented contact position 50. Therefore, in this case, the presentation mode selection unit 14 does not need to select a presentation mode that is different from the current situation.

The recognition determination unit 13 may estimate the gaze range 31 of the worker 30 based on the facial movement of the worker 30 shown in the moving image as the gaze information of the worker 30. For example, if the facial movement of the worker 30 is slow, the worker 30 is considered to be gazing at a certain range, and it can be determined that the recognition range of the worker 30 is narrow and that the worker recognizes details within that recognition range. Therefore, in this case, the recognition determination unit 13 can estimate a gaze range 31 with a large visual distance 33 and a narrow field of view angle 32. Also, for example, if the facial movement of the worker 30 is fast, the worker 30 is considered to be looking at a wide range, and it can be determined that the recognition range of the worker 30 is wide. Therefore, in this case, the recognition determination unit 13 can estimate a gaze range 31 with a wide field of view angle 32.

The recognition determination unit 13 may also determine whether the estimation result of the gaze range 31 of the worker 30 was correct based on the facial expression or movement of the worker 30 shown in the moving image as the gaze information of the worker 30. For example, if the gaze range 31 of the worker 30 was incorrect, the recognition determination unit 13 can re-estimate the gaze range 31. For example, if the facial expression of the worker 30 is a frowning expression or a narrowing expression as in FIG. 12A or FIG. 12B, it can be estimated that the contact position 50 in the gaze range 31 is difficult to see, and it can be determined that the visual distance 33 in the estimated gaze range 31 is too long. Therefore, in this case, it is possible to re-estimate the gaze range 31 with a short visual distance 33. Also, for example, if it is determined that the worker 30 can recognize the contact position 50, but the worker 30 performs an action that does not allow the contact position 50 to be recognized, it can be determined that the estimated gaze range 31 is too wide. An example of an action in which the worker 30 is unable to recognize the contact position 50 is an action in which the worker 30 ignores the contact position 50 and moves the robot 21 closer to a location unrelated to the contact position 50 during a teaching operation, or an action in which the worker attempts to approach the contact position 50 during a collaborative operation. Therefore, in this case, the recognition determination unit 13 can re-estimate the line-of-sight range 31 with a short viewing distance 33 or a narrow viewing angle 32.

Furthermore, when the recognition determination unit 13 determines that the worker 30 cannot recognize the contact position 50, the presentation mode selection unit 14 may select a new presentation mode in addition to the already selected presentation mode, and the presentation unit 15 may present the contact position 50 according to the newly selected presentation mode in addition to the presentation according to the already selected presentation mode. This will be described with reference to Figures 13A and 13B.

FIGS. 13A and 13B are diagrams showing an example of presentation according to a newly selected presentation mode. FIGS. 13A and 13B show a table on which a work object 40 is placed, and an image is projected onto the table by a projector 22. For example, an image is projected onto a contact position 50 on the work object 40, and an additional image is also projected around the work object 40 on the table. The image projected onto the contact position 50 on the work object 40 is an example of presentation according to an already selected presentation mode, and the image projected around the work object 40 on the table is an example of presentation according to a newly selected presentation mode.

For example, if the worker 30 cannot recognize the contact position 50, an additional image 51 may be displayed that indirectly displays the contact position 50 that the worker 30 cannot recognize, as shown in FIG. 13A. The additional image 51 is an image that indirectly displays the location of the contact position 50 that the worker 30 cannot recognize by using an arrow or text. By showing the additional image 51 to the worker 30, the worker 30 can be made to recognize that the contact position 50 that he or she cannot recognize is on the opposite side to the side that the worker 30 can see, and the worker 30 can be prompted to move over and check.

Also, for example, if the worker 30 cannot recognize the contact position 50, an additional image 52 may be displayed that directly displays the contact position 50 that the worker 30 cannot recognize, as shown in FIG. 13B. The additional image 52 is an image of the work object 40 captured from the opposite side of the worker 30, and by looking at the additional image 52, the worker 30 can recognize that the unrecognized contact position 50 is on the opposite side to the side the worker 30 is looking at.

The presentation according to the newly selected presentation mode may be audio. For example, a voice may be output to inform the operator that the contact position 50 that the operator 30 is unable to recognize is on the opposite side to the side the operator 30 is looking at.

As described above, if it is determined that the worker 30 cannot recognize the contact position 50, a presentation mode that makes it easier for the worker 30 to recognize the contact position 50 is selected, and the contact position 50 is presented according to the selected presentation mode. This makes it easier for the worker 30 to recognize the contact position 50 of the robot 21.

(Variation 1)
Next, a presentation system according to a first modification of the embodiment will be described.

FIG. 14 is a block diagram showing an example of a presentation system 10a according to the first variation of the embodiment.

The presentation system 10a differs from the presentation system 10 according to the embodiment in that it includes a concentration level estimation unit 16. In other respects, the presentation system 10a is basically the same as the presentation system 10 according to the embodiment, so the following description will focus on the differences.

The concentration level estimation unit 16 is realized by a processor that executes a program stored in a memory provided in the presentation system 10a, and more specifically, by the computer 25 shown in FIG. 2 and FIG. 7 or the computer 27 shown in FIG. 9. The concentration level estimation unit 16 may be a component having a processor, or a semiconductor chip having a processor. The concentration level estimation unit 16 may also be a computer having a processor, a memory, or a device having a processor, a memory, or a device. The concentration level estimation unit 16 may also be a device having a computer.

The state acquisition unit 12 acquires a video showing the face of the worker 30 (specifically, acquires the state of the worker 30 from the video showing the face of the worker 30), and the concentration estimation unit 16 estimates the concentration level of the worker 30 on the work target 40 based on the acquired video. For example, the concentration estimation unit 16 extracts at least one of facial expression information, pulse wave information, and eye movement information from the video, and estimates the concentration level of the worker 30 using at least one of these pieces of information.

For example, the recognition determination unit 13 determines whether or not the worker 30 can recognize the contact position 50 based on the acquired state of the worker 30, the contact position information, and the concentration level of the worker 30. For example, the recognition determination unit 13 estimates the line of sight 31 of the worker 30 based on the state of the worker 30 and the concentration level of the worker 30, and determines whether or not the contact position 50 indicated by the contact position information is included in the line of sight 31, thereby determining whether or not the worker 30 can recognize the contact position 50.

For example, if the concentration level is estimated to be high, the worker 30 is concentrating on the task at hand without thinking about other things, and is considered to be able to gaze at a wide range. Therefore, in this case, the recognition determination unit 13 can estimate a gaze range 31 with a large viewing distance 33 and a wide field of view angle 32. On the other hand, if the concentration level is estimated to be low, it is considered that the worker 30 has difficulty gazing at a wide range. Therefore, in this case, the recognition determination unit 13 can estimate a gaze range 31 with a small viewing distance 33 and a narrow field of view angle 32. In this way, when the concentration level of the worker 30 is high, it is easy for the worker 30 to recognize the contact position 50, and when the concentration level of the worker 30 is low, it is difficult for the worker 30 to recognize the contact position 50, so the estimated concentration level can be used to determine whether the worker 30 can recognize the contact position 50.

Furthermore, for example, the presentation mode selection unit 14 may select a presentation mode based on the concentration level of the worker 30. If the concentration level is estimated to be low, it is considered that the worker 30 will have difficulty focusing on a wide range. For this reason, in this case, the presentation mode selection unit 14 may select a presentation mode that makes it easier for the worker 30 to recognize the contact position 50 (for example, a display mode in which the color or size is easy to see), and the presentation unit 15 may present the contact position 50 according to the selected presentation mode.

When the concentration level of the worker 30 is high, it can be estimated that the fatigue level of the worker 30 is low, and when the concentration level of the worker 30 is low, it can be estimated that the fatigue level of the worker 30 is high. In other words, there is a correlation between the concentration level and fatigue level of the worker 30. Therefore, the concentration level estimation unit 16 may estimate the fatigue level of the worker 30, the recognition determination unit 13 may determine whether or not the worker 30 can recognize the contact position 50 based on the acquired state of the worker 30, the contact position information, and the worker's fatigue level, and the presentation mode selection unit 14 may select the presentation mode based on the worker's fatigue level.

(Variation 2)
Next, a presentation system according to a second modification of the embodiment will be described.

FIG. 15 is a block diagram showing an example of a presentation system 10b according to the second variation of the embodiment.

The presentation system 10b differs from the presentation system 10 according to the embodiment in that it includes an operation determination unit 17. In other respects, the presentation system 10b is basically the same as the presentation system 10 according to the embodiment, so the following description will focus on the differences.

The operation determination unit 17 is realized by a processor that executes a program stored in a memory provided in the presentation system 10b, and specifically, by the computer 25 shown in FIG. 2 and FIG. 7 or the computer 27 shown in FIG. 9. The operation determination unit 17 may be a component having a processor, or a semiconductor chip having a processor. The operation determination unit 17 may also be a computer having a processor, a memory, or a device having a processor, a memory, or a device having a computer. The operation determination unit 17 may also be a device having a computer.

The state acquisition unit 12 acquires a video in which the worker 30 appears (specifically, acquires the state of the worker 30 from the video in which the worker 30 appears), the movement determination unit 17 determines the movement of the worker 30 with respect to the work object 40 based on the movement, and the recognition determination unit 13 determines whether or not the worker 30 has recognized the contact position 50 due to the presentation of the contact position 50 by the presentation unit 15 based on the movement determination result of the movement determination unit 17. For example, the movement determination unit 17 may determine whether or not the worker 30 has touched the contact position 50 as the movement of the worker 30 with respect to the work object 40, and the recognition determination unit 13 may determine whether or not the worker 30 has recognized the contact position 50 due to the presentation of the contact position 50 by the presentation unit 15 based on whether or not the worker 30 has touched the contact position 50.

For example, when a presentation is made according to the presentation mode selected based on the determination result that the worker 30 can recognize the contact position 50, if the motion determination unit 17 determines that the worker 30 has performed a motion that does not recognize the contact position 50, the recognition determination unit 13 determines that the worker 30 does not recognize the contact position 50 due to the presentation of the contact position 50 made by the presentation unit 15. Therefore, in this case, the recognition determination unit 13 can re-estimate the line of sight range 31 with a short visual distance 33 or a narrow visual angle 32. Note that, as described above, a motion in which the worker 30 is unable to recognize the contact position 50 is, for example, a motion in which the worker 30 ignores the contact position 50 and moves the robot 21 closer to a place unrelated to the contact position 50 during a teaching operation, or a motion in which the worker tries to approach the contact position 50 during a collaborative operation. A motion in which the worker 30 is unable to recognize the contact position 50 may also be a motion in which the worker 30 moves around as if searching for the contact position 50, or a motion in which the worker is confused because he does not know the contact position 50.

Furthermore, for example, the presentation mode selection unit 14 may select a presentation mode based on the judgment result of the action by the action judgment unit 17. For example, when the action judgment unit 17 judges that the worker 30 has performed an action that does not allow the worker 30 to recognize the contact position 50, the presentation mode selection unit 14 may select a presentation mode that makes it easier for the worker 30 to recognize the contact position 50, and the presentation unit 15 may present the contact position 50 according to the selected presentation mode. Note that, in a teaching operation, it is possible that the worker 30 considers the presented contact position 50 to be unnecessary information and deliberately ignores the contact position 50, so the presentation mode selection unit 14 may select a presentation mode that reduces the number of displays or deletes the display.

(Variation 3)
Next, a presentation system according to a third modification of the embodiment will be described.

FIG. 16 is a block diagram showing an example of a presentation system 10c according to the third variation of the embodiment.

The presentation system 10c differs from the presentation system 10 according to the embodiment in that it includes a proficiency acquisition unit 18. In other respects, it is basically the same as the presentation system 10 according to the embodiment, so the following description will focus on the differences.

The proficiency acquisition unit 18 is realized by a processor that executes a program stored in a memory provided in the presentation system 10c, and specifically, by the computer 25 shown in Figures 2 and 7 or the computer 27 shown in Figure 9. The proficiency acquisition unit 18 may be a component equipped with a processor, or a semiconductor chip equipped with a processor. The proficiency acquisition unit 18 may also be a computer equipped with a processor, a memory, or a device equipped with a processor, a memory, or a device equipped with a computer. The proficiency acquisition unit 18 may also be a device equipped with a computer.

The proficiency level acquisition unit 18 acquires the proficiency level of the worker 30 in performing the work on the work object 40, and the presentation mode selection unit 14 further selects the presentation mode based on the proficiency level of the worker 30.

For example, the proficiency level (e.g., the degree of familiarity with the task) of the specific task that the worker 30 is currently performing on the work object 40 is input via an input interface (keyboard, mouse, touch panel, etc.), and the proficiency level acquisition unit 18 acquires the proficiency level of the worker 30. Note that the proficiency level may be the time spent engaged in the specific task, or may be a proficiency level divided into multiple stages.

If the worker 30 is highly skilled, it is considered that in collaborative work, the worker 30 is good at predicting the movements of the robot 21 and has time to work. For this reason, the presentation mode selection unit 14 may select a presentation mode for presenting numerical information such as the speed or pressure at the time of contact, in addition to the presentation mode for presenting the contact position 50.

FIG. 17 shows an example of a presentation according to a presentation mode selected based on the proficiency level of the worker 30.

As shown in FIG. 17, when the worker 30 has a high level of proficiency, numerical information 53 as shown in FIG. 17 may be presented.

If the work sequence for multiple contact positions 50 has been determined, and the worker 30 has a high level of proficiency, a display showing the work sequence may be provided. For example, a display showing the work sequence may be provided using colors or marks, etc.

Furthermore, when the worker 30 is highly skilled, it is considered that the worker 30 has a wealth of knowledge about the work to be taught and about contact (grasping) during the teaching work. For this reason, the presentation mode selection unit 14 may select a presentation mode for presenting many contact positions 50 based on the knowledge of the worker 30 so that the worker 30 can select the optimal position from among many contact positions 50.

On the other hand, if the worker 30 has a low level of proficiency, it is considered that in collaborative work, the worker 30 has a low ability to predict the movements of the robot 21 and is likely to feel anxious. For this reason, the presentation mode selection unit 14 selects a presentation mode that makes it easier for the worker 30 to recognize the contact position 50. For example, a presentation mode that combines display and sound output may be selected. Also, since the worker 30 is not accustomed to the work, there is a risk of confusion if a lot of information is presented. For this reason, the presentation of numerical information may be refrained from, or, if there are multiple contact positions 50, at least one with a high priority or an early task order may be presented.

(Other embodiments)
As described above, the embodiment has been described as an example of the technology according to the present disclosure. However, the technology according to the present disclosure is not limited to this, and can be applied to an embodiment in which appropriate changes, substitutions, additions, omissions, etc. are made. For example, the following modified examples are also included in one embodiment of the present disclosure.

For example, the present disclosure can be realized not only as a presentation system, but also as a presentation method including steps (processing) performed by components that make up the presentation system.

As shown in FIG. 3, the presentation method includes steps S11, S12, S13, S14, and S15. Step S11 is a contact position acquisition step for acquiring contact position information indicating the contact position of a work object of a robot working on the work object. Step S12 is a status acquisition step for acquiring the status of a worker working on the work object. Step S13 is a recognition determination step for determining whether or not the worker can recognize the contact position based on the acquired status of the worker and the contact position information. Step S14 is a presentation mode selection step for selecting a presentation mode for presenting the contact position based on the contact position information and the determination result in the recognition determination step. Step S15 is a presentation step for presenting the contact position according to the selected presentation mode.

For example, the steps in the presentation method may be executed by a computer (computer system). The present disclosure can be realized as a program for causing a computer to execute the steps included in the presentation method.

Furthermore, the present disclosure can be realized as a non-transitory computer-readable recording medium, such as a CD-ROM, on which the program is recorded.

For example, when the present disclosure is realized as a program (software), each step is performed by running the program using hardware resources such as a computer's CPU (Central Processing Unit), memory, and input/output circuits. In other words, each step is performed by the CPU obtaining data from memory or input/output circuits, etc., performing calculations, and outputting the results of the calculations to memory or input/output circuits, etc.

Furthermore, each component included in the presentation system of the above embodiment may be realized as a dedicated or general-purpose circuit.

In addition, each component included in the presentation system of the above embodiment may be realized as an LSI (Large Scale Integration), which is an integrated circuit (IC).

In addition, the integrated circuit is not limited to an LSI, and may be realized by a dedicated circuit or a general-purpose processor. A programmable FPGA (Field Programmable Gate Array) or a reconfigurable processor that allows the connections and settings of circuit cells inside the LSI to be reconfigured may also be used.

Furthermore, if an integrated circuit technology that can replace LSIs emerges due to advances in semiconductor technology or other derived technologies, that technology may naturally be used to integrate the various components included in the presentation system.

In addition, this disclosure also includes forms obtained by applying various modifications to the embodiments that a person skilled in the art may conceive, and forms realized by arbitrarily combining the components and functions of each embodiment within the scope that does not deviate from the spirit of this disclosure.

(Additional Note)
The above description of the embodiments discloses the following techniques.

(Technology 1) A presentation system comprising: a contact position acquisition unit that acquires contact position information indicating the contact position between a robot working on a work object and the work object; a status acquisition unit that acquires the status of a worker working on the work object; a recognition determination unit that determines whether the worker can recognize the contact position based on the acquired status of the worker and the contact position information; a presentation mode selection unit that selects a presentation mode for presenting the contact position based on the contact position information and the determination result of the recognition determination unit; and a presentation unit that presents the contact position according to the selected presentation mode.

If it is determined that the worker cannot recognize the contact position, a presentation mode that will make it easier for the worker to recognize the contact position is selected, and the contact position is presented according to the selected presentation mode. This makes it easier for the worker to recognize the robot's contact position.

(Technology 2) The presentation system described in Technology 1, in which the state acquisition unit acquires line-of-sight information of the worker as the state of the worker, and the recognition determination unit estimates the line-of-sight range of the worker based on the line-of-sight information and determines whether the contact position is included in the line-of-sight range, thereby determining whether the worker can recognize the contact position.

If the contact position is within the line of sight, it can be determined that the worker can recognize the contact position, and if the contact position is not within the line of sight, it can be determined that the worker cannot recognize the contact position.

(Technology 3) The presentation system described in Technology 2, in which the recognition determination unit estimates the line of sight range based on information about the angle of the worker's head as the line of sight information.

The direction of the worker's gaze can be estimated from information about the angle of the worker's head, and it can be inferred that the direction of the worker's gaze has a gaze range.

(Technology 4) The presentation system described in Technology 2 or 3, in which the recognition determination unit estimates the gaze range based on information related to the worker's electrooculography as the gaze information.

The worker's visual field angle and viewing distance can be estimated from the worker's electrooculography, making it possible to estimate the worker's line of sight.

(Technology 5) A presentation system according to any one of technologies 2 to 4, in which the recognition determination unit estimates the gaze range based on information about the movement of the worker's face as the gaze information.

The worker's visual field angle and viewing distance can be estimated from the worker's facial movements, making it possible to estimate the worker's line of sight.

(Technology 6) A presentation system according to any one of techniques 2 to 5, in which the recognition determination unit determines whether the estimation result of the gaze range is correct based on information about the facial expression or movement of the worker as the gaze information.

If the estimated result of the worker's gaze range is incorrect, the gaze range can be re-estimated.

(Technology 7) The presentation system described in Technology 1, in which the state acquisition unit acquires a camera image taken by a camera worn by the worker as the state of the worker, and the recognition determination unit determines whether the contact position is included in the camera image, thereby determining whether the worker can recognize the contact position.

The camera image captured by the camera attached near the worker's eye is an image that is close to the worker's field of vision. Therefore, if the contact position is included in the camera image, it can be determined that the worker can recognize the contact position, and if the contact position is not included in the camera image, it can be determined that the worker cannot recognize the contact position.

(Technology 8) A presentation system according to any one of techniques 1 to 7, in which the state acquisition unit acquires a video showing the face of the worker, and the recognition determination unit further determines whether the worker can recognize the contact position based on the facial expression of the worker shown in the video.

It is possible to determine whether or not the worker is aware of the contact position from the worker's facial expression, and when the worker has an expression that indicates he or she does not recognize the contact position, a presentation mode that makes it easier for the worker to recognize the contact position is selected, and the contact position is presented according to the selected presentation mode. This makes it easier for the worker to recognize the robot's contact position.

(Technology 9) A presentation system according to any one of techniques 1 to 8, in which the presentation mode selection unit selects a new presentation mode in addition to the already selected presentation mode when the recognition determination unit determines that the worker cannot recognize the contact position, and the presentation unit presents the contact position according to the newly selected presentation mode in addition to the presentation according to the already selected presentation mode.

In addition to the presentation mode already selected, a new presentation mode is selected and the contact position is presented anew according to the newly selected presentation mode, making it even easier for the worker to recognize the robot's contact position.

(Technology 10) The state acquisition unit acquires a video image showing the face of the worker, and the presentation system further includes a concentration level estimation unit that estimates the concentration level of the worker on the work target based on the video image. A presentation system according to any one of Technologies 1 to 9.

By estimating the worker's level of concentration, the estimated level of concentration can be used to indicate the contact position, etc.

(Technology 11) The presentation system described in Technology 10, in which the recognition determination unit determines whether the worker can recognize the contact position based on the acquired state of the worker, the contact position information, and the concentration level.

When the worker's level of concentration is high, it is easier for the worker to recognize the contact position, and when the worker's level of concentration is low, it is harder for the worker to recognize the contact position. Therefore, the estimated level of concentration can be used to determine whether the worker can recognize the contact position.

(Technology 12) The presentation system described in Technology 10 or 11, wherein the presentation mode selection unit further selects the presentation mode based on the concentration level.

When the worker's level of concentration is low, it becomes difficult for the worker to recognize the contact position, so a presentation mode that allows the worker to easily recognize the contact position is selected, and the contact position is presented according to the selected presentation mode. This makes it even easier for the worker to recognize the robot's contact position.

(Technology 13) A presentation system according to any one of techniques 1 to 12, in which the state acquisition unit acquires a video image showing the worker, the presentation system further includes a motion determination unit that determines a motion of the worker with respect to the work object based on the video image, and the recognition determination unit further determines, based on a result of the motion determination, whether or not the worker has recognized the contact position by the presentation of the contact position by the presentation unit.

If the worker is performing an action that indicates he/she is aware of the contact position, it is possible to determine that the worker is aware of the contact position based on the already performed presentation of the contact position, and if the worker is performing an action that indicates he/she is not aware of the contact position based on the already performed presentation of the contact position, it is possible to determine that the worker was not able to recognize the contact position.

(Technology 14) The presentation system described in Technology 13, in which the action determination unit determines whether or not the worker has touched the contact position as the action of the worker on the work object, and the recognition determination unit determines whether or not the worker has recognized the contact position by the presentation of the contact position by the presentation unit based on whether or not the worker has touched the contact position.

In this way, depending on whether or not the worker touches the contact position, it is possible to determine whether or not the worker has recognized the contact position through the presentation of the contact position that has already been made.

(Technology 15) The presentation system described in Technology 13 or 14, wherein the presentation mode selection unit further selects the presentation mode based on the result of the judgment of the action.

If the worker is performing an action that does not allow them to recognize the contact position, a presentation mode that allows the worker to easily recognize the contact position is selected, and the contact position is presented according to the selected presentation mode. This makes it even easier for the worker to recognize the robot's contact position.

(Technology 16) The presentation system according to any one of Technologies 1 to 15, further comprising a proficiency acquisition unit that acquires the worker's proficiency in the work performed on the work object, and the presentation mode selection unit further selects the presentation mode based on the proficiency.

Since highly skilled workers are often able to process more information than less skilled workers, for example, if the worker is highly skilled, a presentation mode in which more information is presented to the worker is selected, and if the worker is less skilled, a presentation mode in which less information is presented to the worker is selected. This makes it possible to present information according to the worker's level of skill.

(Technology 17) A presentation system according to any one of techniques 1 to 16, in which the presentation mode selection unit selects a display mode for displaying the contact position as the presentation mode based on the contact position information and the judgment result of the recognition judgment unit, and the presentation unit performs a display on the work object according to the selected display mode.

If it is determined that the worker cannot recognize the contact position, a display mode that makes it easier for the worker to recognize the contact position is selected, and the display position is presented according to the selected display mode. This makes it easier for the worker to recognize the contact position of the displayed robot.

(Technology 18) The presentation system described in Technology 17, in which the presentation mode selection unit, when it is determined that the worker cannot recognize the contact position, selects a display mode that differs in at least one of the display hue, the display color intensity, the display blinking frequency, and the display size compared to the display mode that would be selected when it is determined that the worker can recognize the contact position.

By selecting a display mode that is easy for workers to see, such as the color tone, color intensity, blinking frequency, or size of the display, the worker can easily recognize the displayed contact position of the robot.

(Technology 19) A presentation method including a contact position acquisition step of acquiring contact position information indicating a contact position between a robot performing work on a work object and the work object, a status acquisition step of acquiring a status of a worker performing work on the work object, a recognition determination step of determining whether or not the worker can recognize the contact position based on the acquired status of the worker and the contact position information, a presentation mode selection step of selecting a presentation mode for presenting the contact position based on the contact position information and the determination result in the recognition determination step, and a presentation step of presenting the contact position according to the selected presentation mode.

This provides a presentation method that makes it easier for workers to recognize the robot's contact position.

(Technology 20) A program for causing a computer to execute the presentation method described in Technology 19.

This makes it possible to provide a program that makes it easier for workers to recognize the robot's contact position.

The presentation system, presentation method, and program disclosed herein can be applied to systems that control robots that perform work on work objects. The presentation system, presentation method, and program disclosed herein can make it easier for workers to recognize the contact position of the robot, improving worker safety. In this way, the presentation system, presentation method, and program disclosed herein are industrially useful.

REFERENCE SIGNS LIST 10, 10a, 10b, 10c Presentation system 11 Contact position acquisition unit 12 Status acquisition unit 13 Recognition determination unit 14 Presentation mode selection unit 15 Presentation unit 16 Concentration level estimation unit 17 Action determination unit 18 Skill level acquisition unit 21 Robot 22 Projector 23 Camera 24, 25, 27 Computer 26 Device 28 Gaze camera 29 AR glasses 30 Worker 31 Gaze range 32 View angle 33 Viewing distance 40 Work object 50 Contact position 51, 52 Additional image 53 Numerical information

Claims (20)

1. a contact position acquisition unit that acquires contact position information indicating a contact position between a robot performing a task on a work object and the work object;
   a status acquisition unit that acquires a status of a worker performing work on the work object;
   a recognition determination unit that determines whether or not the worker can recognize the contact position based on the acquired state of the worker and the contact position information;
   a presentation mode selection unit that selects a presentation mode for presenting the contact position based on the contact position information and a determination result of the recognition determination unit;
   a presentation unit that presents the contact position according to the selected presentation mode,
   Presentation system.
2. the state acquisition unit acquires line-of-sight information of the worker as the state of the worker,
   the recognition determination unit estimates a line-of-sight range of the worker based on the line-of-sight information, and determines whether the contact position is included in the line-of-sight range, thereby determining whether the worker can recognize the contact position.
   The presentation system of claim 1 .
3. The recognition determination unit estimates the line of sight range based on angle information of the worker's head.
   The presentation system of claim 2 .
4. The recognition determination unit estimates the line of sight range based on information regarding the operator's electrooculography.
   The presentation system of claim 2 .
5. The recognition determination unit estimates the line of sight range based on information regarding a movement of the face of the worker.
   The presentation system of claim 2 .
6. the recognition determination unit determines whether or not the estimation result of the line of sight range is correct based on information regarding a facial expression or a movement of the worker.
   The presentation system of claim 2 .
7. The status acquisition unit acquires, as the status of the worker, a camera image captured by a camera worn by the worker,
   the recognition determination unit determines whether the contact position is included in the camera image, thereby determining whether the worker can recognize the contact position;
   The presentation system of claim 1 .
8. The state acquisition unit acquires a moving image showing a face of the worker,
   The recognition determination unit further determines whether or not the worker can recognize the contact position based on a facial expression of the worker shown in the moving image.
   A presentation system according to any one of claims 1 to 7.
9. the presentation mode selection unit, when the recognition determination unit determines that the operator cannot recognize the contact position, selects a new presentation mode in addition to the presentation mode already selected;
   the presentation unit presents the contact position according to the newly selected presentation mode in addition to the presentation according to the already selected presentation mode.
   A presentation system according to any one of claims 1 to 7.
10. The state acquisition unit acquires a moving image showing a face of the worker,
    The presentation system further includes a concentration level estimation unit that estimates a concentration level of the worker on the work target object based on the moving image.
    A presentation system according to any one of claims 1 to 7.
11. the recognition determination unit determines whether or not the worker can recognize the contact position based on the acquired state of the worker, the contact position information, and the concentration level.
    The presentation system of claim 10.
12. The presentation mode selection unit further selects the presentation mode based on the concentration degree.
    The presentation system of claim 10.
13. The state acquisition unit acquires a moving image in which the worker is shown,
    The presentation system further includes a motion determination unit that determines a motion of the worker with respect to the work target based on the moving image,
    The recognition determination unit further determines, based on a result of the determination of the motion, whether or not the worker has recognized the contact position through the presentation of the contact position by the presentation unit.
    A presentation system according to any one of claims 1 to 7.
14. the motion determination unit determines, as a motion of the worker on the work object, whether or not the worker has contacted the contact position;
    the recognition determination unit determines whether or not the worker has recognized the contact position through the presentation of the contact position by the presentation unit, based on whether or not the worker has touched the contact position.
    The presentation system of claim 13.
15. The presentation mode selection unit further selects the presentation mode based on a result of the determination of the motion.
    The presentation system of claim 13.
16. The presentation system further includes a proficiency level acquisition unit that acquires a proficiency level of the worker in the work for the work object,
    The presentation mode selection unit further selects the presentation mode based on the proficiency level.
    A presentation system according to any one of claims 1 to 7.
17. the presentation mode selection unit selects, as the presentation mode, a display mode for displaying the contact position based on the contact position information and a determination result of the recognition determination unit;
    The presentation unit performs a display on the work object according to the selected display mode.
    A presentation system according to any one of claims 1 to 7.
18. When it is determined that the worker cannot recognize the contact position, the presentation mode selection unit selects a display mode that is different in at least one of a hue of the display, a color intensity of the display, a blinking frequency of the display, and a size of the display, as compared with the display mode that is selected when it is determined that the worker can recognize the contact position.
    20. The presentation system of claim 17.
19. a contact position acquisition step of acquiring contact position information indicating a contact position between the robot performing a task on a work object and the work object;
    a status acquisition step of acquiring a status of a worker performing work on the work object;
    a recognition determination step of determining whether or not the worker can recognize the contact position based on the acquired state of the worker and the contact position information;
    a presentation mode selection step of selecting a presentation mode for presenting the contact position based on the contact position information and a determination result in the recognition determination step;
    and presenting the contact position according to the selected presentation mode.
    Presentation method.
20. A program for causing a computer to execute the presentation method described in claim 19.

Images