JP2023127176A - Instructor-side apparatus, method, and program - Google Patents

Abstract

To desirably provide a technique capable of enabling an instructor to more effectively give instructions to a worker.SOLUTION: The instructor-side apparatus is provided with a recognition unit that recognizes a specific input pattern from first sensor data, and a switching signal output unit that outputs a switching signal between a still image and a moving image to a worker-side apparatus based on the recognition of the input pattern; and an image display control unit that controls display of the gesture on a display based on gesture information of an instructor and display of the still image or moving image transmitted from the worker-side apparatus based on the switching signal on the display.SELECTED DRAWING: Figure 1

Description

The present invention relates to an instructor side device, method, and program.

As conventional remote work support techniques, techniques for displaying instructions on a small display, techniques for displaying instructions on a transparent display, etc. are known. However, regardless of the technology, as a conventional remote work support technology, a technology that displays instructions on a flat surface is common.

On the other hand, in recent years, with the development of AR (Augmented Reality) technology, work manuals that can be displayed three-dimensionally have appeared. Such a work manual may be created in advance and displayed three-dimensionally using AR technology. This is making it possible for workers to understand their work three-dimensionally. However, even when the work manual is displayed three-dimensionally using AR technology, the work manual itself is created in advance, so it is difficult to provide work support according to the work situation at that time.

On the other hand, in remote work support, it has already been shown that it is effective to use both the output of the voice input by the instructor and the display of hand gestures (for example, see Non-Patent Document 1). However, if gesture display is directly applied to AR technology, there will be issues that need to be improved.

For example, even if gestures are displayed three-dimensionally using AR technology, the gestures do not change as the worker moves, and the worker can only view the gestures from the same viewpoint. In this case, a situation may arise in which it is difficult for the worker to understand instructions using gestures. On the other hand, if the gestures constantly change as the worker moves, situations will frequently occur where the point of view from which the instructor views the gestures and the perspective from which the worker views the gestures do not match. Situations may arise where it is difficult to

Shunsuke Ichihara, Yusuke Suzuki, “Development and issues of a remote work support system with hand gesture transmission function”, Information Processing Society of Japan Interaction 2019, 2B-36

Therefore, it is desired to provide a technique that allows an instructor to more effectively give instructions to a worker.

In order to solve the above problems, one aspect of the present invention includes a recognition unit that recognizes a specific input pattern from first sensor data, and a still image and a moving image based on the recognition of the input pattern. a switching signal output unit that outputs a switching signal between the image and the worker's side device to the worker's side device; and a switching signal output unit that controls display of the gesture on the display based on gesture information of the instructor; An instructor-side device is provided, comprising: an image display control section that controls display of a still image or a moving image transmitted from the display on the display.

The first sensor data may include a gesture, and the specific input pattern may include a first gesture pattern.

The first sensor data may include audio, and the specific input pattern may include a specific audio pattern.

The recognition unit recognizes a second gesture pattern from the gesture of the instructor, and the switching signal output unit determines that when the second gesture pattern is recognized, even if the specific voice pattern is recognized, The switching signal may not be output to the worker side device.

The recognition unit recognizes a third gesture pattern from the gesture of the instructor, and the switching signal output unit determines that when the third gesture pattern is not recognized, even if the specific voice pattern is recognized. , it is not necessary to output the switching signal to the worker side device.

The switching signal output unit may output a switching signal from a still image to a moving image to the worker side device based on recognition of an input pattern indicating switching from a still image to a moving image.

The switching signal output unit may output a switching signal from a moving image to a still image to the worker side device based on recognition of an input pattern indicating switching from a moving image to a still image.

When the worker's behavior recognized from the second sensor data indicates that the worker is moving, the switching signal output unit determines that an input pattern indicating switching from a moving image to a still image has been recognized. Also, it is not necessary to output a switching signal from a moving image to a still image to the worker side device.

According to another aspect of the present invention, a specific input pattern is recognized from the first sensor data, and switching between a still image and a moving image is performed based on the recognition of the input pattern. Outputting a signal to a worker-side device, controlling the display of gestures on a display based on the gesture information of the instructor, and still images or moving images transmitted from the worker-side device based on the switching signal. A method is provided, comprising: controlling a display of a display on the display.

According to another aspect of the present invention, the computer includes a recognition unit that recognizes a specific input pattern from the first sensor data, and a still image and a moving image based on the recognition of the input pattern. a switching signal output unit that outputs a switching signal between the two to the worker device; and a switching signal output unit that controls display of the gesture on the display based on the gesture information of the instructor, and transmits the switching signal from the worker device based on the switching signal. and an image display control section that controls display of still images or moving images that have been displayed on the display.

As described above, according to the present invention, a technique is provided that allows an instructor to more effectively give instructions to a worker.

1 is a diagram showing an example of a functional configuration of a remote work support system according to an embodiment of the present invention. It is a flowchart which shows the example of the switching operation from a still image state to a video state of a remote work support system. It is a flowchart which shows the example of the switching operation from the video state to the still image state of a remote work support system. 7 is a diagram for explaining a coordinate system to which gesture information is converted according to Comparative Example 1. FIG. 7 is a diagram for explaining a coordinate system to which gesture information is converted according to Comparative Example 2. FIG. FIG. 3 is a diagram for explaining a coordinate system to which gesture information is converted according to an embodiment of the present invention. 1 is a diagram showing a hardware configuration of an information processing device as an example of an instructor-side system according to an embodiment of the present invention.

DESCRIPTION OF THE PREFERRED EMBODIMENTS Preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Note that, in this specification and the drawings, components having substantially the same functional configurations are designated by the same reference numerals and redundant explanation will be omitted.

Further, in this specification and the drawings, a plurality of components having substantially the same functional configuration may be distinguished by attaching different numbers after the same reference numeral. However, if there is no particular need to distinguish between a plurality of components having substantially the same functional configuration, only the same reference numerals are given. Further, similar components in different embodiments may be distinguished by attaching different alphabets after the same reference numerals. However, when it is not necessary to particularly distinguish between similar components of different embodiments, only the same reference numerals are given.

(1. Details of embodiment)
Details of embodiments of the present invention will be described.

(1-1. Configuration of remote work support system)
First, a configuration example of a remote work support system according to an embodiment of the present invention will be described. FIG. 1 is a diagram showing an example of the functional configuration of a remote work support system according to an embodiment of the present invention. As shown in FIG. 1, the remote work support system 1 includes a worker side system and an instructor side system.

The worker side system is a system of the remote work support system 1 that is used by the worker. The worker performs the work at a location away from the instructor who gives instructions to the worker (ie, at a remote location). On the other hand, the instructor side system is a system of the remote work support system 1 that is used by the instructor. The worker side system and the instructor side system are connected to a network 30 and are configured to be able to communicate via the network 30.

Note that in remote work support, voice calls and the like are generally conducted between the instructor and the worker. However, the configuration necessary for voice calls and the like is not involved in the explanation of the configuration of the remote work support system 1 according to the embodiment of the present invention, so the description of the configurations necessary for voice calls and the like will be omitted.

As shown in FIG. 1, the worker side system includes an AR display 11, a camera 12, a position and orientation measurement section 13, an arithmetic processing section 14, a speaker 15, and an audio processing section 16. On the other hand, as shown in FIG. 1, the instructor side system includes a display 21, a gesture input device 22, an arithmetic processing section 24, a microphone 25, a gesture recognition device 26, and a voice recognition device 27.

(AR display 11)
The AR display 11 displays gestures. More specifically, the AR display 11 displays the gesture superimposed on the worker's field of view. The AR display 11 is worn by a worker. For example, the AR display 11 may be a head-mounted display mounted on the head of the worker. However, the type of AR display 11 is not limited to a head-mounted display. For example, the AR display 11 may be a display other than a head-mounted display.

(Camera 12)
The camera 12 obtains a moving image (hereinafter also simply referred to as a "moving image") by capturing an image of the worker's environment. It is desirable that the camera 12 be provided so as to face in the same direction as the line of sight of the worker. Therefore, it is desirable that the camera 12 be integrated with the AR display 11. However, the camera 12 may be configured as separate hardware from the AR display 11.

(Position and orientation measurement unit 13)
The position and orientation measurement unit 13 measures the position and direction of the camera 12. For example, if a sensor is incorporated inside the camera 12, the position and orientation measuring unit 13 determines the position and direction of the camera 12 based on sensor data detected by the sensor incorporated inside the camera 12. may be measured. The sensor may be an acceleration sensor, a gyro sensor, or the like, but the type of sensor is not particularly limited.

For example, the position and orientation measurement unit 13 may include a sensor that measures a two-dimensional marker installed in the worker's environment. At this time, the position and orientation measurement unit 13 may measure the position and direction of the camera 12 based on the shape of the measured two-dimensional marker. Alternatively, the position and orientation measurement unit 13 uses a method of measuring the position and direction of the camera 12 based on sensor data detected by a sensor built into the camera 12, and a method of measuring the position and direction of the camera 12 based on the shape of the measured two-dimensional marker. It may be used in combination with a method of measuring the position and direction of the camera 12.

(Calculation processing unit 14)
The arithmetic processing unit 14 is implemented by a computer, and functions as a worker-side device that performs various arithmetic processes. For example, the arithmetic processing unit 14 includes a gesture acquisition unit that acquires gesture information from the arithmetic processing unit 24, a signal acquisition unit that acquires a switching signal from the arithmetic processing unit 24, and an AR display that displays gestures in the field of view of the worker. 11.

Further, the arithmetic processing unit 14 performs processing such as controlling communication via a communication interface with the arithmetic processing unit 24 in the instructor side system. For example, the arithmetic processing unit 14 may function as a transmission control unit that controls transmission of a moving image or a still image (hereinafter also simply referred to as a “still image”) obtained by the camera 12 to the arithmetic processing unit 24.

For example, the arithmetic processing unit 14 includes an arithmetic unit such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit), and a program stored in a ROM (Read Only Memory) is loaded into a RAM and executed by the arithmetic unit. By doing so, the function can be realized. At this time, a computer-readable recording medium on which the program is recorded may also be provided.

(Speaker 15)
The speaker 15 outputs the instruction voice input from the instructor without passing it through the voice processing section 16. Alternatively, the speaker 15 outputs the instruction voice input from the instructor through the voice processing section 16. Note that the speaker 15 is configured by a stereo speaker or an array speaker so that the virtual output position of the instruction voice output from the speaker 15 can be controlled.

(Sound processing unit 16)
The audio processing unit 16 acquires position data and audio data, and performs processing to make the worker hear the audio as if it were being output from that location. As such processing, HRTF (Non-patent document: Fundamentals of head-related transfer functions and applications to three-dimensional acoustic systems, edited by the Acoustical Society of Japan, written by Kazuhiro Iida, Corona Publishing) may be used. More specifically, the audio processing unit 16 acquires the instruction voice from the arithmetic processing unit 14, and controls the output of the acquired instruction voice from the virtual output position of the speaker 15. Note that the instruction voice may be output from the speaker 15 as it is without being processed by the voice processing unit 16.

(Display 21)
Display 21 may display moving images obtained by camera 12. Further, the display 21 can display a snapshot based on a moving image obtained by the camera 12 as a still image. This allows the instructor to view the worker's environment. Furthermore, the display 21 may display the gestures of the instructor. This allows the instructor to confirm what kind of gesture is being transmitted to the worker.

(Gesture input device 22)
The gesture input device 22 is an input device that accepts gestures input from an instructor. The gesture input device 22 outputs the received gesture to the arithmetic processing unit 24 . Further, the gesture input device 22 outputs the received gesture to the gesture recognition device 26. Note that the gesture input device 22 may correspond to an example of a sensor. That is, the gesture input device 22 detects a gesture as an example of sensor data (first sensor data).

For example, gesture input device 22 may be realized by an optical device. As such an optical device, Leap Motion (registered trademark) developed by Ultraleap, etc. can be used. Leap Motion (registered trademark) is a technology that tracks hand movements based on the detection results of infrared rays irradiated onto the hand by a plurality of LEDs (Light Emitting Diodes) using an infrared stereo camera. Alternatively, the gesture input device 22 may be realized by an input device (for example, a sensor glove, etc.) worn by the instructor.

The expression format of the gesture may be any expression format as long as the gesture can be expressed by a structure including a plurality of three-dimensional coordinates and the gesture can be reproduced in a three-dimensional space. For example, a gesture may be expressed by end point data of a skeleton, or may be expressed by mesh data of an epidermis.

(Microphone 25)
The microphone 25 receives audio input from the instructor. The microphone 25 outputs the received voice to the arithmetic processing unit 24 as an instruction voice. Further, the microphone 25 outputs the received voice to the voice recognition device 27. Note that the microphone 25 may correspond to an example of a sensor. That is, the microphone 25 detects audio as an example of sensor data (first sensor data).

(Gesture recognition device 26)
The gesture recognition device 26 performs gesture recognition on the gesture accepted by the gesture input device 22, and attempts to recognize a specific input pattern from the gesture. For example, the specific input pattern may be a predetermined gesture pattern (eg, one or more predetermined gesture movements). For example, the predetermined gesture may be one or more gestures indicating switching.

The gesture recognition device 26 is realized by a computer and functions as an instructor-side device. For example, the gesture recognition device 26 includes an arithmetic device such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit), and a program stored in a ROM (Read Only Memory) is loaded into a RAM and executed by the arithmetic device. By doing so, the function can be realized. At this time, a computer-readable recording medium on which the program is recorded may also be provided.

(Voice recognition device 27)
The speech recognition device 27 performs speech recognition on the speech received by the microphone 25, and attempts to recognize a specific input pattern from the speech. For example, the specific input pattern may be a predetermined speech pattern (eg, one or more predetermined words). For example, the predetermined phrase may be one or more phrases indicating switching.

The voice recognition device 27 is realized by a computer and functions as an instructor-side device. For example, the speech recognition device 27 includes an arithmetic unit such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit), and a program stored in a ROM (Read Only Memory) is loaded into a RAM and executed by the arithmetic unit. By doing so, the function can be realized. At this time, a computer-readable recording medium on which the program is recorded may also be provided.

(Calculation processing unit 24)
The arithmetic processing unit 24 is implemented by a computer and functions as an instructor-side device that performs various arithmetic processes. For example, the arithmetic processing unit 24 outputs a switching signal to output a switching signal to the arithmetic processing unit 14 based on recognition of a specific input pattern by the recognition unit (for example, the gesture recognition device 26 or the voice recognition device 27). can function as a division.

Note that the switching signal is a signal indicating switching from one of two states to the other. In the embodiment of the present invention, the switching signal is based on a state in which a moving image obtained by the camera 12 is displayed on the display 21 (hereinafter also referred to as a "moving image state"), and a state in which a moving image obtained by the camera 12 is displayed. A case is mainly assumed in which a snapshot is displayed as a still image on the display 21 (hereinafter also referred to as a "still image state").

Further, the arithmetic processing unit 24 can function as an image display control unit that controls the display of gestures on the display 21 and controls the display of still images or moving images on the display 21. Further, the arithmetic processing unit 24 performs processes such as controlling communication via a communication interface with the arithmetic processing unit 14 in the worker side system.

For example, the arithmetic processing unit 24 includes an arithmetic unit such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit), and a program stored in a ROM (Read Only Memory) is loaded into a RAM and executed by the arithmetic unit. By doing so, the function can be realized. At this time, a computer-readable recording medium on which the program is recorded may also be provided.

(Network 30)
The network 30 connects the worker side system and the instructor side system. The network 30 can function as a communication path between the worker-side system and the instructor-side system.

The configuration example of the remote work support system 1 according to the embodiment of the present invention has been described above.

(1-2. Operation of remote work support system)
Next, an example of the operation of the remote work support system 1 according to the embodiment of the present invention will be described with reference to FIGS. 1 to 3.

Note that operations such as voice calls are not related to the explanation of the operations of the remote work support system 1 according to the embodiment of the present invention, and therefore, explanations of operations such as voice calls will be omitted. Furthermore, it is assumed that communication delays and the like are negligible.

As described above, the remote work support system 1 can take either a moving image state or a still image state. The state of the remote work support system 1 is switched based on a switching signal. In the moving image state, a moving image obtained by the camera 12 is displayed on the display 21. On the other hand, in the still image state, a snapshot of the moving image at the switching timing is displayed on the display 21 as a still image.

FIG. 2 is a flowchart illustrating an example of the switching operation of the remote work support system 1 from a still image state to a moving image state. FIG. 3 is a flowchart illustrating an example of an operation of switching the remote work support system 1 from a moving image state to a still image state. First, an example of an operation for switching from a still image state to a moving image state will be explained with reference to FIG. 2, and then an example of an operation for switching from a moving image state to a still image state will be explained, mainly referring to FIG. explain.

Here, gesture information whose input from the instructor is accepted by the gesture input device 22 is transmitted from the instructor's system to the worker's system. The gesture information may include information indicating the displacement of the gesture (displacement of each feature point of the hand) from the reference position.

In the worker side system, the arithmetic processing unit 14 controls the AR display 11 so that gestures are arranged based on the reference position, reference direction, and gesture information. On the other hand, in the instructor side system, the arithmetic processing unit 24 controls the display 21 so that gestures are displayed based on the gesture information.

Furthermore, in the system on the instructor's side, an instruction voice is received by the microphone 25 and output to the arithmetic processing section 24 . Then, the instruction voice is constantly transmitted from the arithmetic processing unit 24 to the arithmetic processing unit 14 in the worker side system via the network 30. The arithmetic processing unit 14 always acquires the instruction voice.

(Switching operation from still image state to video state)
In the still image state, the reference position and reference direction in which gestures are placed do not follow the position and direction of the camera 12. That is, in the still image state, the reference position and reference direction in which gestures are placed in the worker side system are fixed to the position and direction of the camera 12 recorded at the timing of switching from the moving image state to the still image state. Ru.

In the instructor side system, the gesture recognition device 26 determines whether a first gesture pattern (a gesture pattern indicating switching from a still image state to a moving image state) is recognized from the instructor's gesture received by the gesture input device 22. try. The specific gesture pattern may be any specific gesture pattern. Note that the gesture pattern indicating switching from a still image state to a moving image state and the gesture pattern indicating switching from a moving image state to a still image state may be the same, but since the two switching instructions are recognized separately, It is desirable that they be different.

Alternatively, the voice recognition device 27 attempts to recognize a specific voice pattern (a voice pattern indicating switching from a still image state to a moving image state) from the instructor's voice received by the microphone 25. The specific audio pattern may be any specific audio pattern. Note that the audio pattern indicating switching from a still image state to a moving image state and the audio pattern indicating switching from a moving image state to a still image state may be the same, but in order to recognize the two switching instructions separately, It is desirable that they be different.

The arithmetic processing unit 24 determines whether an instruction to switch from the still image state to the moving image state has been input from the instructor based on whether the first gesture pattern has been recognized by the gesture recognition device 26 (S11). Alternatively, the arithmetic processing unit 24 determines whether an instruction to switch from the still image state to the moving image state has been input from the instructor, depending on whether a specific voice pattern has been recognized by the voice recognition device 27.

As shown in FIG. 2, if the instruction to switch from the still image state to the moving image state has not been input ("NO" in S11), the arithmetic processing unit 24 shifts the operation to S11. On the other hand, when an instruction to switch from the still image state to the moving image state is input ("YES" in S11), the arithmetic processing section 24 instructs the arithmetic processing section 14 in the worker's system to switch from the still image state to the moving image state. A switching signal is notified via the network 30 (S12).

In the worker side system, upon receiving the switching signal from the still image state to the moving image state, the arithmetic processing unit 14 determines the reference position at which the gesture is placed by the AR display 11 of the camera 12 measured by the position and orientation measurement unit 13. In addition to following the position, the reference direction in which the gesture is placed by the AR display 11 is made to follow the direction of the camera 12 measured by the position and orientation measurement unit 13.

Furthermore, the arithmetic processing unit 14 disables the audio processing unit 16 (that is, stops the operation of the audio processing unit 16), and causes the acquired instruction voice to be output as is from the speaker 15 (S31).

Further, if there is a still image displayed on the display 21, the arithmetic processing unit 24 erases the still image (S13). Then, when the video obtained by the camera 12 is transmitted from the calculation processing unit 14, the calculation processing unit 24 controls the display 21 so that the display 21 resumes displaying the transmitted video (S14).

(Switching operation from video state to still image state)
In the moving image state, the reference position and reference direction in which gestures are placed follow the position and direction of the camera 12.

In the instructor side system, the gesture recognition device 26 determines whether a first gesture pattern (a gesture pattern indicating switching from a moving image state to a still image state) is recognized from the instructor's gesture received by the gesture input device 22. try. Note that, as described above, the gesture pattern indicating switching from a still image state to a moving image state and the gesture pattern indicating switching from a moving image state to a still image state may be the same, but two switching instructions may be used. It is desirable that they be different in order to be recognized separately.

Alternatively, the voice recognition device 27 attempts to recognize a specific voice pattern (a voice pattern indicating switching from a moving image state to a still image state) from the instructor's voice received by the microphone 25. Note that, as described above, the audio pattern indicating switching from the video state to the still image state and the audio pattern indicating switching from the still image state to the video state may be the same, but two switching instructions may be used. It is desirable that they be different in order to be recognized separately.

The arithmetic processing unit 24 determines whether an instruction to switch from the moving image state to the still image state has been input from the instructor based on whether the first gesture pattern has been recognized by the gesture recognition device 26 (S21). Alternatively, the arithmetic processing unit 24 determines whether an instruction to switch from the moving image state to the still image state has been input from the instructor, depending on whether a specific voice pattern has been recognized by the voice recognition device 27.

If the instruction to switch from the moving image state to the still image state has not been input (“NO” in S21), the arithmetic processing unit 24 shifts the operation to S21. On the other hand, when an instruction to switch from the moving image state to the still image state is input ("YES" in S21), the arithmetic processing section 24 instructs the arithmetic processing section 14 in the worker side system to switch from the still image state to the moving image state. A switching signal is notified via the network 30 (S22).

In the worker side system, upon receiving the switching signal from the still image state to the moving image state, the arithmetic processing unit 14 records the position and direction of the camera 12 at the timing of switching from the still image state to the moving image state, and also records ( S26), the reference position and reference direction used by the AR display 11 for arranging gestures are fixed to the position and direction of the camera 12 at the timing when the recorded moving image state was switched to the still image state.

Furthermore, the arithmetic processing unit 14 enables the audio processing unit 16 (that is, starts the operation of the audio processing unit 16) (S41). As a result, the audio processing unit 16 performs processing on the instruction audio (for example, processing using HRTF, etc.) to output the instruction audio from the speaker 15 as if the instruction audio were being output from the reference position where the gesture is placed. Output.

Furthermore, the arithmetic processing unit 24 creates a snapshot of the moving image obtained by the camera 12 as a still image (S23).

If there is a moving image displayed on the display 21, the arithmetic processing unit 24 erases the moving image (S24). Then, the arithmetic processing unit 24 controls the display 21 so that the display 21 resumes displaying the generated still image (S25).

The example of the switching operation from the still image state to the moving image state and the example of the switching operation from the moving image state to the still image state have been described above. Next, the reference position and reference direction in which gestures are placed will be described in more detail.

(1-3. Reference position and reference direction where gesture is placed)
The gesture information input by the gesture input device 22 from the instructor is information expressed by the position and direction in a coordinate system unique to the environment of the instructor. In order to display a gesture on the AR display 11 in the worker's environment based on such gesture information, it is necessary to change from the coordinate system of the instructor (the coordinate system of the transformation source) to the coordinate system of the worker (the coordinate system of the transformation destination). It is necessary to convert the coordinates of the gesture information into .

Here, as examples of coordinate systems to be converted, Comparative Example 1 (world coordinate system), Comparative Example 2 (camera coordinate system), and the coordinate system according to the present embodiment will be described in order. Here, in order to simplify the explanation, it is assumed that the position of the gesture is fixed.

(Comparative example 1 (world coordinate system))
FIG. 4 is a diagram for explaining a coordinate system to which gesture information is converted according to Comparative Example 1. Referring to FIG. 4, a worker environment E11 in an initial state is shown. A work target R1 exists in the worker environment E11 in the initial state. The work target R1 is an object to be worked on by a worker.

Here, in consideration of the ease of viewing the figure, the work target R1 is shown as a square plane having a "front" and "back". However, the shape of the work target R1 is not limited. Typically, the work target R1 is assumed to be a machine such as an ATM (Automated Teller Machine) or a printer, but the work target R1 does not need to be a machine and may be any object used for work. good.

Furthermore, a camera 12 is present in the worker environment E11 in the initial state, and a moving image obtained by the camera 12 is displayed on the screen 121a on the instructor's side. The work target R1 is shown in the video displayed on the screen 121a on the instructor's side. The instructor inputs gesture information while watching a video of the worker environment E11 in the initial state. Gesture J1 is displayed on the screen 121a on the instructor's side based on the gesture information.

Referring to the worker environment E11 in the initial state, a world coordinate system C1 and a camera coordinate system C2 are shown. Based on the gesture information input by the instructor, the AR display on the worker's side is controlled so that the gesture J1 is placed in the world coordinate system C1. The worker can work on the work target R1 while looking at the gesture J1 arranged in the world coordinate system C1.

Here, it is assumed that the worker moves to the back side of the work target R1, as shown in the worker environment E12. The position and direction of gesture J1 in world coordinate system C1 are fixed.

At this time, the "back" of the work target R1 and the gesture J1 are displayed on the instructor's screen 122a. However, the position and direction of gesture J1 also change according to the amount and direction of movement of the worker. Therefore, the instructor is required to input a gesture before the position and direction change (before the gesture J1 is rotated by 180 degrees) compared to the displayed gesture J1, which makes it difficult for the instructor to input the gesture.

On the other hand, suppose that the worker moves to the back side of the work target R1, as shown in the worker environment E13. The position and direction of gesture J1 in world coordinate system C1 are fixed.

At this time, since the position and direction of the gesture J1 in the world coordinate system C1 are fixed, the gesture J1 is located behind the worker, and the gesture J1 is no longer visible to the worker. For example, even if the instructor inputs gesture information for heading to the next work position, the worker will not be able to see the gesture and will not be able to know the next work position.

As described above, Comparative Example 1 is suitable for displaying a work manual created in advance. However, in Comparative Example 1, the gestures made by the instructor may become invisible as the worker moves, so it may not be suitable when it is necessary to give work instructions in real time.

The coordinate system to which gesture information is converted according to Comparative Example 1 has been described above.

(Comparative example 2 (camera coordinate system))
FIG. 5 is a diagram for explaining a coordinate system to which gesture information is converted according to Comparative Example 2. Referring to FIG. 5, similar to the example shown in FIG. 4, a worker environment E21 in an initial state is shown. The work target R1 exists in the worker environment E21 in the initial state.

Furthermore, a moving image obtained by the camera 12 is displayed on the screen 123a on the instructor's side. The work target R1 is shown in the video displayed on the screen 123a on the instructor's side. The instructor inputs gesture information while watching a video of the worker environment E11 in the initial state. Gesture J1 is displayed on the screen 123a on the instructor's side based on the gesture information.

In Comparative Example 2, the AR display on the worker's side is controlled so that gesture J1 is placed in camera coordinate system C2 based on gesture information input by the instructor. The worker can work on the work target R1 while looking at the gesture J1 arranged in the camera coordinate system C2.

Here, it is assumed that the worker moves to the back side of the work target R1, as shown in the worker environment E22. Unlike Comparative Example 1, the position and direction of the gesture J1 in the world coordinate system C1 change, but the position and direction of the gesture J1 in the camera coordinate system C2 are fixed.

At this time, the "back" of the work target R1 and the gesture J1 are displayed on the screen 124a on the instructor's side. At this time, although the worker is moving, the position and direction of gesture J1 displayed on screen 124a are fixed. Therefore, the instructor only needs to input a gesture at the same position and direction as the displayed gesture J1, making it easy for the instructor to input the gesture.

However, since the position and direction of gesture J1 in camera coordinate system C2 are fixed, the position and direction of gesture J1 visible to the operator are constant. Therefore, the operator is no longer able to change the position or angle at which the gesture J1 is viewed (for example, the operator is no longer able to view the gesture J1 from above).

On the other hand, assume that the worker moves to the back side of the work target R1, as shown in the worker environment E23. At this time, the position and direction of the gesture J1 in the world coordinate system C1 change, but the position and direction of the gesture J1 in the camera coordinate system C2 are fixed. At this time, the position and direction of the gesture J1 displayed on the screen 125a on the instructor side are fixed.

Furthermore, since the position of the gesture J1 in the camera coordinate system C2 is fixed, the gesture J1 continues to be located in front of the worker, and the gesture J1 will not become invisible to the worker. For example, when an instructor inputs gesture information for heading to the next work position, the worker can see the gesture and know the next work position.

As described above, Comparative Example 2 is suitable for cases where it is necessary to give work instructions in real time because the gestures made by the instructor do not become invisible as the worker moves. However, in Comparative Example 2, the position and angle of the gesture visible to the worker are not changed, so it can be said that the advantage of displaying the gesture using AR technology is lost.

The coordinate system to which gesture information is converted according to Comparative Example 2 has been described above.

(Coordinate system according to this embodiment)
FIG. 6 is a diagram for explaining a coordinate system into which gesture information is converted according to an embodiment of the present invention. Referring to FIG. 6, similar to the example shown in FIG. 4, a worker environment E31 in an initial state is shown. The work object R1 exists in the worker environment E31 in the initial state. In the initial state, it is assumed that the remote work support system 1 is in a moving image state. For example, in a moving image state, an instructor instructs a worker about a work position using a gesture.

Note that the instruction voice received by the microphone 25 is always transmitted from the arithmetic processing section 24 to the arithmetic processing section 14 via the network 30.

When the remote work support system 1 is in the moving image state, the arithmetic processing unit 24 controls so that the moving image obtained by the camera 12 is displayed on the instructor's screen 126a. The work target R1 is shown in the video displayed on the screen 126a on the instructor's side. The instructor inputs gesture information while watching a video of the worker environment E31 in the initial state. Gesture J1 is displayed on the screen 126a on the instructor's side based on the gesture information.

The arithmetic processing unit 14 acquires gesture information from the arithmetic processing unit 24 . When the state of the remote work support system 1 is a moving image state, the arithmetic processing unit 14 calculates camera coordinates with the position and direction of the camera 12 as a reference position and a reference direction (a second reference position and a second reference direction). A method (second method) is adopted in which the AR display 11 is controlled so that the gesture J1 is placed in the system C2 (second coordinate system). The worker can head to the work position while looking at the gesture J1 arranged in the camera coordinate system C2.

Furthermore, when the remote work support system 1 is in the moving image state, the worker's viewpoint and the instructor's viewpoint are the same. Therefore, when the state of the remote work support system 1 is in the video state, as shown in the worker environment E31, the arithmetic processing unit 14 performs audio processing on the instruction voice received from the arithmetic processing unit 24 in the instructor side system. The output is output from the speaker 15 as it is without being processed by the unit 16.

As an example, assume that the instructor finishes instructing the work position and starts instructing the work content to be performed on the work target R1 by gesture. At this time, the instructor inputs a switching instruction indicating switching from the moving image state to the still image state (that is, switching from the second method to the first method) using a gesture pattern or a voice pattern. When such a switching instruction is input, a switching signal indicating switching from a moving image state to a still image state is notified from the arithmetic processing unit 24 to the arithmetic processing unit 14.

Upon acquiring the switching signal indicating switching from the moving image state to the still image state, the arithmetic processing unit 14 acquired a switching signal indicating switching from the moving image state to the still image state based on the acquisition of the switching signal. The position and direction of the camera 12 at the timing are recorded as a reference position C3 and a reference direction (first reference position and first reference direction) used for gesture placement. Note that the timing at which the switching signal is acquired is an example of a predetermined timing.

Assume that the worker moves to the back of the work target R1, as shown in the worker environment E32. The position and orientation measurement unit 13 measures the movement amount and movement direction of the worker based on the reference position and reference direction recorded by the arithmetic processing unit 14.

The arithmetic processing unit 14 calculates a gesture based on the movement amount and movement direction of the worker measured by the position and orientation measurement unit 13 in a coordinate system (first coordinate system) based on the recorded reference position and reference direction. A method (first method) is adopted in which the AR display 11 is controlled so that J1 is placed. In other words, the arithmetic processing unit 14 arranges the gesture J1 at the reference position and reference direction in the AR space.

At this time, as shown on the screen 127a, while the position and direction of the gesture J1 in the world coordinate system C1 are fixed, the amount and movement of the worker are displayed from the recorded reference position and direction. Gesture J1 is displayed at the position and direction of gesture J1 that can be seen from the position and direction where the gesture J1 is moved and rotated by the direction. Note that the "back" of the work target R1 that exists on the screen 127a may be a through image or may be the actual object.

The arithmetic processing unit 14 calculates a virtual output position of the instruction voice in a coordinate system based on the recorded reference position and reference direction, based on the movement amount and movement direction of the worker measured by the position and orientation measurement unit 13. and determine the virtual output direction. In other words, the arithmetic processing unit 14 determines the reference position and reference direction in the AR space as the virtual output position and output direction of the instruction voice.

The audio processing unit 16 outputs the sound from the speaker 15 as if it were being output in a virtual output direction from the virtual output position determined by the arithmetic processing unit 14 (as if it were being output from the virtual speaker K1). Controls the output of voice instructions. This virtual output position may correspond to the point of view of the instructor. Therefore, the worker can recognize the point of view of the instructor based on the instruction voice, and can understand the instructions while experiencing a sense of realism.

At this time, as shown in the worker environment E32, while the reference position and reference direction are fixed to the world coordinate system C1, the reference position and reference direction are moved and rotated by the amount and direction of movement of the worker from the reference position and reference direction. The instruction voice is output as if the instruction voice were being output from the reference position and reference direction that can be seen from the direction.

On the other hand, in the instructor side system, the arithmetic processing unit 24 generates a snapshot of the moving image at the timing when a switching signal indicating switching from the moving image state to the still image state is input as a still image. Then, as shown on the screen 127b, the arithmetic processing unit 24 controls the generated still image to be displayed on the display 21. Gesture J1 is displayed on the screen 127b on the instructor's side based on the gesture information.

At this time, although the worker is moving, the position and direction of gesture J1 displayed on screen 127b are fixed. Therefore, the instructor only needs to input a gesture at the same position and direction as the displayed gesture J1, making it easy for the instructor to input the gesture.

Furthermore, as the worker moves, the position and direction of the gesture J1 visible to the worker may change. Therefore, the operator can change the position or angle at which the gesture J1 is viewed (for example, the operator can now view the gesture J1 from above).

On the other hand, suppose that the worker moves to the back side of the work target R1, as shown in the worker environment E33. At this time, the position and direction of the gesture J1 in the world coordinate system C1 change, but the position and direction of the gesture J1 in the camera coordinate system C2 are fixed. At this time, the position and direction of the gesture J1 displayed on the screen 128a on the instructor side are fixed.

Therefore, similarly to Comparative Example 2, the gesture J1 continues to be located in front of the worker, and the gesture J1 will not become invisible to the worker. For example, when an instructor inputs gesture information for heading to the next work position, the worker can see the gesture and know the next work position.

Although not shown in FIG. 6, it is assumed that the instructor finishes instructing the work to be performed on the work target R1 by gesture and starts instructing the next work position. At this time, the instructor inputs a switching instruction indicating switching from the still image state to the moving image state (that is, switching from the first method to the second method) using a gesture pattern or a voice pattern. When such a switching instruction is input, a switching signal indicating switching from a still image state to a moving image state is notified from the arithmetic processing unit 24 to the arithmetic processing unit 14.

Upon acquiring a switching signal indicating switching from a still image state to a moving image state, the arithmetic processing unit 14 calculates camera coordinates with the position and direction of the camera 12 as a reference position and a reference direction based on the acquisition of the switching signal. The AR display 11 is controlled so that gesture J1 is placed in system C2. The worker can head to the work position while looking at the gesture J1 arranged in the camera coordinate system C2.

Furthermore, when the remote work support system 1 is in the moving image state, the worker's viewpoint and the instructor's viewpoint are the same. Therefore, when the state of the remote work support system 1 is in the video state, the arithmetic processing unit 14 performs audio processing on the instruction voice received from the arithmetic processing unit 24 in the instructor side system so that the state is set to the worker environment E31. The output is output from the speaker 15 as it is without being processed by the unit 16.

The reference positions and reference directions in which gestures are placed have been described above in detail.

(1-4. Effect)
As described above, according to the embodiment of the present invention, the position and angle of the gesture visible to the worker can be changed, so the advantages of displaying the gesture using AR technology are not lost. Furthermore, according to the embodiment of the present invention, the gestures made by the instructor will not become invisible as the worker moves. Therefore, the technology according to the embodiment of the present invention is suitable for cases where it is necessary to issue work instructions in real time.

Further, according to the embodiment of the present invention, when the state of the remote work support system 1 is a still image state, the virtual output position of the instruction voice is controlled to be the viewpoint of the instructor, so that the work The person can understand the point of view of the person giving the instruction based on the instruction voice.

Further, according to an embodiment of the present invention, an instruction to switch between a moving image state and a still image state may be given by voice or gesture. Therefore, the instructor does not need to reach for the changeover switch when instructing switching between the moving image state and the still image state. This prevents the instructor's gesture instruction from being interrupted every time the instructor issues a switching instruction (or it becomes possible to shorten the time period during which the instructor's gesture instruction is interrupted).

The effects of the remote work support system 1 according to the embodiment of the present invention have been described above.

(2. Hardware configuration example)
Next, an example of the hardware configuration of the worker-side system according to the embodiment of the present invention will be described.

An example of the hardware configuration of the information processing device 900 will be described below as an example of the hardware configuration of the instructor-side system according to the embodiment of the present invention. Note that the hardware configuration example of the information processing device 900 described below is only an example of the hardware configuration of the instructor side system. Therefore, for the hardware configuration of the instructor-side system, unnecessary configurations may be deleted from the hardware configuration of the information processing apparatus 900 described below, or new configurations may be added. Note that the hardware configuration of the worker-side system can also be realized in the same way as the hardware configuration of the instructor-side system.

FIG. 7 is a diagram showing a hardware configuration of an information processing device 900 as an example of an instructor-side system according to an embodiment of the present invention. The information processing device 900 includes a CPU (Central Processing Unit) 901, a ROM (Read Only Memory) 902, a RAM (Random Access Memory) 903, a host bus 904, a bridge 905, an external bus 906, and an interface. With 907 , an input device 908, an output device 909, a storage device 910, and a communication device 911.

The CPU 901 functions as an arithmetic processing device and a control device, and controls overall operations within the information processing device 900 according to various programs. Further, the CPU 901 may be a microprocessor. The ROM 902 stores programs used by the CPU 901, calculation parameters, and the like. The RAM 903 temporarily stores programs used in the execution of the CPU 901 and parameters that change as appropriate during the execution. These are interconnected by a host bus 904 composed of a CPU bus and the like.

The host bus 904 is connected via a bridge 905 to an external bus 906 such as a PCI (Peripheral Component Interconnect/Interface) bus. Note that the host bus 904, bridge 905, and external bus 906 do not necessarily need to be configured separately, and these functions may be implemented in one bus.

The input device 908 includes input means for the user to input information, such as a mouse, keyboard, touch panel, button, microphone, switch, and lever, and an input control circuit that generates an input signal based on the user's input and outputs it to the CPU 901. It is composed of etc. By operating the input device 908, a user operating the information processing device 900 can input various data to the information processing device 900 and instruct processing operations.

The output device 909 includes, for example, a CRT (Cathode Ray Tube) display device, a liquid crystal display (LCD) device, an OLED (Organic Light Emitting Diode) device, a display device such as a lamp, and an audio output device such as a speaker.

The storage device 910 is a device for storing data. The storage device 910 may include a storage medium, a recording device that records data on the storage medium, a reading device that reads data from the storage medium, a deletion device that deletes data recorded on the storage medium, and the like. The storage device 910 is configured with, for example, an HDD (Hard Disk Drive). This storage device 910 drives a hard disk and stores programs executed by the CPU 901 and various data.

The communication device 911 is, for example, a communication interface configured with a communication device for connecting to a network. Further, the communication device 911 may support either wireless communication or wired communication.

The example hardware configuration of the instructor-side system according to the embodiment of the present invention has been described above.

(3. Summary)
Although preferred embodiments of the present invention have been described above in detail with reference to the accompanying drawings, the present invention is not limited to such examples. It is clear that a person with ordinary knowledge in the technical field to which the present invention pertains can come up with various changes or modifications within the scope of the technical idea stated in the claims. It is understood that these also naturally fall within the technical scope of the present invention.

For example, in the above description, there is one instructor and one worker. However, there may be multiple instructors. In such a case, gestures by each of a plurality of instructors may be handled. Alternatively, there may be multiple workers. In such a case, by specifying a camera corresponding to the camera coordinate system in which the gesture is placed, the gesture may be presented to the operator of the specified camera.

Furthermore, the above description has mainly been given to the case where the reference position and reference direction in which gestures are placed are switched based on the switching signal. However, based on the switching signal, the reference position and reference direction where objects other than gestures (for example, written instructions) are placed may also be switched in the same manner as the reference position and reference direction where gestures are placed.

Furthermore, in the above, when the switching signal indicates switching from a moving image to a still image, the arithmetic processing unit 24 in the instructor side system creates a snapshot from the moving image, and the created snapshot is displayed as a still image on the display 21. The explanation has mainly been given to the case where the control is performed so that the screen is displayed using . However, the arithmetic processing unit 24 may display a moving image on the display 21 together with the still image. At this time, the still image and the moving image may be juxtaposed.

Alternatively, in the still image state, the still image and the moving image may not be juxtaposed. At this time, the process of creating a snapshot from the video may be performed by the arithmetic processing section 14 in the worker's system instead of the arithmetic processing section 24 in the instructor's system. In such a case, the camera 12 may stop capturing a moving image in the worker's system, and may stop transmitting the moving image from the worker's system to the instructor's system. As a result, not only the power consumption by the camera 12 can be reduced, but also the communication band can be reduced.

In the above, the case where the reference position where the virtual instruction voice is placed is the position of the camera 12 at the timing when the switching signal indicating switching from the moving image state to the still image state is acquired has been mainly described. However, the reference position where the virtual instruction voice is placed may be a position shifted (offset position) from the position of the camera 12 at the timing when the switching signal was acquired.

Further, in the above description, the case where the gesture of the instructor and the voice of the instructor are used separately for switching between the still image state and the moving image state has been mainly assumed. However, a combination of the instructor's gesture and the instructor's voice may be used to switch between the still image state and the moving image state.

For example, the gesture recognition device 26 may attempt to recognize a gesture pattern (second gesture pattern) indicating prohibition of switching from the gestures of the instructor received by the gesture input device 22. The gesture pattern indicating prohibition of switching may be any specific gesture pattern.

Then, when a gesture pattern indicating prohibition of switching is recognized, the arithmetic processing unit 24 does not need to output a switching signal to the arithmetic processing unit 14 even if a specific voice pattern is recognized by the speech recognition device 27. good. That is, the arithmetic processing unit 24 outputs a switching signal to the arithmetic processing unit 14 only when a specific voice pattern is recognized by the speech recognition device 27 and a gesture pattern indicating prohibition of switching is not recognized. good.

Alternatively, the gesture recognition device 26 may attempt to recognize a gesture pattern (third gesture pattern) indicating execution of switching from the gestures of the instructor received by the gesture input device 22. The gesture pattern indicating execution of switching may be any specific gesture pattern.

Then, when the gesture pattern indicating execution of switching is not recognized, the arithmetic processing unit 24 does not need to output a switching signal to the arithmetic processing unit 14 even if a specific voice pattern is recognized by the speech recognition device 27. . That is, the arithmetic processing unit 24 outputs a switching signal to the arithmetic processing unit 14 only when a gesture pattern indicating execution of switching is recognized when a specific voice pattern is recognized by the speech recognition device 27. Good too.

In the above, when the first gesture pattern is recognized by the gesture recognition device 26 or when a specific voice pattern is recognized by the voice recognition device 27, the calculation processing unit 24 The main assumption is that a switching signal between 1 and 2 is output to the arithmetic processing unit 14. However, whether or not to output the switching signal to the arithmetic processing unit 14 may be determined by taking additional conditions into consideration.

For example, a sensor (for example, an acceleration sensor, a vibration sensor, a gyro sensor, etc.) that detects the state of the worker and an action recognition device that recognizes the worker's actions from such sensor data (second sensor data) are provided. You can leave it there. For example, the sensor may be attached to the worker's body (for example, it may be attached to the AR display 11). The behavior recognition device may be provided in the worker-side system, the instructor-side system, or outside the worker-side system and the instructor-side system.

For example, when the worker's behavior recognized from the sensor data indicates that the worker is moving (for example, walking), it is assumed that the worker is not working. Therefore, in such a case, even if the gesture recognition device 26 or the voice recognition device 27 recognizes the input pattern indicating switching from a moving image to a still image, the arithmetic processing unit 24 will not be able to switch from a moving image to a still image. It is not necessary to output the signal to the arithmetic processing section 14.

Further, in the above description, the case where the gesture recognition device 26 and the voice recognition device 27 are provided exclusively for switching between the moving image state and the still image state has been mainly assumed. However, the gesture recognition device and voice recognition device provided to realize functions other than switching between the video state and the still image state (for example, disconnecting communication, changing the volume, etc.) The present invention may also be used as a gesture recognition device 26 and a voice recognition device 27 for switching between the image state and the image state.

1 Remote work support system 11 AR display 12 Camera 13 Position and orientation measurement unit 14 Arithmetic processing unit 15 Speaker 16 Voice processing unit 21 Display 22 Gesture input device 24 Arithmetic processing unit 25 Microphone 26 Gesture recognition device 27 Voice recognition device 30 Network

Claims (10)

a recognition unit that recognizes a specific input pattern from the first sensor data;
a switching signal output unit that outputs a switching signal between a still image and a moving image to a worker-side device based on the recognition of the input pattern;
an image display control unit that controls the display of gestures on the display based on gesture information of the instructor, and controls the display of still images or moving images transmitted from the worker side device on the display based on the switching signal; and,
An instructor-side device comprising: The first sensor data includes a gesture,
the specific input pattern includes a first gesture pattern;
The instructor side device according to claim 1. The first sensor data includes audio,
the specific input pattern includes a specific audio pattern;
The instructor side device according to claim 1 or 2. The recognition unit recognizes a second gesture pattern from the gesture of the instructor,
The switching signal output unit does not output the switching signal to the worker side device when the second gesture pattern is recognized, even if the specific voice pattern is recognized.
The instructor side device according to claim 3. The recognition unit recognizes a third gesture pattern from the gesture of the instructor,
The switching signal output unit does not output the switching signal to the worker side device when the third gesture pattern is not recognized, even if the specific voice pattern is recognized.
The instructor side device according to claim 3. The switching signal output unit outputs a switching signal from a still image to a moving image to the worker side device based on recognition of an input pattern indicating switching from a still image to a moving image.
The instructor-side device according to any one of claims 1 to 5. The switching signal output unit outputs a switching signal from a moving image to a still image to the worker side device based on recognition of an input pattern indicating switching from a moving image to a still image.
The instructor-side device according to any one of claims 1 to 6. When the worker's behavior recognized from the second sensor data indicates that the worker is moving, the switching signal output unit determines that an input pattern indicating switching from a moving image to a still image has been recognized. Also, a switching signal from a moving image to a still image is not output to the worker side device.
The instructor side device according to claim 7. recognizing a specific input pattern from the first sensor data;
outputting a switching signal between a still image and a moving image to a worker-side device based on the recognition of the input pattern;
controlling the display of the gesture on the display based on gesture information of the instructor, and controlling the display of the still image or moving image transmitted from the worker side device on the display based on the switching signal;
A method of providing. computer,
a recognition unit that recognizes a specific input pattern from the first sensor data;
a switching signal output unit that outputs a switching signal between a still image and a moving image to a worker-side device based on the recognition of the input pattern;
an image display control unit that controls the display of gestures on the display based on gesture information of the instructor, and controls the display of still images or moving images transmitted from the worker side device on the display based on the switching signal; and,
A program that functions as an instructor-side device equipped with.

Images