JP2019159936A - First information processing system, second information processing system and third information processing system - Google Patents

Abstract

To provide a technique capable of more suitably generating instruction information from a first user to a second user by grasping the first user's intention.SOLUTION: A first information processing system is provided, which includes an estimation part for estimating the type of an instruction operation of a first user on the basis of the position of the first user's hand part in a three-dimensional space measured on the basis of a first image and an area set in each type of an instruction operation in advance, and an instruction information generation part for generating instruction information to a second user on the basis of the type of the first user's instruction operation.SELECTED DRAWING: Figure 3

Description

  The present invention relates to a first information processing system, a second information processing system, and a third information processing system.

  In recent years, techniques for detecting user actions are known. For example, an apparatus having both a function of detecting a contact position on a touch panel by a user and a function of detecting a user's hand shape by non-contact is disclosed (for example, refer to Patent Document 1). According to such an apparatus, the contact position on the touch panel of the user and the hand shape of the user are associated, and the same operation can be performed by either the touch on the touch panel or the hand shape based on the association.

International Publication No. 2013/136776

  However, it is also assumed that a certain user (first user) wants to notify instruction information to another user (second user) by an operation. In such a case, various types of instruction information are assumed. For example, there are cases where it is desired to notify the position as instruction information, and there are cases where it is desired to notify the direction. That is, it is useful to know the intention of the first user (what kind of instruction information the first user wants to notify the second user) and generate instruction information according to the intention of the first user. It is.

  Therefore, it is desired to provide a technique that enables the first user to understand the intention of the first user and more appropriately generate instruction information from the first user to the second user.

  In order to solve the above-described problem, according to an aspect of the present invention, the position of the hand of the first user in the three-dimensional space measured based on the first image and the type of instruction operation are set in advance. And an instruction for generating instruction information for the second user based on the type of instruction operation of the first user based on the type of instruction operation of the first user based on the type of instruction operation of the first user A first information processing system including an information generation unit is provided.

  The first information processing system may include a transmission unit that transmits the instruction information to a second information processing system used by the second user.

  The first information processing system presents a receiving unit that receives a second image from a second information processing system used by the second user, and presents the second image to the first user. An image output unit.

  The first information processing system may include an image output unit that presents the instruction information to the first user.

  When the position of the hand part belongs to the first area, the estimation unit estimates that the type of the instruction operation is the first type, and the position of the hand part is different from the first area. When belonging to the second area, the type of the instruction operation may be estimated to be a second type different from the first type.

  The first region may include a space from a position of the touch panel to a position separated by a predetermined length in front of the touch panel with reference to the position of the touch panel.

  The estimation unit estimates that the type of the pointing operation is the second type when the position of the hand unit belongs to the second region and the shape of the hand unit is a predetermined shape. May be.

  The predetermined shape may include a pointing shape.

  When the position of the hand portion belongs to a third region different from the first region and the second region, the estimation unit determines the type of the instruction operation as the first type and the second type. It may be estimated that the third type is different from the type.

  The instruction information generation unit generates first instruction information when the instruction operation type is estimated to be the first type, and the instruction operation type is different from the first type. If it is estimated that the second type, the second instruction information different from the first instruction information may be generated.

  When it is estimated that the type of the instruction operation is the first type, the instruction information generation unit may generate a touch position on the touch panel by the hand unit as the first instruction information. .

  The instruction information generation unit may generate, as the second instruction information, a direction indicated by the pointing of the hand part when it is estimated that the type of the instruction operation is the second type. .

  When the instruction information generation unit estimates that the instruction operation type is a third type different from the first type and the second type, the instruction information generation unit Third instruction information different from the second instruction information may be generated.

  The instruction information generation unit may generate the shape of the hand part as the third instruction information when the type of the instruction operation is estimated to be the third type.

  Further, according to another aspect of the present invention, the second user generated based on the position of the hand of the first user in the three-dimensional space and the region set in advance for each type of instruction operation is provided. A second information processing system is provided that includes a receiving unit that receives instruction information and an image output unit that presents the instruction information to the second user.

  Moreover, according to another viewpoint of this invention, based on the position of the 1st user's hand part in the three-dimensional space measured based on the 1st image, and the area | region preset for every kind of instruction | indication operation | movement An estimation unit that estimates the type of instruction operation of the first user, an instruction information generation unit that generates instruction information to the second user based on the type of instruction operation of the first user, A second information processing system comprising: a first information processing system comprising: a receiving unit that receives the instruction information; and an image output unit that presents the instruction information to the second user. A third information processing system is provided.

  As described above, according to the present invention, there is provided a technique capable of grasping the intention of the first user and more appropriately generating instruction information from the first user to the second user. .

It is a figure for demonstrating a general remote work assistance system. It is a figure showing an example of composition of a remote work support system (work) concerning an embodiment of the present invention. It is a figure showing an example of composition of a remote work support system (finger) concerning the embodiment. It is a figure for demonstrating the position of an instructor's hand part measured by a hand part measurement part. It is a figure for demonstrating the position of an instructor's hand part measured by a hand part measurement part. It is a top view of a remote work support system (finger). It is a front view of the image | video output part (finger) in a remote work assistance system (finger). It is a block diagram which shows the function structural example of the remote operation assistance system which concerns on the embodiment. It is a figure which shows the example of a mode that a position instruction | indication operation | movement is performed by the instructor. It is a figure which shows the example of a mode when the kind of instruction | indication operation | movement by an instruction | indication is estimated that it is a position instruction | indication operation | movement, and the instruction | indication person is not contacting the contact information acquisition part. It is a figure which shows the example of a mode when the kind of instruction | indication operation | movement by an instructor is estimated that it is a position instruction | indication operation | movement, and the instructor is contacting the contact information acquisition part. It is a figure which shows the example of a mode that a direction instruction | indication operation | movement is performed by the instruction person. It is a figure which shows the example of a mode when the pointing person performs the pointing to left direction as an example of direction instruction | indication operation | movement. It is a figure for demonstrating the case where a direction is expressed by the hand part of an instructor. It is a figure which shows the example of a mode that shape instruction | indication operation | movement is performed by the instructor. It is a figure which shows the example of a mode when the gesture which spreads both hands as an example of shape instruction | indication operation | movement is performed by the instruction person. It is a flowchart which shows the operation example of the remote operation assistance system which concerns on the embodiment. It is a figure for demonstrating the 1st modification of the display object which shows a direction. It is a figure for demonstrating the 2nd modification of the display object which shows a direction. It is a figure for demonstrating the 2nd modification of the display object which shows a direction. It is a figure for demonstrating the 3rd modification of the display object which shows a direction. It is a figure for demonstrating the 3rd modification of the display object which shows a direction.

  Exemplary embodiments of the present invention will be described below in detail with reference to the accompanying drawings. In addition, in this specification and drawing, about the component which has the substantially same function structure, duplication description is abbreviate | omitted by attaching | subjecting the same code | symbol.

  In the present specification and drawings, a plurality of constituent elements having substantially the same functional configuration may be distinguished by attaching different numerals to the same reference numerals. However, when there is no need to particularly distinguish each of a plurality of constituent elements having substantially the same functional configuration, only the same reference numerals are given. In addition, similar components in different embodiments may be distinguished by attaching different alphabets after the same reference numerals. However, in the case where it is not necessary to particularly distinguish each of similar components of different embodiments, only the same reference numerals are given.

(1. Overview)
First, the outline | summary which concerns on embodiment of this invention is demonstrated.

  In general, when a person explains something to a nearby person, not only by words, but also by gestures and other actions (including hand posture and movement) There are many. In particular, when a person (instructor) explains work to a nearby person (worker), the explanation by hand movement is more information than the explanation by words. Is easy to explain the work and easy for the operator to understand the work. On the other hand, in recent years, an instruction (information support) is being given by an instructor from a remote place to an on-site worker by a remote work support system.

  FIG. 1 is a diagram for explaining a general remote work support system. Referring to FIG. 1, an operator U91 exists on the operator side, and an instructor U92 exists on the instructor side (corresponding to a remote place from the operator side). The worker U91 uses the worker terminal 91, and the instructor U92 uses the instructor terminal 92. The worker terminal 91 and the instructor terminal 92 can communicate with each other via the network 90. Although FIG. 1 shows a case where the worker terminal 91 is a tablet terminal, the worker terminal 91 is also assumed to be a wearable terminal or the like. 1 shows a case where the instructor terminal 92 is a PC (Personal Computer), but various types of the instructor terminal 92 are assumed.

  For example, the worker terminal 91 includes a camera, and transmits an image (worker side image) captured by the camera to the instructor terminal 92. As a result, the worker and the instructor can see the same worker-side video (the same worker-side video is shared). The instructor grasps the situation on the worker side through the worker-side video, and inputs instruction information to the worker by voice or a drawing line on the worker-side video. The instruction information is transmitted from the instructor terminal 92 to the worker terminal 91. As a result, the instructor U92 can explain the work to the worker U91 from a remote place.

  As described above, in the general remote work support system as shown in FIG. 1, an instruction can be given by operations such as drawing a line and voice input. However, an instruction from the instructor U92 to the operator U91 by an instruction operation (for example, gesture by hand, pointing, etc.) that can be performed when the instructor U92 and the operator U91 are close to each other is not possible. Can not. Therefore, for example, the instructor U92 has to explain the work using many words, so the instructor U92 is difficult to explain and the operator U91 is difficult to receive explanation (both the instructor U92 and the worker U91) Difficult to share instruction information).

  In order to facilitate explanation, the instructor U92 may want to notify the operator U91 of the shape of the hand part directly as instruction information by a gesture by the hand part. Alternatively, the instructor U92 may want to notify the worker U91 of the direction as instruction information by pointing. Thus, various types of instruction information that the instructor U92 wants to notify are assumed. Therefore, it is useful to grasp the intention of the instructor U92 (what kind of instruction information the instructor U92 wants to notify the worker U91) and generate the instruction information according to the intention of the instructor U92.

  Therefore, in the embodiment of the present invention, a technique for grasping the intention of the instructor U92 and generating instruction information from the instructor U92 to the worker U91 more appropriately will be mainly described. In the following, it is mainly assumed that the instructor U92 gives an instruction regarding work to the worker U91. However, the worker U91 may not actually perform the work, and the instructor U92 may give an instruction related to other than the work. That is, the “instructor” described below may be a “first user” who gives some instruction, and the “worker” described below receives an instruction from the “first user” “second user”. User ".

  Hereinafter, the information processing system (first information processing system) used by the “instructor” is mainly expressed as “remote work support system (finger)”. The information processing system (second information processing system) used by the “worker” is mainly expressed as “remote work support system (work)”. Furthermore, an information processing system (third information processing system) in which the first information processing system and the second information processing system are integrated is mainly expressed as a “remote work support system”.

  Similarly, in the following description, there is a case where a character string “finger” is added to the end of the component used by the “instructor”. Further, there may be a case where a character string “saku” is added to the end of the component used by the “worker” for explanation. Thus, even if there is a component having the same name between the component used by the “instructor” and the component used by the “worker”, these components can be distinguished.

  The outline according to the embodiment of the present invention has been described above.

(2. System configuration example)
Subsequently, a configuration example of the remote operation support system according to the embodiment of the present invention will be described.

  FIG. 2 is a diagram showing a configuration example of a remote work support system (work) according to the embodiment of the present invention. As shown in FIG. 2, the remote work support system (work) 10 according to the embodiment of the present invention is a system used by a worker U1, and a remote work support system (finger) 20 via a network 30. (FIG. 3) can be communicated. The remote work support system (work) 10 includes a remote work support device (work) 11, and the remote work support device (work) 11 includes a video acquisition unit 101, a video output unit (work) 102, and audio. An input / output unit (operation) 103 is provided.

  The video acquisition unit 101 is configured by an image sensor, and acquires a video (worker side video) by imaging with the image sensor. In the embodiment of the present invention, it is assumed mainly that the video acquisition unit 101 is provided on the back surface of the remote work support device (work) 11. However, the position where the video acquisition unit 101 is provided is not limited. For example, the video acquisition unit 101 may be provided on the front surface of the remote work support device (work) 11 or may be provided on the side surface of the remote work support device (work) 11.

  The video output unit (product) 102 is configured by a display and has a function of outputting the video (worker side video) obtained by the video acquisition unit 101. The video output unit (product) 102 also has a function of outputting instruction information received from the remote work support system (finger) 20 (FIG. 3). Here, the form of the display is not particularly limited. For example, the display may be a CRT (Cathode Ray Tube) display device, a liquid crystal display (LCD) device, an OLED (Organic Light Emitting Diode) device, a lamp, etc. The display device may be used.

  The voice input / output unit (product) 103 includes a microphone and a speaker. The microphone detects the sound emitted by the worker U1. On the other hand, the speaker outputs the voice of the instructor U2 received from the remote work support system (finger) 20 (FIG. 3). In the example shown in FIG. 2, for convenience, a voice input / output unit (work) 103 including a microphone and a speaker is provided in one place. However, the microphone and the speaker may be provided at remote locations. When the remote work support device (product) 11 is a wearable terminal that can be worn on the body of the worker U1, the voice input / output unit (product) 103 may be configured as a headset.

  In addition, the remote work support device (work) 11 has a control unit (not shown). The control unit (not shown) includes a CPU (Central Processing Unit) and the like, and the function is realized by a program stored in the storage unit (not shown) being expanded and executed by the CPU in a RAM (Random Access Memory). obtain. At this time, a computer-readable recording medium that records the program can also be provided. Or the control part which is not illustrated may be comprised by the hardware for exclusive use, and may be comprised by the combination of several hardware.

  Such a storage unit (not shown) is a storage device capable of storing a program and data for operating a control unit (not shown). Further, the storage unit (not shown) can temporarily store various data required in the process of operation of the control unit (not shown). For example, the storage device may be a non-volatile storage device.

  Further, the remote work support device (work) 11 includes a communication unit (work) 104 (FIG. 8). The communication unit (work) 104 may be a communication interface that performs communication with the remote work support system (finger) 20 via the network 30. In the embodiment of the present invention, it is mainly assumed that the communication unit (work) 104 performs wired communication, but the communication unit (work) 104 may perform wireless communication.

  In the embodiment of the present invention, as shown in FIG. 2, the remote work support device (production) 11 includes a video acquisition unit 101, a video output unit (production) 102, and an audio input / output unit (production) 103. Is mainly assumed. However, some or all of the video acquisition unit 101, the video output unit (production) 102, and the audio input / output unit (production) 103 may exist outside the remote work support device (production) 11.

  In the embodiment of the present invention, it is mainly assumed that the remote work support device (work) 11 is a tablet terminal. However, the remote work support device (work) 11 is not limited to a tablet terminal. For example, the remote work support device (product) 11 may be a smartphone or a PC (Personal Computer). Alternatively, the remote work support device (work) 11 may be a wearable terminal that can be worn on the body of the worker U1, or may be another terminal.

  For example, the worker U1 is going to do some work. The type of work performed by the worker U1 is not limited. In the embodiment of the present invention, it is mainly assumed that the work performed by the worker U1 is an installation work of an electrical appliance. In such a case, the video acquisition unit 101 is directed to the electrical appliance by the worker U1, the video of the electrical appliance is captured by the video acquisition unit 101, and the video of the electrical appliance is used as the instructor-side video by the remote work support device (product) 11. Sent to the instructor. Thereby, the instructor U2 (FIG. 3) can give an instruction by an instruction operation while viewing the video of the appliance.

  When the remote work support device (product) 11 receives the instruction information from the instructor U2, and the video output unit (product) 102 displays a display object corresponding to the instruction information, the worker U1 displays the display object. It is possible to work while watching. Thereby, the worker U1 can work efficiently. Note that the voice uttered by the worker U1 received by the voice input / output unit 103 is transmitted to the instructor side, and the voice uttered by the instructor U2 is received from the instructor side and received by the voice input / output unit ) 103. Such voice input / output can also support efficient work.

  The configuration example of the remote work support system (work) according to the embodiment of the present invention has been described above.

  FIG. 3 is a diagram illustrating a configuration example of the remote operation support system (finger) 20 according to the embodiment of the present invention. As shown in FIG. 3, the remote work support system (finger) 20 according to the embodiment of the present invention is a system used by the instructor U <b> 2, and via the network 30, the remote work support system (work) 10. (FIG. 2) can be communicated with. The remote work support system (finger) 20 includes a remote work support device (finger) 21, a hand measurement unit 201, a contact information acquisition unit 203, a video output unit (finger) 205, and a voice input / output unit (finger) 206. Have.

  The hand measurement unit 201 includes, for example, an infrared LED (Light Emitting Diode) and an infrared stereo camera. At this time, the hand measurement unit 201 acquires an image (instructor-side image) by irradiating infrared rays with an infrared LED and capturing the reflected infrared rays with an infrared stereo camera. Further, the hand measurement unit 201 measures the position of the hand of the instructor U2 in the three-dimensional space based on the instructor side image. In the embodiment of the present invention, it is mainly assumed that the hand measurement unit 201 is provided obliquely forward above the instructor U2. However, the position where the hand measurement unit 201 is provided is not limited. For example, the hand measurement unit 201 may be provided above the front of the instructor U2, or may be provided obliquely above and behind the instructor U2.

  Here, the position of the hand of the instructor U2 measured by the hand measuring unit 201 will be described. FIG. 4 is a diagram for explaining the position of the hand of the instructor U2 measured by the hand measurement unit 201. Referring to FIG. 4, the left hand Lh of the instructor U <b> 2 and the right hand Rh of the instructor U <b> 2 exist in the measurement target area by the hand measuring unit 201. As shown in FIG. 4, the hand measurement unit 201 measures the positions (three-dimensional positions) of the vertices of the left hand Lh and the right hand Rh in a three-dimensional space (that is, “skeleton model information” is measured). )

  Specifically, the vertices whose positions are measured by the hand measurement unit 201 include fingertips, finger joints, polygonal vertices constituting the palm, the center of gravity of the vertex group constituting the palm, bending / extension information for each of the five fingers, etc. Can be included. In FIG. 4, the tip P1 of the thumb of the right hand Rh is shown as an example of the vertex.

  FIG. 5 is a diagram for explaining the position of the hand of the instructor U2 measured by the hand measurement unit 201. Referring to FIG. 5, the right hand Rh of the instructor U <b> 2 exists in the measurement target area by the hand measurement unit 201. That is, the measurement target region shown in FIG. 5 corresponds to an enlarged view of the measurement target region shown in FIG.

  As shown in FIG. 5, the hand measurement unit 201 measures the position (three-dimensional position) in the three-dimensional space of the center of gravity RhP of each vertex of the right hand Rh. Similarly, the position (three-dimensional position) of the center of gravity of each vertex of the left hand Lh in the three-dimensional space is also measured. In the embodiment of the present invention, the centroid of the vertex group constituting the palm of the right hand Rh and the centroid of the vertex group constituting the palm of the left hand Lh are used as examples of the position of the hand portion. Referring to FIG. 5, the diameter r of the circumscribed circle of the palm is shown. The diameter r of the circumscribed circle of the palm may be used for determining the size of the region as will be described later.

  Returning to FIG. 3, the description will be continued. The video output unit (finger) 205 is configured by a display and has a function of outputting video (worker side video) received from the remote work support system (work) 10. The video output unit (product) 102 also has a function of outputting instruction information generated by the remote work support device (finger) 21. Here, the form of the display is not particularly limited. For example, the display may be a CRT (Cathode Ray Tube) display device, a liquid crystal display (LCD) device, an OLED (Organic Light Emitting Diode) device, a lamp, etc. The display device may be used.

  The voice input / output unit (finger) 206 includes a microphone and a speaker. The microphone detects the voice emitted by the instructor U2. On the other hand, the speaker outputs the voice of the worker U1 received from the remote work support system (work) 10. In the example shown in FIG. 3, for convenience, a voice input / output unit (finger) 206 including a microphone and a speaker is provided in the same housing. However, the microphone and the speaker may be provided in different housings. Further, the position where the voice input / output unit (finger) 206 is provided is not limited.

  The contact information acquisition unit 203 is configured by a touch panel, and detects a touch position on the touch panel by the instructor U2. For example, the touch panel method may be a capacitance method, but is not particularly limited. In the embodiment of the present invention, it is mainly assumed that the remote work support system (finger) 20 has a contact information acquisition unit 203 (touch panel). However, the remote work support system (finger) 20 may have another input device that can detect the position indicated by the instructor U2 instead of the contact information acquisition unit 203 (touch panel). For example, when an icon exists at a position where the instructor U2 touches the touch panel, a function corresponding to the icon may be executed.

  The remote work support device (finger) 21 has a control unit (not shown). The control unit (not shown) includes a CPU (Central Processing Unit) and the like, and the function is realized by a program stored in the storage unit (not shown) being expanded and executed by the CPU in a RAM (Random Access Memory). obtain. At this time, a computer-readable recording medium that records the program can also be provided. Or the control part which is not illustrated may be comprised by the hardware for exclusive use, and may be comprised by the combination of several hardware.

  Such a storage unit (not shown) is a storage device capable of storing a program and data for operating a control unit (not shown). Further, the storage unit (not shown) can temporarily store various data required in the process of operation of the control unit (not shown). For example, the storage device may be a non-volatile storage device.

  The remote work support device (finger) 21 includes a communication unit (finger) 207 (FIG. 8). The communication unit (finger) 207 may be a communication interface that performs communication with the remote work support system (work) 10 via the network 30. In the embodiment of the present invention, it is mainly assumed that the communication unit (finger) 207 performs wired communication, but the communication unit (finger) 207 may perform wireless communication.

  In the embodiment of the present invention, as shown in FIG. 3, the hand measurement unit 201, the contact information acquisition unit 203, the video output unit (finger) 205, and the voice input / output unit (finger) 206 are remote. The case where it exists in the exterior of the work assistance apparatus (finger) 21 is mainly assumed. However, some or all of the hand measurement unit 201, the contact information acquisition unit 203, the video output unit (finger) 205, and the voice input / output unit (finger) 206 exist inside the remote work support device (finger) 21. May be.

  For example, the instructor U2 tries to give an instruction to the worker U1 in order to support the work performed by the worker U1. In the embodiment of the present invention, it is mainly assumed that the support performed by the instructor U2 is support for the installation work of the electrical appliance. In this case, when the video of the electrical appliance is received from the worker side by the remote work support device (finger) 21, the video of the electrical appliance is displayed by the video output unit (finger) 205. Thereby, the instructor U2 can give an instruction by an instruction operation while viewing the video of the appliance.

  The type of instruction operation by the instructor U2 is not limited. In the embodiment of the present invention, as the instruction operation, an instruction operation for instructing a position (hereinafter also referred to as “position instruction operation”) and an instruction operation for instructing a direction (hereinafter also referred to as “direction instruction operation”). )), A case where three of the instruction operations for instructing the shape (hereinafter also referred to as “shape instruction operation”) are used (the three instruction operations are different from each other). However, any two of the three instruction operations (from the first type to the third type) may be used as the instruction operation.

  In the embodiment of the present invention, the touch position of the hand touching the touch panel is used as the position indicated by the position indicating operation, and the direction (pointing direction) indicated by the pointing hand is used as the direction specified by the direction indicating operation. Assume that a hand gesture (shape of the hand) is used as the shape indicated by the shape indicating operation. However, there is no particular limitation on what information is used as the position indicated by the position indicating operation, the direction specified by the direction indicating operation, and the shape specified by the shape indicating operation.

  In the embodiment of the present invention, a region is set in advance for each type of instruction operation (each region is different from each other). Here, the area set for each type of instruction operation is preferably an area where the instruction operation is predicted to be performed by the instructor U2, and is automatically set by the remote work support system (finger) 20. It is better. For example, these areas may be executed as one of initial processes when the remote work support system (finger) 20 is installed. Here, examples of the position, size, and shape of each region will be described with reference to FIGS. 6 and 7.

  FIG. 6 is a top view of the remote work support system (finger) 20. FIG. 7 is a front view of the video output unit (finger) 205 in the remote work support system (finger) 20. Referring to FIGS. 6 and 7, a position indicating region R1 (first region), a direction indicating region R2 (second region), and a shape indicating region R3 (third region) are shown.

  The position indicating region R1 is a region in which the position of the hand portion is predicted to belong when a position indicating operation is performed by the instructor U2, that is, when the instructor U2 touches the contact information acquisition unit 203 (touch panel). It is. Specifically, as illustrated in FIGS. 6 and 7, the position indication region R1 acquires contact information from the position of the contact information acquisition unit 203 (touch panel) with reference to the position of the contact information acquisition unit 203 (touch panel). A space (a rectangular parallelepiped) up to a position separated by a predetermined length L1 may be included in front of the unit 203 (touch panel). However, the position and shape of the position indicating region R1 are not limited. The length L1 may be approximately the same as the diameter r of the circumscribed circle of the palm, but is not limited.

  The direction indicating area R2 is an area where the position of the hand portion is predicted to belong when a direction indicating operation is performed by the instructor U2, that is, when pointing is performed by the instructor U2. Specifically, as shown in FIG. 6 and FIG. 7, the direction indicating region R <b> 2 is a spherical interior centered on the intersection of the imaging direction of the hand measurement unit 201 and the front direction of the video output unit (finger) 205. However, the position and shape of the direction indicating region R2 are not limited. Further, the diameter L2 of the direction indicating region R2 may be approximately the same as the diameter r of the circumscribed circle of the palm, but is not limited.

  The shape instruction region R3 is a region where the position of the hand portion is predicted to belong when a shape instruction operation is performed by the instructor U2, that is, when a gesture is performed by the instructor U2. As illustrated in FIG. 6, the shape instruction area R3 may be an area after the position instruction area R1 and the direction instruction area R2 are excluded from the measurement target area R by the hand measurement unit 201. That is, the position, size, and shape of the shape indicating region R3 are not limited, as are the position, size, and shape of the position indicating region R1 and the direction indicating region R2.

  When the position of the hand of the instructor U2 is measured by the hand measuring unit 201, the remote work support device (finger) 21 determines the position of the hand, the position indicating area R1, the direction indicating area R2, and the shape indicating area R3. Based on the above, the type of instruction operation of the instructor U2 (intention of the instructor U2) is estimated. Then, the remote work support device (finger) 21 generates instruction information to the worker U1 based on the type of instruction operation of the instructor U2. The remote work support device (finger) 21 transmits instruction information for the worker U1 to the remote work support device (work) 11. Thereby, the instructor U2 can efficiently support the work.

  The voice uttered by the instructor U2 received by the voice input / output unit (finger) 206 is transmitted to the worker side, and the voice uttered by the worker U1 is received from the worker side and received by the voice input / output unit (finger ) 206 is output. Such voice input / output can also support efficient work.

  The configuration example of the remote work support system (finger) according to the embodiment of the present invention has been described above.

(3. Functional configuration example of remote work support system)
Subsequently, a functional configuration example of the remote operation support system 1 according to the embodiment of the present invention will be described. FIG. 8 is a block diagram illustrating a functional configuration example of the remote operation support system 1 according to the embodiment of the present invention. As shown in FIG. 8, the remote work support system 1 according to the embodiment of the present invention includes a remote work support system (work) 10 and a remote work support system (finger) 20.

  The remote work support system (production) 10 includes a video acquisition unit 101, a video output unit (production) 102, a voice input / output unit (production) 103, and a communication unit (production) 104. The remote operation support system (finger) 20 includes a hand measurement unit 201, a hand analysis unit 202 (estimation unit), a contact information acquisition unit 203, an instruction information generation unit 204, a video output unit (finger) 205, a voice input unit, An output unit (finger) 206 and a communication unit (finger) 207 are included. The communication unit (work) 104 and the communication unit (finger) 207 are configured to be communicable via the network 30. In the following description, FIG. 8 will be referred to as appropriate.

  The function configuration example of the remote operation support system 1 according to the embodiment of the present invention has been described above.

(4. Detailed examples of remote work support system functions)
Next, functional details of the remote operation support system 1 according to the embodiment of the present invention will be described.

(4-1. When position indicating operation is performed)
First, a case where the position instruction operation is performed by the instructor U2 will be described with reference to FIGS. FIG. 9 is a diagram illustrating an example of a state in which a position indicating operation is performed by the instructor U2. Referring to FIG. 9, a state where the instructor U2 is looking at the video output unit (finger) 205 is shown. Here, the worker video (second image) is received by the communication unit (finger) 207, the worker side video is displayed by the video output unit (finger) 205, and the instructor U2 is watching the worker video. Assume a case.

  For example, it is assumed that the instructor U2 wants to instruct the worker U1 of the position in the worker-side video. In such a case, the hand U touches the contact information acquisition unit 203 (touch panel) by the instructor U2. At this time, as shown in FIG. 9, the position of the hand of the instructor U2 is predicted to belong to the position indicating region R1 corresponding to the position indicating operation. Therefore, the hand measurement unit 201 captures a video (first image) of the measurement target region, and measures the position of the hand of the instructor U2 in the three-dimensional space based on the captured video. Here, as an example, it is assumed that the position of the right hand of the instructor U2 is measured.

  Subsequently, based on the position of the hand of the instructor U2 measured by the hand measuring unit 201 and the position indicating region R1, the direction indicating region R2, and the shape indicating region R3, the hand analyzing unit 202 determines the instruction U2 Estimate the type of instruction action. Then, the instruction information generation unit 204 generates instruction information for the worker U1 based on the type of instruction operation of the instructor U2. Specifically, as shown in FIG. 9, when the position of the hand of the instructor U2 belongs to the position instruction area R1, the hand analysis unit 202 specifies the type of instruction operation by the instructor U2 as a position instruction. Presumed to be motion.

  The instruction information generation unit 204 generates instruction information (first instruction information) corresponding to the position instruction operation when it is estimated that the type of instruction operation by the instructor U2 is the position instruction operation. More specifically, when it is estimated that the type of the instruction operation by the instructor U2 is the position instruction operation, the instruction information generation unit 204 proceeds to the contact information acquisition unit 203 (touch panel) by the hand of the instructor U2. Is generated as instruction information (first instruction information).

  The communication unit (finger) 207 uses the hand of the instructor U2 as a contact position to the contact information acquisition unit 203 (touch panel) as instruction information (first instruction information) via the network 30 to make a remote work support system (operation). ) Send to 10. In the remote work support system (work) 10, the communication unit (work) 104 receives the contact position as instruction information (first instruction information) via the network 30. Then, the video output unit (product) 102 superimposes and displays a display object corresponding to the instruction information on the worker-side video (presents the worker U1).

  FIG. 10 shows an example of a state in which it is estimated that the type of the pointing operation by the instructor U2 is a position indicating operation, and when the instructor U2 is not in contact with the contact information acquisition unit 203 (touch panel). FIG. As illustrated in FIG. 10, it is assumed that the type of the instruction operation is estimated to be a position instruction operation and that the instructor U2 is not in contact with the contact information acquisition unit 203 (touch panel). In such a case, the position of the foot of the perpendicular line (hereinafter referred to as “proximity”) from the vertex closest to the contact information acquisition unit 203 (touch panel) among the vertexes of the hand of the instructor U2 to the contact information acquisition unit 203 (touch panel). Is also transmitted to the remote work support system (work) 10 via the network 30.

  In the remote work support system (work) 10, the communication unit (work) 104 receives the proximity position via the network 30. Then, as shown in FIG. 10, the video output unit (product) 102 superimposes and displays the display object B11 at a close position in the worker-side video (presented to the worker U1). The worker U1 can confirm the position instructed by the instructor U2 based on the position of the display object B11, and can grasp a sign that the instructor U2 gives an instruction. In the example shown in FIG. 10, the display object B11 is indicated by a circle (circular pointer), but the shape of the display object B11 is not particularly limited.

  As shown in FIG. 10, not only the video output unit (product) 102 in the remote work support system (work) 10 but also the video output unit (finger) 205 in the remote work support system (finger) 20 The display object B11 may be superimposed and displayed at a close position in the person-side video (may be presented to the instructor U2). Then, since the instructor U2 can see the display object B11 at the same position as the position of the display object B11 that the operator U1 is viewing, it is possible to confirm the position instructed by the operator U1. It becomes.

  FIG. 11 shows an example of a state in which it is estimated that the type of the pointing operation by the instructor U2 is a position indicating operation, and the instructor U2 is in contact with the contact information acquisition unit 203 (touch panel). FIG. As illustrated in FIG. 11, it is assumed that the type of instruction operation is estimated to be a position instruction operation and that the instructor U2 is in contact with the contact information acquisition unit 203 (touch panel). In this case, the contact position on the contact information acquisition unit 203 (touch panel) by the instructor U2 is transmitted to the remote work support system (work) 10 via the network 30.

  In the remote work support system (work) 10, the communication unit (work) 104 receives the contact position via the network 30. Then, as shown in FIG. 11, the video output unit (product) 102 superimposes and displays the display object B12 on the contact position in the worker-side video (presented to the worker U1). The worker U1 can confirm the position instructed by the instructor U2 based on the position of the display object B12, and can work efficiently. In the example shown in FIG. 11, since the instructor U2 moves the hand while making contact with the contact information acquisition unit 203 (touch panel), the video output unit (product) 102 moves from the movement start position to the movement end position. The trajectory is also displayed.

  As shown in FIG. 11, not only the video output unit (work) 102 in the remote work support system (work) 10 but also the video output unit (finger) 205 in the remote work support system (finger) 20 The display object B12 may be superimposed and displayed at the contact position in the person-side video (presented to the instructor U2). Then, since the instructor U2 can see the display object B12 at the same position as the position of the display object B12 that the operator U1 is looking at, it is possible to confirm the position instructed by the operator U1. It becomes.

(4-2. When direction instruction operation is performed)
Next, a case where the direction instruction operation is performed by the instructor U2 will be described with reference to FIGS. FIG. 12 is a diagram illustrating an example in which a direction instruction operation is performed by the instructor U2. Referring to FIG. 12, a state where the instructor U2 is looking at the video output unit (finger) 205 is shown. Also here, the worker video (second image) is received by the communication unit (finger) 207, the worker side video is displayed by the video output unit (finger) 205, and the instructor U2 is watching the worker video. Assume a case.

  For example, it is assumed that the instructor U2 wants to instruct the worker U1 in the direction in the worker-side video. In such a case, pointing is performed by the instructor U2. At this time, as shown in FIG. 12, the position of the hand of the instructor U2 is predicted to belong to the direction indicating region R2 corresponding to the direction indicating operation. Therefore, the hand measurement unit 201 captures a video (first image) of the measurement target region, and measures the position of the hand of the instructor U2 in the three-dimensional space based on the captured video. Here, as an example, a case where the position of the right hand of the instructor U2 is measured is assumed.

  Subsequently, based on the position of the hand of the instructor U2 measured by the hand measuring unit 201 and the position indicating region R1, the direction indicating region R2, and the shape indicating region R3, the hand analyzing unit 202 determines the instruction U2 Estimate the type of instruction action. Then, the instruction information generation unit 204 generates instruction information for the worker U1 based on the type of instruction operation of the instructor U2. Specifically, as shown in FIG. 12, when the position of the hand of the instructor U2 belongs to the direction indicating area R2, the hand analyzing unit 202 indicates the direction of the instruction operation performed by the instructor U2. Presumed to be motion.

  The instruction information generation unit 204 generates instruction information (second instruction information) corresponding to the direction instruction operation when the type of instruction operation by the instructor U2 is estimated to be the direction instruction operation. More specifically, when it is estimated that the type of the instruction operation by the instructor U2 is the direction instruction operation, the instruction information generation unit 204 performs an instruction based on the position of each vertex of the hand of the instructor U2. The direction indicated by the pointing by the hand of the person U2 is detected, and the direction indicated by the pointing is generated as instruction information (second instruction information).

  Alternatively, just because the position of the hand of the instructor U2 belongs to the direction indicating region R2, the shape of the hand is not necessarily the pointing shape. Therefore, when the position of the hand of the instructor U2 belongs to the direction indicating area R2, the instruction information generation unit 204 changes the shape of the hand of the instructor U2 based on the position of each vertex of the hand of the instructor U2. You may recognize it. When the shape of the hand of the instructor U2 is not the pointing shape, the instruction information generation unit 204 does not generate instruction information (second instruction information) corresponding to the direction instruction operation, and the instruction information U2 When the shape of the hand portion is a pointing shape, instruction information (second instruction information) corresponding to the direction instruction operation may be generated.

  Here, the pointing operation is described as an example of the pointing operation, but the pointing operation may not be a pointing operation. That is, a predetermined direction indicating operation capable of indicating a direction may be used as the direction finger operation. At this time, a predetermined shape corresponding to a predetermined direction indicating operation may be used instead of the pointing shape.

  The communication unit (finger) 207 transmits the direction indicated by the pointing of the instructor U2 as instruction information (second instruction information) to the remote work support system (work) 10 via the network 30. In the remote work support system (work) 10, the communication unit (work) 104 receives the direction as instruction information (second instruction information) via the network 30. Then, the video output unit (product) 102 displays a display object corresponding to the instruction information.

  FIG. 13 is a diagram illustrating an example of a state in which a pointing in the left direction is performed by the instructor U2 as an example of the direction instruction operation. As shown in FIG. 13, a case is assumed in which a pointing in the left direction is performed by the instructor U2 as an example of the direction indicating operation. In such a case, the direction indicated by the pointing (left direction) is transmitted to the remote work support system (work) 10 via the network 30.

  In the remote work support system (work) 10, the communication unit (work) 104 receives the direction (left direction) indicated by the pointing device via the network 30. Then, as shown in FIG. 13, the video output unit (product) 102 superimposes and displays the display object B2 indicating the same direction as the direction indicated by the pointing (left direction) on the worker side video (to the worker U1). Present). The worker U1 can confirm the direction instructed by the instructor U2 based on the direction of the display object B2, and can work efficiently. In the example shown in FIG. 13, the display object B2 is indicated by an arrow. The advantage that the direction is expressed by an arrow will be described.

  FIG. 14 is a diagram for describing a case where the direction is expressed by the hand of the instructor U2. Referring to FIG. 14, the right direction is indicated by right hand Rh-1 and right hand Rh-2. Moreover, the left direction is shown by the right hand Rh-3 and the right hand Rh-4. However, since there is a limit to the range of motion of the hand, the shape of these hands is asymmetric with respect to the direction indicated by the hand. On the other hand, if the direction is indicated by a symmetrical figure such as an arrow, it is expected that the direction can be easily understood by the worker U1, and the burden of the instruction operation in the direction of the instructor U2 is expected to be reduced. The However, the shape of the display object B2 is not limited to the arrow.

  As shown in FIG. 13, not only the video output unit (product) 102 in the remote work support system (product) 10 but also the video output unit (finger) 205 in the remote work support system (finger) 20 is pointed. The display object B2 indicating the same direction as indicated by may be superimposed on the worker video (may be presented to the instructor U2). Then, since the instructor U2 can see the display object B2 indicating the same direction as the display object B2 that the operator U1 is looking at, the instructor U2 can confirm the direction instructed by the operator U1. It becomes possible.

(4-3. When shape instruction operation is performed)
Next, a case where the shape instructing operation is performed by the instructor U2 will be described with reference to FIGS. 15 and 16. FIG. 15 is a diagram illustrating an example in which the shape instructing operation is performed by the instructor U2. Referring to FIG. 15, a state where the instructor U2 is looking at the video output unit (finger) 205 is shown. Also here, the worker video (second image) is received by the communication unit (finger) 207, the worker side video is displayed by the video output unit (finger) 205, and the instructor U2 is watching the worker video. Assume a case.

  For example, assume that the instructor U2 wants to instruct the worker U1 of the shape. In such a case, a gesture is performed by the instructor U2. At this time, as shown in FIG. 15, the position of the hand of the instructor U2 is predicted to belong to the shape instruction region R3 corresponding to the shape instruction operation. Therefore, the hand measurement unit 201 captures a video (first image) of the measurement target region, and measures the position of the hand of the instructor U2 in the three-dimensional space based on the captured video. Here, as an example, a case where the position of the right hand of the instructor U2 is measured is assumed.

  Subsequently, based on the position of the hand of the instructor U2 measured by the hand measuring unit 201 and the position indicating region R1, the direction indicating region R2, and the shape indicating region R3, the hand analyzing unit 202 determines the instruction U2 Estimate the type of instruction action. Then, the instruction information generation unit 204 generates instruction information for the worker U1 based on the type of instruction operation of the instructor U2. Specifically, as shown in FIG. 15, when the position of the hand part of the instructor U2 belongs to the shape instruction area R3, the hand part analysis unit 202 specifies the type of instruction operation by the instructor U2 as a shape instruction. Presumed to be motion.

  Then, the instruction information generation unit 204 generates instruction information (third instruction information) corresponding to the shape instruction operation when it is estimated that the type of the instruction operation by the instructor U2 is the shape instruction operation. More specifically, the instruction information generation unit 204, when it is estimated that the type of instruction operation by the instructor U2 is a shape instruction operation, the hand of the instructor U2 measured by the hand measurement unit 201 ( The position of each vertex of the left hand and the right hand) is converted into a position seen in the direction facing the video output unit (finger) 205, and the position of each vertex after the conversion (the shape of the hand) is indicated in the instruction information (third (Instruction information).

  Alternatively, as described above, when the position of the hand of the instructor U2 belongs to the direction instruction area R2, the instruction information generation unit 204, based on the position of each vertex of the hand of the instructor U2, You may recognize the shape of a hand part. Then, when the shape of the hand of the instructor U2 is not the pointing shape, the instruction information generation unit 204 does not generate instruction information (second instruction information) corresponding to the direction instruction operation, and the instruction information U 204 The instruction information (third instruction information) may be generated by estimating that the type of instruction operation is a shape instruction operation.

  The communication unit (finger) 207 uses the position (the shape of the hand part) of each vertex of the hand part (left hand and right hand) of the instructor U2 (the shape of the hand part) as instruction information (third instruction information). To the remote work support system (work) 10. In the remote work support system (work) 10, the communication unit (work) 104 receives the shape of the hand part as instruction information (third instruction information) via the network 30. Then, the video output unit (product) 102 displays a display object corresponding to the instruction information.

  FIG. 16 is a diagram illustrating an example of a state in which a gesture with both hands extended is performed by the instructor U2 as an example of the shape instruction operation. As illustrated in FIG. 16, it is assumed that the instructor U2 performs a gesture with both hands extended as an example of the shape instruction operation. In such a case, the shape of the hand indicated by the gesture is transmitted to the remote work support system (work) 10 via the network 30.

  In the remote work support system (work) 10, the communication unit (work) 104 receives the shape of the hand part indicated by the gesture via the network 30. And as FIG. 16 shows, the video output part (production) 102 displays the display object B3 according to the same shape as the shape of the hand part shown by gesture (it shows to the operator U1). The worker U1 can confirm the shape instructed by the instructor U2 based on the shape of the display object B3, and can work efficiently. In the example shown in FIG. 16, the display object B3 is indicated by a hand fleshing at the position of each vertex of the hand portion. However, the display object B3 may be the position of each vertex of the hand.

  As shown in FIG. 16, not only the video output unit (product) 102 in the remote work support system (work) 10 but also the video output unit (finger) 205 in the remote work support system (finger) 20 is used for the gesture. The display object B3 corresponding to the same shape as the shape indicated by is preferably displayed (presented to the instructor U2). Then, the instructor U2 can see the display object B3 corresponding to the same shape as the shape of the display object B2 that the operator U1 is viewing, and therefore confirms the shape instructed to the operator U1. Is possible.

  The above mainly assumes the case where the position of the hand (right hand) of the instructor U2 is used for estimating the type of instruction operation of the instructor U2. However, when the instructor U2 is left-handed or the like, instead of the position of the hand (right hand) of the instructor U2, the position of the hand (left hand) of the instructor U2 is changed to the type of instruction operation of the instructor U2. It may be used for estimation. Alternatively, both the hand (right hand) and the hand (left hand) of the instructor U2 may be used for estimating the type of instruction operation of the instructor U2.

  At this time, for example, when at least one of the hand (right hand) and the hand (left hand) of the instructor U2 belongs to the position instruction area R1, the hand analysis unit 202 determines the type of instruction operation performed by the instructor U2. May be estimated as a position indicating operation. Further, the hand part analysis unit 202 has a predetermined shape when the hand part (right hand) and the hand part (left hand) of the instructor U2 belong to the direction instruction area R2 and the shape of the hand part belonging to the direction instruction area R2 is predetermined. The shape of the pointing operation by the instructor U2 may be estimated to be a direction indicating operation.

  In addition, the hand part analysis unit 202 indicates the type of instruction operation by the instructor U2 when at least one of the hand part (right hand) and the hand part (left hand) of the instructor U2 belongs to the shape instruction region R3. It may be estimated that the operation. In addition, the hand part analysis unit 202 has a predetermined shape when the hand part (right hand) and the hand part (left hand) of the instructor U2 belong to the direction instruction area R2 and the shape of the hand part belonging to the direction instruction area R2 is predetermined. When the shape is not a shape (for example, a pointing shape), the type of instruction operation performed by the instructor U2 may be estimated as a shape instruction operation. Furthermore, when both the hand (right hand) and the hand (left hand) of the instructor U2 belong to the direction indicating region R2, the hand analysis unit 202 indicates that the type of the instruction operation by the instructor U2 is the shape indicating operation. It may be estimated.

  Heretofore, detailed functional examples of the remote operation support system 1 according to the embodiment of the present invention have been described.

(5. Operation example of remote work support system)
Next, an operation example of the remote work support system 1 according to the embodiment of the present invention will be described. FIG. 17 is a flowchart showing an operation example of the remote operation support system 1 according to the embodiment of the present invention. Note that the operation example shown in FIG. 17 is only an example of the operation of the remote operation support system 1 according to the embodiment of the present invention. Therefore, the operation of the remote operation support system 1 according to the embodiment of the present invention is not limited to the operation example shown in FIG.

  As shown in FIG. 17, on the worker side, the video acquisition unit 101 acquires a video (worker-side video) by imaging (S11). Then, the communication unit (product) 104 transmits the worker-side video to the instructor side (S12). On the instructor side, the communication unit (finger) 207 receives the worker-side video (S21). Then, the hand measurement unit 201 acquires a video (instructor side video) by imaging. In addition, the hand measurement unit 201 measures the position of the hand of the instructor U2 in the three-dimensional space based on the instructor side image (S22).

  When the position of the hand of the instructor U2 is not measured by the hand measuring unit 201 (“No” in S23), the operation is shifted to S27. On the other hand, when the position of the hand of the instructor U2 is measured by the hand measurement unit 201 (“Yes” in S23), the hand analysis unit 202 analyzes the hand (S24). Specifically, the hand part analysis unit 202 estimates the type of instruction operation of the instructor U2 based on the position of the hand part of the instructor U2 and the position instruction area R1, the direction instruction area R2, and the shape instruction area R3. To do.

  The instruction information generation unit 204 generates instruction information for the worker U1 based on the type of instruction operation of the instructor U2 (S25). The communication unit (finger) 207 transmits instruction information to the worker side. When the transmission of the instruction information is finished, the operation is shifted to S27.

  On the worker side, the communication unit (product) 104 receives the instruction information when the instruction information is transmitted from the instruction side (S13). The video output unit (product) 102 outputs information on the worker side (S14). Specifically, when the instruction information is received from the instructor side, the image output unit (product) 102 displays the operator side image and the instruction information superimposed on the operator side image. On the other hand, when the instruction information is not received from the instructor side, the video output unit (product) 102 displays the operator side image. When the information output on the worker side is completed, the operation proceeds to S11.

  On the instructor side, the video output unit (finger) 205 outputs information on the instructor side (S27). Specifically, when instruction information is transmitted to the worker side, the video output unit (finger) 205 displays the worker side video and the instruction information superimposed on the worker side video. On the other hand, the video output unit (finger) 205 displays the worker-side video when the instruction information is not transmitted to the worker side. When the information output on the instructor side ends, the operation proceeds to S21.

  The operation example of the remote work support system 1 according to the embodiment of the present invention has been described above.

(6. Modified examples of display objects)
As described above, the display object corresponding to the instruction information is displayed by each of the video output unit (product) 102 and the video output unit (finger) 205. Here, the display object is not limited to the above-described example. For example, in the above description, the case where the display object indicating the direction is represented by an arrow has been described. However, the display object indicating the direction is not limited to the arrow. Hereinafter, modifications of the display object indicating the direction will be described.

  FIG. 18 is a diagram for explaining a first modification of the display object indicating the direction. Referring to FIG. 18, a display area B21 corresponding to the left direction, a display area B22 corresponding to the upper direction, a display area B23 corresponding to the right direction, and a display area B24 corresponding to the lower direction are shown. Here, as in the example shown in FIG. 13, a case is assumed in which the instructor U2 points to the left as an example of the direction instruction operation. In such a case, the video output unit (product) 102 displays the display object in the display area (display area B21 corresponding to the left direction) corresponding to the direction (left direction) indicated by the pointing. The worker U1 can confirm the direction instructed from the instructor U2 by the display area of the display object, and can work efficiently.

  It is also assumed that the direction indicated by the pointing finger is an oblique direction. In such a case, the video output unit (product) 102 may display a display object in display areas corresponding to two directions close to the oblique direction. At this time, the video output unit (product) 102 may darken the display object or increase the size of the display object in a direction closer to the oblique direction out of the two directions.

  As shown in FIG. 13, not only the video output unit (product) 102 in the remote work support system (product) 10 but also the video output unit (finger) 205 in the remote work support system (finger) 20 is pointed. The display object may be displayed in a display area corresponding to the same direction as indicated by (display area B21 corresponding to the left direction). Then, since the instructor U2 can see the display object in the same display area as the display area of the display object that the operator U1 is viewing, it is possible to confirm the direction instructed by the operator U1. It becomes.

  19 and 20 are diagrams for explaining a second modification of the display object indicating the direction. 19 and 20, as in the example shown in FIG. 18, the display area B21 corresponding to the left direction, the display area B22 corresponding to the upper direction, the display area B23 corresponding to the right direction, and the downward direction A corresponding display area B24 is shown. Further, referring to FIGS. 19 and 20, unlike the example shown in FIG. 18, a display area B25 corresponding to the depth direction and a display area B26 corresponding to the front direction are shown.

  In the example shown in FIG. 19, it is assumed that the instructor U2 has pointed in the depth direction as an example of the direction instruction operation. In such a case, the video output unit (product) 102 displays a display object in a display area (display area B25 corresponding to the depth direction) corresponding to the direction (depth direction) indicated by the pointing. The worker U1 can confirm the direction instructed from the instructor U2 by the display area of the display object, and can work efficiently.

  Further, as shown in FIG. 19, the display area B22 corresponding to the upward direction and the display area B25 corresponding to the depth direction are made different (for example, corresponding to the upward direction as shown in FIG. 19). By making the vertical width of the display area B25 corresponding to the depth direction shorter than the vertical width of the display area B22), the direction indicated by the instructor U2 can be intuitively understood.

  As shown in FIG. 19, not only the video output unit (product) 102 in the remote work support system (product) 10 but also the video output unit (finger) 205 in the remote work support system (finger) 20 is pointed. The display object may be displayed in a display area corresponding to the same direction as indicated by (display area B25 corresponding to the depth direction). Then, since the instructor U2 can see the display object in the same display area as the display area of the display object that the operator U1 is viewing, it is possible to confirm the direction instructed by the operator U1. It becomes.

  In the example shown in FIG. 20, it is assumed that the instructor U2 has pointed forward as an example of the direction instruction operation. In such a case, the video output unit (product) 102 displays the display object in a display area (display area B26 corresponding to the front direction) corresponding to the direction (front direction) indicated by the pointing. The worker U1 can confirm the direction instructed from the instructor U2 by the display area of the display object, and can work efficiently.

  Further, as shown in FIG. 20, the display area B24 corresponding to the lower direction and the display area B26 corresponding to the front direction are made different in shape (for example, corresponding to the lower direction as shown in FIG. 20). By making the vertical width of the display area B26 corresponding to the front direction smaller than the vertical width of the display area B24, the direction instructed by the instructor U2 can be understood intuitively.

  As shown in FIG. 20, not only the video output unit (product) 102 in the remote work support system (work) 10 but also the video output unit (finger) 205 in the remote work support system (finger) 20 is pointed. The display object may be displayed in a display area corresponding to the same direction as indicated by (display area B26 corresponding to the front direction). Then, since the instructor U2 can see the display object in the same display area as the display area of the display object that the operator U1 is viewing, it is possible to confirm the direction instructed by the operator U1. It becomes.

  21 and 22 are diagrams for describing a third modification of the display object indicating the direction. In the example shown in FIG. 21, it is assumed that the instructor U2 has pointed in the depth direction as an example of the direction indicating operation. In such a case, the video output unit (product) 102 displays the display object B27 having a shape (three-dimensional shape) corresponding to the direction (depth direction) indicated by the pointing. The worker U1 can confirm the direction instructed from the instructor U2 by the display object B27, and can work efficiently.

  As shown in FIG. 21, not only the video output unit (product) 102 in the remote work support system (product) 10 but also the video output unit (finger) 205 in the remote work support system (finger) 20 is pointed. The display object B27 having a shape (three-dimensional shape) corresponding to the same direction as the direction indicated by (depth direction) may be displayed. Then, since the instructor U2 can see the display object in the same display area as the display area of the display object that the operator U1 is viewing, it is possible to confirm the direction instructed by the operator U1. It becomes.

  In the example shown in FIG. 22, it is assumed that the instructor U2 has pointed forward as an example of the direction instruction operation. In such a case, the video output unit (product) 102 displays the display object B28 having a shape (three-dimensional shape) corresponding to the direction (front direction) indicated by the pointing. The worker U1 can confirm the direction instructed from the instructor U2 by the display object B28 and can work efficiently.

  As shown in FIG. 22, not only the video output unit (product) 102 in the remote work support system (product) 10 but also the video output unit (finger) 205 in the remote work support system (finger) 20 is pointed. A display object B28 having a shape (three-dimensional shape) corresponding to the same direction as the direction indicated by (frontward direction) may be displayed. Then, since the instructor U2 can see the display object in the same display area as the display area of the display object that the operator U1 is viewing, it is possible to confirm the direction instructed by the operator U1. It becomes.

  In the above, the modification of the display object which shows a direction was demonstrated.

(7. Summary)
As described above, according to the embodiment of the present invention, the position of the hand of the first user in the three-dimensional space measured based on the first image and the type of instruction operation are set in advance. An instruction unit that estimates the type of instruction operation of the first user based on the region, and instruction information that generates instruction information to the second user based on the type of instruction operation of the first user A first information processing system including a generation unit is provided.

  According to such a configuration, the first user's intention (what kind of instruction information the first user wants to notify the second user) is grasped, and instruction information corresponding to the first user's intention is generated. It becomes possible to do. That is, according to such a configuration, it is possible to grasp the intention of the first user and generate instruction information from the first user to the second user more appropriately.

  Further, according to the embodiment of the present invention, the first user can give an instruction to the second user without performing an operation of changing the input device. Is reduced.

  Further, according to the embodiment of the present invention, if the position or direction is indicated by a symmetric graphic such as a pointer and an arrow, it is expected that the position or direction can be easily understood by the second user. It is expected that the burden of instructing the user's position or direction will be reduced. Further, according to the embodiment of the present invention, the second user is instructed by the shape of the hand only when the first user wants to instruct by the shape of the hand (gesture). Therefore, it is expected that the shape can be easily understood, and the burden of the first user's shape designating operation is expected to be reduced.

  The preferred embodiments of the present invention have been described in detail above with reference to the accompanying drawings, but the present invention is not limited to such examples. It is obvious that a person having ordinary knowledge in the technical field to which the present invention pertains can come up with various changes or modifications within the scope of the technical idea described in the claims. Of course, it is understood that these also belong to the technical scope of the present invention.

  In the above description, an example in which a display object (for example, a circular pointer) is displayed when it is estimated that the type of instruction operation by the instructor U2 is a position instruction operation has been described. At this time, when the type of the instruction operation by the instructor U2 is changed from the position instruction operation to another instruction operation, the video output unit (product) 102 immediately erases the display of the display object (for example, a circular pointer). The display object corresponding to the new instruction operation may be displayed immediately.

  Alternatively, the video output unit 102 can detect the predetermined operation after a predetermined time has elapsed when the type of the instruction operation by the instructor U2 is changed from the position instruction operation to another instruction operation. Then, the display of the display object (for example, a circular pointer) may be turned off. As a result, the instructor U2 can continuously instruct the position to the worker U1 even if the position instruction operation is stopped halfway. Alternatively, the instructor U2 can start the shape instructing operation while instructing the worker U1 of the position.

  Similarly, in the above description, an example in which a display object (for example, an arrow) is displayed when it is estimated that the type of instruction operation by the instructor U2 is a direction instruction operation has been described. At this time, when the type of the instruction operation by the instructor U2 is changed from the direction instruction operation to another instruction operation, the video output unit (product) 102 immediately deletes the display object (for example, an arrow) and displays a new one. A display object corresponding to a specific instruction operation may be displayed immediately.

  Alternatively, the video output unit (product) 102 detects the predetermined operation after a predetermined time elapses when the type of the instruction operation by the instructor U2 is changed from the direction instruction operation to another instruction operation. Then, the display of the display object (for example, an arrow) may be turned off. Thus, the instructor U2 can continuously instruct the operator U1 in the direction even if the direction instruction operation is stopped halfway. Alternatively, the instructor U2 can start the shape instruction operation while instructing the worker U1 in the direction.

  In the above, the case where the remote work support system (finger) 20 has the contact information acquisition unit 203 (touch panel) has been mainly described. However, the remote work support system (finger) 20 can detect the position indicated by the instructor U2 instead of the contact information acquisition unit 203 (touch panel) or in addition to the contact information acquisition unit 203 (touch panel). You may have other input devices. For example, the other input device may be a mouse or a keyboard. When a mouse, a keyboard, or the like is used as the input device, the region where the instructor U2 operates the input device may be excluded from the measurement target region by the hand measurement unit 201.

  In the above description, an example in which a display object corresponding to the same shape as the shape of the hand portion is displayed when it is estimated that the type of the instruction operation by the instructor U2 is the shape instruction operation has been described. However, when the shape of the hand portion has a movement in the front-rear direction, there is a possibility that the movement cannot be sufficiently transmitted by the two-dimensional expression. Therefore, in such a case, the video output unit (product) 102 may display the display object with the expression emphasized by increasing the magnification of the shape of the hand.

1 Remote work support system 10 Remote work support system (work)
101 Video acquisition unit 102 Video output unit (work)
103 Voice input / output unit
104 Communication Department (work)
20 Remote work support system (finger)
DESCRIPTION OF SYMBOLS 201 Hand part measurement part 202 Hand part analysis part 203 Contact information acquisition part 204 Instruction information generation part 205 Image | video output part (finger)
206 Voice input / output (finger)
207 Communication part (finger)
30 Network R1 Position indicating area R2 Direction indicating area R3 Shape indicating area

Claims (16)

Based on the position of the hand of the first user in the three-dimensional space measured based on the first image and the region set in advance for each type of instruction operation, the type of instruction operation of the first user An estimation unit for estimating
An instruction information generating unit that generates instruction information to the second user based on the type of instruction operation of the first user;
A first information processing system. The first information processing system includes:
A transmission unit configured to transmit the instruction information to a second information processing system used by the second user;
The first information processing system according to claim 1. The first information processing system includes:
A receiving unit for receiving a second image from a second information processing system used by the second user;
An image output unit that presents the second image to the first user,
The first information processing system according to claim 1. The first information processing system includes:
An image output unit for presenting the instruction information to the first user;
The first information processing system according to claim 1. When the position of the hand part belongs to the first area, the estimation unit estimates that the type of the instruction operation is the first type, and the position of the hand part is different from the first area. When belonging to a second area, the type of the instruction operation is estimated to be a second type different from the first type.
The first information processing system according to claim 1. The first area includes a space from a position of the touch panel to a position separated by a predetermined length in front of the touch panel with respect to the position of the touch panel.
The first information processing system according to claim 5. The estimation unit estimates that the type of the pointing operation is the second type when the position of the hand unit belongs to the second region and the shape of the hand unit is a predetermined shape. To
The first information processing system according to claim 5. The predetermined shape includes a pointing shape,
The first information processing system according to claim 7. When the position of the hand portion belongs to a third region different from the first region and the second region, the estimation unit determines the type of the instruction operation as the first type and the second type. Estimating a third type different from the type,
The first information processing system according to claim 5. The instruction information generation unit generates first instruction information when the instruction operation type is estimated to be the first type, and the instruction operation type is different from the first type. If it is estimated that the second type, second instruction information different from the first instruction information is generated.
The first information processing system according to claim 1. The instruction information generation unit generates, as the first instruction information, a touch position on the touch panel by the hand part when it is estimated that the type of the instruction operation is the first type.
The first information processing system according to claim 10. The instruction information generation unit generates, as the second instruction information, the direction indicated by the pointing of the hand part when the instruction operation type is estimated to be the second type.
The first information processing system according to claim 10. When the instruction information generation unit estimates that the instruction operation type is a third type different from the first type and the second type, the instruction information generation unit Generating third instruction information different from the instruction information of 2,
The first information processing system according to claim 10. The instruction information generation unit generates the shape of the hand part as the third instruction information when the instruction operation type is estimated to be the third type.
The first information processing system according to claim 13. A receiving unit that receives instruction information for the second user generated based on the position of the hand of the first user in the three-dimensional space and a region set in advance for each type of instruction operation;
An image output unit for presenting the instruction information to the second user;
A second information processing system comprising: Based on the position of the hand of the first user in the three-dimensional space measured based on the first image and the region set in advance for each type of instruction operation, the type of instruction operation of the first user An estimation unit for estimating
An instruction information generating unit that generates instruction information to the second user based on the type of instruction operation of the first user;
A first information processing system comprising:
A receiving unit for receiving the instruction information;
An image output unit for presenting the instruction information to the second user;
A second information processing system comprising:
A third information processing system.

Images