WO2022049707A1 - Somatosensory interface system, and action somatosensation system - Google Patents

Abstract

A VR system S comprises: a virtual space generation unit 10 that generates a virtual space in which a first avatar and a second avatar are present; an avatar control unit 15 that controls an action of the first avatar in accordance with an action of a user, and controls an action of the second avatar in accordance with the action of the first avatar; an HMD 4 that allows the user to perceive the virtual space; a possible action recognition unit 14 that recognizes an action or actions which a control target can perform; and a target control unit 17 that controls an action of the control target in accordance with the action of the second avatar. The avatar control unit 15 restricts the actions which the second avatar can perform, on the basis of the action or actions that the control target can perform in a second region.

Description

Experience-based interface system and motion experience system

The present invention is an experience-based interface system for controlling the operation of a controlled object existing in an area different from the area in which the user exists in the real space via a virtual space, and an operation that the controlled object can execute. It is related to the motion experience system that allows you to experience.

Conventionally, a user is made to recognize an image of a virtual space generated by a server or the like and an image of an avatar corresponding to the user in the virtual space via a head-mounted display (hereinafter, may be referred to as “HMD”). As a result, there is a so-called immersive virtual space experience system in which the user himself / herself recognizes that he / she exists in the virtual space and experiences virtual reality.

In this kind of virtual space experience system, a motion capture device or the like recognizes a user's movement in the real space (for example, body movement, coordinate movement, posture change, etc.), and according to the recognized movement, There is something that controls the behavior of the avatar in the virtual space (see, for example, Patent Document 1).

The virtual space experience system of Patent Document 1 generates a first avatar corresponding to a user, a game pad type object, and a predetermined character type second avatar in the virtual space. Then, this system operates the first avatar according to the movement of the user, operates the gamepad type object according to the movement of the first avatar, and operates according to the movement of the gamepad type object. Activate the second avatar.

Japanese Unexamined Patent Publication No. 2019-033881

By the way, the virtual space is interposed as an interface between the two real spaces, and the second avatar is operated in the virtual space through the operation of the first avatar corresponding to the user. And in a real space where at least one of the positions is different, an interface system that controls the operation of a controlled object such as a robot corresponding to the second avatar is desired.

In the interface system, it exists in a predetermined area of the real space in a virtual space created corresponding to an area where it is difficult for a user to actually enter (for example, a nuclear power plant during decommissioning work). Generate a first avatar corresponding to the user and a second avatar corresponding to a controlled object (for example, a robot existing inside the nuclear power plant) existing in a real space area different from the area where the user exists. ..

According to this, the user operates the second avatar corresponding to the controlled object via the first avatar corresponding to the user in the virtual space, thereby performing the operation of the controlled object existing in the area different from the user. , You will be able to control it as if you were operating it in direct contact with your own hands.

Here, when constructing such an interface system, it is conceivable to use the system of Patent Document 1. However, what the user controls in the system of Patent Document 1 is an avatar existing in the virtual space.

Therefore, when such an interface system is constructed by using the system of Patent Document 1, the user can use the avatar corresponding to the controlled object even though the controlled object exists in the real space. In some cases, the user tries to move without considering the restrictions in the real space (for example, the surrounding environment of the controlled object, the function of the controlled object, etc.).

The present invention has been made in view of the above points, and the user can control the operation of a controlled object that exists in a region different from the region in which he / she exists in the real space via a virtual space. It is an object of the present invention to provide an interface system that can be controlled in accordance with the above, and an operation experience system that allows the controlled object to experience an executable operation.

The experience-based interface system of the present invention is
Through the virtual space, a user existing in the first region of the real space controls the operation of the controlled object existing in the second region of the real space where at least one of the time and the position is different from the first region. It is a hands-on interface system for
A virtual space generation unit that generates a virtual space in which a first avatar corresponding to the user and a second avatar corresponding to the controlled object exist.
A user motion recognition unit that recognizes the user's motion in the first region,
An avatar control unit that controls the operation of the first avatar according to the operation of the user and controls the operation of the second avatar according to the operation of the first avatar.
An image determination unit that determines an image of the virtual space to be recognized by the user based on the state of the first avatar and the state of the second avatar.
An image display that causes the user to recognize the determined image of the virtual space, and
A possible motion recognition unit that recognizes an action that the controlled object can execute in the second region,
A target control unit that controls the operation of the controlled object according to the operation of the second avatar is provided.
The avatar control unit is characterized in that it controls while limiting the actions that the second avatar can perform, based on the actions that the control target can perform in the second region.

As described above, the experience-based interface system of the present invention controls the operation of the first avatar corresponding to the user according to the operation of the user, and controls the operation of the second avatar according to the operation of the first avatar. ing. As a result, the user can operate the control target corresponding to the second avatar in the real space by operating the second avatar via the first avatar corresponding to the user in the virtual space. ing.

However, the operation of the second avatar in the virtual space is limited based on the operation that can be executed by the controlled object existing in the second area having at least one of the time and position different from the first area in which the user exists. While being controlled. That is, the operation is controlled while being restricted based on the constraints in the real space in which the controlled object exists (for example, the surrounding environment of the controlled object, the function of the controlled object, etc.).

As a result, even if the user tries to operate the second avatar via the first avatar, the operation corresponds to the operation that the controlled object cannot execute (that is, the operation ignoring the constraint in the real space). ), The user cannot naturally execute the operation even in the virtual space. For example, the second avatar may not be able to perform a predetermined action or may not be able to move at a predetermined speed or higher.

Therefore, according to the experience-based interface system of the present invention, the operation of the second avatar corresponding to the operation that the controlled object cannot execute cannot be executed naturally even in the virtual space, so that the user has the controlled object. The operation of the controlled object in the second region can be controlled according to the second region without considering the restrictions in the second region of the real space.

Further, in the experience-based interface system of the present invention,
A space recognition unit that recognizes the shape of the first region and the shape of the second region is provided.
The virtual space generation unit generates the virtual space so that the shape of the virtual space corresponds to the shape of the first region.
The avatar control unit can execute the second avatar based on the operation that the controlled object can execute in the second region and the difference between the shape of the first region and the shape of the second region. It is preferable to control while limiting the operation.

As a situation in which the experience-based interface system of the present invention is used, there are cases where the shape of the first region and the shape of the second region cannot be matched, or it is difficult to match them. For example, the first area may be a narrow space such as a room in a home or an office, and the second area may be a large space such as a construction site.

In such a case, if a virtual space is created so as to correspond to the shape of the second region, the user may mistakenly recognize his / her operable range. As a result, when the user tries to operate the first avatar corresponding to himself / herself, he / she comes into contact with the surrounding environment (for example, furniture installed in his / her room) in the first area where he / she exists. There is a risk that it will end up.

Therefore, when the virtual space is generated so that the shape of the virtual space corresponds to the shape of the first region in this way, the virtual space corresponds to the range in which the original user can operate. You will be able to operate the first avatar without making a mistake in recognizing the range in which you can operate. As a result, even if the shape of the first region and the shape of the second region are different, the user can perform the operation of the controlled object in the second region according to the second region while being in the first region. Can be controlled.

However, simply generating the virtual space based on the shape of the first region makes it easier for the user to make the operation corresponding to the operation that the controlled object cannot execute to the second avatar corresponding to the controlled object. There is a risk that it will end up. For example, when the first area is wider than the second area, the movement amount of the controlled object with respect to the movement amount of the second avatar becomes large, so that when the user moves the second avatar as in the case where there is no difference in shape. , There is a risk that the upper limit of the movement speed of the controlled object will be exceeded.

Therefore, in this way, the movement of the second avatar is restricted based on the difference between the shape of the first region and the shape of the second region as well as the movement that the controlled object can execute in the second region. It is preferable to do so.

As a result, even when the virtual space is generated based on the shape of the first region, the operation of the second avatar corresponding to the operation that makes the controlled object unexecutable can be naturally restricted. As a result, even in such a case, the user controls the operation of the controlled object in the second region according to the second region without considering the difference between the shape of the first region and the shape of the second region. can do.

Further, in the experience-based interface system of the present invention,
A period recognition unit that recognizes the length of the first period, which is the operable period of the second avatar in the virtual space, and the length of the second period, which is the operable period of the controlled object in the second region. ,
A motion storage unit for storing the motion of the second avatar is provided.
The avatar control unit can execute the second avatar based on the operation that the controlled object can execute in the second region and the difference between the length of the first period and the length of the second period. Control while limiting movement,
It is preferable that the target control unit controls the control target in the second period according to the operation of the second avatar stored in the motion storage unit in the first period.

As a situation in which the experience-based interface system of the present invention is used, there is a situation in which the operation of the second avatar and the operation of the controlled object do not have to correspond in real time. In such a situation, if the motion storage unit is provided in this way, the motion of the second avatar in a certain time zone is stored (that is, the motion of the controlled object is set), and the motion is stored. Depending on the situation, the controlled object can actually operate in a later time zone.

By the way, it is not always necessary to match the length of the first period, which is the operable period of the second avatar in the virtual space, with the length of the second period, which is the operable period of the controlled object in the second region. For example, by setting the first period longer than the second period, the operation to be executed by the controlled object in a short time (for example, surgery by a robot) is set while being verified in advance over time. You can do something like that.

However, if the length of the first period and the length of the second period are different, it becomes easy for the user to make the operation corresponding to the operation that the controlled object cannot execute to the second avatar corresponding to the controlled object. There is a risk that it will end up. For example, when the first period is longer than the second period, the operation speed of the controlled object with respect to the operation speed of the second avatar becomes faster, so that when the user operates the second avatar in the same manner as when there is no difference in the period. , There is a risk of exceeding the upper limit of the operating speed of the controlled object.

Therefore, in this way, the movement of the second avatar is restricted based on the difference between the length of the first period and the length of the second period as well as the movement that the controlled object can execute in the second region. It is preferable to do so.

As a result, even if the length of the first period and the length of the second period are different, the operation of the second avatar corresponding to the operation that makes the controlled object unexecutable can be naturally restricted. As a result, even in such a case, the user controls the operation of the controlled object in the second period according to the second region without considering the difference between the length of the first period and the length of the second period. can do.

In addition, the motion experience system of the present invention is
Through the virtual space, a user existing in the first region of the real space can perform an operation that can be executed by a controlled object existing in the second region of the real space where at least one of the time and the position is different from the first region. It is a movement experience system for experiencing,
A virtual space generation unit that generates a virtual space in which a first avatar corresponding to the user and a second avatar corresponding to the controlled object exist.
A user motion recognition unit that recognizes the user's motion in the first region,
An avatar control unit that controls the operation of the first avatar according to the operation of the user and controls the operation of the second avatar according to the operation of the first avatar.
A mode switching unit that switches between the first mode and the second mode,
An image determination unit that determines an image of the virtual space to be recognized by the user based on the state of the first avatar and the state of the second avatar.
An image display that causes the user to recognize the determined image of the virtual space, and
A target function recognition unit that recognizes an operation that the control target can execute as a function,
It is provided with a constraint recognition unit that recognizes constraints when the control target operates in the second region.
In the first mode, the avatar control unit controls an operation that can be executed by the second avatar based only on the function of the controlled object, and in the second mode, the function of the controlled object and the second. It is characterized in that the second avatar controls while limiting the actions that can be performed based on the constraint in the area.

As described above, the motion experience system of the present invention controls the motion of the first avatar corresponding to the user according to the motion of the user in any of the switchable modes, and according to the motion of the first avatar. It controls the operation of the second avatar.

At this time, in the first mode, the operation that the second avatar can execute in the virtual space is controlled based only on the function to be controlled. Specifically, in the first mode, at least one of the time and the position is different from the first region in which the user exists, and the surrounding environment of the second region in which the control target exists is not considered. Thereby, in the first mode, the user can experience the operation that the controlled object can execute as a function.

In the first mode, when the controlled object is actually operated in the second region, there is a possibility that the controlled object comes into contact with the surrounding environment in the second region. Therefore, in principle, it is recommended that the controlled object is not actually operated in the first mode, except when there are almost no restrictions in the second region.

On the other hand, in the second mode, the operation that the second avatar can execute in the virtual space is controlled while being restricted not only based on the function to be controlled but also based on the constraint in the second area.

That is, in the second mode, the actions that can be executed by the second avatar correspond to the actions that the controlled object can actually execute in the second region, which is the real space. For example, the second avatar may not be able to perform a predetermined action or may not be able to move at a predetermined speed or higher. Thereby, in the second mode, the user can experience the operation that the controlled object can actually execute in the second region which is the real space.

In the second mode, even if the controlled object is actually operated in the second area, there is no possibility that the controlled object comes into contact with the surrounding environment of the second area. However, from the viewpoint of experiencing the operation that the controlled object can actually perform, it is not always necessary to actually operate the controlled object. Therefore, in the second mode, the controlled object may or may not be actually operated.

Therefore, according to the motion experience system of the present invention, the user compares the operation that the controlled object can execute based only on the function with the operation that the controlled object can execute based on the function and the constraint in the second region. You can experience it while doing it. As a result, according to this system, the user can intuitively verify the operation of fully exerting the function of the controlled object under a predetermined environment.

The schematic diagram which shows the schematic structure of the VR system which concerns on embodiment. The block diagram which shows the structure of the processing part of the VR system of FIG. The schematic diagram of the virtual space which the VR system of FIG. 1 makes a user recognize. It is explanatory drawing which shows the relationship between the virtual space area generated in the VR system of FIG. 1 and the area of the 1st area and the area of 2nd area in a real space. A timing chart showing an example of the relationship between the time zone in which the user's action is executed in the VR system of FIG. 1 and the time zone in which the action of the avatar, the object, and the drone is executed. The flowchart which shows the process to execute when the VR system of FIG. 1 is started to use. FIG. 3 is a flowchart showing a process executed based on the recognized information among the processes executed when the VR system of FIG. 1 recognizes an executable operation of an object. FIG. 3 is a flowchart showing a process executed based on newly recognized information among the processes executed when the VR system of FIG. 1 recognizes an executable operation of an object. The flowchart which shows the process which the VR system of FIG. 1 executes when operating an object and operating a drone.

Hereinafter, the VR system S (experience-based interface system, motion-experience system) according to the embodiment will be described with reference to the drawings.

The VR system S is a system that makes the user U recognize that the user U itself exists in the virtual space and allows the user U to experience virtual reality (so-called VR (virtual reality)).

Further, the VR system S is a controlled object that exists in the second region RS2 in the real space where at least one of the time and the position is different from the first region RS1 in the real space where the user U exists via the virtual space. This is a system for allowing the user U to experience an operation that controls the operation or that the controlled object can execute.

In the following, a case where the VR system S is used to cause the drone to perform a predetermined work (for example, transportation of luggage) at a construction site will be described. Therefore, the own room of the user U in which the user U exists is set as the first region RS1, and the first drone D1 and the second drone D2 (hereinafter, collectively referred to as "drone D") to be controlled exist. However, the construction site located away from the first area RS1 is referred to as the second area RS2.

However, the first region and the second region of the present invention and the control target are not limited to such a configuration. The first area may be an area in the real space in which the user exists, and the second area may have or may have a controlled object, and at least one of the first area and the time and position. May be different real-world areas. Further, the control target may be one in which the user controls the operation or verifies the operation via the virtual space.

Therefore, for example, when the system of the present invention is applied to a service for experiencing nature from the viewpoint of small organisms such as small animals and insects, the first area is a room for providing the service, and the second area is the room. It may be a breeding box in which small animals, insects, etc. that are smaller than the actual breeding box are actually bred, and the control target may be a robot with a camera that can move inside the breeding box.

Further, for example, when the system of the present invention is applied to a robotic operation, the first area is set as an operating room in a predetermined time zone, and the second area is set as a time zone after the predetermined time zone. In addition to being an operating room, the controlled object may be a robot that actually performs surgery.

[Overview of system configuration]
First, a schematic configuration of the VR system S will be described with reference to FIGS. 1 and 2.

As shown in FIG. 1, the VR system S includes a device existing in the first region RS1 in the real space where the user U exists, a device existing in the second region RS2 where the drone D exists, and a device. , Is configured by a server 1 installed by a service provider or the like using the VR system S.

The system of the present invention is not limited to such a configuration. For example, a terminal instead of the server may be installed in the first area or the second area, or a plurality of servers, terminals, and the like may be used to realize the function of the server 1 of the present embodiment (processing unit described later). You may do it.

The devices existing in the first region RS1 include a plurality of signs 2 attached to the user U, a first camera 3 for photographing the user U (strictly speaking, the sign 2 attached to the user U), and a virtual device. A head-mounted display (hereinafter referred to as "HMD4") that allows the user U to recognize the image and sound of the space VS (see FIG. 3), and a controller 5 used by the user U (not shown in FIG. 1, see FIG. 2). Applies to.

The sign 2 is attached to each of the user U's head, both hands, and both feet via the HMD4, gloves, shoes, and the like worn by the user U. However, the sign 2 is used to recognize the movement of the user U in the first region RS1 (for example, the movement of each part of the body, the movement of coordinates, the change of posture, etc.) as described later. Therefore, the mounting position may be appropriately changed according to other devices and the like constituting the VR system S.

A plurality of first cameras 3 are installed in the first area RS1 so that the user U itself and the range in which the user U can operate in the first area RS1 can be photographed from multiple directions. Only one first camera 3 may be installed depending on the performance of the first camera 3, the shape of the first region RS1, and the like, and the installation location thereof may be appropriately changed.

The HMD4 is attached to the head of the user U. As shown in FIG. 2, the HMD 4 has a monitor 40 (image display) for causing the user U to recognize the image of the virtual space VS determined by the server 1, and a virtual space determined by the server 1. It has a speaker 41 (voice generator) for causing the user U to recognize the voice of VS.

The controller 5 transmits to the server 1 a mode switching instruction, which will be described later, and an operation instruction to the first object O1 corresponding to the first drone D1 and the second object O2 corresponding to the second drone D2. Used for.

When the user U is made to experience the virtual reality by using this VR system S, the user U is made to recognize only the image and the sound of the virtual space VS by the HMD4, and the user U himself / herself is made to recognize the avatar A described later (see FIG. 3). ) To be recognized as existing in the virtual space. That is, the VR system S is configured as a so-called immersive system.

The VR system S employs a so-called motion capture device configured by using the sign 2 and the first camera 3 as a system for recognizing the operation of the user U in the first region RS1.

However, the system of the present invention is not limited to such a configuration. For example, when using a motion capture device, in addition to the above configuration, one having a different number of signs and cameras from the above configuration (for example, one is provided for each) may be used.

Further, for example, a device other than the motion capture device may be used to recognize the operation in the first region of the user's real space. Specifically, a sensor such as GPS may be mounted on the HMD, gloves, shoes, or the like worn by the user, and the movement of the player may be recognized based on the output from the sensor. Further, such a sensor may be used in combination with a motion capture device as described above.

Further, for example, the controller may be omitted in the real space, and an object corresponding to the controller (for example, the fifth object O5 described later in the present embodiment) may be generated only in the virtual space. Also, both such controllers and objects may be omitted, or used in combination with at least one of them, to recognize instructions given using them based on the user's voice or gesture. ..

The device existing in the second area RS2 corresponds to the second camera 6 (not shown in FIG. 1, see FIG. 2) installed in the second area RS2. Although omitted in the present embodiment, a relay device for controlling the drone D may be installed in the second region RS2 based on the instruction received from the server 1.

A plurality of second cameras 6 are installed so that the drone D itself and the range in which they can operate in the second area can be photographed from multiple directions. Only one second camera 6 may be installed depending on the performance of the second camera 6, the shape of the second region RS2, and the like, and the installation location thereof may be appropriately set.

The server 1 is composed of one or a plurality of electronic circuit units including a CPU, RAM, ROM, an interface circuit, and the like.

The server 1 communicates with the first camera 3, the HMD 4, and the controller 5 existing in the first area RS1 and the second camera 6 existing in the second area RS1 by short-range wireless communication and wired communication. , Internet network, public line, etc., are configured to enable mutual information communication. Further, the server 1 is also configured to enable information communication with the drone D in the same manner.

[Configuration of each processing unit]
Next, the configuration of the processing unit included in the server 1 will be described in detail with reference to FIGS. 1 to 5.

As shown in FIG. 2, the server 1 has a virtual space generation unit 10 (spatial recognition unit, constraint recognition unit), a user motion recognition unit 11, and a mode as functions realized by the implemented hardware configuration or program. Switching unit 12, period recognition unit 13, possible motion recognition unit 14 (target function recognition unit), avatar control unit 15, output information determination unit 16 (image determination unit), target control unit 17 (operation storage unit). ) And.

The virtual space generation unit 10 includes a virtual space VS (strictly speaking, the background of the virtual space VS), an avatar A (first avatar) corresponding to the user U existing in the virtual space VS, and a plurality of objects. Generate an image. The virtual space generation unit 10 also generates sounds related to those images.

As shown in FIGS. 1 and 3, the objects generated by the virtual space generation unit 10 include the first object O1 (second avatar) and the second drone D2 corresponding to the first drone D1 existing in the second region RS2. The second object O2 (second avatar) corresponding to, the third object O3 corresponding to the work machine W, the fourth object O4 corresponding to the building material M, and the controller 5 existing in the first region RS1. The fifth object O5 corresponding to is included.

Further, the virtual space generation unit 10 (spatial recognition unit) recognizes the shape of the first region RS1 based on the image taken by the first camera 3, and the second camera 6 is based on the image taken by the second camera 6. Recognize the shape of the two-region RS2. Then, the virtual space generation unit 10 generates the virtual space VS so that the shape of the virtual space VS corresponds to the shape of the first region RS1.

Specifically, as shown in FIG. 4, for example, the first region RS1 is recognized as a narrow space of a plan view square, and the second region RS2 is recognized as a vertically long wide space of a plan view rectangle. do. In that case, the virtual space generation unit 10 generates the shape of the virtual space VS so as to match the shape of the first region RS1. On the other hand, the image of the virtual space VS is generated as an image based on the captured image of the second region RS2, reduced to fit the shape of the first region RS1.

At this time, the aspect ratio of the first region RS1 (that is, the virtual space VS) in the plan view is different from the aspect ratio of the second region RS2. Therefore, the reduction ratios of the images of the virtual space VS in the vertical direction and the horizontal direction of the drawing are different.

The system of the present invention is not limited to such a configuration, and when the shape of the first region and the shape of the second region always match (for example, when only the time is different), the system is not limited to such a configuration. The virtual space generation unit may not include a function of deforming the image of the second region based on the shape of the first region.

Further, the virtual space generation unit 10 (constraint recognition unit) restricts the operation of the user U (and by extension, the avatar A corresponding to the user U) in the first region RS1 based on the image taken by the first camera 3. Recognize. Then, the virtual space generation unit 10 generates the virtual space VS in consideration of the constraint.

Specifically, for example, as shown in FIGS. 1, 3 and 4, the virtual space generation unit 10 recognizes the furniture F existing in the first region RS1, and the furniture F recognizes the furniture F existing in the first region RS1. The first constraint area LA1 in which the avatar A cannot enter is generated at the position of the virtual space VS corresponding to the position existing in. At this time, the furniture F is generated in the virtual space VS as a ghost in a semi-transparent state.

Further, the virtual space generation unit 10 recognizes the restriction when the drone D operates in the second region RS2 based on the image taken by the second camera 6 and the function of the work machine W input in advance. do. Then, the virtual space generation unit 10 generates the virtual space VS in consideration of the constraint.

Specifically, for example, as shown in FIGS. 1, 3 and 4, the virtual space generation unit 10 recognizes the building material M existing in the second region RS2. Then, the second constraint area LA2 is generated at the position of the virtual space VS corresponding to the position where the building material M exists in the second region RS2. The second restricted area LA2 is an area in which the first object O1 and the second object O2 (hereinafter, collectively referred to as “target objects O1 and O2”) corresponding to the drone D cannot enter.

Similarly, the virtual space generation unit 10 recognizes the work machine W existing in the second region RS2. Then, the third constraint area LA3 is generated in the area of the virtual space VS corresponding to the area where the work machine W can exist in the second area RS2 (that is, the area where the work machine W operates). The third restricted area LA3 is an area where entry of the target objects O1 and O2 is prohibited (for example, an area where entry is possible but a warning is displayed).

The first constraint area LA1, the second constraint area LA2, and the third constraint area LA3 generated in this way are shown as a semi-transparent three-dimensional object in the virtual space VS, for example, as shown in FIG.

The virtual space generation unit 10 limits the shapes of the first region RS1 and the second region RS2, and the restrictions in the first region RS1 and the second region RS2, not necessarily the images of the first camera 3 and the second camera 6. It is not necessary to recognize based on the above, and it may be recognized based on a model, numerical value, etc. separately input by the user U himself or the like.

As shown in FIG. 2, the user motion recognition unit 11 recognizes the motion of the user U based on the image data captured by the first camera 3. Specifically, the user motion recognition unit 11 extracts the sign 2 attached to the user U from the image data of the user U, and based on the extraction result, the motion and coordinates of each part of the body of the user U are Recognize movements and changes in posture.

The mode switching unit 12 recognizes the user U's mode change instruction based on the input to the controller 5, and changes the mode executed by the VR system S based on the instruction.

In the VR system S, a mode for controlling the drone D in real time (second mode) and a mode for controlling the drone D in a different time zone after the time zone in which the target objects O1 and O2 are operated (second mode). The drone D is not actually controlled and can be changed to one of three modes (first mode) for verifying its operation.

The system of the present invention is not limited to such a configuration, and may include at least one of these three modes. Therefore, when configuring to execute only one mode, the mode switching unit may be omitted.

The period recognition unit 13 recognizes the length of the first period, which is the operable period of the target objects O1 and O2 in the virtual space VS, and the time zone thereof, based on the input to the controller 5. That is, the operable period of the avatar A (and by extension, the user U) that operates the target objects O1 and O2 is recognized.

Further, the period recognition unit 13 recognizes the length of the second period, which is the operable period of the drone D in the second area RS2, and the time zone thereof, based on the input to the controller 5 and the like.

The system of the present invention is not limited to such a configuration. For example, when the system has only a mode for controlling a controlled object in real time, the period recognition unit may be omitted. good.

The possible motion recognition unit 14 (target function recognition unit) recognizes the motion that the drone D can execute in the second area RS2.

Specifically, first, the possible motion recognition unit 14 (target function recognition unit) recognizes the motion that the drone D can execute as a function based on the information or the like input in advance by the user U or the like.

The operation that can be executed as a function may be recognized only at the stage when the VR system S is started to be used, and may be recognized at any time based on the information fed back from the drone D while using the VR system S. You may try to recognize it.

Further, the possible motion recognition unit 14 has a time zone of the second period recognized by the period recognition unit 13, an image of the second region RS2 recognized by the virtual space generation unit 10, and a plan of construction to be executed in the second region RS2. The drone D recognizes an action that can be executed even based on the environment (weather, etc.) of the second region RS2.

For example, the possible motion recognition unit 14 recognizes the work schedule of the work machine W in the second period separately input, and within that period, the drone D can be executed as a function, but with the work machine W. Recognize actions that are infeasible to avoid contact. After that, the possible motion recognition unit 14 recognizes the action that the drone D can perform in the second period with reference to the recognized action that becomes infeasible.

The avatar control unit 15 controls the operation of the avatar A according to the operation of the user U, and controls the operation of the target objects O1 and O2 according to the operation of the avatar A. Specifically, as shown in FIG. 3, the user U operates the target object in the virtual space VS through the operation of the avatar A corresponding to the user U or by operating the object O5 corresponding to the controller 5. Controls the operation of O1 and O2.

At this time, the avatar control unit 15 limits the operations of the target objects O1 and O2 based on the operations that the drone D corresponding to them can execute as a function.

For example, when the user U tries to move the corresponding first object O1 at a speed corresponding to a speed exceeding the actually feasible movement speed of the first drone D1, the first object O1 Operation is restricted. Specifically, when the user U moves the first object O1 by pushing it with the hand of the avatar A, and the moving speed becomes equal to or higher than a predetermined speed, the hand of the avatar A moves. Processing such as passing through the first object O1 is performed.

Further, the avatar control unit 15 limits the operations of the target objects O1 and O2 based on the operations that the drone D corresponding to them can execute in the second region RS2.

For example, the drone D cannot enter the position where the building material M exists in the second region RS2. Therefore, the target objects O1 and O2 cannot be moved to the second constraint area LA2 corresponding to the position.

Further, for example, if the drone D is moved to a region where the work machine W can exist in the second region RS2, there is a possibility that the work machine W will come into contact with the work machine W. Therefore, when the target objects O1 and O2 are moved to the third constraint area LA3 corresponding to the area, their colors change and a warning text is displayed in the vicinity of the avatar A in the virtual space VS.

Further, the avatar control unit 15 limits the operation of the target objects O1 and O2 even based on the difference between the shape of the first region RS1 and the shape of the second region RS2.

For example, when the shape of the first region RS1 and the shape of the second region RS2 are different, the virtual space VS is generated based on the shape of the first region RS1. However, in such a virtual space VS, there is a possibility that the user U tends to make the target objects O1 and O2 corresponding to the operations corresponding to the operations that the drone D cannot execute.

Specifically, when the second region RS2 is wider than the first region RS1, when the virtual space VS is generated based on the shape of the first region RS1, the movement of the drone D with respect to the movement amount of the target objects O1 and O2. The amount will be large. Therefore, if the user U moves the target objects O1 and O2, the upper limit of the moving speed of the drone D may be exceeded.

Therefore, when the second region RS2 is wider than the first region RS1 (that is, when the virtual space VS is a reduced version of the second region RS2 as shown in FIG. 4), the avatar control unit 15 is the target. The movable speed of the objects O1 and O2 in each direction is limited for each direction according to the degree of reduction thereof.

As a result, the target objects O1 and O2 can move only at a speed or less specified for each direction. Specifically, the movable speed of the target objects O1 and O2 in the vertical direction on the paper surface of FIG. 4 is slower than the moving speed in the horizontal direction.

Further, the avatar control unit 15 limits the operation of the target objects O1 and O2 even based on the difference between the length of the first period and the length of the second period.

For example, the length of the first period and the length of the second period may be different. However, in such a case, there is a possibility that the user U tends to make the target objects O1 and O2 corresponding to the actions corresponding to the actions that the drone D cannot execute.

Specifically, when the first period is longer than the second period, the operating speed of the drone D becomes faster than the operating speed of the target objects O1 and O2. Therefore, if the user U operates the target objects O1 and O2 in the same manner as when there is no difference in the period, the upper limit of the operating speed of the drone D may be exceeded.

Therefore, when the first period is longer than the second period, the avatar control unit 15 uniformly limits the movable speeds of the target objects O1 and O2 in all directions according to the difference in the periods.

As a result, the target objects O1 and O2 can move only at a predetermined speed or less in any direction. Specifically, the movable speed of the target objects O1 and O2 is slower than the normal speed.

The system of the present invention is not limited to such a configuration. For example, when the shape of the first region and the shape of the second region match, the avatar control unit operates the second avatar based on the difference between the shape of the first region and the shape of the second region. Does not have to be restricted. Further, for example, when the system is configured to have only a mode for controlling the controlled object in real time, the avatar control unit may use the second avatar based on the difference between the first period and the second period. It is not necessary to limit the operation.

In the case of the mode in which the drone D is controlled in real time or the mode in which the drone D is controlled in different time zones, the drone D can be executed in the second region RS2 in consideration of the surrounding environment as described above. The operation of the target objects O1 and O2 in the virtual space VS based on the operation, the difference between the shape of the first region RS1 and the shape of the second region RS2, and the difference between the length of the first period and the length of the second period. Is restricted.

On the other hand, in the case of the mode for verifying the operation, the operation is not restricted based on the difference in the surrounding environment and the period, and the target objects O1 and O2 are based only on the difference in shape and the function of the drone D. The operation in the virtual space VS is limited.

The output information determination unit 16 (image determination unit) determines the image and sound of the virtual space VS to be recognized by the user U via the HMD 4.

The target control unit 17 (operation storage unit) controls the operation of the drone D corresponding to the operation of the target objects O1 and O2 in the virtual space VS in the second region RS2.

Further, the target control unit 17 can store the operations of the target objects O1 and O2 executed in the first period. This makes it possible to control the drone D not only in real time but also in the second period, which is a period after the first period (at least a period in which the start time is different).

Further, the target control unit 17 can individually store each operation of the first object O1 and the second object O2. This makes it possible for one user U to control the operations of the first drone D1 and the second drone D2 corresponding to them in the second region RS2 at different time zones.

For example, as shown in FIG. 5, first, it is assumed that the user operates the first object O1 via the avatar A in the first period (the period from t1 to t2). The operation of the first object O1 at that time is stored in the target control unit 17. After that, in the second period (the period from t4 to t7) after the first period, the first drone D1 is operated according to the operation of the first object O1 stored in the target control unit 17.

At this time, the lengths of the first period and the second period in this case do not necessarily have to be the same. As a result, for example, by setting the first period to be longer than the second period, the operation to be executed by the controlled object in a short time (for example, surgery by a robot) is set while being verified in advance over time. You can do something like that. On the contrary, by setting the second period longer than the first period, it is possible to set the operation that must be performed slowly in consideration of the situation of the second region in a short time.

Further, for example, first, it is assumed that the user operates the second object O2 via the avatar A in the first period (the period from t3 to t6). The operation of the second object O2 at that time is stored in the target control unit 17. After that, in the second period (the period from t5 to t8) after the first period, the second drone D2 is operated according to the operation of the second object O2 stored in the target control unit 17.

In this way, by making the first period (the period from t1 to t2) for the first object O1 different from the first period (the period from t3 to t6) for the second object O2, one user U can be used. It is possible to control multiple control targets. Further, the first period (for example, the period from t3 to t6) may partially overlap with the second second period (for example, the period from t5 to t8). As a result, the user U can set the subsequent operation while taking into account the situation in real time to some extent.

The system of the present invention is not limited to such a configuration. For example, if the system is configured to have only a mode for controlling the controlled object in real time, the operation storage unit may be omitted.

[Processes executed by each processing unit]
Next, the process executed by the VR system S will be described with reference to FIGS. 2 to 8.

In the following, the processing in the mode in which the drone D is controlled in a different time zone after the time zone in which the target objects O1 and O2 are operated will be described in detail. The processing in the mode that controls the drone D in real time and the processing in the mode that does not actually control the drone D and verifies its operation are different from the processing in the mode that controls the drone D in different time zones. Only explain.

[Process to be executed when use starts]
First, with reference to FIGS. 2, 3, 4, and 6, a process to be executed by each processing unit of the VR system S when the use of the VR system S is started will be described.

In this process, first, the virtual space generation unit 10 determines the shape of the first region RS1 and the second region based on the images taken by the first camera 3 and the second camera 6, the model input in advance, the numerical values, and the like. Recognize the shape of RS (Fig. 6 / STEP100).

Next, the virtual space generation unit 10 determines whether or not the recognized shape of the first region RS1 and the shape of the second region RS2 are different (FIG. 6 / STEP101).

When the shape of the first region RS1 and the shape of the second region RS2 are different (YES in STEP 101), the virtual space generation unit 10 has, for example, the shape of the first region RS1 as shown in FIG. Based on this, the image of the second region RS2 is corrected so as to match the shape of the first region RS1 (FIG. 6 / STEP102).

Next, the virtual space generation unit 10 generates a virtual space VS (strictly speaking, an image that becomes the background of the virtual space VS) based on the corrected image of the second region RS2 (FIG. 6 / STEP103).

On the other hand, when the shape of the first region RS1 and the shape of the second region RS2 match (NO in STEP101), the virtual space generation unit 10 does not correct the image of the second region RS2. A virtual space VS (strictly speaking, an image that is a background of the virtual space VS) is generated based on the uncorrected image (FIG. 6 / STEP104).

Next, the virtual space generation unit 10 sets the initial state (coordinates, posture) in the first region RS1 of the user U based on the images taken by the first camera 3 and the second camera 6, the model input in advance, the numerical values, and the like. Etc.) and the initial state (coordinates, posture, etc.) in the second region RS2 of the drone D is recognized (FIG. 6 / STEP105).

In the present embodiment, the initial state of the user U is the initial state in the first period and is the current state. Further, in the present embodiment, the initial state of the drone D is the initial state in the second period, and is the initial state in the second period.

Next, the virtual space generation unit 10 generates the avatar A in the virtual space VS so as to correspond to the initial state of the user U, and causes the target objects O1 and O2 to correspond to the initial state of the drone D. (Fig. 6 / STEP106).

Next, the virtual space generation unit 10 creates the surrounding environment of the first region RS1 and the second region RS2 based on the images taken by the first camera 3 and the second camera 6, the model input in advance, the numerical values, and the like. Recognize (Fig. 6 / STEP107).

Specifically, the virtual space generation unit 10 recognizes the furniture F existing in the first region RS1 as the surrounding environment of the first region RS1, and the building material M and the building material M existing in the second region RS2. The work machine W is recognized as the surrounding environment of the second region RS2.

Next, the virtual space generation unit 10 generates objects and constraint areas corresponding to them in the virtual space VS based on the recognized surrounding environment.
(Fig. 6 / STEP108).

Specifically, the virtual space generation unit 10 generates the ghost of the furniture F and the first constraint area LA1 at the position of the virtual space VS corresponding to the position where the furniture F exists in the first region RS1. Further, the virtual space generation unit 10 places the building material M and the work machine W at the position of the virtual space VS corresponding to the position where the building material M and the work machine W exist in the second region RS2, and the third object corresponding to the building material M and the work machine W. The fourth object O4 corresponding to the O3 and the work machine, and the second constraint area LA2 and the third constraint area LA3 are generated.

Next, the output information determination unit 16 determines the image and sound to be recognized by the user based on the state of the avatar A (FIG. 6 / STEP109).

Next, the output information determination unit 16 displays the determined image on the monitor 40 of the HMD 4 worn by the user U, and generates the determined voice on the speaker 41 (FIG. 6 / STEP110). , End this process.

Immediately after the use of the VR system S is started by the above processing, the user U is in a state of recognizing, for example, the virtual space VS as shown in FIG.

[Process to be executed when recognizing the executable action of the object after it is started to be used]
Next, with reference to FIGS. 2 to 4, when each processing unit of the VR system S recognizes the executable operation of the target objects O1 and O2 after the use of the VR system S is started. The process to be executed will be described.

In this process, first, the possible motion recognition unit 14 can execute the drone D corresponding to the target objects O1 and O2 as a function in the second region RS2 based on the information or the like input in advance by the user U or the like. Recognize the operation (Fig. 7A / STEP200).

Next, the period recognition unit 13 recognizes the first period and the second period based on the information or the like input in advance by the user U or the like (FIG. 7A / STEP201).

Next, the possible motion recognition unit 14 recognizes the surrounding environment of the second region RS2 in the second period (FIG. 7A / STEP202).

Specifically, the possible motion recognition unit 14 recognizes the surrounding environment that may change in the second region RS2 in the second period. In the present embodiment, the amount of the building material M (that is, the arrangement space thereof) and the operation of the work machine W may change in the second period. Therefore, the possible motion recognition unit 14 changes the status of the building material M and the work machine W at each time point in the second period based on the construction plan or the like input in advance by the user U or the like to the surrounding environment of the second region RS2. Recognize as.

Next, the target objects O1 and O2 corresponding to the drone D are the virtual space VS based on the recognized function of the drone D and the surrounding environment of the second region RS1 in the second period. Recognizes the actions that can be performed in (Fig. 7A / STEP203).

Next, the avatar control unit 15 determines whether or not the shape of the first region RS1 and the shape of the second region RS2 are different based on the information recognized by the virtual space generation unit 10 (FIG. 7A / STEP204). ..

When the shape of the first region RS1 and the shape of the second region RS2 are different (YES in STEP 204), the avatar control unit 15 refers to the image of the second region RS2 when generating the virtual space VS. The content of the correction made is recognized, and the operation of the target objects O1 and O2 limited by the position of the virtual space VS is recognized based on the content (FIG. 7A / STEP205).

Specifically, as shown in FIG. 4, for example, the first region RS1 is a narrow space of a plan view square, and the second region RS2 is a vertically long wide space of a plan view rectangle, and the virtual space generation unit 10 However, it is assumed that the image of the virtual space VS is corrected so that the image of the second region RS2 is reduced to fit the shape of the first region RS1.

In that case, the avatar control unit 15 limits the movable speed of the target objects O1 and O2 for each direction according to the degree of reduction thereof. As a result, the movable speed of the target objects O1 and O2 in the vertical direction on the paper surface of FIG. 4 becomes slower than the moving speed in the horizontal direction.

After recognizing the limitation based on the difference between the shape of the first region RS1 and the shape of the second region RS2, or when the shape of the first region RS1 and the shape of the second region RS2 match (NO in STEP 204). In the case of), the avatar control unit 15 determines whether or not the length of the first period and the length of the second period are different (FIG. 7A / STEP206).

When the length of the first period and the length of the second period are different (YES in STEP206), the avatar control unit 15 sets the execution limit of each operation of the target objects O1 and O2 based on the difference amount. (Fig. 7A / STEP207).

Specifically, the speed at which the drone D can perform as a function in the second period is uniformly accelerated or decelerated based on the length of the first period and the ratio between the second period and the length.

The processing up to this point is the processing executed before the user U starts to operate the target objects O1 and O2 via the avatar A (that is, before the start of the first period). The process described below is a process executed after the user U starts operating the target objects O1 and O2 via the avatar A (that is, after the start of the first period).

After recognizing the limitation based on the difference between the length of the first period and the length of the second period, or when the length of the first period and the length of the second period match (NO in STEP 206). The virtual space generation unit 10 determines whether or not the surrounding environment of the drone D in the second region RS2 has changed at the time of the second period corresponding to the present time (that is, a predetermined time of the first period). (Fig. 7B / STEP208).

When the surrounding environment has changed (YES in STEP 208), the virtual space generation unit 10 modifies the second constraint area LA2 or the third constraint area LA3 based on the change in the surrounding environment of the second region RS1. (FIG. 7B / STEP209).

Specifically, for example, when the amount of the building material M is smaller than that shown in FIG. 3, the virtual space generation unit 10 sets the second constraint area LA2 corresponding to the building material M. Shrink as it decreases. Further, for example, when the posture or the like has changed from the state of FIG. 3 as a result of the operation of the work machine W, the virtual space generation unit 10 sets the third constraint area LA3 corresponding to the work machine W. Deform according to changes.

Next, the avatar control unit 15 recognizes the operation of the target objects O1 and O2 newly restricted as a result of the modification based on the modification of the second constraint area LA2 or the third constraint area LA3 (FIG. 7B / STEP210). ).

After recognizing the movements of the newly restricted target objects O1 and O2, or when the surrounding environment has not changed (NO in STEP 208), the possible movement recognition unit 14 has changed the state of the drone D. It is determined whether or not (FIG. 7B / STEP211).

When the state of the drone D has changed (YES in STEP211), the avatar control unit 15 recognizes the action that the target objects O1 and O2 can execute again based on the change in the state of the drone D (FIG. FIG. 7B / STEP212).

Specifically, for example, it is assumed that the state of the drone D has changed from a mere flight state to a state in which the building material M is being transported. In that case, the movable speed and movable area of the drone D change as the state of the drone D changes. Therefore, when such a change occurs, the avatar control unit 15 also changes the executable operation of the target objects O1 and O2 corresponding to the drone D according to the change in the state.

After recognizing the executable operation of the target objects O1 and O2 again, or when the state of the drone D has not changed (NO in STEP211), the VR system S is instructed to end by the user U. It is determined whether or not it was present (Fig. 7B / STEP 213).

If there is no end instruction (NO in STEP 213), the process returns to STEP 208 and the processes of STEP 208 to STEP 213 are executed again.

On the other hand, if there is an instruction to end (YES in STEP 213), the VR system S ends this process.

[Process to be executed when operating an object and when operating a drone]
Next, with reference to FIGS. 2 and 8, each processing unit of the VR system S performs a process to be executed when the user U operates the target objects O1 and O2 via the avatar A, and a drone D. The process to be executed when operating is described.

In this process, first, the user motion recognition unit 11 determines whether or not the user U has operated (FIG. 8 / STEP300).

If the user is not operating (NO in STEP300), the determination of STEP300 is executed again in a predetermined control cycle.

On the other hand, when the user is operating (YES in STEP300), the avatar control unit 15 operates the avatar A based on the operation of the user U (FIG. 8 / STEP301).

Next, the avatar control unit 15 determines whether or not the operation of the avatar A is such that the target objects O1 and O2 are operated (FIG. 8 / STEP302).

Specifically, the movement of the avatar A is such that the target objects O1 and O2 corresponding to the drone D are moved by using the body such as a hand, or the fifth object O5 corresponding to the controller 5 is used. , The avatar control unit 15 determines whether or not the operation is such that a predetermined operation is performed on the target objects O1 and O2.

When the operation of the avatar A is such that the target objects O1 and O2 are operated (YES in STEP 302), the avatar control unit 15 operates the target objects O1 and O2 based on the operation of the avatar A. (FIG. 8 / STEP303).

Next, the target control unit 17 stores the operations of the target objects O1 and O2 (FIG. 8 / STEP304).

After memorizing the actions of the target objects O1 and O2, or when the action of the avatar A is not an action that causes the target objects O1 and O2 to operate (NO in STEP302), the output information determination unit 16 determines the user. The image and sound to be recognized by U are determined based on the states of the avatar A and the target objects O1 and O2 (FIG. 8 / STEP305).

Next, the output information determination unit 16 displays the determined image on the monitor 40 of the HMD 4 worn by the user U, and generates the determined voice on the speaker 41 (FIG. 8 / STEP306).

Next, the VR system S determines whether or not the user U has instructed the termination (FIG. 8 / STEP307).

If there is no end instruction (NO in STEP307), the process returns to STEP300 and the processes of STEP300 to STEP307 are executed again.

On the other hand, when there is an instruction to end (YES in STEP 307), the target control unit 17 determines whether or not the second period has started (FIG. 8 / STEP 308).

If the second period has not started (NO in STEP308), the determination of STEP308 is executed again in a predetermined control cycle.

On the other hand, when the second period is started (YES in STEP308), the target control unit 17 controls the operation of the drone D based on the stored operations of the target objects O1 and O2 (FIG. 8 / STEP309), this process is completed.

[Difference between the mode controlled in different time zone and the mode controlled in real time]
In the above, the processing in the mode in which the loan D is controlled in a different time zone after the time zone in which the target objects O1 and O2 are operated has been described in detail.

In contrast to the mode in which the drone D is controlled in these different time zones, the mode in which the drone D is controlled in real time starts a process of storing the operation of the target objects O1 and O2 (FIG. 8 / STEP304) and the start of the second period. The difference is that the process of determining (FIG. 8 / STEP308) is omitted, and the process of controlling the drone D (FIG. 8 / STEP309) is executed before the process of determining the end (FIG. 8 / STEP307).

Then, in both the mode of controlling the drone D in different time zones and the mode of controlling the drone D in real time, the VR system S corresponds to the user U as an experience-based interface system according to the operation of the user U. The operation of the avatar A is controlled, and the operation of the target objects O1 and O2 is controlled according to the operation of the avatar A.

As a result, the user U operates the target objects O1 and O2 in the virtual space VS via the avatar A corresponding to the user U, thereby operating the drone D corresponding to the target objects O1 and O2 in the second region RS2. It is designed to be able to be made to.

However, the operation of the target objects O1 and O2 in the virtual space VS is an operation that can be executed by the drone D existing in the second area RS2 having at least one of the time and the position different from the first area RS1 in which the user U exists. It is controlled while limiting based on. That is, the operation is controlled while being restricted based on the constraints in the real space in which the drone D exists (for example, the surrounding environment of the drone D, the function of the drone D, etc.).

As a result, even if the user U tries to operate the target objects O1 and O2 via the avatar A, the operation seems to ignore the operation corresponding to the operation that the drone D cannot execute (that is, the constraint in the real space is ignored). In the case of (operation), the user U cannot naturally execute the operation even in the virtual space VS.

Therefore, according to the VR system S, the operations of the target objects O1 and O2 corresponding to the operations that the drone D cannot execute cannot be executed naturally even in the virtual space VS, so that the user U has the drone D. The operation of the drone D in the second region RS2 can be controlled according to the second region RS2 without considering the restrictions in the second region RS2 in the real space.

Further, in the VR system S, when the shape of the first region RS1 and the shape of the second region RS2 are different, the shape of the virtual space VS is generated so as to correspond to the shape of the first region RS1. As a result, even in such a case, the virtual space VS corresponds to the original operable range of the user U, so that the user U does not mistakenly recognize the operable range by himself / herself. (For example, contact with the furniture F existing in the first region RS1 is suppressed), and the avatar A can be operated.

Further, in such a case, in the VR system S, the target is based not only on the operation that the drone D can execute in the second region RS2 but also on the difference between the shape of the first region RS1 and the shape of the second region RS2. The operation of objects O1 and O2 is restricted.

Thereby, even in such a case, the operation of the target objects O1 and O2 corresponding to the operation that makes the drone D infeasible can be naturally restricted. As a result, even in such a case, the user U can perform the operation of the drone D in the second region RS2 in the second region RS2 without considering the difference between the shape of the first region RS1 and the shape of the second region RS2. It can be controlled according to.

Further, in the VR system S, by providing the motion storage unit 17, the length of the first period, which is the operable period of the avatar A in the virtual space VS, and the operable period of the target objects O1 and O2 in the second region RS2 Even if the length of a certain second period is different, the operation of the drone D can be controlled.

Further, in such a case, in the VR system S, the target object O1 is based not only on the operation that the drone D can execute in the second region RS2 but also on the difference between the length of the first period and the length of the second period. , The operation of O2 is restricted.

Thereby, even in such a case, the operation of the target objects O1 and O2 corresponding to the operation that makes the drone D infeasible can be naturally restricted. As a result, even in such a case, the user U adjusts the operation of the drone D in the second period to the second region RS2 without considering the difference between the length of the first period and the length of the second period. Can be controlled.

[Difference between the mode to control in different time zone and the mode to verify the operation]
Further, in contrast to the mode in which the drone D is controlled in different time zones described in detail, the mode in which the drone D is not actually controlled and its operation is verified is the surrounding environment of the first region RS1 and the second region RS2. Processing that considers the first period and the second period (FIG. 6 / STEP107, 108, FIG. 7A / STEP201, 202, 204 to 207, FIG. 7B / STEP208 to 2013), and processing that actually controls the drone D (FIG. 8 / STP304,308,309) is omitted.

Then, in the mode for verifying the operation, the VR system S performs the operation that the target objects O1 and O2 can execute as the operation experience system, the difference between the shape of the first region RS1 and the shape of the second region RS2, and It is controlled based only on the function of the drone D. That is, the surrounding environment of the second region RS2 in which the drone D exists, the difference between the length of the first period and the length of the second period, etc. are not taken into consideration.

As a result, in the mode for verifying the operation, the user U can experience the operation that the drone D can execute as a function.

On the other hand, in the mode other than the mode for verifying the operation, the operations that the target objects O1 and O2 can execute in the virtual space VS are the surrounding environment of the second region RS2, the length of the first period and the length of the second period. It is controlled while limiting based on the constraints caused by differences and the like. That is, in the mode other than the mode for verifying the operation, the operation that the target objects O1 and O2 can execute corresponds to the operation that the target objects O1 and O2 can actually execute in the second region RS2.

Thereby, in the mode other than the mode for verifying the operation, the user U can experience the operation that the drone D can actually execute in the second area RS2. From the viewpoint of experiencing the operation that the drone D can actually perform, it is not always necessary to actually operate the drone D. Therefore, when the purpose is only to experience the operation, the drone D may or may not be actually operated.

Therefore, according to the VR system S, the user U switches the operation of the target objects O1 and O2 (and thus the drone D) by switching between the mode for verifying the operation and the mode for actually controlling the operation of the drone D. , You can experience it while comparing. As a result, according to the VR system S, the user U can sensuously verify the operation of fully exerting the function of the drone D under a predetermined environment.

Note that, depending on the user U, the drone D may have sufficient knowledge about the operations that can be executed as a function. In such a case, there is a case where only one of a mode for controlling the drone D in real time and a mode for controlling the drone D in a different time zone after the time zone in which the target objects O1 and O2 are operated is provided. Even if there is, the VR system S can be used as an operation experience system.

Further, for example, when there are a plurality of types of drones and the operation that each drone can perform is verified in order to select a drone that is easy to operate in the second region RS2 from among them, it is a candidate. The VR system S is used to generate an object corresponding to the virtual space VS based only on the function of each drone, thereby verifying the action that each object (and thus each candidate drone) can execute. You may.

In such a case, it is not always necessary to actually operate each candidate drone in the second area RS2. This is because, in such a case, when the drone is actually operated in the second region, there is a possibility that the drone may come into contact with the surrounding environment in the second region. Therefore, in principle, it is recommended not to actually operate the drone in such a case, except when there are almost no restrictions in the second area.

1 ... Server, 2 ... Marker, 3 ... First camera, 4 ... HMD, 5 ... Controller, 6 ... Second camera, 10 ... Virtual space generation unit (space recognition unit, constraint recognition unit), 11 ... User motion recognition unit , 12 ... mode switching unit, 13 ... period recognition unit, 14 ... possible motion recognition unit (target function recognition unit), 15 ... avatar control unit, 16 ... output information determination unit (image determination unit), 17 ... target control unit ( Operation storage unit), 40 ... monitor (image display), 41 ... speaker (voice generator), A ... avatar (first avatar), D1 ... first drone, D2 ... second drone, F ... furniture, LA1 ... 1st constraint area, LA2 ... 2nd constraint area, LA3 ... 3rd constraint area, M ... building materials, O1 ... 1st object (2nd avatar), O2 ... 2nd object (2nd avatar), O3 ... 3rd Object, O4 ... 4th object, O5 ... 5th object, RS1 ... 1st area, RS2 ... 2nd area, S ... VR system, U ... user, VS ... virtual space, W ... work machine.

Claims (4)

1. Through the virtual space, a user existing in the first region of the real space controls the operation of the controlled object existing in the second region of the real space where at least one of the time and the position is different from the first region. It is a hands-on interface system for
   A virtual space generation unit that generates a virtual space in which a first avatar corresponding to the user and a second avatar corresponding to the controlled object exist.
   A user motion recognition unit that recognizes the user's motion in the first region,
   An avatar control unit that controls the operation of the first avatar according to the operation of the user and controls the operation of the second avatar according to the operation of the first avatar.
   An image determination unit that determines an image of the virtual space to be recognized by the user based on the state of the first avatar and the state of the second avatar.
   An image display that causes the user to recognize the determined image of the virtual space, and
   A possible motion recognition unit that recognizes an action that the controlled object can execute in the second region,
   A target control unit that controls the operation of the controlled object according to the operation of the second avatar is provided.
   The avatar control unit is a sensory interface system characterized in that the second avatar controls the operations that can be executed by the second avatar based on the operations that can be executed by the controlled object in the second region.
2. In the experience-based interface system according to claim 1,
   A space recognition unit that recognizes the shape of the first region and the shape of the second region is provided.
   The virtual space generation unit generates the virtual space so that the shape of the virtual space corresponds to the shape of the first region.
   The avatar control unit can execute the second avatar based on the operation that the controlled object can execute in the second region and the difference between the shape of the first region and the shape of the second region. A hands-on interface system that is characterized by controlling operations while limiting them.
3. In the experience-based interface system according to claim 1 or 2.
   A period recognition unit that recognizes the length of the first period, which is the operable period of the second avatar in the virtual space, and the length of the second period, which is the operable period of the controlled object in the second region. ,
   A motion storage unit for storing the motion of the second avatar is provided.
   The avatar control unit can execute the second avatar based on the operation that the controlled object can execute in the second region and the difference between the length of the first period and the length of the second period. Control while limiting movement,
   The target control unit is a sensory interface system characterized in that the control target is controlled in the second period according to the operation of the second avatar stored in the motion storage unit in the first period. ..
4. Through the virtual space, a user existing in the first region of the real space can perform an operation that can be executed by a controlled object existing in the second region of the real space where at least one of the time and the position is different from the first region. It is a movement experience system for experiencing,
   A virtual space generation unit that generates a virtual space in which a first avatar corresponding to the user and a second avatar corresponding to the controlled object exist.
   A user motion recognition unit that recognizes the user's motion in the first region,
   An avatar control unit that controls the operation of the first avatar according to the operation of the user and controls the operation of the second avatar according to the operation of the first avatar.
   A mode switching unit that switches between the first mode and the second mode,
   An image determination unit that determines an image of the virtual space to be recognized by the user based on the state of the first avatar and the state of the second avatar.
   An image display that causes the user to recognize the determined image of the virtual space, and
   A target function recognition unit that recognizes an operation that the control target can execute as a function,
   It is provided with a constraint recognition unit that recognizes constraints when the control target operates in the second region.
   In the first mode, the avatar control unit controls an operation that can be executed by the second avatar based only on the function of the controlled object, and in the second mode, the function of the controlled object and the second. A motion experience system characterized in that the second avatar controls while limiting the actions that can be performed based on the constraints in the area.

Images