JP2020177347A - Information processing apparatus and information processing method - Google Patents

Abstract

To provide an information processing apparatus and an information processing method capable of ensuring a degree of freedom of operation of a virtual object and achieving consistency between the virtual object and a real object in a region, in particular, where a high degree of consistency is required.SOLUTION: An information processing apparatus 100 includes a posture acquisition unit 110 for detecting a posture of a first user, a video output unit 120 for outputting an image of an avatar 10 of the first user, a real space information acquisition unit 141 for detecting a shape of a real object in a real space, an actuator position acquisition unit 142 for determining an operation unit 2 as a high consistency object and acquiring a position of the operation unit 2, a high consistency region setting unit 143 for setting an area around the operation unit 2 as a high consistency region, and a display control unit 150 for correcting a movement or position of at least a part of the avatar 10 when the avatar 10 and the high consistency region 3 approach or overlap with each other.SELECTED DRAWING: Figure 2

Description

The present invention relates to an information processing device and an information processing method.

When a user wearing an HMD (head-mounted display) visually recognizes a virtual object in mixed reality space, the image of the virtual object may interfere with the real object such as the ground or a building in the real space. Patent Document 1 describes an information processing device capable of detecting an inconsistency when there is a contradiction in the geometrical consistency between a virtual object and a real object.

JP-A-2018-106661

However, in the information processing device of Patent Document 1, since the movement of the virtual object operated by human operation is not taken into consideration, the interference between the virtual object and the real object is predicted in advance, and the operation or position of the virtual object is determined. There was a problem that it could not be corrected.

On the other hand, if the operation or position of the virtual object is corrected in order to be consistent with all real objects, the degree of freedom of the operation of the virtual object may be excessively limited.

The problem to be solved by the present invention is an information processing device capable of ensuring the degree of freedom of operation of a virtual object and achieving consistency between a virtual object and a real object in an area where particularly high consistency is required. And to provide an information processing method.

The present invention sets a region that requires particularly high consistency as a high consistency region, and corrects the operation or position of the virtual object when the virtual objects approach or overlap the high consistency region. Solve the above problems.

According to the present invention, it is possible to secure the degree of freedom of operation of the virtual object and to obtain the consistency between the virtual object and the real object in the high consistency area.

It is a figure which shows an example of the display mode of the avatar of a 1st user in the interior space of a vehicle. It is a block diagram which shows the structure of the information processing apparatus which concerns on embodiment of this invention. It is a figure which shows the flow of the setting method of the high consistency region among the information processing methods by the information processing apparatus shown in FIG. It is a figure which shows the high consistency area recognized by using the information processing apparatus shown in FIG. It is a figure which shows an example of the aspect which sets the high consistency region using the information processing apparatus shown in FIG. It is a figure which shows an example of the aspect which sets the high consistency region using the information processing apparatus shown in FIG. It is a figure which shows the flow which shows the method of the combination processing of a high consistency area among the information processing methods by the information processing apparatus shown in FIG. It is a figure which shows an example of the mode which connects two adjacent highly consistent regions by using the information processing apparatus shown in FIG. It is a figure which shows an example of the mode which connects two overlapping highly consistent regions by using the information processing apparatus shown in FIG. It is a figure which shows the flow of the correction method of the operation of the avatar of the first user among the information processing methods by the information processing apparatus shown in FIG. It is a figure which shows an example of the mode which corrects the finger movement of the avatar of a 1st user by using the information processing apparatus shown in FIG. It is a figure which shows an example of the mode which corrects the position of the finger of the avatar of a 1st user by using the information processing apparatus shown in FIG.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.
As shown in FIG. 1, the interior space 31 of the vehicle 30 is provided with a driver's seat 21, a passenger seat 11, a dashboard 32, and the like. The driver 20 is seated on the driver's seat 21. The driver 20 is equipped with the HMD 130. Through the HMD 130, the driver 20 can visually recognize the avatar 10 of a virtual person who does not actually exist in the indoor space 31. The avatar 10 is displayed in a posture of being seated on the passenger seat 11. The avatar 10 is displayed on the HMD 130 so as to superimpose on the scenery of the actual indoor space 31. The avatar 10 is a virtual object indicating an outside user 71 in a remote space 70 away from the vehicle 30.
Here, the outside user 71 is the first user, and the driver 20 is the second user. Further, the interior and equipment of the interior space 31 including the driver's seat 21, the passenger seat 11, the dashboard 32, the inner wall of the vehicle 30, the ceiling, the floor, and the like constitute a real object. Further, the indoor space 31 constitutes a real space. Further, the HMD 130 mounted on the driver 20 constitutes a display device.

The outside user 71 wears the HMD 111 and holds the tracker 112 in each of his hands. On the HMD 111, a three-dimensional image of the scenery of the interior space 31 of the vehicle 30, including the driver 20, is displayed as a virtual reality. The posture of the outside user 71 is detected by the HMD 111 and the tracker 112, and is reflected in the posture of the avatar 10. The posture of the outside user 71 detected by the HMD 111 and the tracker 112 includes the movement and position of the hand of the outside user 71. In the example shown in FIG. 1, the avatar 10 extends the right arm toward the operation unit 2 of the dashboard 32 by operating the tracker 112 held by the outside user 71 in the right hand. The operation unit 2 is an operation button for operating the in-vehicle device of the vehicle 30. The in-vehicle device operated via the operation unit 2 is either a driving operation-related device, an indoor environment adjusting device, or an information device. Specifically, the driving operation-related equipment includes a steering wheel, an accelerator, a brake, a shift lever, and the like. Indoor environment adjustment equipment includes power windows, air conditioning systems, room lamps, seat position adjustment units, seat heaters, and the like. The information device is a navigation system, a radio, a volume control unit or other in-vehicle information terminal.
Here, the operation unit 2 constitutes a highly consistent object.

The overall configuration of the information processing apparatus 100 that outputs the image of the avatar 10 to the HMD 130 mounted by the driver 20 will be described with reference to FIG.
The information processing device 100 includes a posture acquisition unit 110, a video output unit 120, an HMD 130, a real space information processing unit 140, and a display control unit 150. The video output unit 120, the real space information processing unit 140, and the display control unit 150 are mounted on the HMD 130. Further, the posture acquisition unit 110 includes an HMD 111 and a tracker 112 mounted on the user 71 outside the vehicle. The information regarding the posture of the outside user 71 detected by the HMD 111 and the tracker 112 is transmitted to the display control unit 150 mounted on the HMD 130 mounted by the driver 20.

The real space information processing unit 140 includes a real space information acquisition unit 141, an operation unit position acquisition unit 142, a high consistency area setting unit 143, and an output unit 144. The real space information acquisition unit 141 is a laser scanner attached to the HMD 130, and acquires the three-dimensional shape of the real object in the interior space 31 of the vehicle 30 as a point cloud. Further, the operation unit position acquisition unit 142 is an optical sensor attached to the HMD 130. The operation unit position acquisition unit 142 recognizes the operation unit 2 which is a part of the real object by image recognition, point cloud segmentation, or marker recognition, and determines the operation unit 2 as a highly consistent object. Further, the operation unit position acquisition unit 142 acquires the position of the operation unit 2. It should be noted that one optical sensor attached to the HMD 130 may simultaneously have a function of forming the real space information acquisition unit 141 and a function of forming the operation unit position acquisition unit 142.
Here, the operation unit position acquisition unit 142 constitutes a highly consistent object position acquisition unit.

The high consistency area setting unit 143 sets a high consistency area including the operation unit 2 around the operation unit 2 based on the data acquired from the real space information acquisition unit 141 and the operation unit position acquisition unit 142. The output unit 144 outputs information such as the interior of the interior space 31 of the vehicle 30, the position information of the operation unit 2, and the information of the high consistency area to the display control unit 150 of the HMD 111 of the outside user 71 and the HMD 130 of the driver 20. ..

The display control unit 150 has a correction determination unit 151 that determines whether or not the position and operation of the avatar 10 should be corrected, and a correction processing unit 152 that corrects the position and operation of the avatar 10 based on the determination by the correction determination unit 151. And have.

Hereinafter, the information processing method by the information processing apparatus 100, particularly the setting method of the high consistency region 3 will be described with reference to FIGS. 3 to 10.
As shown in FIG. 3, the real space information acquisition unit 141 of the information processing apparatus 100 recognizes the shape of the real object in the interior space 31 of the vehicle 30 in step S1. Next, the operation unit position acquisition unit 142 of the information processing device 100 recognizes the position of the operation unit 2 in step S2. Further, in step S3, the operation unit position acquisition unit 142 of the information processing device 100 determines whether or not the operation unit 2 is among the real objects in the indoor space 31. When the operation unit 2 is located in the indoor space 31, the high consistency area setting unit 143 sets the high consistency area 3 around the operation unit 2.

According to the information processing method shown in FIG. 3, as shown in FIG. 4, the real space information acquisition unit 141 mounted on the HMD 130 acquires the three-dimensional shape of the real object in the indoor space 31 as the point cloud 1. Although the point cloud 1 is represented as a plane in FIG. 4 for convenience of explanation, it actually has a concave-convex shape along the surface shape of each real object. Further, the operation unit position acquisition unit 142 recognizes the position of the operation unit 2 in the point cloud 1. Then, as shown in FIGS. 5A and 5B, the high consistency area setting unit 143 sets the high consistency area 3 including the operation unit 2 in correspondence with the point cloud 1 acquired by the real space information acquisition unit 141. To do. The high consistency region 3 is a region formed on the point cloud 1. Further, the high consistency region 3 may have a rectangular shape as shown in FIG. 5A, or may have a circular shape as shown in FIG. 5B. The range of the high consistency region 3 is determined based on a predetermined ratio according to the size of each operation unit 2.

Next, a method of combining the high consistency regions 3 when a plurality of high consistency regions 3 are adjacent to each other or overlap each other will be described with reference to FIGS. 6, 7A and 7B.
As shown in FIG. 6, first, the real space information processing unit 140 of the information processing device 100 acquires the shape of the real object in the indoor space 31 and the position of the operation unit 2 in step S11, and in step S12, the height is increased. Consistency area 3 is set. Further, in step S13, the high consistency area setting unit 143 of the real space information processing unit 140 determines whether or not a plurality of high consistency areas 3 are adjacent to each other, or the high consistency areas 3 overlap each other. Judge whether or not. When a plurality of high-consistency regions 3 are adjacent to each other or the high-consistency regions 3 overlap each other, the high-consistency region setting unit 143 of the plurality of high-consistency regions 3 in step S14. Perform the join process.

Specifically, as shown in FIG. 7A, when the distance between the high consistency regions 3a and 3b set around each of the operation units 2a and 2b is a predetermined distance D or less, the high consistency regions 3a and 3b They are determined to be adjacent to each other and are combined. Further, as shown in FIG. 7B, even when the high consistency regions 3c and 3d set around each of the operation units 2c and 2d overlap each other, the high consistency regions 3c and 3d are combined to form one height. It is set as the consistency area 3. Further, in a place where three or more operation units are densely packed, the high consistency regions may be combined together to form one high consistency region. In addition, in FIGS. 7A and 7B, the two-dimensional positional relationship of each high consistency region is shown, but when a plurality of high consistency regions are separated from each other in the depth direction, the high consistency region is shown. It is not necessary to combine them. Further, when the method of correction processing of each high consistency region is different, even if they are adjacent to each other or overlap each other, these high consistency regions are not combined with each other.

Next, a method of restricting the operation of the avatar 10 with respect to the high consistency region 3 will be described with reference to FIGS. 8 to 10.
First, in step S21, the video output unit 120 outputs the video of the avatar 10 reflecting the posture of the user 71 outside the vehicle to the HMD 130 mounted by the driver 20, and the display of the avatar 10 is started. Next, in step S22, the display control unit 150 acquires the position of the avatar 10 from the posture acquisition unit 110. Further, the display control unit 150 acquires the position of the high consistency region 3 from the real space information processing unit 140, and in step S23, determines whether or not the distance X between the avatar 10 and the high consistency region 3 is equal to or less than the reference distance. judge. When the distance X between the avatar 10 and the high consistency region 3 is equal to or less than the reference distance, the display control unit 150 limits the operation of the avatar 10 in step S24. On the other hand, when the distance X between the avatar 10 and the high consistency region 3 is larger than the reference distance, the display control unit 150 releases the restriction on the operation of the avatar 10 in step S25.
The reference distance is a distance at which the avatar 10 is close to the high consistency region 3 to the extent that the intention of the outside user 71 to access the operation unit 2 can be clearly recognized.

Specifically, as shown in FIG. 9, when the distance X between the finger 10a of the hand of the avatar 10 and the high consistency region 3 is equal to or less than the reference distance, the movement of the finger 10a moving in the direction of the arrow is corrected. , Limited. As a result, the finger 10a of the avatar 10 is prevented from penetrating the high consistency region 3, and the HMD 130 of the driver 20 has an image in which the finger 10a of the avatar 10 is in natural contact with the high consistency region 3. Is displayed. Further, as shown in FIG. 10, when the distance X between the finger 10a of the avatar 10 and the high consistency region 3 is equal to or less than the reference distance, the finger 10a of the avatar 10 is attracted to the operation unit 2 and comes into contact with the operation unit 2. The display control unit 150 may correct the position of the finger 10a.

Further, when the finger 10a of the avatar 10 and the high consistency area 3 overlap, that is, when the finger 10a of the avatar 10 penetrates the high consistency area 3, the display control unit 150 controls the finger 10a of the avatar 10. Corrects the position of the finger 10a so that is in contact with the high consistency region 3.

Further, as shown in FIG. 8, the information processing apparatus 100 determines in step S26 whether or not to continue displaying the avatar 10 on the HMD 130. For example, when the power of the HMD 111 of the outside user 71 or the HMD 130 of the driver 20 is turned off, it is determined that the display of the avatar 10 is not continued, and the display of the avatar 10 ends in step S27. On the other hand, if it is determined in step S26 that the display of the avatar 10 is continued, the control returns to step S22.

Based on the above, the information processing apparatus 100 of the present embodiment sets a predetermined region including the operation unit 2 as a highly consistent region 3 that requires particularly high consistency. The information processing device 100 corrects the movement or position of the finger 10a of the avatar 10 when the finger 10a of the hand of the avatar 10 and the high consistency region 3 are close to each other or overlap each other. As a result, the information processing device 100 can make the avatar 10 and the real object consistent around the operation unit 2. Therefore, the intention of the outside user 71 to access the operation unit 2 is visually and more reliably transmitted to the driver 20. Further, the driver 20 can clearly visually recognize and identify which device is operated by the outside user 71 via the avatar 10. Further, when the driver 20 sees the avatar 10 via the HMD 130, the operation of the avatar 10 is felt more naturally, and the immersive feeling in the augmented reality can be enhanced. Further, since only the periphery of the operation unit 2 that requires particular consistency is set as the high consistency area 3, the limited range of the operation of the avatar 10 can be minimized, and the operation of the avatar 10 even in the narrow indoor space 31. The degree of freedom can be secured. Further, since the range for correcting the operation or position of the avatar 10 is limited, it is possible to suppress an increase in the calculation load of the information processing device 100.

Further, the in-vehicle device operated via the operation unit 2 is a driving operation-related device, an indoor environment adjusting device, or an information device. As a result, the user 71 outside the vehicle can operate the in-vehicle device to adjust the environment of the indoor space 31 in place of the driver 20 during driving or assist the driver 20 in driving.

The display control unit 150 corrects the movement or position of the finger 10a of the avatar 10 so that the finger 10a of the avatar 10 comes into contact with the surface of the operation unit 2. As a result, the intention of the outside user 71 to access the operation unit 2 can be reliably conveyed to the driver 20 without any discomfort.

Further, the video output unit 120 outputs the video of the avatar 10 so as to be displayed on the HMD 130 mounted by the driver 20. As a result, the driver 20 can recognize the operation of the avatar 10 via the HMD 130, and can visually confirm the intention of the outside user 71 to access the operation unit 2.

In addition, the display control unit 150 combines a plurality of highly consistent regions 3 that are adjacent to each other or overlap each other. As a result, the calculation load of the information processing apparatus 100 can be reduced. Further, when the user 71 outside the vehicle operates a plurality of adjacent operation units 2 in order, if the plurality of highly consistent regions 3 are connected to each other, the finger 10a of the avatar 10 is one combined high-matched region. It moves along the sex region 3. Therefore, while the outside user 71 is operating the plurality of operation units 2, the consistency between the finger 10a of the avatar 10 and the real object is always maintained, so that the operation of the avatar 10 as seen from the driver 20 is easier to understand. , Will be natural.

In the present embodiment, the actual space is the interior space 31 of the vehicle 30, but the space is not limited to this and may be the interior space inside the building. The highly consistent object of the indoor space is preferably an operation unit that controls the environment of the indoor space, such as a temperature controller of an air conditioner. Further, the real space is not limited to the indoor space, but may be an outdoor space.

The virtual object is not limited to the avatar of a person, and may imitate a character such as an animal or a robot.

The display control unit 150 displays at least a part of the avatar 10 when the distance between the avatar 10 and the high consistency area 3 is equal to or less than a predetermined reference distance, or when the avatar 10 and the high consistency area 3 overlap each other. The morphology may be corrected. Specifically, when the distance between the hand of the avatar 10 in the fully opened state and the highly consistent region 3 becomes equal to or less than a predetermined reference distance, the display control unit 150 causes the hand of the avatar 10 to hold only the index finger. The display form of the hand of the avatar 10 is corrected so that the shape is grasped in the protruding state. As a result, it becomes clear that the finger 10a of the avatar 10 points to the operation unit 2, and the driver 20 can more accurately confirm the intention of the outside user 71 to touch the operation unit 2.

The target for which the display control unit 150 corrects the operation or position may be only a part that actually executes the operation, such as the hand or finger of the avatar 10, or the entire avatar 10 may be the target for correction.

The highly consistent region 3 shown in FIGS. 4, 5 and 9 is described as a region having a two-dimensional range, but is not limited to this, and a spherical shape or a cube shape centered on the operation unit 2 or the like. A three-dimensional range that extends three-dimensionally may be used as a highly consistent region.

When the finger 10a of the avatar 10 moves between two or more highly consistent regions 3 separated from each other, the movement of the finger 10a is corrected so as to be smoother, and the high consistency region 3 and the other regions are corrected. The sudden change in the movement of the finger 10a when passing through the boundary of the finger 10a may be prevented.

The display device worn by the driver 20 is not limited to the HMD 130, and may be AR goggles or smart contact lenses.

In FIGS. 1, 4, 5A, 5B, 7A and 7B, only one operation button of the operation unit 2 is shown, but the operation unit 2 is not limited to this, and the operation unit 2 has two or more operation buttons or a touch panel. It may be composed of.

In the present embodiment, the posture of the outside user 71 is detected by the HMD 111 and the tracker 112, but the posture of the outside user 71 may be detected by the controller used for operating the avatar 10.

The operation unit 2 described above corresponds to a highly consistent object according to the present invention. The avatar 10 described above corresponds to a virtual object according to the present invention. The above-mentioned outside user 71 corresponds to the first user according to the present invention. The above driver 20 corresponds to a second user according to the present invention. The above-mentioned indoor space 31 corresponds to the real space according to the present invention. The above-mentioned HMD 111 and tracker 112 correspond to the posture acquisition unit according to the present invention. The above-mentioned HMD 130 corresponds to the display device according to the present invention. The operation unit position acquisition unit 142 corresponds to the highly consistent object position acquisition unit according to the present invention.

100 ... Information processing device 2 ... Operation unit (highly consistent object)
3 ... Highly consistent area 10 ... Avatar (virtual object)
10a ... Avatar's finger 20 ... Driver (second user)
30 ... Vehicle 31 ... Interior space (real space)
71 ... Outside user (first user)
110 ... Posture acquisition unit 111 ... HMD (Posture acquisition unit)
112 ... Tracker (posture acquisition section)
120 ... Video output unit 130 ... HMD (display device)
141 ... Real space information acquisition unit 142 ... Operation unit position acquisition unit (highly consistent object position acquisition unit)
143 ... High consistency area setting unit 150 ... Display control unit

Claims (9)

A posture acquisition unit that detects the posture of the first user,
A video output unit that outputs a video of a virtual object that can be seen by a second user based on the posture of the first user detected by the posture acquisition unit.
A real space information acquisition unit that detects the shape of a real object in the real space in which the second user exists, and
A highly consistent object position acquisition unit that determines at least a part of the real object detected by the real space information acquisition unit as a highly consistent object and acquires the position of the highly consistent object.
A high-consistency area setting unit that sets a predetermined area including the high-consistency object as a high-consistency area,
When the distance between the virtual object and the highly consistent area is equal to or less than a predetermined reference distance, or when the virtual object and the highly consistent area overlap, at least a part of the operation or position of the virtual object is performed. An information processing device including a display control unit for correction. The real space in which the second user exists is an indoor space.
The information processing device according to claim 1, wherein the highly consistent object is an operation unit that controls the environment of the indoor space. The real space in which the second user exists is the interior space of the vehicle.
The highly consistent object is an operation unit for operating an in-vehicle device of the vehicle.
The information processing device according to claim 1, wherein the in-vehicle device is any of a driving operation-related device, an indoor environment adjusting device, and an information device. Any one of claims 1 to 3, wherein the display control unit corrects the operation or position of at least a part of the virtual object so that at least a part of the virtual object comes into contact with the surface of the highly consistent object. The information processing device according to paragraph 1. The virtual object is an avatar of a person indicating the first user.
The display control unit determines the avatar when the distance between the finger of the avatar's hand and the highly consistent region is equal to or less than a predetermined reference distance, or when the finger of the avatar and the highly consistent region overlap. The information processing apparatus according to any one of claims 1 to 4, which corrects the movement or position of the finger. The information processing device according to any one of claims 1 to 5, wherein the video output unit outputs a video of the virtual object so as to be displayed on a display device worn by the second user. The display control unit is at least one of the virtual objects when the distance between the virtual object and the high consistency area is equal to or less than a predetermined reference distance, or when the virtual object and the high consistency area overlap. The information processing apparatus according to any one of claims 1 to 6, wherein the display form of the unit is corrected. The information processing apparatus according to any one of claims 1 to 7, wherein the display control unit combines a plurality of the high consistency regions that are adjacent to each other or overlap each other. Steps to detect the posture of the first user and
A step of outputting a video of a virtual object that can be seen by a second user based on the posture of the first user, and
The step of detecting the shape of the real object in the real space where the second user exists, and
A step of determining at least a part of the real object as a highly consistent object and acquiring the position of the highly consistent object, and
A step of setting a predetermined area including the highly consistent object as a high consistency area, and
When the distance between the virtual object and the highly consistent area is less than or equal to a predetermined distance, or when the virtual object and the highly consistent area overlap, at least a part of the operation or position of the virtual object is corrected. An information processing method that includes steps to perform.

Images