WO2019130991A1

WO2019130991A1 - Information processing device

Info

Publication number: WO2019130991A1
Application number: PCT/JP2018/044278
Authority: WO
Inventors: 敬幸古田; 雄太樋口; 和輝東
Original assignee: 株式会社Ｎｔｔドコモ
Priority date: 2017-12-26
Filing date: 2018-11-30
Publication date: 2019-07-04
Also published as: JP2021043476A

Abstract

An information processing device 10 according to an embodiment is for providing an image of a virtual space V to be displayed on an HMD 1 worn by a user. The information processing device 10 comprises: an image acquisition unit 12 for acquiring a real-space image obtained by imaging a real space in the vicinity of the user; a virtual object generation unit 13 for recognizing an object 6 included in the real-space image and generating, in the virtual space V, a virtual object 8 corresponding to the object 6; and an image generation unit 16 for generating a virtual space image to be displayed on the HMD 1 that shows at least a part of the virtual space V including the virtual object 8.

Description

Information processing device

One aspect of the present invention relates to an information processing apparatus.

Conventionally, by providing a user with a head mounted display (HMD) or the like with a view as if the user were present in the virtual space, the virtual reality (VR: Virtual Reality) is embedded with the virtual space. A technique called is known (see, for example, Patent Document 1). In such VR technology, for example, a user object (avatar, character, etc.) interlocked with the user's action (for example, the action of a part of the body such as the head and hands) is generated on the virtual space and the user's action It is controlled according to Then, by displaying an image showing a view seen from the user object on the HMD, the user is provided with an experience as if the user object exists in the virtual space.

JP, 2017-55851, A

However, for example, when performing business communication such as a meeting between a plurality of users via a virtual space, it is impossible for the user to recognize at all things (for example, proceedings memo, notebook PC, etc.) at hand. It may result in the loss of convenience.

Therefore, one aspect of the present invention is to provide an information processing apparatus capable of improving the convenience of a virtual reality experience of a user.

An information processing apparatus according to an aspect of the present invention is an information processing apparatus for providing an image of a virtual space displayed on a display device worn by a user, the real space image obtained by imaging a real space in the vicinity of the user An image acquisition unit to acquire, a virtual object generation unit that recognizes an object included in a real space image and generates a virtual object corresponding to the object in a virtual space, and at least a part of a virtual space including the virtual object And an image generation unit configured to generate the virtual space image to be displayed on the display device.

According to the information processing apparatus according to one aspect of the present invention, an object included in the real space image obtained by imaging the real space near the user is generated as a virtual object in the virtual space, and a virtual space image including the virtual object (A virtual space image in which the virtual object is reflected) is generated. By such processing, the user wearing the display device can visually recognize an object present in the vicinity of the user via the virtual space image. Therefore, according to the information processing apparatus, the convenience of the virtual reality experience of the user can be improved.

According to one aspect of the present invention, it is possible to provide an information processing apparatus capable of improving the convenience of the user's virtual reality experience.

It is a figure showing functional composition of an information processing system containing an information processor concerning one embodiment. It is a figure for demonstrating acquisition of a real space image, and generation of a virtual object. It is a sequence diagram which shows an example of operation | movement of an information processing system. It is a sequence diagram which shows an example of operation | movement of an information processing system. It is a sequence diagram which shows an example of operation | movement of an information processing system. It is a block diagram showing an example of the hardware constitutions of an information processor.

Hereinafter, an embodiment of the present invention will be described in detail with reference to the attached drawings. In the description of the drawings, the same or corresponding elements will be denoted by the same reference symbols, without redundant description.

FIG. 1 is a diagram showing a functional configuration of an information processing system 100 including an information processing apparatus 10 according to an embodiment of the present invention. The information processing apparatus 10 is an apparatus for providing a user with a virtual space in which arbitrary VR contents such as a game space and a chat space are expanded, via a head mounted display (HMD) 1 (display device) mounted to the user. is there. That is, the information processing apparatus 10 is an apparatus that provides a user with a virtual reality (VR) experience through an image of a virtual space displayed on the HMD 1. The information processing apparatus 10 has a function of generating a virtual object corresponding to the object from the object existing in the real space on the virtual space.

In the present embodiment, a case where a plurality of user objects (avatar, characters, etc.) operated by each of a plurality of users exist in the same virtual space will be described. However, the processing of the information processing apparatus 10 described later can be applied even when only a user object of one user exists in the same virtual space.

As illustrated in FIG. 1, the information processing apparatus 10 includes a communication unit 11, an image acquisition unit 12, a virtual object generation unit 13, a virtual object storage unit 14, a sharing setting unit 15, and an image generation unit 16. , An object detection unit 17 and a virtual object update unit 18. The information processing apparatus 10 is, for example, a game terminal, a personal computer, a tablet terminal or the like that can communicate with the plurality of HMDs 1 attached by each of the plurality of users. However, the implementation form of the information processing apparatus 10 is not limited to a specific form. For example, the information processing device 10 may be a computer device incorporated in the same device as the HMD 1. Further, the information processing apparatus 10 may be a server device or the like that can communicate with each of the HMDs 1 (or each computer terminal that controls the operation of each HMD 1) of each of a plurality of users via a communication line such as the Internet. Further, the information processing apparatus 10 may be physically configured by a single device or may be configured by a plurality of devices. For example, the information processing apparatus 10 is realized by a computer terminal (a computer terminal provided for each HMD 1) that controls the operation of each HMD 1 with some functions (for example, the functions of the image generation unit 16), and other functions are performed. You may be comprised as a distributed system implement | achieved by the server apparatus which can communicate with the said computer terminal.

The HMD 1 is a display device mounted on the body (for example, the head) of the user. The HMD 1 includes, for example, a display unit that displays an image (an image for the left eye and an image for the right eye) in front of each eye of the user in a state of being worn on the head of the user. By displaying different images (videos) on the left-eye image and the right-eye image, a stereoscopic image (three-dimensional image) is recognized by the user. The display unit described above may be a display integrally configured with a main unit mounted on the user's body, such as a glasses type or helmet type, or a device detachably attachable to the main unit of the HMD 1 (For example, a display of a terminal such as a smartphone attached to the main unit) may function as the display unit.

The HMD 1 includes, for example, a sensor (eg, an acceleration sensor, an angular velocity sensor, a geomagnetic sensor, a gyro sensor, etc.) capable of detecting the position, orientation (tilt), velocity, acceleration, etc. of the user's head (ie, the HMD 1). There is. The HMD 1 periodically transmits information on the motion (position, orientation, velocity, acceleration, etc.) of the head of the user detected by such a sensor to the information processing apparatus 10 as motion information on the head of the user.

Further, the HMD 1 includes, for example, a sensor such as an infrared camera that detects an action of the user's eyes (for example, the position and the movement of the black eye portion, etc.). The sensor is, for example, a sensor having a known eye tracking function. The said sensor detects the operation | movement of each eyeball, for example by receiving the reflected light from the cornea etc. of the infrared light irradiated to the user's right eye and left eye. The HMD 1 periodically transmits the operation information of the user's eyes detected as described above to the information processing apparatus 10.

The HMD 1 also includes a microphone (not shown) for inputting the voice of the user wearing the HMD 1 and a speaker (not shown) for outputting voice and the like of each user as accessories. The voice acquired by the microphone is transmitted to the information processing apparatus 10. In addition, the speaker outputs the voice or the like of the other user received from the information processing device 10. With such a microphone and a speaker, it is possible to make conversation (chat) between a plurality of users. The microphone and the speaker may be devices integrated with the HMD 1 or may be devices different from the HMD 1.

The HMD 1 also includes a camera 2 (photographing device) for photographing a space in the vicinity of the user wearing the HMD 1 (in the present embodiment, the space in front of the user) as an accessory. The HMD 1 and the camera 2 can communicate with each other. The camera 2 may be a camera integrally configured with the main body of the HMD 1 or may be a camera provided to a main body of the HMD 1 and a device (e.g., a smartphone or the like) that is detachable.

In the present embodiment, as shown in FIG. 2, the camera 2 of the HMD 1 recognizes a specific area 4 on the desk 3 in front of the user 5 wearing the HMD 1, and an object present on the specific area 4 is Take an image. For example, the specific area 4 is defined by a mat of green back placed on the desk 3 or the like. For example, a sensor (or a marker that can be recognized by the camera 2) that can communicate with the camera 2 is embedded at a specific position (for example, a central position or four corners) of the mat, and the camera 2 communicates with the sensor (or The specific area 4 may be recognized based on the position of the sensor (or the marker) grasped by the recognition of the marker). Note that the camera 2 is not necessarily a device attached to the HMD 1 and may be a camera (a separate device from the HMD 1) fixedly disposed at a position capable of photographing a space including the specific area 4 Good. Further, the camera 2 may be configured by a plurality of fixed cameras that capture a space including the specific area 4 from a plurality of different angles. In this case, it is possible to obtain a three-dimensional image of an object present on the specific region 4 based on images of different angles taken by different fixed cameras.

For example, the camera 2 starts capturing an image of the real space including the specific area 4 in response to an operation by the user on a controller attached to the HMD 1 (or a controller separate from the HMD 1). The video taken by the camera 2 is transmitted to the HMD 1 as needed, and displayed superimposed on the virtual space image displayed on the HMD 1. Here, the virtual space image is an image of the virtual space of the angle specified based on the motion information of the head and eyes of the user wearing the HMD 1. For example, the video captured by the camera 2 may be displayed on a small window (so-called wipe) provided at a corner (for example, the upper right corner or the like) of the virtual space image. In this case, the user can grasp the state of the space (real space) including the specific area 4 by viewing the virtual space image and simultaneously experiencing the virtual reality and confirming the small window-like screen. it can. At this stage, the object on the specific area 4 is not generated as a virtual object. Therefore, the object in the specific area 4 can not be treated (eg, carried) as a thing in the virtual space, and can not be recognized by users other than the user.

The communication unit 11 transmits / receives data to / from an external device such as the HMD 1 (including a microphone, a speaker, a camera 2, a controller, and the like that are accessories of the HMD 1) via a wired or wireless communication network. In the present embodiment, the communication unit 11 receives from the HMD 1 the motion information of the head and eyes of the user acquired in the HMD 1 as described above. Also, the communication unit 11 transmits the image generated by the image generation unit 16 described later to the HMD 1. By this processing, in each HMD 1 worn by each user, an image of a virtual space of an angle determined based on the motion information of the head and eyes of each user is displayed. The communication unit 11 also receives the voice of each user input to the above-described microphone, and transmits the received voice of each user to the speaker of each user. By such processing, the voice is shared among the users, and the above-described chat is realized.

The image acquisition unit 12 acquires a real space image obtained by imaging a real space near the user. For example, the image acquisition unit 12 acquires an image (details will be described later) acquired by the above-described camera 2 as a real space image through the communication unit 11. Also, the virtual object generation unit 13 recognizes an object included in the real space image, and generates a virtual object corresponding to the object in the virtual space. In the present embodiment, the virtual object generation unit 13 generates a virtual object corresponding to an object designated by the user among a plurality of objects included in the real space image. That is, the virtual object generation unit 13 does not immediately generate virtual objects corresponding to all the objects included in the real space image, but generates only virtual objects corresponding to the object designated by the user. By such processing, only the virtual object desired by the user can be generated, and the processing load of generation of the virtual object (hereinafter also referred to as “objectification”) can be reduced. That is, the processing load and usage of hardware resources such as processors and memories can be reduced.

FIG. 2B shows a state in which two objects 6 (6A, 6B) exist on the specific area 4 in front of the user 5. The object 6A is a plastic bottle containing a beverage, and the object 6B is a notebook PC operated by the user 5. For example, in response to an operation (image acquisition instruction) from the user on the controller or the like described above, the camera 2 acquires a real space image including the specific area 4 and transmits it to the HMD 1. By such processing, the real space image including the

objects

6A and 6B is displayed on the HMD 1. Then, for example, the user 5 designates a target area including the object 6 (here, the object 6B as an example here) which is the object of objectification in the real space image by the operation using the above-mentioned controller or the like. Subsequently, the real space image and the information indicating the target area are transmitted from the HMD 1 to the information processing apparatus 10.

The image acquisition unit 12 acquires these pieces of information (information indicating the real space image and the target area) via the communication unit 11. Then, the virtual object generation unit 13 performs known image recognition on the target area in the real space image. By such processing, appearance information of the object 6B included in the target area is extracted. As shown in FIG. 2B, the virtual object generation unit 13 generates a virtual object 8 corresponding to the object 6B based on the appearance information extracted in this manner.

In the example of FIG. 2B, the user object 7 associated with the user 5 is disposed on the virtual space V. In the present embodiment, as an example, the virtual object generation unit 13 is configured such that the relative position of the virtual object 8 to the user object 7 in the virtual space V matches the relative position of the object 6B to the user 5 in the real space. Determine the position of 8. With such processing, the user can perform an operation on the object 6B in the real space by performing an operation (for example, an operation to carry) on the virtual object 8 in the virtual space V via the user object 7 . However, the relative position of the virtual object 8 to the user object 7 may not coincide with the relative position of the object 6B to the user 5. That is, the virtual object generation unit 13 may generate the virtual object 8 at an arbitrary position (for example, a position designated by the user 5) in the virtual space.

The virtual object storage unit 14 stores information on the virtual object generated by the virtual object generation unit 13 (hereinafter, “virtual object information”). In this embodiment, as an example, virtual object information includes, for each virtual object, a virtual object ID for uniquely identifying a virtual object, appearance information for drawing a virtual object, generation time at which a virtual object is generated, virtual A camera ID for uniquely identifying the camera 2 (or the user 5 of the camera 2 or the like) who acquired the real space image on which the object is generated, a user (or the HMD 1 or the like who is permitted to share the virtual object) Share setting information indicating the device). The camera ID is associated with the real space image as additional information, for example, when the real space image is photographed by the camera 2.

Here, the virtual space V is a space shared by a plurality of users. That is, the virtual space V includes at least a first user (user 5 in this case) wearing the first HMD (HMD 1, first display device) and a second user (user 5) wearing the second HMD (HMD 1, second display device) Is a space shared with different users). The virtual space V is, for example, a chat space for conducting business communication such as a meeting among a plurality of users. In such a case, while making the object in the real space into an object, the first user may not want the contents of the virtual object generated by the objectification to be known to users other than the specific user. For example, a virtual object corresponding to a memo or the like in which confidential information is described may be desired to be viewed only by a user having a specific job title or higher.

Therefore, the sharing setting unit 15 shares the virtual object with the second user according to the operation content received from the first user for the virtual object generated by the virtual object generating unit 13 based on the specification by the first user. Set whether or not to In the example of FIG. 2B, for example, a sharing setting screen for setting a user who is permitted to share the virtual object 8 is displayed on the first HMD. On the sharing setting screen, for example, information indicating the appearance and the like of the virtual object 8 which is the target of the sharing setting, a screen for setting a user who permits sharing of the virtual object 8 and the like are displayed. When a plurality of virtual objects are generated by the virtual object generation unit 13, the sharing setting screen may be a setting screen capable of performing the sharing setting of each of the plurality of virtual objects.

The user 5 (first user) designates a user (or a user who does not permit sharing) who permits sharing of the virtual object 8 by performing an operation using the above-described controller or the like on the sharing setting screen. Do. The sharing setting unit 15 acquires setting information generated by such processing, and sets sharing setting information of the virtual object 8 based on the setting information. Specifically, the sharing setting unit 15 accesses virtual object information of the virtual object 8 stored in the virtual object storage unit 14, and sets or updates sharing setting information of the virtual object information.

The image generation unit 16 generates a virtual space image indicating at least a part of the virtual space V including the virtual object 8 generated by the virtual object generation unit 13. Specifically, the image generation unit 16 displays the virtual object 8 in the virtual space image (an image of an angle determined based on the motion information of the head and eyes of the user wearing the HMD 1) displayed on the HMD 1 In the case of including, a virtual space image including the virtual object 8 is generated. When the virtual space V is shared by a plurality of users, the image generation unit 16 generates a virtual space image for each user (for each HMD 1).

Here, in the case where sharing with a certain user (hereinafter, the second user) is not permitted for a certain virtual object (for example, the virtual object 8) by the processing of the sharing setting unit 15 described above (that is, There may be a case where viewing is not permitted to the second user. In such a case, the image generation unit 16 does not display the virtual object 8 that is not permitted to share with the second user in the virtual space image displayed on the HMD 1 (second HMD) of the second user. That is, even if the virtual object 8 is included in the virtual space image for the second HMD, the image generation unit 16 hides the virtual object 8 in the virtual space image. On the other hand, when the image generation unit 16 is permitted to share the virtual object 8 with the second user, and the virtual object 8 is included in the virtual space image for the second HMD, the virtual space for the second HMD is Display the virtual object 8 on the image. By such processing, only the user who is permitted to share the virtual object 8 can recognize the virtual object 8.

The virtual space image for each user (for each HMD 1) generated by the image generation unit 16 is transmitted to the HMD 1 of each user. Through such processing, each user visually recognizes, via the HMD 1, a virtual space image in which display or non-display of the virtual object 8 according to the above-described sharing setting is reflected.

The object detection unit 17 detects an object corresponding to the virtual object from the real space image acquired by the image acquisition unit 12 after the virtual object is generated by the virtual object generation unit 13.

Specifically, the object detection unit 17 detects the object when the same object as the already-objectized object is included in the further acquired real space image. For example, among the one or more pieces of virtual object information stored in the virtual object storage unit 14, the object detection unit 17 may be appearance information (e.g., similar to the appearance of an object included in the real space image further acquired). Appearance information that is recognized as having a certain degree of similarity or more by known image recognition based on the outline, color, shape, etc. of an object, a camera ID indicating the camera 2 that has captured the real space image obtained further Virtual object information is searched in which the generation time past the time when the acquired real space image is acquired is associated. When virtual object information meeting such conditions is extracted, the object detection unit 17 corresponds an object included in the further acquired real space image to a virtual object indicated by the extracted virtual object information. Detect as an object.

The process of the object detection unit 17 described above will be specifically described using the example of FIG. In this example, at a time after the virtual object 8 corresponding to the object 6B (including the screen portion of the notebook PC) is generated, the image acquisition unit 12 acquires a real space image including the object 6B. . In this case, among the one or more pieces of virtual object information stored in the virtual object storage unit 14, the object detection unit 17 selects appearance information similar to the appearance of the object 6B included in the real space image, and the real space The virtual object information in which the camera ID indicating the camera 2 that captured the image and the generation time past the time when the real space image was captured is associated with each other is searched. By such processing, virtual object information of the virtual object 8 is extracted. As a result, the object detection unit 17 detects the object 6B included in the real space image as an object corresponding to the virtual object 8.

The virtual object update unit 18 updates the state of the virtual object corresponding to the object based on the state of the object detected by the object detection unit 17. In the example of FIG. 2B described above, the virtual object update unit 18 updates the state of the virtual object 8 corresponding to the object 6B based on the state of the object 6B detected by the object detection unit 17. For example, the state of the object 6B included in the real space image acquired at a time later than the time when the virtual object 8 is first generated may be different from the state of the object 6B at the time of generation of the virtual object 8. Specifically, the screen (a part of the appearance of the object 6B) of the object 6B (notebook PC) at the later time may be different from the screen of the object 6B at the time of generation of the virtual object 8. Therefore, the virtual object updating unit 18 updates the state of the virtual object 8 corresponding to the object 6B (here, the content of the screen) to the content of the screen of the object 6B captured in the real space image acquired at the later time point Do. Specifically, the virtual object update unit 18 adds the appearance information of the virtual object information of the virtual object 8 stored in the virtual object storage unit 14 to the real space image captured at the later time point. Update based on the contents of the screen. In addition, the virtual object update unit 18 changes the generation time of the virtual object information of the virtual object 8 to the time when the update is performed. By the processing as described above, the content of the latest object 6B in the real space can be reflected on the virtual object 8 corresponding to the object 6B.

Here, there may be a case where the virtual object corresponding to the object in the old state and the virtual object corresponding to the object in the new state coexist on the virtual space V for comparison. Therefore, the virtual object generation unit 13, the object detection unit 17, and the virtual object update unit 18 described above may execute the following processing.

That is, when the object detection unit 17 detects an object corresponding to a virtual object already generated from the real space image further acquired as described above, the first process of updating the virtual object (that is, the above-described process) Selection by the user as to which of the process of the virtual object update unit 18 and the second process of generating a new virtual object corresponding to the object (that is, the process of the virtual object generation unit 13 described above) Accept For example, the object detection unit 17 causes the HMD 1 of the user to display a selection screen for selecting one of the first process and the second process, and the result of the selection operation by the user from the user (user selection) To get

When the object detection unit 17 receives a user selection indicating that the first process is to be performed, the virtual object update unit 18 executes the first process. On the other hand, when the object detection unit 17 receives a selection of a user indicating execution of the second process, the virtual object generation unit 13 executes the second process. According to such a configuration, the first process of updating a virtual object that has already been created and the second process of creating a new new virtual object are appropriately switched and executed according to the user's request. be able to.

Next, an example of an operation of the information processing system 100 will be described with reference to FIGS. FIG. 3 is a sequence diagram showing processing until a virtual object is generated. FIG. 4 is a sequence diagram showing processing from generation of a virtual object to display of a virtual space image corresponding to the sharing setting on each HMD 1. FIG. 5 is a sequence diagram showing processing (updating of a virtual object or creation of a new virtual object) when an object which has already been turned into an object from a real space image is detected.

As shown in FIG. 3, first, the information processing apparatus 10 generates a virtual space V shared by a plurality of users (step S1). Specifically, a virtual space V in which various objects such as user objects associated with each user are arranged at an initial position is generated. Virtual space data (an image of the virtual space viewed from each user object) indicating the virtual space V generated in this manner is transmitted to the HMD 1 (here, the first HMD and the second HMD) of each user (step S2) . Thereby, each user experiences virtual reality as if it were in the virtual space V via each HMD 1.

Subsequently, the first HMD instructs the camera 2 to start shooting in response to an operation from the user 5 (first user) of the first HMD with respect to the controller etc. (step S3). The camera 2 having received the shooting start instruction starts shooting of the real space including the specific area 4 (see FIG. 2), and acquires an image of the real space (step S4). The video taken by the camera 2 is transmitted to the first HMD at any time (step S5), and displayed superimposed on the virtual space image displayed on the first HMD (step S6). For example, an image captured by the camera 2 is displayed on a small window-like screen (wipe) provided at a corner of the virtual space image.

Subsequently, the first HMD instructs the camera 2 to acquire a real space image in response to an operation from the user 5 on the controller etc. (step S7). Here, the real space image is a still image as a basis for extracting a virtual object. The camera 2 having received the image acquisition instruction acquires a real space image obtained by imaging the real space including the specific area 4 (step S8). The real space image acquired by the camera 2 is transmitted to the first HMD (step S9) and displayed on the first HMD (step S10).

Subsequently, the first HMD indicates a target region (here, a region including the object 6B as an example) including an object to be objectified in the real space image by receiving an operation on the controller or the like by the user 5. Information is acquired (step S11). Subsequently, the real space image acquired in step S9 and the information indicating the target area acquired in step S11 are transmitted to the information processing apparatus 10 (step S12).

Subsequently, the image acquisition unit 12 acquires information indicating the real space image and the target area transmitted in step S12 (step S13). Subsequently, the virtual object generation unit 13 generates a virtual object 8 corresponding to the object 6B included in the target area by executing known image recognition on the target area in the real space image (step S14). . At this time, virtual object information on the virtual object 8 is stored in the virtual object storage unit 14.

Subsequently, as illustrated in FIG. 4, the sharing setting unit 15 transmits data such as the appearance of the virtual object 8 to the first HMD (step S15), and the sharing setting screen described above (for example, the target of the sharing setting) The setting screen including the appearance of the virtual object 8 is displayed on the first HMD (step S16). The first HMD (for example, the controller attached to the first HMD) acquires setting information indicating the content of the sharing setting input by the user 5 on the sharing setting screen (step S17), and transmits the setting information to the information processing apparatus 10 (Step S18). Subsequently, the sharing setting unit 15 sets sharing setting information of the virtual object 8 based on the setting information (step S19).

Subsequently, the image generation unit 16 generates a virtual space image indicating at least a part of the virtual space V including the virtual object 8 generated by the virtual object generation unit 13 (step S20). Here, the image generation unit 16 generates a virtual space image for each user (for each HMD 1), transmits a virtual space image for the first HMD to the first HMD, and transmits a virtual space image for the second HMD to the second HMD (Steps S21 and S22). Subsequently, a virtual space image is displayed in each of the first HMD and the second HMD (steps S23 and S24).

Here, in the case where sharing of the virtual object 8 with the second HMD user (second user) is not permitted, in step S20, the image generation unit 16 converts the virtual object 8 into a virtual space image for the second HMD. Do not display That is, the image generation unit 16 generates a virtual space image in which the virtual object 8 is not displayed. In this case, the virtual object 8 is not displayed in the virtual space image displayed on the second HMD in step S24. On the other hand, if sharing of the virtual object 8 with the second user is permitted, the image generation unit 16 displays the virtual object 8 in the virtual space image for the second HMD in step S20. As a result, the virtual object 8 is displayed on the virtual space image displayed on the second HMD in step S24.

Subsequently, with reference to FIG. 5, an example of processing in the case where the real space image is acquired again after the virtual object 8 is generated will be described.

The processes of steps S31 to S36 are the same as the processes of steps S8 to S13, and thus detailed description will be omitted. Subsequently, the object detection unit 17 detects an object 6B corresponding to the virtual object 8 already generated from the real space image acquired in step S36 (step S37).

Subsequently, the object detection unit 17 is different from the first process (the process of the virtual object update unit 18) for updating the virtual object 8 and the new virtual object (the virtual object 8 already generated) corresponding to the object 6B. The selection of the user 5 as to which of the second process (process of the virtual object generation unit 13) for generating a new object) is to be executed is received. For this purpose, for example, the object detection unit 17 notifies the first HMD that the object 6B corresponding to the virtual object 8 already generated is detected from the real space image (step S38). For example, the object detection unit 17 causes the first HMD to display a notification pop-up or the like.

Subsequently, the first HMD (the controller or the like) accepts the selection of the user 5 as to which of the first processing and the second processing is to be executed (step S39), and transmits the result of the selection to the information processing apparatus 10. (Step S40). Subsequently, the information processing apparatus 10 executes a process according to the selection of the user 5 (step S41). Specifically, when the object detection unit 17 receives the selection of the user 5 indicating execution of the first process, the virtual object update unit 18 executes the first process. In this example, the virtual object update unit 18 updates the state of the virtual object 8 based on the state of the object 6B detected from the real space image acquired in step S36. On the other hand, when the object detection unit 17 receives the selection of the user 5 indicating execution of the second process, the virtual object generation unit 13 executes the second process. In this example, the virtual object generation unit 13 generates a new virtual object based on the state of the object 6B detected from the real space image acquired in step S36. In this case, the new virtual object generated in this manner and the virtual object 8 already generated coexist in the virtual space V.

According to the information processing apparatus 10 described above, the object 6 (in the present embodiment, the object 6B) included in the real space image obtained by imaging the real space near the user 5 is generated as the virtual object 8 in the virtual space V A virtual space image including the virtual object 8 (a virtual space image in which the virtual object 8 is captured) is generated. By such processing, the user 5 wearing the HMD 1 can visually recognize the object 6 present in the vicinity of the user via the virtual space image. Therefore, according to the information processing apparatus 10, the convenience of the virtual reality experience of the user 5 can be improved.

The information processing apparatus 10 further includes an object detection unit 17 and a virtual object update unit 18. Thus, when a real space image including the already-objectified object 6B is acquired again, the virtual object 8 corresponding to the object 6B is updated based on the state of the object 6B included in the real space image. be able to. As a result, the latest state of the object 6B in the real space can be recognized through the virtual object 8 on the virtual space V. Further, by not generating a plurality of virtual objects corresponding to the same object, it is possible to suppress the memory usage.

Further, when the object detection unit 17 detects an object 6B corresponding to the virtual object 8 from the acquired real space image, the first process of updating the virtual object 8 and a new virtual object corresponding to the object 6B Accept the user's selection as to which of the second processes to generate. When the object detection unit 17 receives the selection of the user indicating that the first process is to be performed, the virtual object update unit 18 performs the first process, and the object detection unit 17 indicates that the second process is to be performed. When the selection of is received, the virtual object generation unit 13 executes the second process. According to this configuration, it is possible to appropriately switch and execute the update of the existing virtual object 8 and the generation of a new virtual object according to the user's request.

In addition, the virtual object generation unit 13 selects an object 6 (the object 6B in the example of FIG. 2) specified by the user 5 among the plurality of objects 6 (the

objects

6A and 6B in the example of FIG. 2) included in the real space image. Create a virtual object 8 corresponding to. According to this configuration, since only virtual objects corresponding to objects desired by the user can be generated, unnecessary object formation processing can be omitted to reduce the processing amount of the processor, and memory used by the unnecessary virtual objects can be reduced. It is possible to suppress the increase of the amount.

The virtual space V is a space shared by at least the first user wearing the first HMD and the second user wearing the second HMD, and the information processing apparatus 10 includes the sharing setting unit 15 described above. . The image generation unit 16 does not display a virtual object that is not permitted to share with the second user in the virtual space image displayed on the second HMD. According to this configuration, by performing the sharing setting as described above for each virtual object, a specific virtual object (for example, an object corresponding to a document or the like in which confidential information is described) can be It can be viewed only by the above users. This makes it possible to more smoothly carry out business communication such as a meeting via the virtual space V.

In addition, the block diagram (FIG. 1) used for description of the said embodiment has shown the block of a function unit. These functional blocks (components) are realized by any combination of hardware and / or software. Moreover, the implementation means of each functional block is not particularly limited. That is, each functional block may be realized by one physically and / or logically coupled device, or directly and / or indirectly two or more physically and / or logically separated devices. It may be connected (for example, wired and / or wirelessly) and realized by the plurality of devices.

For example, the information processing apparatus 10 in the above embodiment may function as a computer that performs the processing of the information processing apparatus 10 in the above embodiment. FIG. 6 is a diagram showing an example of the hardware configuration of the information processing apparatus 10 according to the present embodiment. The above-described information processing apparatus 10 may be physically configured as a computer apparatus including a processor 1001, a memory 1002, a storage 1003, a communication device 1004, an input device 1005, an output device 1006, a bus 1007, and the like.

In the following description, the term "device" can be read as a circuit, a device, a unit or the like. The hardware configuration of the information processing device 10 may be configured to include one or more of the devices illustrated in FIG. 6 or may be configured without including some devices.

Each function in the information processing apparatus 10 causes the processor 1001 to perform an operation by reading predetermined software (program) on hardware such as the processor 1001, the memory 1002, etc., communication by the communication device 1004, the memory 1002 and the storage 1003. This is realized by controlling the reading and / or writing of data in

The processor 1001 operates, for example, an operating system to control the entire computer. The processor 1001 may be configured by a central processing unit (CPU) including an interface with a peripheral device, a control device, an arithmetic device, a register, and the like.

Also, the processor 1001 reads a program (program code), a software module, and / or data from the storage 1003 and / or the communication device 1004 to the memory 1002, and executes various processing according to these. As a program, a program that causes a computer to execute at least a part of the operations described in the above embodiments is used. For example, the virtual object generation unit 13 of the information processing apparatus 10 may be realized by a control program stored in the memory 1002 and operated by the processor 1001, and similarly realized for other functional blocks shown in FIG. It is also good. The various processes described above have been described to be executed by one processor 1001, but may be executed simultaneously or sequentially by two or more processors 1001. The processor 1001 may be implemented by one or more chips. The program may be transmitted from the network via a telecommunication line.

The memory 1002 is a computer readable recording medium, and includes, for example, at least one of a ROM (Read Only Memory), an EPROM (Erasable Programmable ROM), an EEPROM (Electrically Erasable Programmable ROM), and a RAM (Random Access Memory). It may be done. The memory 1002 may be called a register, a cache, a main memory (main storage device) or the like. The memory 1002 may store a program (program code), a software module, etc. that can be executed to execute the information processing method (for example, the procedure shown in the sequence diagrams of FIGS. 3 to 5) according to the above embodiment. it can.

The storage 1003 is a computer readable recording medium, and is, for example, an optical disk such as a CD-ROM (Compact Disc ROM), a hard disk drive, a flexible disk, a magneto-optical disk (for example, a compact disk, a digital versatile disk, Blu-ray A (registered trademark) disk, a smart card, a flash memory (for example, a card, a stick, a key drive), a floppy (registered trademark) disk, a magnetic strip, and the like may be used. The storage 1003 may be called an auxiliary storage device. The above-described storage medium may be, for example, a database including the memory 1002 and / or the storage 1003, a server, or any other suitable medium.

The communication device 1004 is hardware (transmission / reception device) for performing communication between computers via a wired and / or wireless network, and is also called, for example, a network device, a network controller, a network card, a communication module, or the like.

The input device 1005 is an input device (for example, a keyboard, a mouse, a microphone, a switch, a button, a sensor, and the like) that receives external input. The output device 1006 is an output device (for example, a display, a speaker, an LED lamp, etc.) that performs output to the outside. The input device 1005 and the output device 1006 may be integrated (for example, a touch panel).

Also, each device such as the processor 1001 and the memory 1002 is connected by a bus 1007 for communicating information. The bus 1007 may be configured by a single bus or may be configured by different buses among the devices.

Further, the information processing apparatus 10 includes hardware such as a microprocessor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a programmable logic device (PLD), and a field programmable gate array (FPGA). And part or all of each functional block may be realized by the hardware. For example, processor 1001 may be implemented in at least one of these hardware.

Although the present invention has been described above in detail, it is apparent to those skilled in the art that the present invention is not limited to the embodiments described herein. The present invention can be implemented as a modified and changed embodiment without departing from the spirit and scope of the present invention defined by the description of the claims. Accordingly, the description in the present specification is for the purpose of illustration and does not have any limiting meaning on the present invention.

As long as there is no contradiction, the processing procedure, sequence, flow chart, etc. of each aspect / embodiment described in this specification may be reversed. For example, for the methods described herein, elements of the various steps are presented in an exemplary order and are not limited to the particular order presented.

The input / output information may be stored in a specific place (for example, a memory) or may be managed by a management table. Information to be input or output may be overwritten, updated or added. The output information etc. may be deleted. The input information or the like may be transmitted to another device.

The determination may be performed by a value (0 or 1) represented by one bit, may be performed by a true / false value (Boolean: true or false), or may be compared with a numerical value (for example, a predetermined value). Comparison with the value).

Each aspect / embodiment described in the present specification may be used alone, may be used in combination, and may be switched and used along with execution.

Software may be called software, firmware, middleware, microcode, hardware description language, or any other name, and may be instructions, instruction sets, codes, code segments, program codes, programs, subprograms, software modules. Should be interpreted broadly to mean applications, software applications, software packages, routines, subroutines, objects, executables, threads of execution, procedures, functions, etc.

Also, software, instructions and the like may be transmitted and received via a transmission medium. For example, software may use a wireline technology such as coaxial cable, fiber optic cable, twisted pair and digital subscriber line (DSL) and / or a website, server or other using wireless technology such as infrared, radio and microwave When transmitted from a remote source, these wired and / or wireless technologies are included within the definition of transmission medium.

The information, signals, etc. described herein may be represented using any of a variety of different techniques. For example, data, instructions, commands, information, signals, bits, symbols, chips etc that may be mentioned throughout the above description may be voltage, current, electromagnetic waves, magnetic fields or particles, light fields or photons, or any of these May be represented by a combination of

The terms described in the present specification and / or the terms necessary for the understanding of the present specification may be replaced with terms having the same or similar meanings.

In addition, the information, parameters, and the like described in the present specification may be represented by an absolute value, may be represented by a relative value from a predetermined value, or may be represented by corresponding other information. .

The names used for the parameters described above are in no way limiting. In addition, the formulas etc. that use these parameters may differ from those explicitly disclosed herein.

As used herein, the phrase "based on" does not mean "based only on," unless expressly stated otherwise. In other words, the phrase "based on" means both "based only on" and "based at least on."

As long as “include”, “including”, and variations thereof are used in the present specification or claims, these terms are as used in the term “comprising”. Is intended to be comprehensive. Furthermore, it is intended that the terms "or" and "or" as used in the present specification or in the claims are not exclusive ORs.

In the present specification, a plurality of devices are also included unless a context or technically apparent device is shown as having only one.

The term "determining" as used herein may encompass a wide variety of operations. “Decision” may be, for example, judging, calculating, computing, processing, deriving, investigating, looking up (eg table, database or other (Searching in the data structure of (a)), ascertaining it may be regarded as “decided”, and the like. Also, "determination" may be receiving (e.g., receiving information), transmitting (e.g., transmitting information), input (input), output (output), accessing (accessing) (e.g. For example, it can be regarded as "determining" access to data in memory. Also, "determining" may include considering "resolving", selecting, choosing, establishing, comparing, etc., as "determining". That is, "determination" may include considering that some action is "decision".

Throughout this disclosure, unless the context clearly indicates otherwise, it is intended to include the plural.

DESCRIPTION OF SYMBOLS 1 ... HMD (display apparatus), 5 ... user, 6, 6A, 6B ... object, 7 ... user object, 8 ... virtual object, 10 ... information processor, 12 ... image acquisition part, 13 ... virtual object production | generation part, 15 ... shared setting unit, 16 ... image generation unit, 17 ... object detection unit, 18 ... virtual object update unit, V ... virtual space.

Claims

An information processing apparatus for providing an image of a virtual space displayed on a display device mounted on a user, comprising:
An image acquisition unit that acquires a real space image obtained by imaging a real space near the user;
A virtual object generation unit that recognizes an object included in the real space image and generates a virtual object corresponding to the object in the virtual space;
An virtual space image showing at least a part of the virtual space including the virtual object, the image generation unit generating the virtual space image to be displayed on the display device;
An information processing apparatus comprising:
An object detection unit for detecting an object corresponding to the virtual object from the real space image further acquired by the image acquisition unit after the virtual object is generated;
A virtual object update unit that updates the state of the virtual object corresponding to the object based on the state of the object detected by the object detection unit;
The information processing apparatus according to claim 1, further comprising:
When the object detection unit detects an object corresponding to the virtual object from the further acquired real space image, the object detection unit generates a first process of updating the virtual object and a new virtual object corresponding to the object. Accept the user's choice as to which of the second processes to be performed;
When the object detection unit receives the selection of the user indicating that the first process is to be performed, the virtual object update unit executes the first process,
When the object detection unit receives the selection of the user indicating execution of the second process, the virtual object generation unit executes the second process.
The information processing apparatus according to claim 2.
The virtual object generation unit generates the virtual object corresponding to an object designated by the user among a plurality of objects included in the real space image.
The information processing apparatus according to any one of claims 1 to 3.
The virtual space is a space shared by a first user wearing at least a first display device and a second user wearing a second display device,
Whether the information processing apparatus shares the virtual object with the second user according to the operation content received from the first user for the virtual object generated based on the specification by the first user And a sharing setting unit for setting
The image generation unit does not display the virtual object, for which sharing with the second user is not permitted, in the virtual space image displayed on the second display device.
The information processing apparatus according to claim 4.