KR20200100046A - Information processing device, information processing method and program - Google Patents

Abstract

It becomes possible to generate interest in content provided outside the home. An acquisition unit that includes image information of a virtual object and position information of the virtual object in real space, and acquires content information added to map information indicating the real space, and the position based on the content information There is provided an information processing apparatus comprising a first control unit for displaying the image information on a first client terminal so as to overlap with an image of the virtual space that can be visually recognized from a first-person view at a location in a virtual space corresponding to information. .

Description

Information processing device, information processing method and program

The present disclosure relates to an information processing apparatus, an information processing method, and a program.

In recent years, with progress in information processing technology, various technologies for providing content such as games to users have been developed. For example, in Patent Document 1 below, by logging in from a mobile terminal device to a game server, items related to a game that can be enjoyed by a game device at home, etc., can be obtained according to the location of the mobile terminal device when going out. Is disclosed. In addition, a technique for reflecting the acquired item or the like to a game executed by a game device at home or the like is also disclosed. With these techniques, the user's interest in the game is caused, and a game with higher entertainment is provided.

Japanese Patent Publication No. 2016-087017

However, the technology disclosed in Patent Literature 1 mainly causes interest in games that the user enjoys at home or the like, and does not cause interest in games that users enjoy outside their home. Therefore, there is still room for improvement in terms of causing interest in content provided outside the home.

Therefore, the present disclosure has been made in view of the above problems, and an object of the present invention is a novel and improved information processing device capable of inducing user interest in content such as games provided outside the home, It is to provide information processing methods and programs.

According to the present disclosure, an acquisition unit that includes image information of a virtual object and position information of the virtual object in real space, and acquires content information added to map information indicating the real space, and the content information Information, comprising: a first control unit configured to display the image information on a first client terminal such that the image information is superimposed on a position in the virtual space corresponding to the location information with respect to an image of the virtual space that can be visually recognized from a first-person view. A processing device is provided.

Further, according to the present disclosure, including image information of a virtual object and position information of the virtual object in real space, acquiring content information added to map information indicating the real space, and acquiring content information added to the content information Based on the location in the virtual space corresponding to the location information, having the image information displayed on the first client terminal so as to be superimposed on the image of the virtual space that can be visually recognized from a first-person view, executed by a computer Information processing methods are provided.

Further, according to the present disclosure, including image information of a virtual object and position information of the virtual object in real space, acquiring content information added to map information indicating the real space, and acquiring content information added to the content information On the basis of, there is a program for realizing in a computer that the image information is displayed on a first client terminal so as to be superimposed on an image of the virtual space that is visually recognizable from a first-person viewpoint at a position in the virtual space corresponding to the location information Is provided.

As described above, according to the present disclosure, it is possible to generate interest in content provided outside the home.

In addition, the above effect is not necessarily limited, and any effect shown in the present specification or other effects that can be grasped from the present specification may be exhibited in addition to or instead of the above effect.

1 is a diagram showing an example of a system configuration of an information processing system according to the present embodiment.
2 is a diagram illustrating an example of the client 200.
3 is a diagram illustrating an example of the client 200.
4 is a diagram showing an example of the client 200.
5 is a block diagram showing an example of the functional configuration of the server 100.
6 is a diagram showing an example of a GUI screen used for creating content.
7 is a diagram showing an example of a GUI screen used for creating contents.
8 is a diagram showing an example of a GUI screen used for creating contents.
9 is a diagram showing an example of a GUI screen used for creating content.
10 is a diagram showing an example of a GUI screen used for creating contents.
11 is a diagram for explaining a display example of AR content.
12 is a diagram for describing a display example of VR content.
13 is a block diagram showing an example of the functional configuration of the client 200.
14 is a flowchart showing an example of a processing flow related to provision of content.
15 is a flowchart showing an example of a processing flow related to provision of content.
16 is a block diagram showing an example of a hardware configuration of an information processing device 900 that implements the server 100 or the client 200.
17 is a diagram showing an example of providing content according to the first embodiment.
18 is a diagram showing an example of providing content according to the first embodiment.
19 is a diagram showing an example of providing content according to the first embodiment.
Fig. 20 is a diagram showing an example of providing content according to the second embodiment.

Hereinafter, preferred embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In addition, in the present specification and drawings, elements having substantially the same functional configuration are denoted by the same reference numerals, so that redundant descriptions are omitted.

In addition, explanation will be made in the following order.

1. Embodiment

1.1. summary

1.2. System configuration example

1.3. Example of functional configuration of the server 100

1.4. Example of functional configuration of the client 200

1.5. Processing flow example

1.6. Hardware configuration example

2. Examples

2.1. Embodiment 1

2.2. Embodiment 2

2.3. Embodiment 3

2.4. Variation

3. Finish

<1. Embodiment>

(1.1. Overview)

First, an outline of an embodiment according to the present disclosure will be described.

As described above, with advances in information processing technology, various technologies have been developed for providing content including games to users. However, there is still room for improvement as to causing the user's interest in content provided outside the home.

Therefore, the initiator of this case came to create the description of this case in consideration of this situation. The present disclosure may provide a platform capable of providing Virtual Reality (VR) content by using at least part of information used to provide Augmented Reality (AR) content.

More specifically, the information processing device (server 100) according to the present disclosure can create VR content by using at least a part of information used for provision of AR content. Then, the information processing device can provide the created VR content to the user device (client 200). In addition, the information processing device may create AR content or provide the created AR content to a user device. In the present disclosure, AR contents may be regarded as contents provided outdoors, and VR contents may be regarded as contents provided indoors. In addition, when the content is a content provided indoors having a larger area than a general private house such as a commercial facility or a building, the content may be regarded as AR content. That is, the AR content may be regarded as content corresponding to the movement of the user's position in real space on a one-to-one basis with the movement of the user's position in the AR content. On the other hand, VR content may be regarded as content in which the user's position in VR content can be arbitrarily moved independently of the movement of the user's position in real space.

Accordingly, the present disclosure allows users to experience VR content corresponding to AR content, thereby causing user interest in AR content that was originally experienced only in the local area, so that AR content can be more efficiently distributed. .

In addition, the AR content according to the present embodiment refers to content that can superimpose a virtual object on an image in real space, and the VR content is a content that can superimpose a virtual object on an image in a virtual space (or Refers to content that constitutes the entire display screen as virtual objects). In the following, AR content or VR content may be simply referred to as "content" or "these content". In addition, the "real-space image" in the present disclosure may include a composite image generated based on a real-space image acquired by imaging a real space. That is, the composite image is a depth image having depth information corresponding to real space based on the analysis result of the real space image, a correction image processed based on the tone (appearance information such as color, contrast, brightness, etc.) of a virtual object, etc. May be. On the other hand, the "image of a virtual space" may be interpreted as an image created without referring to information on real space.

(1.2. System configuration example)

In the above, an outline of an embodiment according to the present disclosure has been described. Next, an example of the system configuration of the information processing system according to the present embodiment will be described with reference to FIGS. 1 to 4.

As shown in FIG. 1, the information processing system according to the present embodiment includes a server 100 and a client 200.

(Server(100))

The server 100 is an information processing device capable of creating AR contents and VR contents and providing these contents to the client 200. In more detail with respect to the creation of contents, the server 100 provides a user with a development platform capable of creating AR contents and VR contents. For example, the server 100 provides a GUI (Graphical User Interface) screen in which these contents can be created to the client 200. Then, by making various inputs to the GUI screen, the user can create content information to be used for AR content, including image information of a virtual object and position information of a virtual object in real space. Then, the user can also create content information used for VR content by using at least a part of the virtual object.

In addition, to describe the provision of content in more detail, the server 100 determines whether to provide AR content or VR content based on the location information of the user in real space and the property information of the client 200. do. For example, when the location information of the user indicates "home" and the property information of the client 200 indicates "a terminal for residence (non-AR-compliant terminal)", the server 100 transmits the VR content to the client ( 200). In addition, when the location information of the user indicates a specific location in real space and the property information of the client 200 indicates “AR-adaptive terminal”, the server 100 sends AR content to the client 200. to provide. Here, the property information of the client 200 refers to any information about the client 200, including product information or setting information of the client 200. In addition, the above is only an example, and the method of controlling the provision of content by the server 100 may be appropriately changed. For example, when the client 200 is a mobile terminal, the server 100 provides VR content when the location information of the user of the mobile terminal indicates "outside the AR content area", and the location of the user of the mobile terminal When the information indicates "in the AR content area", it may be controlled to provide AR content. Further, the server 100 may present an icon indicating which of the AR content and the VR content can be played with respect to the same content to the portable terminal.

Here, the communication method between the server 100 and the client 200 is not particularly limited. For example, the network connecting the server 100 and the client 200 may be either a wired transmission path or a wireless transmission path. For example, the network may include a public line network such as the Internet, various local area networks (LANs) including Ethernet (registered trademarks), and wide area networks (WANs). Further, the network may include a dedicated line network such as IP-VPN (Internet Protocol-Virtual Private Network) or a short-range wireless communication network such as Bluetooth (registered trademark).

In addition, the type of the server 100 is not particularly limited, and may be any information processing device including, for example, a general-purpose computer, a desktop PC, a notebook PC, a tablet PC, and a smartphone. .

(Client (200))

The client 200 is an information processing device used by a user when creating AR content and VR content, or when playing these content. To describe the creation of content in more detail, the client 200 displays a GUI screen provided from the server 100. The user can set information necessary for content creation (eg, image information of a virtual object, position information of a virtual object in real space, event information by a virtual object, etc.) using the GUI screen. Then, the client 200 realizes creation of these contents by providing the server 100 with input information from the user.

In more detail with respect to content playback, the client 200 provides the server 100 with location information of the user and property information of the client 200. Then, the client 200 receives the content information provided by the server 100 based on the information, and provides the content to the user by outputting it in a predetermined method. In addition, a method of outputting content by the client 200 is not particularly limited. For example, the client 200 may display the content on a display or the like, or may make it sound from a speaker or the like. In addition, information provided from the client 200 to the server 100 when the content is reproduced is not limited to the above.

In addition, as shown in Fig. 1, the client 200 includes various devices. For example, the client 200 includes an optically transmissive head mounted display 201 (hereinafter referred to as ``HMD''), a shielded (or video transmissive) HMD 202, a smartphone 203, or a tablet. Type PC 204, etc. are included. For the reproduction of AR contents, an optically transmissive HMD 201, a video transmissive HMD 202, a smartphone 203, or a tablet PC 204 are used, and for the reproduction of VR contents, a shielded HMD 202 and a smart Although it is assumed that the phone 203 or the tablet PC 204 is used, it is not limited to these. Further, the client 200 may include any display device other than these. For example, the client 200 may include a television or the like.

Further, the client 200 may include an arbitrary information processing device that does not have a display function. For example, the client 200 may include a speaker 205 that does not cover the ears (hereinafter referred to as "open ear speaker") as shown in Figs. 2A and 2B. As shown in 2B, the open-ear speaker 205 is used in a state that is caught on the user's neck, and does not block the ear, so it does not block external sound. As a result, the open-ear speaker 205 can perform sound output with a sense of presence by positioning the sound image so as to overlap the sound in the real space when reproducing content (especially AR content).

In addition, the client 200 may include an open-ear speaker 206 shown in Figs. 3A and 3B. The open-ear speaker 206 is attached by being inserted into the user's ear as shown in 3B, but does not block the ear and does not block external sound because it has a hole through which the ear is inserted. As a result, similarly to the above, the open-ear speaker 206 can also perform sound output with a sense of presence by positioning the sound image so as to superimpose the sound in the real space when reproducing content (especially AR content).

Further, the client 200 may include a wearable terminal 207 shown in FIGS. 4A and 4B. As shown in 4B, the wearable terminal 207 is a device that is attached to the user's ear, and can perform sound output with a sense of presence by positioning the sound image. In addition, the wearable terminal 207 is equipped with various sensors, and based on sensor information from these sensors, the user's posture (for example, the inclination of the head), speech, position, or behavior can be estimated. It is a device.

(1.3. Example of functional configuration of server 100)

In the above, an example of the system configuration of the information processing system according to the present embodiment has been described. Next, an example of the functional configuration of the server 100 will be described with reference to FIG. 5.

As shown in FIG. 5, the server 100 includes a content creation unit 110, a content providing unit 120, a communication unit 130, and a storage unit 140.

(Content creation unit (110))

The content creation unit 110 is a functional configuration for creating AR content or VR content. As shown in FIG. 5, the content creation unit 110 includes a location processing unit 111, an object processing unit 112, an event processing unit 113, and a content creation control unit 114.

The position processing unit 111 is a functional configuration that performs processing regarding positional information of a virtual object in AR content or VR content. More specifically, the position processing unit 111 sets positional information of a virtual object in real space based on an input from a user when creating these contents. Here, the location information includes, for example, latitude information and longitude information. By setting latitude information and longitude information as the location information of the virtual object, the location processing unit 111 can display the virtual object at a location in real space corresponding to the location information for AR content, and VR content With respect to, the virtual object can be displayed at a location in a virtual space corresponding to the location information.

In addition, the content of the location information is not limited to the above. For example, the location information may include altitude information. The position processing unit 111 also sets altitude information as position information of the virtual object, so that the virtual object can be displayed at a different altitude even at a position having the same latitude and longitude. With this configuration, for example, different virtual object groups can be displayed for each floor plan of the building. Further, the location information may include any information indicating a location, area, or building in real space, such as address information, spot name information, or spot code information. In addition, the positional information may include information about the orientation or posture of the virtual object.

The object processing unit 112 is a functional configuration that performs processing on a virtual object in AR content or VR content. More specifically, the object processing unit 112 sets virtual object information including image information of the virtual object based on an input from the user. For example, the object processing unit 112 manages image information of a plurality of virtual objects, and may cause the user to select image information of a virtual object to be used for content from among image information of the virtual object. Further, the object processing unit 112 may collect image information scattered on an external network (for example, the Internet), and may cause the user to select image information to be used for content from among these image information. Further, the object processing unit 112 may use image information input from a user for content.

In addition, the content of the image information of the virtual object is not particularly limited. For example, the image information of the virtual object may be any illustration information (or animation information), or may be still image information (or moving image information) of an object existing in real space.

The event processing unit 113 is a functional configuration that performs processing related to an event performed by a virtual object in AR content or VR content. More specifically, the event processing unit 113 sets event information including contents of the event and occurrence conditions based on an input from the user. Here, the event includes, but is not limited to, an arbitrary action performed by one or two or more virtual objects, an event performed in a specific place, or an event generated by an action of a virtual object. In addition, the content of an event includes, but is not limited to, a virtual object that performs an event, a location where the event is performed, a timing at which the event is performed, the purpose of the event, or a method of performing the event. Incidentally, the conditions for occurrence of an event are those specified by date and time, location, behavior or situation of a user or a virtual object, etc., but may be specified by factors other than these.

The content creation control unit 114 is a functional configuration that controls creation of AR content or VR content. More specifically, the content creation control unit 114 provides, to the client 200, a GUI screen to be used for creating (or editing, etc.) these contents.

Here, an example of a GUI screen provided by the content creation control unit 114 to the client 200 (client terminal) will be described with reference to FIGS. 6 to 10. Further, the data related to the GUI screen may be temporarily used offline by the client 200, and the data related to the GUI screen may be distributedly managed by the server 100 and the client 200. 6 is an example of a GUI screen used to create position information. As shown in Fig. 6, on the GUI screen, a content tab 300 for managing content information related to a plurality of contents (contents A to E in Fig. 6) is displayed, and the user receives the client 200 ) To select the content tab 300 to designate the content to be created (or edited, etc.).

When the user selects the content tab 300, the content tab 300 is opened and the map tab 301 is displayed. In addition, although not shown in Fig. 6, a plurality of map information may be used for each content, and when a plurality of map information is used, a plurality of map tabs 301 may be displayed (a plurality of map information referred to herein). May be generated by dividing a single map information, or may be a plurality of different map information). Then, when the user selects the map tab 301, the content creation control unit 114 displays the virtual object selection area 310 and the map display area 320 on the screen. The virtual object selection area 310 is an area in which virtual objects that can be placed on the map (in Fig. 6, virtual objects A to virtual objects D. Event A is also displayed) are displayed, and the user By operation, a virtual object to be placed on the map can be selected.

The map display area 320 is an area in which a map representing a real space selected as a stage of content is displayed, and a user enlarges, reduces, or moves the world map displayed in the map display area 320 By doing so, a map of a desired range can be displayed. Then, the user can place the virtual object selected in the virtual object selection area 310 on the map display area 320 by a predetermined operation. In addition, a predetermined operation content for realizing selection of a virtual object from the virtual object selection area 310 and placement of the virtual object on the map display area 320 is not particularly limited. For example, the user may implement selection and arrangement of a virtual object by dragging a virtual object in the virtual object selection area 310 and dropping it at a desired position in the map display area 320. Thereby, the user can edit the location information more easily. The location processing unit 111 generates location information based on the arrangement of virtual objects in the map display area 320.

In addition, the icon of the virtual object to be dragged has a predetermined size in order to ensure the user's visibility and operability. For this reason, it may be difficult to arrange a virtual object at an intended position according to the scale of the map to be displayed.

In order to solve the above problem, the user may display the content being created as VR content from a first-person view, and appropriately adjust the position of the virtual object according to the user's operation in the VR content. The result of adjusting the position of the virtual object by user manipulation in the VR content is reflected in the position information of the virtual object on the map display area 320.

Instead of adjustment in the VR content, the position of the dragged and dropped virtual object in the map display area 320 may be appropriately automatically adjusted based on road information, section information, building information, etc. included in the map information. . For example, when an icon of a person, which is a dropped virtual object, is substantially included in the road area, the position of the person icon is adjusted to be set along the right or left side of the road area outside the road area. Alternatively, when the furniture icon as the dropped virtual object is substantially included in the building area, the position of the furniture icon may be set along the wall in the building. That is, when the combination of the properties of the virtual object, the drop position of the virtual object, and map information corresponding to the drop position of the virtual object meets a predetermined condition, the position of the virtual object is automatically adjusted based on the map information. In addition, when the combination does not satisfy a predetermined condition, the position of the virtual object may be set as a drop position by the user. Further, such automatic adjustment is prohibited when the scale of the displayed map is greater than or equal to the threshold value, and the position of the virtual object may be arbitrarily set according to the user's drop position. When the scale of the displayed map is relatively large, the displayed map may have a sufficient size with respect to the size of the icon of the virtual object. Therefore, since the user can appropriately manipulate the drop position, in such a case, automatic adjustment of the position of the virtual object may be appropriately prohibited.

7 is an example of a GUI screen used for creation of virtual object information. As shown in FIG. 7, when the user selects the content tab 300, the content tab 300 is expanded and the virtual object tab 302 is displayed. In addition, as shown in FIG. 7, when a plurality of virtual objects are used in the content, the virtual object tab 302 may be further expanded and the virtual object tab 302a may be displayed. Then, when the user selects the virtual object tab 302 or the virtual object tab 302a, the content creation control unit 114 causes the virtual object display area 330 to be displayed on the screen. The virtual object display area 330 is an area in which image information 331 of a virtual object selected (or input) by a user for use in the content is displayed, and the user displays image information 331 of the selected virtual object. It can be seen in the virtual object display area 330. In addition, the content creation control unit 114 not only displays the image information 331 of the virtual object on the virtual object display area 330, but also displays the image information 331 of the virtual object through the virtual object display area 330. An editing function (for example, an editing function such as shape, shape, or color) may be provided to the user. In addition, although not shown in FIG. 7, the virtual object information may include sound image information, and confirmation and editing of sound image information may be realized on the GUI screen of FIG. 7. The object processing unit 112 generates virtual object information based on the contents of edits made through the virtual object display area 330.

8 and 9 are examples of GUI screens used for creating event information. As shown in Fig. 8, when the user selects the content tab 300, the content tab 300 is expanded and the event tab 303 is displayed. In addition, as shown in Fig. 8, when a plurality of events are set in the content, the event tab 303 may be further expanded and the event tab 303a may be displayed. Then, when the user selects the event tab 303 or the event tab 303a, the content creation control unit 114 displays the event display area 340 on the screen. The event display area 340 is an area in which a user can edit an event. Editing of the event may be described using a unified modeling language (UML). For example, in the event display area 340, a text box 341 or the like is displayed in advance, and the user inputs the text box 341 to perform processing in an event or a virtual object. You can define actions, etc. In addition, the user can define transitions such as processing in an event using the arrow 342, and can define transition conditions and the like using the transition condition 343. In addition, the content shown in FIG. 8 is a part of the entire event, and the user's work area can be enlarged, dragged, or scrolled. In addition, by pressing the icon 344 of the event display area 340, the user can switch from the screen shown in Fig. 8 (text-based screen) to the screen shown in Fig. 9 (GUI-based screen). have.

9 is a screen (a GUI-based screen) in which content corresponding to the event information set in FIG. 8 is displayed (information is appropriately changed). In other words, in the event display area 350 of FIG. 9, the event information edited in FIG. 8 is displayed on a map using icons or the like. The arrow 342 and the transition condition 343 in FIG. 8 correspond to the arrow 351 and the transition condition 352 in FIG. 9. In other words, when the transition condition 352 in Fig. 9 is satisfied, the previous processing indicated by the arrow 351 or the like is performed. A user can draw an arrow 351 between virtual objects or the like by clicking or dragging in the event display area 350. Further, for example, in order to correspond the achievement of the transition condition A to a plurality of virtual objects, a rectangular broken line 353 may be used. In addition, a star-shaped object 354 is used in FIG. 9 as an event segment. When the user sets the object 354 as the end of a series of events, a segment may be automatically added in the event progress 355 shown in the lower part of FIG. 9. The event progress 355 is a configuration in which a step-by-step event progress is displayed on a slider. By changing the position of the slider, the user can display the arrangement of virtual objects according to the progress of the event on the event display area 350. This makes it possible to easily and visually confirm the progress of the event. To the slider, the name of the event or the like is added as an annotation. In addition, by pressing the icon 356 of the event display area 350, the user can switch from the screen shown in Fig. 9 (GUI-based screen) to the screen shown in Fig. 8 (text-based screen). have. In addition, regarding the arrangement of the virtual objects, as described with reference to FIG. 6, when the virtual object moves in the event display area 350, the location information of the virtual object is updated and the location information is updated for each event. Is managed.

10 is an example of a GUI screen used for checking and editing content information including location information, virtual object information, or event information. As shown in FIG. 10, when the user selects the content tab 300, the content tab 300 is expanded and the data table tab 304 is displayed. A data table representing content information is displayed in the data table display area 360.

In addition, in the data table, "virtual object No", "name" and "supplement" included in the virtual object information, and "whether or not the position is fixed", "latitude" and "longitude" included in the position information are displayed. have. "Virtual object No" indicates a number that can identify a virtual object, "Name" indicates a name of a virtual object, and "Supplementary" indicates supplementary information about the virtual object. In addition, "whether or not the position is fixed" indicates whether the position of the virtual object is fixed or variable, "latitude" indicates the latitude information where the virtual object is placed, and "longitude" indicates the longitude where the virtual object is placed. Display information. The content displayed in the data table display area 360 is not limited to this. For example, arbitrary information other than position information, virtual object information, or event information may be displayed in the data table display area 360.

Then, the user can edit the content information including location information, virtual object information, or event information by editing the data table displayed on the data table display area 360. Also, the content edited using the data table is automatically reflected on the separate screen described above. For example, when the user edits the latitude information or longitude information of the virtual object in the data table, the position of the virtual object in the map display area 320 of FIG. 6 is in the latitude information or longitude information after editing. It has been changed to the corresponding location.

In addition, the GUI screen provided by the content creation control unit 114 to the client 200 is not limited to the examples of FIGS. 6 to 10.

Then, the content creation control unit 114 determines the format, size, and security settings of the content information (eg, access rights, etc.), and performs integration and packaging of location information, virtual object information and event information, etc. Create content information constituting AR content or VR content. Further, the content creation control unit 114 stores the created content information in the storage unit 140 in a state in which the created content information is added to the map information indicating the real space. Accordingly, the content can be appropriately executed based on the map information indicating the real space.

(Contents provider 120)

The content providing unit 120 is a functional configuration for providing AR content or VR content to the client 200. As shown in FIG. 5, the content providing unit 120 includes an acquisition unit 121, a route determining unit 122, and a content providing control unit 123.

The acquisition unit 121 is a functional configuration for acquiring arbitrary information used for provision of AR content or VR content. For example, the acquisition unit 121 includes content information created by the content creation unit 110 and stored in the storage unit 140 (either content information about AR content or content information about VR content). It is okay).

Further, the acquisition unit 121 acquires information indicating the user's situation (or state) (hereinafter referred to as "user situation information"). For example, the client 200 uses various sensors provided in the own device to estimate the posture, gaze, speech, position, or behavior of the user who is equipped with the own device, and based on the estimation result, the user Context information is generated and provided to the server 100. In addition, the content of the user situation information is not limited to this, and any information may be used as long as it is information about the user's situation (or state) that can be output based on a sensor or the like. Further, the user context information may include contents of an input made by a user using an input means provided in the client 200.

Further, the acquisition unit 121 also acquires a user's behavior log. The action log contains, for example, the history of the user's location information. Additionally, the behavior log may include an image acquired along with the user's behavior, the context of the user's behavior, and the like. Here, the user who provides the behavior log to the acquisition unit 121 may include one or more users different from the users who use AR content or VR content. Further, the acquisition unit 121 also acquires property information, which is any information about the client 200, including product information or setting information of the client 200. In addition, the method of acquiring these information is not particularly limited. For example, the acquisition unit 121 may acquire these information from the client 200 via the communication unit 130.

Then, the acquisition unit 121 provides these acquired information to the route determination unit 122 and the content provision control unit 123. Thereby, the route determination unit 122 can determine a route in the content (for example, a recommended route, etc.) based on the user's location information or an action log, and the content provision control unit 123 The provision of the content can be controlled based on.

The route determination unit 122 is a functional configuration for determining a route (for example, a recommended route, etc.) in AR contents or VR contents. More specifically, the route determination unit 122 determines the route in the content based on the content of the content provided to the user and the location information included in the user context information acquired by the acquisition unit 121. Decide. For example, when AR content is provided, the route determination unit 122, from the user's current position in real space, to the next destination position in the AR content (for example, the event occurrence position Etc.). In addition, when VR content is provided, the route determination unit 122 outputs the shortest route from the user's current position in the virtual space to the next destination position in the VR content. In addition, the route output method is not particularly limited.

In addition, the information used to determine the route is not limited to the above. For example, the route determination unit 122 may determine a route based on an action log or the like. Thereby, the route determination unit 122 can determine a more appropriate route based on the user's past behavior. For example, the route determination unit 122 may determine a route that the user frequently passes as a route in the content, or conversely, may determine a route that the user has never passed as a route in the content. The route determining unit 122 provides information indicating the determined route (hereinafter, referred to as “root information”) to the content providing control unit 123.

The content providing control unit 123 is a functional configuration that functions as a first control unit that controls provision of VR contents and a second control unit that controls provision of AR contents. More specifically, the content providing control unit 123 is based on the user context information (including location information) acquired by the acquisition unit 121 and the route information output by the route determination unit 122, and the like, Or, it controls the provision of VR content.

For example, the content providing control unit 123 determines whether to provide AR content or VR content based on the location information of the user in real space and property information of the client 200 included in the user context information. do. For example, when the location information of the user indicates "home" and the property information of the client 200 indicates "a terminal for residence (non-AR-compliant terminal)", the content providing control unit 123 Provided to the client 200. In addition, when the location information of the user indicates a specific location in real space and the property information of the client 200 indicates “AR-adaptive terminal”, the content providing control unit 123 transfers the AR content to the client 200 ). In addition, the above is only an example, and the method of controlling the provision of content by the content providing control unit 123 may be appropriately changed. For example, the content providing control unit 123 may determine whether to provide AR content or VR content based on an action log or the like.

In addition, the content providing control unit 123 can also propose a content. For example, the content providing control unit 123, based on the history of the user's location information included in the behavior log, the user has been provided at a location (or a location close to the location) where the user has been You may suggest a plurality of contents. Then, the content providing control unit 123 provides the user with the content selected by the user from among the plurality of proposed contents.

Here, a display example of AR content and VR content provided by the content providing control unit 123 will be described with reference to FIGS. 11 and 12. 11 is a display example of the client 200 when the content providing control unit 123 provides AR content. 11B in FIG. 11 shows that, as shown in 11A, in a state in which the user is located at a position in the real space where the virtual object 10 of the train is arranged, the user is in the forward direction of the virtual object 10 A display example of the client 200 in the case of facing the client 200 is shown (in addition, in the example of FIG. 11, the client 200 is the smartphone 203). At this time, the content providing control unit 123 displays the virtual object 10 on the client 200 so as to overlap with the image (background 11) in real space.

12 is a display example of the client 200 when the content providing control unit 123 provides VR content. 12B in Fig. 12 is a state in which the user is located at a position in the virtual space corresponding to the position in the real space in which the virtual object 12 of the train is arranged, as shown in 12A. Shows an example of display of the client 200 in the case where is facing the front direction of the virtual object 12 (in addition, in the example of FIG. 12, the client 200 is the shielded HMD 202). At this time, the content providing control unit 123 displays the virtual object 12 on the client 200 so as to be superimposed on an image (background image 13) of a virtual space that can be visually recognized from a first-person view. Further, the background image 13 is an image corresponding to a real space. In other words, the background image 13 is an image that reproduces an image in real space, and may be an all-celestial image or a free viewpoint image. For example, an all celestial sphere image captured within a predetermined distance from the location (latitude, longitude) of the virtual object may be searched on the network, and the extracted all celestial sphere image may be used as the background image 13. Further, the tone of the virtual object to be arranged may be analyzed, and the tone of the all-celestial image may be adjusted based on the analysis result. For example, when an animated virtual object is displayed, an all-celestial sphere image used as the background image 13 may also be processed in an animation manner. The display mode of AR content and VR content provided by the content providing control unit 123 is not limited to FIGS. 11 and 12.

Further, the content providing control unit 123 may provide not only image information of the virtual object but also sound image information of the virtual object included in the content information to the client 200. For example, with respect to AR content, the content providing control unit 123 may output sound image information to the client 200 at a position in the real space corresponding to the position information of the virtual object in the real space. In addition, with respect to VR content, the content providing control unit 123 may output sound image information to the client 200 at a position in the virtual space corresponding to the position information of the virtual object in the real space. Further, the content providing control unit 123 may output only sound image information to the client 200, or may output both image information and sound image information.

(Communication Department (130))

The communication unit 130 is a functional configuration for controlling various types of communication with the client 200. For example, when creating AR content or VR content, the communication unit 130 transmits GUI screen information used for various settings to the client 200 and receives input information or the like from the client 200. In addition, when providing AR content or VR content, the communication unit 130 receives user context information (including location information, etc.) or property information of the client 200 from the client 200, and receives content information. It transmits to the client 200. In addition, information communicated by the communication unit 130 and a case for communicating are not limited thereto.

(Memory (140))

The storage unit 140 is a functional configuration for storing various types of information. For example, the storage unit 140 includes location information generated by the location processing unit 111, virtual object information generated by the object processing unit 112, event information generated by the event processing unit 113, and content creation. Content information generated by the control unit 114, various types of information acquired by the acquisition unit 121 (for example, user context information, behavior log, or property information of the client 200), route determination unit 122 The route information determined by or the content information provided to the client 200 by the content providing control unit 123 is stored. Further, the storage unit 140 stores programs or parameters used by each functional configuration of the server 100. In addition, the contents of the information stored by the storage unit 140 are not limited to these.

In the above, an example of the functional configuration of the server 100 has been described. In addition, the functional configuration described with reference to FIG. 5 is only an example, and the functional configuration of the server 100 is not limited to this example. For example, the server 100 does not necessarily have to have all the configurations shown in FIG. 5. In addition, the functional configuration of the server 100 can be flexibly modified according to specifications and operation.

(1.4. Example of functional configuration of client 200)

In the above, an example of the functional configuration of the server 100 has been described. Subsequently, an example of the functional configuration of the client 200 will be described with reference to FIG. 13. Fig. 13 is a functional configuration example assuming an optically transmissive HMD 201 for executing AR contents and a shielded HMD 202 for executing VR contents. In addition, when the client 200 is other than the optically transmissive HMD 201 or the shielded HMD 202, the addition or deletion of functional configurations can be appropriately performed.

As shown in Fig. 13, the client 200 includes a sensor unit 210, an input unit 220, a control unit 230, an output unit 240, a communication unit 250, and a storage unit 260. ).

(Control unit 230)

The control unit 230 functions as an arithmetic processing unit and a control unit, and controls overall operations in the client 200 according to various programs. The control unit 230 is implemented by, for example, an electronic circuit such as a CPU (Central Processing Unit) or a microprocessor. Further, the control unit 230 may include a ROM (Read Only Memory) for storing programs to be used, operation parameters, and the like, and a RAM (Random Access Memory) for temporarily storing parameters that change appropriately.

Further, the control unit 230 according to the present embodiment includes a recognition engine 231 and a content processing unit 232 as shown in FIG. 13. The recognition engine 231 has a function of recognizing a user or various situations around the user by using various types of sensor information sensed by the sensor unit 210. More specifically, the recognition engine 231 includes a head posture recognition engine 231a, a depth recognition engine 231b, a Simultaneous Localization and Mapping (SLAM) recognition engine 231c, and a gaze recognition engine 231d. And, a speech recognition engine 231e, a location recognition engine 231f, and a behavior recognition engine 231g are provided. In addition, these recognition engines shown in FIG. 13 are examples, and this embodiment is not limited to this.

The head posture recognition engine 231a recognizes the user's head posture (including the direction or inclination of the face relative to the body) using various sensor information sensed by the sensor unit 210. For example, the head posture recognition engine 231a includes a surrounding image captured by the outward camera 211, gyro information acquired by the gyro sensor 214, and acceleration information acquired by the acceleration sensor 215. , And the orientation information acquired by the orientation sensor 216 can be analyzed to recognize the attitude of the user's head. In addition, the algorithm for recognizing the head posture may use a generally known algorithm, and is not particularly limited in this embodiment.

The depth recognition engine 231b uses various types of sensor information sensed by the sensor unit 210 to recognize depth information in a space around the user. For example, the depth recognition engine 231b can analyze a captured image of the surroundings acquired by the outward camera 211, and recognize the distance information of the object in the surrounding space and the plane position of the object. In addition, as for the depth recognition algorithm, a generally known algorithm may be used, and the present embodiment is not particularly limited.

The SLAM recognition engine 231c uses various sensor information sensed by the sensor unit 210 to estimate its own position and create a map of the surrounding space at the same time, thereby identifying its own position in the surrounding space. have. For example, the SLAM recognition engine 231c can analyze the surrounding captured image acquired by the outward camera 211 and identify the self-position of the client 200. In addition, as for the SLAM recognition algorithm, a generally known algorithm may be used, and the present embodiment is not particularly limited.

Further, the recognition engine 231 can perform spatial recognition (space recognition) based on the recognition result of the depth recognition engine 231b and the recognition result of the SLAM recognition engine 231c described above. Specifically, the recognition engine 231 can recognize the position of the client 200 in the surrounding three-dimensional space.

The gaze recognition engine 231d uses various types of sensor information sensed by the sensor unit 210 to detect the gaze of the user. For example, the gaze recognition engine 231d analyzes the captured image of the user's eyes acquired by the inward camera 212 to recognize the user's gaze direction. In addition, the algorithm of gaze detection is not particularly limited, but the user's gaze direction can be recognized based on, for example, the positional relationship between the eyeball and the iris, or the positional relationship between the corneal reflection and the pupil.

The speech recognition engine 231e recognizes a user or environmental sound using various types of sensor information sensed by the sensor unit 210. For example, the speech recognition engine 231e can perform noise removal, sound source separation, and the like on the sound-receiving information acquired by the microphone 213, and perform speech recognition, morpheme analysis, sound source recognition, or noise level recognition.

The location recognition engine 231f uses various types of sensor information sensed by the sensor unit 210 to recognize the absolute location of the client 200. For example, the location recognition engine 231f, based on the location information positioned by the positioning unit 217 and the map information acquired in advance, the location of the client 200 (for example, a station, school, home, company, Train, theme park, etc.)

The behavior recognition engine 231g recognizes a user's behavior using various types of sensor information sensed by the sensor unit 210. For example, the behavior recognition engine 231g includes a captured image of the outward camera 211, a sound-receiving voice of the microphone 213, angular velocity information of the gyro sensor 214, acceleration information of the acceleration sensor 215, and orientation sensor 216. ) Of the orientation information and the absolute position information of the positioning unit 217 are used to recognize a user's behavioral state (an example of an activity state). The user's behavior includes, for example, a stationary state, a walking state (slowly walking, jogging), a driving state (dash, high-speed driving), a sitting state, a standing state, a sleeping state, a bicycle riding state, and an electric vehicle. You can recognize the state you are riding in and the state you are in a car. Further, more specifically, the behavior recognition engine 231g may recognize a state situation according to an activity amount measured based on angular velocity information and acceleration information.

The content processing unit 232 is a functional configuration that executes processing related to AR content or VR content. More specifically, the content processing unit 232 performs processing related to creation of content or reproduction of content.

When the content is created more specifically, when the user inputs using the input unit 220 (for example, input to the GUI screen provided from the server 100), the content processing unit 232, Input information is acquired from the input unit 220 and the input information is provided to the server 100.

When the content is reproduced in more detail, the content processing unit 232 generates user context information that indicates the user's situation (or state) recognized by the recognition engine 231, and the user context information Is provided to the server 100. Thereafter, when the server 100 has provided content information for AR content or VR content based on user context information, the content processing unit 232 controls the output unit 240 to output the content information. . For example, the content processing unit 232 displays a virtual object on a display or the like, or outputs a sound image to a speaker or the like.

(Sensor unit 210)

The sensor unit 210 has a function of acquiring various types of information about the user or the surrounding environment. For example, the sensor unit 210 includes an outward camera 211, an inward camera 212, a microphone 213, a gyro sensor 214, an acceleration sensor 215, an orientation sensor 216, and , And a positioning portion 217. In addition, the specific example of the sensor part 210 illustrated here is an example, and this embodiment is not limited to this. In addition, there may be multiple sensors.

The outward camera 211 and the inward camera 212 are a lens system composed of an imaging lens, an aperture, a zoom lens, a focus lens, and the like, a driving system for performing a focus operation or a zoom operation on the lens system, and photoelectric conversion of imaged light obtained from the lens system. And a solid-state imaging element array that generates an imaging signal. The solid-state image sensor array may be realized by, for example, a CCD (Charge Coupled Device) sensor array or a CMOS (Complementary Metal Oxide Semiconductor) sensor array.

The microphone 213 receives the user's voice and surrounding environmental sound, and outputs the voice information to the control unit 230.

The gyro sensor 214 is realized by a three-axis gyro sensor, for example, and detects an angular speed (rotation speed).

The acceleration sensor 215 is realized by a three-axis acceleration sensor (also referred to as a G sensor), for example, and detects acceleration during movement.

The orientation sensor 216 is realized by, for example, a 3-axis geomagnetic sensor (compass), and detects an absolute direction (orientation).

The positioning unit 217 has a function of detecting the current position of the client 200 based on an externally acquired signal. Specifically, for example, the positioning unit 217 is realized by a GPS (Global Positioning System) positioning unit, receives radio waves from GPS satellites, detects the location where the client 200 exists, and detects One location information is output to the controller 230. In addition, the positioning unit 217 is capable of detecting a location by transmitting/receiving, for example, Wi-Fi (registered trademark), Bluetooth (registered trademark), mobile phone, PHS, smartphone, etc., or short-range communication other than GPS. You can do it. Further, the positioning unit 217 may indirectly specify the location of the client 200 by recognizing barcode information (eg, QR code, etc.) manually installed in advance. In addition, the positioning unit 217 records images of various points in real space on a database in advance, and matches the feature points of these images with the feature points of the image captured by the outward camera 211, etc. 200) may be specified.

(Input unit 220)

The input unit 220 is realized by an operation member having a physical structure such as a switch, a button, or a lever.

(Output unit 240)

The output unit 240 is a functional configuration for outputting various types of information. For example, the output unit 240 includes display means such as a display or audio output means such as a speaker, and realizes output of content information based on the control of the control unit 230. Further, the output means provided in the output unit 240 is not particularly limited.

(Communication Department (250))

The communication unit 250 is a functional configuration that controls various types of communication with the server 100. For example, when creating AR content or VR content, the communication unit 250 receives GUI screen information to be used for various settings from the server 100 and transmits input information or the like to the server 100. In addition, when reproducing AR content or VR content, the communication unit 250 transmits user context information (including location information, etc.) or property information of the own device to the server 100, and transmits the content information to the server ( 100). In addition, information communicated by the communication unit 250 and a case for communicating are not limited thereto.

(Memory Department (260))

The storage unit 260 is a functional configuration for storing various types of information. For example, the storage unit 260 stores property information of the own device, user context information generated by the content processing unit 232, content information provided from the server 100, and the like. Further, the storage unit 260 stores programs or parameters used by each functional configuration of the client 200. In addition, the contents of the information stored by the storage unit 260 are not limited to these.

In the above, the functional configuration example of the client 200 has been described. In addition, the functional configuration described above using FIG. 13 is only an example, and the functional configuration of the client 200 is not limited to this example. For example, the client 200 does not necessarily have all of the functional configurations shown in FIG. 13. In addition, the functional configuration of the client 200 can be flexibly modified according to specifications and operation.

(1.5. Example of processing flow)

In the above, an example of the functional configuration of the client 200 has been described. Subsequently, with reference to Figs. 14 and 15, an example of a processing flow related to provision of contents will be described.

14 is an example of a processing flow related to provision of VR content by the server 100. In step S1000, the acquisition unit 121 of the server 100 acquires, from the client 200, various types of information including user context information (including location information, etc.) and property information of the client 200. In step S1004, the content providing control unit 123 proposes one or more VR contents based on the acquired various types of information. For example, when the location information included in the user context information indicates “home” and the property information of the client 200 indicates “a dwelling terminal (non-AR-compliant terminal)”, the content providing control unit 123 , Propose one or more VR contents to the user. In addition, the content providing control unit 123 proposes one or more VR contents to the user based on a location that the user has been to in the past based on an action log or the like.

In step S1008, the user uses the input unit 220 of the client 200 to select a desired VR content from among the proposed VR contents (if the proposed VR content is one, the user plays the VR content. Enter whether to do it or not). In step S1012, the content provision control unit 123 of the server 100 provides the VR content selected by the user (a more detailed processing flow for provision of the VR content will be described with reference to FIG. 15). When the provision of the VR content is ended, in step S1016, the acquisition unit 121 acquires the action log from the client 200 and stores it in the storage unit 140, thereby ending a series of processing.

Fig. 15 shows in more detail the VR content providing process performed in step S1012 in Fig. 14. In step S1100, the acquisition unit 121 of the server 100 continuously acquires user context information (including location information). In addition, the frequency or timing at which the acquisition unit 121 acquires user situation information during content provision is not particularly limited.

Then, the content providing control unit 123 checks whether or not an event occurrence condition in the VR content is satisfied based on the user context information. When the event occurrence condition is satisfied (step S1104/Yes), in step S1108, the content provision control unit 123 generates an event from the VR content.

In step S1112, the content provision control unit 123 checks whether the VR content has ended, and when the VR content has not ended (step S1112/"No"), the processing of steps S1100 to S1108 is continuously executed. . When the VR content is ended (step S1112/"Yes"), the VR content provision process is ended.

In addition, each step in the flowcharts shown in Figs. 14 and 15 does not necessarily need to be processed in time series according to the described procedure. That is, each step in the flowchart may be processed in a different order from the described order or may be processed in parallel.

(1.6. Example of hardware configuration)

In the above, an example of a processing flow related to provision of content has been described. Subsequently, an example of the hardware configuration of the server 100 or the client 200 will be described with reference to FIG. 16.

16 is a block diagram showing an example of a hardware configuration of an information processing device 900 that implements the server 100 or the client 200. The information processing device 900 includes a CPU (Central Processing Unit) 901, a ROM (Read Only Memory) 902, a RAM (Random Access Memory) 903, a host bus 904, and a bridge ( 905, an external bus 906, an interface 907, an input device 908, an output device 909, a storage device (HDD) 910, a drive 911, and a communication device ( 912).

The CPU 901 functions as an arithmetic processing device and a control device, and controls overall operations in the information processing device 900 according to various programs. Further, the CPU 901 may be a microprocessor. The ROM 902 stores programs, operation parameters, and the like to be used by the CPU 901. The RAM 903 temporarily stores programs to be used in the execution of the CPU 901, parameters that change appropriately in the execution, and the like. These are connected to each other by a host bus 904 composed of a CPU bus or the like. By cooperation of the CPU 901, ROM 902, and RAM 903, the content creation unit 110 or the content provision unit 120 of the server 100, or the sensor unit 210 of the client 200 Alternatively, each function of the control unit 230 is realized.

The host bus 904 is connected to an external bus 906 such as a Peripheral Component Interconnect/Interface (PCI) bus through a bridge 905. In addition, it is not always necessary to separate the host bus 904, the bridge 905, and the external bus 906, and these functions may be mounted on one bus.

The input device 908 generates input means for a user to input information, such as a mouse, a keyboard, a touch panel, a button, a microphone, a switch, and a lever, and an input signal based on an input by the user, and the CPU 901 It is composed of an input control circuit that outputs to. A user who uses the information processing device 900 can input various data or instruct a processing operation to each device by operating the input device 908. The input device 908 implements the function of the input unit 220 of the client 200.

The output device 909 includes, for example, a cathode ray tube (CRT) display device, a liquid crystal display (LCD) device, an OLED (Organic Light Emitting Diode) device, and a display device such as a lamp. Further, the output device 909 includes an audio output device such as a speaker and a headphone. The output device 909 outputs, for example, the reproduced content. Specifically, the display device displays various types of information such as reproduced video data as text or images. On the other hand, the audio output device converts the reproduced audio data or the like into audio and outputs it. By the output device 909, the function of the output unit 240 of the client 200 is realized.

The storage device 910 is a device for storing data. The storage device 910 may include a storage medium, a recording device that writes data to the storage medium, a reading device that reads data from the storage medium, and a deletion device that deletes data recorded on the storage medium. The storage device 910 is configured of, for example, a hard disk drive (HDD). This storage device 910 drives a hard disk and stores programs and various data to be executed by the CPU 901. The storage device 910 implements the function of the storage unit 140 of the server 100 or the storage unit 260 of the client 200.

The drive 911 is a reader/writer for a storage medium, and is built-in or external to the information processing device 900. The drive 911 reads information recorded in a removable storage medium 913 such as a mounted magnetic disk, optical disk, magneto optical disk, or semiconductor memory, and outputs it to the RAM 903. Further, the drive 911 can also write information to the removable storage medium 913.

The communication device 912 is a communication interface composed of, for example, a communication device for connecting to the communication network 914. The communication device 912 realizes the functions of the communication unit 130 of the server 100 or the communication unit 250 of the client 200.

<2. Example>

In the above, an embodiment according to the present disclosure has been described. In the above, a main example related to the provision of contents has been described, but the server 100 can provide contents in various aspects other than the above. In the following, various embodiments will be described with respect to variations in content provision by the server 100.

(2.1. First embodiment)

For example, when the user is using the client 200 such as the shielded HMD 202, the acquisition unit 121 of the server 100 acquires user situation information or an action log. In addition, as described above, the content providing control unit 123 may propose VR content based on location information included in user context information or an action log. At this time, as shown in FIG. 17, the content providing control unit 123 determines the position where the proposed VR content is provided with respect to a bird's-eye view (bird view) of the map of the virtual space corresponding to the real space. Interest) (in the example of FIG. 17, POI 14 and POI 15 are displayed).

At that time, the content providing control unit 123 may display together an image representing the content of the VR content (for example, a poster image of the VR content or an image representing a scene of the VR content), like the POI 15. . Further, the content providing control unit 123 may display information other than an image indicating the contents of the VR content, like the POI 14. For example, in the example of the POI 14, "total number of users", "price", "time required", and "difficulty" are displayed. "Total number of users" represents the total number of users playing VR content corresponding to the POI 14 at the time of display, and "Price" represents the play fee of the VR content, and "Time required "" represents the time required from the start to the end of the VR content (or the average value of the time required from the start to the end), and the "difficulty" represents the difficulty of the VR content. In addition, the information displayed on the POI is not limited to the above.

The user selects the VR content to be played by a predetermined method (for example, input to the input unit 220 of the client 200, or a gesture or gaze of the content while the client 200 is mounted). After that, the route determination unit 122 of the server 100 determines the recommended route 16 from the current location in the virtual space (or the location selected by the user in a predetermined method) to the location where the selected VR content is provided. ), and the content providing control unit 123 displays the recommended route 16 on the client 200. Further, the content provision control unit 123 may provide advertisement information such as stores and events on the recommended route 16 in a real space, for example.

When the user moves the recommended route 16 in the virtual space, the content providing control unit 123 is the user in the virtual space shown in 18B from a bird's-eye view (bird view) of the map shown in Fig. 18A. All celestial sphere images corresponding to the position (eg, all celestial sphere images reproduced in real space) are displayed. More specifically, the content providing control unit 123, based on the location information of the user in the virtual space, acquired from the client 200, sends a global image corresponding to each location on the recommended route 16 to the client. Mark at (200). At this time, the content providing control unit 123 may reproduce a hyperlapse moving image in which all celestial images are continuously reproduced on the client 200 so as to be linked to the movement of the recommended route 16 by the user (in other words, time lapse It is not necessary to reproduce the moving image on the client 200). Thereby, the content providing control unit 123 can provide a smoother and more dynamic display to the user. Further, the reproduction speed of the hyperlapse moving image may be appropriately adjusted by the user. In addition, the content providing control unit 123 may display a free viewpoint image instead of an all-round image.

Then, when the user arrives at the provision position of the selected VR content and plays the VR content, the content provision control unit 123 may display the character 17 as a virtual object as shown in FIG. 19. Further, the content providing control unit 123 may make the user recognize the route by moving the character 17 toward the inside with respect to the screen as the event progresses.

In addition, when a plurality of users are simultaneously playing the VR content, the content providing control unit 123 may display avatars 18 to 20 representing other users based on the location information of each user. Thereby, the content providing control unit 123 can indicate to the user how another user is playing the VR content. In addition, the content providing control unit 123 may express the busyness of VR content. Further, the content providing control unit 123 may display an avatar of a user who has played the VR content in the past (for example, within the last one week). Further, the content providing control unit 123 may adjust the number of avatars to be displayed based on the congestion situation of the avatars. Further, the content providing control unit 123 may display the image as an avatar by randomly selecting an image prepared in advance, or may display an image input by the user as an avatar. The content providing control unit 123 may display an avatar on the client 200 playing the VR content based on the location information of the client 200 playing (or playing) the AR content.

In addition, the content providing control unit 123 may cause each avatar to perform an action (eg, waving a hand, tilting his head for a quiz) according to a condition set for an event in progress by each user. Accordingly, the content providing control unit 123 can make the user feel the presence of VR content in more detail.

(2.2. Second embodiment)

The second embodiment is an embodiment of content (raid content) that can be played by a plurality of users at the same time.

In this embodiment, the content providing control unit 123 is based on the user context information (including location information) from the client 200 used by a plurality of users, the situation of each user (including the location relationship). ), it is possible to reflect each user's situation (including location relationship) to AR content or VR content.

For example, as shown in FIG. 20, the content providing control unit 123 provides content in which the avatar 21 corresponding to the user A and the avatar 22 corresponding to the user B fight the monster 23, It can be provided to user A and user B at the same time. The content provision control unit 123 generates an event based on the location information of the user A and the user B, or makes a virtual object (for example, the monster 23, etc.) common to these users. It can provide high sense of presence and entertainment.

Here, it is preferable from the viewpoint of a sense of immersion that the behavior or position of the monster 23 changes according to the position or behavior of each of the users A and B that change every moment. To this end, based on the user context information (including location information) transmitted from the client 200 of user A, and the user context information (including location information) transmitted from the client 200 of user B, the server The common content information processed on (100) needs to be transmitted to the clients 200 of users A and B in real time.

However, the user context information of each user is acquired by a plurality of sensors provided in each client 200, and is usually transmitted to the server 100 at a communication speed having an upper limit. Therefore, there is a possibility that the progress of the content may be delayed, and the degree of delay may vary for each user.

Therefore, the client 200 may solve the problem by performing short-range wireless communication such as Bluetooth with other clients 200 as well as communication with the server 100. For example, first, it is assumed that the client 200 of the user A acquires the current position of the monster 23 from the server 100. The location information of the monster 23 is also acquired in real time by the client 200 of the user B. The user A performs a gesture of waving his hand from the right to the left, for example, to the monster 23. The user A's client 200 determines whether the monster 23 has been bounced or not, based on the user situation information regarding the user A's gesture and the location information of the monster 23 obtained from the server 100. It is judged without passing through ). Then, the client 200 of the user A generates content information (or event information included in the content information) that "the monster 23 has been bounced" according to the determination result.

Content information generated by the client 200 of user A is transmitted to the server 100 by a predetermined communication method and transmitted to the client 200 of user B by short-range wireless communication such as Bluetooth. Thereby, the client 200 of the user B can recognize the occurrence of the event "the monster 23 was bounced by the user A".

After that, the client 200 of the user B controls the behavior of the monster 23 based on the content information. In other words, the user B's client 200 can provide the user B with an event of "the monster 23 was bounced by user A" without going through the server 100.

The behavior or position of the monster 23 processed without going through the server 100 is corrected according to the result of processing by the server 100. At this time, the server 100 prioritizes the processing result of the user A's client 200 in order to realize the reflection of the actions of the user A and the user B into the content as possible in real time, and performs correction processing. Also works. For example, at the time of user A's gesture, the position of the monster 23 changes according to the position or behavior of user B, so that if the gesture of waving a hand from right to left by user A is originally, it will not touch the monster 23. There may be cases that do not. Even in this case, the server 100 prioritizes the result of "the monster 23 was bounced by user A" processed by the client 200 of user A, and corrects the progress of the event according to the result. You can do it. Specifically, the server 100 assumes the result that "the monster 23 was bounced by user A", the position of the monster 23 in the client 200 of the user A and the user B is In order to match, the positions of the monsters 23 in the clients 200 of the users A and B are corrected.

Thereby, gestures by each of the user A and the user B can be reflected with a small delay with respect to the content executed by each client 200. In addition, as a result of taking into account the sensor information of each client 200, it is possible to prevent rewinding of events such as correction so that a gesture that touches the monster 23 (a gesture that shakes a hand from right to left) does not touch. Can be suppressed.

(2.3. Third embodiment)

The third embodiment is an embodiment of real-time interaction between AR content and VR content.

Consider the case where there is a user A playing AR content and a user B playing VR content corresponding to the AR content at the same timing as the user A. In this case, the server 100 receives user context information (including location information) from the client 200 used by each user, and grasps each user's situation (including the location relationship), The user's situation can be reflected in each of AR content and VR content.

As a result, the server 100 can synchronize the events in the AR content played by the user A with the events in the VR content played by the user B, so that the server 100 has a high sense of presence and entertainment for these users. Can provide. For example, each user can cooperate to conduct an event (eg, solving a problem, etc.).

In this case, the movement speed and movement range of the user B playing the VR content are more limited than when the VR content unit is played without synchronizing the AR content and the event. More specifically, the moving speed in the virtual space is limited to about 4 to 10 [km] per hour so as to correspond to the speed when the real space is moved on foot. Further, with respect to the movement range of the user B playing the VR content, walking in the center of the road in a virtual space may be prohibited, or the movement range may be limited so that the road cannot be crossed without using a crosswalk.

On the client 200 of user A playing AR content, the position in real space of user B playing VR content may be displayed. For example, if the client 200 of the user A is a smartphone (or a transmissive HMD), the avatar representing the user B is placed on the image of the real space by pointing the angle of view of the camera of the smartphone to the range where the user B is located. The display of the smartphone is controlled to overlap. On the other hand, on the client 200 of user B playing VR content, the position in real space of user A playing AR content may be displayed. The position of user A in real space is detected by GPS, for example. When the client 200 of the user B is an HMD, the display of the HMD is controlled so that the avatar representing the user A is superimposed on, for example, an entire celestial sphere image by pointing the HMD in the range where the user A is located.

By the way, the location and direction of the client 200 of user A playing AR content may be specified by a combination of various sensors such as GPS, acceleration sensor, and gyro sensor. On the other hand, since various sensors each have a detection error, the position and direction of user A indicated by sensor values acquired by the various sensors may be different from the actual position and direction of user A. Accordingly, the position and direction of the avatar representing the user A based on the sensor value in the client 200 of the user B may be different from the actual position and direction of the user A. As a result, there is a possibility that the gaze of the user A who plays the AR content and that of the user B who plays the VR content are unintentionally deviated during communication.

In view of the above problems, when it is estimated that communication between the user B and the user A will be performed, for example, the user A's gaze on the client 200 of the user B is preferably corrected appropriately. Specifically, the angle formed by the straight line connecting the position of user A and the position of user B in the real space and the direction of the client 200 of user A is decreasing, that is, the user A and the user B face each other. When it is estimated that the user B's client 200, the sensor value acquired by the user A's client 200 indicates that the user A is not facing the user B directly, the user The avatar representing A may be faced to the user B. That is, the direction of the avatar is changed significantly more than the case where communication between the user B and the user A is estimated to be performed, rather than the case where communication between the user B and the user A is not estimated to be performed. This front-facing display process may be similarly executed in the client 200 of user A. As a result, it is possible to alleviate or eliminate the discrepancy in the gaze in communication between the user A and the user B.

(2.4. Modification)

In the above embodiment, the AR content and VR content on the earth have been basically described, but the server 100 according to the present disclosure may also be used to provide AR content and VR content about the outside of the earth. For example, VR content may be created by associating a virtual object with a map on the surface of a celestial body such as the moon or Mars. Alternatively, VR content may be created using a three-dimensional map of outer space. Such extraterrestrial content may be provided, for example, as content for astronaut training or pseudo-space travel by civilians. Alternatively, it may be provided as astronomical observation content available on the earth.

In the above embodiment, an example has been described in which a user who creates a content properly places a virtual object on the map information. In general, for areas such as temples, shrines, and temples, or areas with high risk such as on railroad tracks, prior authorization is required for the provision of AR contents in the area from the organization (or manager) that manages the area. It is desirable to receive. Accordingly, the AR content management organization related to the play area may be searched through the network based on the setting of the play area of each content by the user, and authorization-related information based on the search result may be presented to the user on the GUI. Alternatively, the platform may be configured to automatically transmit an authorization application to the AR content management organization by E-mail or the like based on the search result. In addition, when the platform does not receive permission to provide the content from the AR content management organization, the server 100 may be controlled to prohibit the provision of AR content from the server 100 to the client 200. On the other hand, for approval review, provision of produced VR content to at least the AR content management organization may be permitted.

Through the platform of the present disclosure, 3D modeling of a specific real object may be performed using 2D images acquired by the clients 200 of a plurality of users. Each user who plays AR content may receive an instruction (event) stating "Next, photograph a stuffed toy", for example. In accordance with this instruction, a plurality of different 2D images including a stuffed toy photographed at different viewpoints and angles are transmitted to the server 100 from the client 200 of each user. The server 100 can generate a 3D virtual object of a stuffed toy by analyzing the feature points of the plurality of 2D images. The server 100 determines, from a plurality of 2D images received, candidates for 2D images to be used for 3D modeling of real objects, specific event information that the client 200 that transmitted the 2D image has been progressing, and the client 200 It can be specified (limited) based on location information. Therefore, the server 100 can efficiently reduce the analysis load applied to 3D modeling. The generated 3D virtual object may be shared on the platform by giving location information. Accordingly, 3D virtual objects that can be used for AR content can be collected along with the provision of AR content. The collected 3D virtual object may be appropriately provided as a 3D virtual object displayed on a platform, for example, in VR content. In addition, even if the collected 3D virtual object is given property information of each user or client 200, which is obtained from the calculation result by the client 200 of a plurality of users transmitting 2D images using blockchain technology. do. This property information is suitably used for calculating the contribution rate of each user regarding the generation of the 3D virtual object, and a reward may be provided to each user according to the calculated contribution rate. Payment of the reward can be performed by, for example, provision of currency information including virtual currency based on blockchain technology or provision of privilege data related to AR content.

<3. Finishing>

As described above, the present disclosure may provide a platform capable of providing VR content by using at least part of information used for providing AR content. More specifically, the server 100 according to the present disclosure can create VR content by using at least a part of information used to provide AR content. Then, the server 100 may provide the created VR content to the client 200. In addition, the server 100 may create AR content or provide the created AR content to the client 200.

Accordingly, the present disclosure allows users to experience VR content corresponding to AR content, thereby causing user interest in AR content that was originally experienced only in the local area, so that AR content can be more efficiently distributed. .

In the above, preferred embodiments of the present disclosure have been described in detail with reference to the accompanying drawings, but the technical scope of the present disclosure is not limited to these examples. It is clear that a person having ordinary knowledge in the technical field of the present disclosure can conceive of various modifications or corrections within the scope of the technical idea described in the claims. It is understood to be within the technical scope.

For example, out of the context of AR content and VR content, a predetermined operation by the user (for example, a simple operation such as drag & drop) can be used in real space (or any object in real space). A system capable of associating objects and sharing the virtual objects regardless of the type of the client 200 may be provided. Thereby, for example, the user uses the GUI screen displayed on the client 200 to place a virtual material (virtual object) in an empty tenant by dragging and dropping, and the user (or another user) is actually an empty tenant. When it reaches to, the experience of recognizing the virtual object in the client 200 may be provided.

Further, as a virtual object, an object associated with the positional information and set as a creative common, a public pay object, an object limited only to a specific user, and the like may be appropriately searched and used.

In addition, the effects described in this specification are illustrative or illustrative to the last, and are not limiting. That is, the technology according to the present disclosure can exhibit other effects that are clear to those skilled in the art from the description of the present specification in addition to or instead of the above effects.

In addition, the following configuration also belongs to the technical scope of the present disclosure.

(One)

An acquisition unit that includes image information of a virtual object and position information of the virtual object in real space, and acquires content information added to map information indicating the real space;

Based on the content information, a first control unit for displaying the image information on a first client terminal such that the image information is superimposed on a position in the virtual space corresponding to the location information with respect to an image of the virtual space that can be visually recognized from a first-person view. Equipped,

Information processing device.

(2)

The image of the virtual space is a merchant corresponding to the real space based on a captured image of the real space,

The information processing device according to (1) above.

(3)

The first control unit, based on the location information on the map information of the second client terminal different from the first client terminal, so that the avatar corresponding to the second client terminal overlaps the image of the virtual space. Displaying on the first client terminal,

The information processing device according to (2) above.

(4)

The first control unit, based on the location information of the first client terminal regarding the map information, the location information of the second client terminal, and the posture information of the second client terminal, the user of the first client terminal When it is estimated that communication between the user of the second client terminal and the second client terminal is to be performed, the direction of the avatar is changed significantly, compared to a case where the communication is not estimated to be performed,

The information processing device according to (3) above.

(5)

The content information further includes information on an event to be performed by the virtual object,

The information processing device according to any one of (1) to (4) above.

(6)

The content information further includes sound image information of the virtual object,

The first control unit outputs the sound image information at a location in a virtual space corresponding to the location information based on the content information,

The information processing device according to any one of (1) to (5) above.

(7)

Based on the content information, further comprising a second control unit for displaying the image information so as to overlap the image of the real space at a position in the real space corresponding to the position information,

The information processing device according to any one of (1) to (6) above.

(8)

Further comprising a content creation unit for creating the content information based on an input from a user,

The information processing device according to any one of (1) to (7) above.

(9)

The content creation unit creates AR (Augmented Reality) content and VR (Virtual Reality) content corresponding to the AR content,

The information processing device according to (8) above.

(10)

The content creation unit creates the VR content by using at least part of the information used to create the AR content,

The information processing device according to (9) above.

(11)

The content creation unit provides a user with a GUI screen used for the input,

The information processing device according to any one of (8) to (11) above.

(12)

The content creation unit provides the user with, as the GUI screen, an input screen of the image information, an input screen of the location information, or an input screen of information about an event performed by the virtual object,

The information processing device according to (11) above.

(13)

The content creation unit,

Receiving drag operation information and drop operation information for the virtual object by the user,

When a combination of the property of the virtual object, the drop position of the virtual object corresponding to the drag operation information, and the map information corresponding to the drop position satisfies a predetermined condition, the position of the virtual object is mapped Automatically adjust based on information,

When the combination does not satisfy the predetermined condition, setting the position of the virtual object as a drop position by the user,

The information processing device according to (12) above.

(14)

The content creation unit includes a plurality of captured images acquired from a plurality of client terminals that play AR content, content information of the AR content acquired from the plurality of client terminals, and locations of the map information of the plurality of client terminals. Based on the information, performing 3D modeling of a specific real object in the real space,

The information processing device according to any one of (8) to (13) above.

(15)

Acquiring content information added to map information indicating the real space, including image information of a virtual object and position information of the virtual object in real space;

Based on the content information, displaying the image information on a first client terminal such that the image information is superimposed on a position in the virtual space corresponding to the location information with respect to an image of the virtual space that can be visually recognized from a first-person view,

A method of processing information executed by a computer.

(16)

Acquiring content information added to map information indicating the real space, including image information of a virtual object and position information of the virtual object in real space;

Based on the content information, displaying the image information on a first client terminal such that the image information is superimposed on a position in a virtual space corresponding to the location information with respect to an image of the virtual space that can be visually recognized from a first-person view.

Program for realizing the computer.

100: server
110: content creation unit
111: location processing unit
112: object processing unit
113: event processing unit
114: content creation control unit
120: content provider
121: acquisition unit
122: route decision section
123: content provision control unit
130: communication department
140: memory
200: client
210: sensor unit
211: outward camera
212: introverted camera
213: microphone
214: gyro sensor
215: acceleration sensor
216: orientation sensor
217: positioning portion
220: input unit
230: control unit
231: recognition engine
231a: Head posture recognition engine
231b: Depth recognition engine
231c: SLAM recognition engine
231d: gaze recognition engine
231e: speech recognition engine
231f: location awareness engine
231g: behavior recognition engine
232: content processing unit
240: output
250: communication department
260: memory

Claims (16)

An acquisition unit that includes image information of a virtual object and position information of the virtual object in real space, and acquires content information added to map information indicating the real space;
Based on the content information, a first control unit for displaying the image information on a first client terminal such that the image information is superimposed on a position in a virtual space corresponding to the location information with respect to an image of the virtual space that can be visually recognized from a first person viewpoint Equipped,
Information processing device. The method of claim 1,
The image of the virtual space is a merchant corresponding to the real space based on a captured image of the real space,
Information processing device. The method of claim 2,
The first control unit, based on the location information on the map information of the second client terminal different from the first client terminal, so that the avatar corresponding to the second client terminal overlaps the image of the virtual space. Displaying on the first client terminal,
Information processing device. The method of claim 3,
The first control unit, based on the location information of the first client terminal regarding the map information, the location information of the second client terminal, and the posture information of the second client terminal, the user of the first client terminal When it is estimated that communication between the user of the second client terminal and the second client terminal is to be performed, the direction of the avatar is changed significantly, compared to a case where the communication is not estimated to be performed
Information processing device. The method of claim 1,
The content information further includes information on an event to be performed by the virtual object,
Information processing device. The method of claim 1,
The content information further includes sound image information of the virtual object,
The first control unit outputs the sound image information at a location in a virtual space corresponding to the location information based on the content information,
Information processing device. The method of claim 1,
Based on the content information, further comprising a second control unit for displaying the image information so as to overlap the image of the real space at a position in the real space corresponding to the position information,
Information processing device. The method of claim 1,
Further comprising a content creation unit for creating the content information based on an input from a user,
Information processing device. The method of claim 8,
The content creation unit creates AR (Augmented Reality) content and VR (Virtual Reality) content corresponding to the AR content,
Information processing device. The method of claim 9,
The content creation unit creates the VR content by using at least part of the information used to create the AR content,
Information processing device. The method of claim 8,
The content creation unit provides a user with a GUI screen used for the input,
Information processing device. The method of claim 11,
The content creation unit provides the user with, as the GUI screen, an input screen of the image information, an input screen of the location information, or an input screen of information about an event performed by the virtual object,
Information processing device. The method of claim 12,
The content creation unit,
Receiving drag operation information and drop operation information for the virtual object by the user,
When a combination of the property of the virtual object, the drop position of the virtual object corresponding to the drag operation information, and the map information corresponding to the drop position satisfies a predetermined condition, the position of the virtual object is mapped Automatically adjust based on information,
When the combination does not satisfy the predetermined condition, setting the position of the virtual object as a drop position by the user,
Information processing device. The method of claim 8,
The content creation unit includes a plurality of captured images acquired from a plurality of client terminals that play AR content, content information of the AR content acquired from the plurality of client terminals, and locations of the map information of the plurality of client terminals. Based on the information, performing 3D modeling of a specific real object in the real space,
Information processing device. Acquiring content information added to map information indicating the real space, including image information of a virtual object and position information of the virtual object in real space;
Based on the content information, displaying the image information on a first client terminal such that the image information is superimposed on a position in the virtual space corresponding to the location information with respect to an image of the virtual space that can be visually recognized from a first-person view,
Information processing method executed by a computer. Acquiring content information added to map information indicating the real space, including image information of a virtual object and position information of the virtual object in real space;
Based on the content information, displaying the image information on a first client terminal such that the image information is superimposed on a position in a virtual space corresponding to the location information with respect to an image of the virtual space that can be visually recognized from a first-person view.
Program for realizing the computer.

Images