WO2023281803A1

WO2023281803A1 - Information processing device, information processing method, and storage medium

Info

Publication number: WO2023281803A1
Application number: PCT/JP2022/007217
Authority: WO
Inventors: 孝悌清水
Original assignee: ソニーグループ株式会社
Priority date: 2021-07-08
Filing date: 2022-02-22
Publication date: 2023-01-12

Abstract

[Problem] To further improve entertainment properties in a virtual space. [Solution] Provided is an information processing device comprising: an acquisition unit for acquiring feature information from content included in a first virtual object arranged in a virtual space; and a video generation unit for, on the basis of the feature information acquired by the acquisition unit, controlling display of a second virtual object in at least the surroundings of the content arranged in the virtual space.

Description

Information processing device, information processing method and storage medium

The present disclosure relates to an information processing device, an information processing method, and a storage medium.

VR (Virtual Reality) applications, which have become popular in recent years, allow users to view a virtual space in which a 3D model is placed from any viewpoint. Such a VR world can be provided mainly using a non-transmissive HMD (Head Mounted Display) that covers the user's field of vision with a display unit.

In addition, regarding the technology for providing virtual space, technology has been developed for displaying live video obtained by capturing events in real space in virtual space. For example, Patent Literature 1 listed below discloses a technique for arranging a virtual object such as a pillar in the vicinity of the boundary between a live video displayed in the virtual space and the virtual space.

WO2020/129115

However, with the technology described in Patent Document 1, the virtual object merely hides the boundary between the live video and the virtual space.

Therefore, the present disclosure proposes a new and improved information processing device, information processing method, and storage medium that can further improve entertainment in virtual space.

According to the present disclosure, an acquisition unit for acquiring feature information from content included in a first virtual object arranged in a virtual space, and at least arranged in the virtual space based on the feature information acquired by the acquisition unit. and a video generation unit that controls display of a second virtual object around the content to be processed.

Further, according to the present disclosure, acquiring feature information from content included in a first virtual object placed in a virtual space, and placing at least the virtual object in the virtual space based on the acquired feature information. and controlling the display of a second virtual object around said content.

Further, according to the present disclosure, there is provided a storage medium that non-temporarily stores a computer-executable program, the program extracting feature information from content included in a first virtual object arranged in a virtual space. and a display control function of controlling display of a second virtual object around at least the content arranged in the virtual space based on the feature information acquired by the acquisition function. A storage medium is provided.

1 is an explanatory diagram for explaining a configuration example of an information processing system according to the present disclosure; FIG. FIG. 4 is an explanatory diagram for explaining an example of a virtual space V to which XR live is distributed; 1 is an explanatory diagram for explaining a functional configuration example of a distribution system 10 according to the present disclosure; FIG. FIG. 4 is an explanatory diagram for explaining a functional configuration example of the HMD 40 according to the present disclosure; FIG. 4 is an explanatory diagram for explaining an image example of the first embodiment according to the present disclosure; FIG. FIG. 4 is an explanatory diagram for explaining an example of operation processing of the first embodiment according to the present disclosure; FIG. 11 is an explanatory diagram for explaining an example of an image in the second embodiment according to the present disclosure; FIG. FIG. 11 is an explanatory diagram for explaining an example of an image in the second embodiment according to the present disclosure; FIG. FIG. 11 is an explanatory diagram for explaining an example of operation processing of the second embodiment according to the present disclosure; FIG. 11 is an explanatory diagram for explaining an example of an image of the third embodiment according to the present disclosure; FIG. FIG. 11 is an explanatory diagram for explaining an example of operation processing of the third embodiment according to the present disclosure; FIG. 5 is an explanatory diagram for explaining a modification according to the present disclosure; 2 is an explanatory diagram showing a hardware configuration of an HMD 40 according to the present disclosure; FIG.

Preferred embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. In the present specification and drawings, constituent elements having substantially the same functional configuration are denoted by the same reference numerals, thereby omitting redundant description.

In addition, the "Mode for Carrying Out the Invention" will be explained according to the order of items shown below.
1. Overview 1.1. Outline of information processing system 1.2. Example of virtual space where XR live is delivered 1.3. Functional configuration example of distribution system 10 2 . Functional configuration example of HMD 40 3 . Example 3.1. First embodiment 3.2. Second embodiment 3.3. Third embodiment 4. Modification 5. Hardware configuration example6. supplement

<<1. Overview＞＞
As an embodiment of the present disclosure, a mechanism for further improving entertainment in virtual space will be described.

<1.1. Outline of information processing system>
FIG. 1 is an explanatory diagram for explaining a configuration example of an information processing system according to the present disclosure. As shown in FIG. 1, the information processing system according to the present disclosure includes a network 1, a distribution system 10, and an HMD 40.

(Network 1)
A network 1 is a wired or wireless transmission path for information transmitted from devices connected to the network 1 . For example, the network 1 may include a public line network such as the Internet, a telephone line network, a satellite communication network, various LANs (Local Area Networks) including Ethernet (registered trademark), WANs (Wide Area Networks), and the like. The network 1 may also include a dedicated line network such as IP-VPN (Internet Protocol-Virtual Private Network).

The distribution system 10 and the HMD 40 are connected via the network 1. The HMD 40A used by a certain user and the HMD 40B used by another user are also connected via the network 1, respectively.

(Distribution system 10)
The distribution system 10 is a system used by a user on the side of a distributor who distributes various types of event information such as live video and live audio (hereinafter sometimes referred to as a distribution user). For example, the distribution system 10 transmits to the HMD 40 a virtual space including live video captured by a camera and live audio captured by a sensor, which will be described later. In the following description, an event in which live video and live audio are delivered in a virtual space may be collectively referred to as XR (Extended Reality) live.

(HMD40)
The HMD 40 is an example of an information processing device, and is a terminal used by a user who views live video and live audio in virtual space. The HMD 40 is, for example, a head-mounted display to which applications such as VR (Virtual Reality) or AR (Augmented Reality) are applied.

A user wearing the HMD 40 can use an input device such as a hand controller to move the avatar, which is the alter ego of the user, in the virtual space.

Also, the user wearing the HMD 40 can view the live video from the first-person viewpoint based on the eyes of the avatar from the display of the HMD 40 .

Also, the HMD 40 according to the present disclosure acquires feature information from content included in virtual objects placed in the virtual space. In addition, the HMD 40 according to the present disclosure controls display of images representing the virtual space based on the acquired feature information. Note that display control includes displaying a virtual object in the virtual space, dynamically displaying the virtual object in the virtual space, and the like. Various details such as images representing virtual objects, content, and virtual space will be described later.

Next, with reference to FIG. 2, an example of a virtual space where a certain XR live is distributed will be described.

<1.2. Example of virtual space where XR live is delivered>
FIG. 2 is an explanatory diagram for explaining an example of the virtual space V in which XR live is delivered. A virtual space V in which a certain XR live is delivered includes, for example, a two-dimensional screen VC1, a two-dimensional video CT1, and an avatar U participating in the virtual space.

The two-dimensional screen VC1 is the first virtual object arranged to output the two-dimensional image CT1. The two-dimensional image CT1 is an example of content, and is a live image captured by a camera. The two-dimensional image CT1 may include images of equipment and facilities such as the performer P1 and the speaker C1 as shown in FIG. Also, the two-dimensional image CT1 is inserted on the two-dimensional screen VC1 arranged in the virtual space V. FIG. At this time, the two-dimensional image CT1 may be subjected to image processing according to the size and shape of the two-dimensional screen VC1. Note that the content includes the above-described two-dimensional image CT1 and various types of information such as sound information output in conjunction with the two-dimensional image CT1.

Also, the avatar U is the clone of the user who participated in the virtual space V where the XR live is delivered. For example, when a user in the virtual space V where the XR live is distributed participates, the user's avatar U1 is displayed in the virtual space V. FIG. In addition, when other users (for example, two users) also participate in the virtual space V where the XR live is distributed, the number of avatars U2 corresponding to the number of other users who participated in the virtual space V displayed above. Each avatar can move and act based on each user's operation.

An example of the virtual space V in which XR live is distributed has been described above, but the virtual space V according to the present disclosure is not limited to such an example. For example, virtual avatars such as NPCs (Non Player Characters) may be placed in the virtual space V, and various virtual objects such as buildings and mobile objects may be placed in the virtual space V.

The outline of the information processing system according to the present disclosure has been described above. Next, a functional configuration example of the distribution system 10 according to the present disclosure will be described with reference to FIG. 3 .

<<1.3. Functional Configuration Example of Distribution System 10>>
FIG. 3 is an explanatory diagram for explaining a functional configuration example of the distribution system 10 according to the present disclosure. As shown in FIG. 3 , distribution system 10 according to the present disclosure includes camera 20 , sensor 25 , and distribution server 30 .

(Camera 20)
The camera 20 captures an image of an object and obtains an image. For example, the camera 20 captures a live venue and acquires live video including performers and equipment or facilities at the live venue. The video captured by the camera 20 is transmitted to the distribution server 30 by any communication method.

(Sensor 25)
The sensor 25 is a sensor that acquires live sound at a live venue. Further, the sensor 25 may be a sensor that acquires body position information of the performer. Various information (for example, live sound) acquired by the sensor 25 is transmitted to the distribution server 30 by any communication method.

(Distribution server 30)
The distribution server 30 is a server used by the distributor. The distribution server 30 includes a control unit 310 and a communication unit 320, as shown in FIG.

{control unit 310}
The control unit 310 controls overall operations of the distribution server 30 . For example, the control unit 310 synchronizes live video and live audio received from the camera 20 and the sensor 25 and causes the communication unit 320 to transmit them to the HMD 40 used by the user.

{Communication unit 320}
Under the control of the control unit 310, the communication unit 320 performs various communications with the HMD 40 participating in the virtual space where the XR live is distributed. For example, the communication unit 320 streams a virtual space including live video captured by the camera 20 and live audio captured by the sensor 25 to the HMD 40 .

The above is an overview of the present disclosure. Next, a functional configuration example of the HMD 40 according to the present disclosure will be described with reference to FIG. 4 .

<<3. Example of functional configuration of HMD 40 >>
FIG. 4 is an explanatory diagram for explaining a functional configuration example of the HMD 40 according to the present disclosure. As shown in FIG. 4 , the HMD 40 according to the present disclosure includes a communication section 410 , a storage section 420 , a display section 430 , an operation section 440 and a control section 450 .

(Communication unit 410)
The communication unit 410 performs various communications with the distribution server 30 and other HMDs 40 under the control of the control unit 450 . A plurality of HMDs 40 participating in the virtual space V to which the same XR live is distributed are bi-directionally connected via the network 1 .

(storage unit 420)
Storage unit 420 holds software and various data. For example, the storage unit 420 holds learning data obtained by learning using pairs of object types and object feature information as teacher data. The object feature information may include information about the shape of the object and information about the color of the object. Also, the object to be learned as teacher data is not limited to an object. For example, the storage unit 420 may hold learning data obtained by learning using sets of various objects such as performers and lights and feature information of the objects as teacher data.

In addition, the storage unit 420 holds three-dimensional data representing images representing the virtual space, such as virtual objects and particles. Specific examples of virtual objects and particles will be described later.

(Display unit 430)
The display unit 430 displays an image in the virtual space from the first-person viewpoint based on the eyes of the avatar. Note that the first-person viewpoint image based on the avatar's eyes is generated based on sensor data acquired by the orientation sensor mounted on the HMD 40 . The functions of the display unit 430 are implemented by, for example, a CRT (Cathode Ray Tube) display device, a liquid crystal display (LCD) device, or an OLED (Organic Light Emitting Diode) device. In addition, the display unit 430 may display various images such as a screen for selecting a virtual space to participate in and an image from a bird's-eye view of the virtual space in which the participant participates.

(Operation unit 440)
The operation unit 440 controls movement and actions of the avatar. Functions of the operation unit 440 are implemented by a device such as a hand controller. Note that the avatar's action includes, for example, various actions of the avatar such as waving and jumping.

(control unit 450)
The control unit 450 controls overall operations of the HMD 40 according to the present disclosure. As shown in FIG. 3, the control unit 450 includes a color information detection unit 451, an object/body recognition unit 455, a correction processing unit 459, a correlation processing unit 463, a music analysis processing unit 467, and a space drawing processing unit. 471.

{Color information detector 451}
The color information detection unit 451 is an example of an acquisition unit and detects color information of a 2D image. For example, the color information detection unit acquires color information from the background image of the two-dimensional image. More specifically, the color information detection unit 451 acquires color information near the boundary between the 2D video and the virtual space in the background video included in the 2D video. It should be noted that the background image here indicates an area in which the performer and the object included in the two-dimensional image do not exist.

Also, the color information detection unit 451 may detect color information of an object included in the two-dimensional image.

{Object/body recognition unit 455}
The object/body recognition unit 455 is an example of an acquisition unit, and acquires feature information from a two-dimensional image. For example, the object/body recognition unit 455 recognizes objects and performers included in the two-dimensional video. The object/body recognition unit 455 also acquires feature information from objects and performers included in the two-dimensional video, and recognizes the objects and performers based on the feature information and learning data held in the storage unit 420 .

{Correction processing unit 459}
The correction processing unit 459 performs correction processing such as shape interpolation and noise removal on the images of the objects and performers recognized by the object/body recognition unit 455 .

{Correlation processing unit 463}
The correlation processing unit 463 calculates the degree of correlation between the image of the object or the performer corrected by the correction processing unit 459 and the three-dimensional data of the virtual object held by the storage unit 420 .

{music analysis processing unit 467}
The music analysis processing unit 467 is an example of an acquisition unit, and acquires sound information of live audio included in the two-dimensional video output in the virtual space. For example, the music analysis processing unit 467 analyzes the acquired sound information and acquires various kinds of information such as rhythm, tempo, volume change (crescendo and decrescendo) of the live sound.

{Spatial drawing processing unit 471}
The space drawing processing unit 471 is an example of an image generation unit, and controls display of an image representing a virtual space based on the feature information acquired by the object/body recognition unit 455 . A specific example of the video representing the virtual space will be described later.

Also, the space drawing processing unit 471 may control the display of the image representing the virtual space based on the color information of the background image detected by the color information detection unit 451 .

Also, the space drawing processing unit 471 may control the display of the video representing the virtual space based on the sound information acquired by the music analysis processing unit 467 .

The functional configuration example of the HMD 40 according to the present disclosure has been described above. Concrete examples of the “image representing the virtual space” whose display is controlled by the space drawing processing unit 471 will be sequentially described.

<<3. Example>>
<3.1. First embodiment>
In the first embodiment according to the present disclosure, the space drawing processing unit 471 controls the display of an image that expresses a virtual space with three-dimensional data of a virtual object similar to an object displayed on a two-dimensional image.

(Video example)
FIG. 5 is an explanatory diagram for explaining a video example of the first embodiment according to the present disclosure. First, the object/body recognition unit 455 recognizes an object included in the two-dimensional image CT1. In the example shown in FIG. 5, the object/body recognition unit 455 recognizes the speaker C1 installed at the live venue. The correlation processing unit 463 then calculates the degree of correlation between the speaker C<b>1 and each of the virtual objects held by the storage unit 420 .

Then, the space drawing processing unit 471 controls the display of the three-dimensional data of the virtual object whose degree of correlation satisfies a predetermined criterion by the correlation processing unit 463 as the image of the second virtual object, and arranges it in the virtual space V. In the example shown in FIG. 5, the space drawing processing unit 471 controls the display of the speaker model VC2 having a high degree of correlation with the speaker C1 as the second virtual object. This allows the user to feel more like they are participating in a live performance. A virtual object whose degree of correlation satisfies a predetermined criterion may be, for example, a virtual object whose degree of correlation is equal to or greater than a predetermined value.

Also, the space drawing processing unit 471 may control the display of the video of the speaker model VC2 in an area that does not interfere with other virtual objects placed in the virtual space V. At this time, the number of speaker models VC2 whose display is controlled may be singular or plural.

Also, objects included in the two-dimensional image CT1 may change over time, but when the speaker C1 disappears from the two-dimensional image CT1, the space drawing processing unit 471 renders the once-displayed speaker model VC2 into the virtual space. You can erase it from the top, or you can leave it on. Further, the space drawing processing unit 471 may erase the image of the speaker model VC2 when the speaker C1 is not displayed from the two-dimensional image CT1 for a certain period of time.

(Operation example)
FIG. 6 is an explanatory diagram for explaining an operation processing example of the first embodiment according to the present disclosure. First, the object/body recognition unit 455 recognizes an object on a two-dimensional image (S101). For example, when a speaker is included in the two-dimensional image, the object/body recognition unit 455 recognizes the object type of the speaker on the two-dimensional image as “speaker”.

Subsequently, the color information detection unit 451 acquires the color information of the object (S105).

Then, the correlation processing unit 463 calculates the degree of correlation between the virtual object held by the storage unit 420 and the color information of the virtual object, the type of the recognized object, and the color information of the object (S113). At this time, the correlation processing unit 463 determines a virtual object that has a high degree of similarity to the object based on the degree of correlation between the type of object and the color information of the object. may be prepared. More specifically, when determining the degree of similarity, the weighting parameter for the type of object may be set larger than the weighting parameter set for the color information of the object. As a result, it is possible to improve the determination accuracy of the product type of the virtual object, which has a particularly large impact on the user's visual recognition.

Then, based on the degree of correlation calculated in S113, the control unit 450 determines whether or not there is a virtual object similar to the recognized object (S113). If it is determined that there is a similar virtual object (S113/Yes), the process proceeds to S117, and if it is determined that there is no similar virtual object (S113/No), the control unit 450 according to the present disclosure processes exit.

When it is determined that there are similar virtual objects (S113/Yes), the control unit 450 acquires position information of each virtual object in the virtual space (S117).

Then, the space drawing processing unit 471 arranges the virtual objects determined to be similar in S113 at positions that do not interfere with each virtual object (S121), and the control unit 450 according to the present disclosure ends the processing.

The first embodiment according to the present disclosure has been described above. Next, a second embodiment according to the present disclosure will be described with reference to FIGS. 7 to 9. FIG.

<3.2. Second embodiment>
In the second embodiment according to the present disclosure, the space drawing processing unit 471 creates virtual objects (for example, lighting) or particles that have the same color as the color indicated by the color information of the background image of the dynamically changeable two-dimensional image. The display of the image is controlled as an image expressing the virtual space.

(Video example)
FIG. 7 is an explanatory diagram for explaining a video example of the second embodiment according to the present disclosure. First, the color information detection unit 451 detects color information from the background image of the two-dimensional image CT1.

Then, based on the color information acquired by the color information detection unit 451, the space drawing processing unit 471 controls the display of the video representing the virtual space. More specifically, the space drawing processing unit 471 controls display of an image having a similar color to the color indicated by the color information acquired by the color information detection unit 451 .

For example, as shown in FIG. 7, the color information detection unit 451 acquires color information near the boundary between the two-dimensional image CT1 and the virtual space V. Then, the space drawing processing unit 471 controls the display of the image VC3 of the light emitting light having the same color as the color indicated by the color information as the third virtual object. As a result, the visibility of the boundary between the two-dimensional image CT1 and the virtual space V is reduced, and the user can feel more immersed in the virtual space V. FIG.

Further, the space drawing processing unit 471 may control the display of the light image VC3 so as to hide the boundaries with the virtual space V at both ends of the two-dimensional screen VC1, as shown in FIG. For example, the space drawing processing unit 471 controls the display of the image based on the color information near the left edge boundary between the two-dimensional image CT1 and the virtual space V so as to hide the boundary with the virtual space V at the left edge of the two-dimensional screen VC1. do. Further, the space drawing processing unit 471 controls the display of the image based on the color information near the right edge boundary between the two-dimensional image CT1 and the virtual space V so as to hide the boundary between the two-dimensional screen VC1 and the virtual space V at the right edge. do. As a result, even when the left and right sides of the two-dimensional screen VC1 have different background colors, the visibility of the boundary between the two-dimensional image CT1 and the virtual space V is reduced, and the user can feel more immersed in the virtual space V.

Also, the image based on the color information acquired by the color information detection unit 451 is not limited to the image of a virtual object such as a light. For example, it may be an image of particles such as light or particles.

FIG. 8 is an explanatory diagram for explaining a video example of the second embodiment according to the present disclosure. As shown in FIG. 8, the space drawing processing unit 471 may control the display of the particle image E1 so as to hide the boundaries with the virtual space V at both ends of the two-dimensional screen VC1.

In addition, the space drawing processing unit 471 may control display of an image obtained by combining the light image VC3 and the particle image E1 described above.

(Operation example)
FIG. 9 is an explanatory diagram for explaining an operation processing example of the second embodiment according to the present disclosure. First, the color information detection unit 451 acquires color information from the background image included in the two-dimensional image (S201).

Next, the space drawing processing unit 471 controls the display of lights that emit light of the same color as the color indicated by the color information so as to hide the boundaries at both ends of the two-dimensional image (S205).

Then, the control unit 450 determines whether or not the degree of similarity between the background color of the 2D video included in the 2D screen and the background color of the virtual space is equal to or greater than a threshold (S209). If the degree of similarity is greater than or equal to the threshold (S209/Yes), the process ends, and if the degree of similarity is less than the threshold (S209/No), the process proceeds to S213.

If the degree of similarity is less than the threshold (S209/No), the space drawing processing unit 471 controls the display of the image of the light with the expanded light irradiation range (S213). Then, the processing of S209 to S213 is repeated until it is determined in S209 that the degree of similarity is equal to or greater than the threshold.

The second embodiment according to the present disclosure has been described above. Next, a third embodiment according to the present disclosure will be described with reference to FIGS. 10 and 11. FIG.

<3.3. Third embodiment>
In the third embodiment according to the present disclosure, the space drawing processing unit 471 expresses images such as NPC actions, virtual object actions, particles, etc. in the virtual space based on the sound information acquired by the music analysis processing unit 467. Control display as video.

(Video example)
FIG. 10 is an explanatory diagram for explaining a video example of the third embodiment according to the present disclosure. First, the music analysis processing unit 467 analyzes the sound information M1 of the live audio included in the two-dimensional image CT1 output in the virtual space V, and acquires various information such as the rhythm of the live audio. In FIG. 10, the sound information M1 is illustrated as a musical note mark, but the sound information M1 is actually included in the voice, so it is not displayed in the video.

Then, the space drawing processing unit 471 may control the display of the video of the virtual object that moves in response to the rhythm of the live sound as the video of the fourth virtual object. The image of the virtual object that moves in response to the rhythm of sound may be, for example, an image that moves as if the psyllium VC 4 shown in FIG. 10 is swung from side to side.

Further, the space drawing processing unit 471 may control the display of an image E2 of particles such as light and particles that move in response to the rhythm of sound as an image representing the virtual space V. By displaying such video linked with the live sound in the virtual space V, the user can improve the sense of participation in the live performance and the sense of unity with the surrounding audience.

(Operation example)
FIG. 11 is an explanatory diagram for explaining an operation processing example of the third embodiment according to the present disclosure. First, the music analysis processing unit 467 acquires sound information output in conjunction with a two-dimensional image (S301).

Next, the music analysis processing unit 467 analyzes the rhythm, volume, etc. from the acquired sound information (S305).

Then, the space drawing processing unit 471 displays the virtual object corresponding to the analysis result in S305 in the virtual space (S309), and the control unit 450 according to the present disclosure ends the processing.

In the above-described embodiments, an example in which the two-dimensional screen is a planar screen has been described, but the two-dimensional screen according to the present disclosure is not limited to such an example. Next, a modification according to the present disclosure will be described with reference to FIG. 12 .

<<4. Modification>>
FIG. 12 is an explanatory diagram for explaining a modification according to the present disclosure. For example, the two-dimensional screen VC1 may have a curved shape as shown in FIG. For example, the two-dimensional screen VC1 may have a curved shape with a curvature radius corresponding to a predetermined area range on the virtual space V. FIG.

It should be noted that the predetermined area range may be the crowd area PA1 as shown in FIG. This makes it possible to reduce the effects of video distortion that may occur depending on the viewing position in the crowd area PA1, and provides the user with the feeling of participating in an actual live performance.

The modified example according to the present disclosure has been described above. Next, a hardware configuration example of the HMD 40 according to the present disclosure will be described with reference to FIG. 13 .

<<5. Hardware configuration example >>
The embodiment according to the present disclosure has been described above. The information processing described above is realized by cooperation between software and the hardware of the HMD 40 described below. The hardware configuration described below can also be applied to the camera 20, the sensor 25, or the distribution server 30 included in the distribution system 10. FIG.

FIG. 13 is an explanatory diagram showing the hardware configuration of the HMD 40 according to the present disclosure. As shown in FIG. 13, the HMD 40 includes a CPU (Central Processing Unit) 4201, a ROM (Read Only Memory) 4202, a RAM (Random Access Memory) 4203, an input device 4208, an output device 4210, and a storage device 4211. , a drive 4212 , an imaging device 4213 , and a communication device 4215 .

The CPU 4201 functions as an arithmetic processing device and a control device, and controls the overall operation of the HMD 40 according to various programs. Alternatively, the CPU 4201 may be a microprocessor. The ROM 4202 stores programs and calculation parameters used by the CPU 4201 . The RAM 4203 temporarily stores programs used in the execution of the CPU 4201, parameters that change as appropriate during the execution, and the like. These are interconnected by a host bus comprising a CPU bus or the like.

The input device 4208 generates an input signal based on input means for the user to input information, such as a hand controller, mouse, keyboard, touch panel, button, microphone, switch, and lever, and an input by the user, and outputs the signal to the CPU 4201 . It consists of an input control circuit, etc. A user using the HMD 40 can input various data to the HMD 40 and instruct processing operations by operating the input device 4208 .

Output devices 4210 include display devices such as liquid crystal display (LCD) devices, OLED devices, and lamps, for example. In addition, output device 4210 includes audio output devices such as speakers and headphones. For example, the display device displays an image of an avatar viewpoint in virtual space. The audio output device converts audio data including live audio into audio and outputs the audio.

The storage device 4211 is a data storage device configured as an example of the storage unit 420 of the HMD 40 according to the present disclosure. The storage device 4211 may include a storage medium, a recording device that records data on the storage medium, a reading device that reads data from the storage medium, a deletion device that deletes data recorded on the storage medium, and the like. The storage device 4211 stores programs executed by the CPU 4201 and various data.

A drive 4212 is a reader/writer for storage media, and is built in or externally attached to the HMD 40 . The drive 4212 reads information recorded in the removable storage medium 44 such as a semiconductor memory and outputs it to the RAM 4203 . Drive 4212 can also write information to removable storage medium 44 .

The imaging device 4213 includes an imaging optical system such as an imaging lens and a zoom lens for condensing light, and a signal conversion element such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor). The imaging optical system collects light emitted from a subject to form a subject image in a signal conversion section, and the signal conversion element converts the formed subject image into an electrical image signal.

The communication device 4215 is, for example, a communication interface configured with a communication device or the like for connecting to the network 1. Also, the communication device 4215 may be a wireless LAN compatible communication device, an LTE compatible communication device, or a wire communication device that performs wired communication.

The hardware configuration example of the HMD 40 according to the present disclosure has been described above. Next, supplements related to the present disclosure will be described.

<<6. Supplement >>
Although the preferred embodiments of the present disclosure have been described in detail above with reference to the accompanying drawings, the present disclosure is not limited to such examples. It is clear that a person having ordinary knowledge in the technical field to which the present disclosure belongs can conceive of various modifications or modifications within the scope of the technical idea described in the claims. It is understood that these also naturally belong to the technical scope of the present disclosure.

For example, some functions included in the HMD 40 may be configured separately from the HMD 40. For example, the HMD 40 transmits information about two-dimensional video and audio in the virtual space to the server, and the server executes various processes such as object recognition included in the two-dimensional video and analysis of rhythms included in sound information. may be This can reduce the burden on the HMD 40 .

Also, if the audience at the live venue is included in the two-dimensional video, the object/body recognition unit 455 may recognize the audience. Then, the space drawing processing unit 471 may control display of images such as particles that move according to the amount of movement of the spectator.

Also, each step in the processing of the HMD 40 in this specification does not necessarily have to be processed in chronological order according to the order described as the flowchart. For example, each step in the processing of the HMD 40 may be processed in an order different from the order described as the flowchart or in parallel.

In addition, hardware such as CPU, ROM, and RAM built in HMD 40, camera 20, sensor 25, and distribution server 30 exhibits the same functions as each configuration of HMD 40, camera 20, sensor 25, and distribution server 30 described above. A computer program can also be written to do so. A storage medium storing the computer program is also provided.

Also, the effects described in this specification are merely descriptive or exemplary, and are not limiting. In other words, the technology according to the present disclosure can produce other effects that are obvious to those skilled in the art from the description of this specification, in addition to or instead of the above effects.

Note that the following configuration also belongs to the technical scope of the present disclosure.
(1)
an acquisition unit that acquires feature information from content included in a first virtual object placed in the virtual space;
a video generation unit that controls display of a second virtual object around at least the content arranged in the virtual space based on the feature information acquired by the acquisition unit;
An information processing device.
(2)
The acquisition unit
obtaining the feature information from an object included in the content;
The information processing device according to (1) above.
(3)
The acquisition unit
Recognizing an object based on the acquired feature information,
The video generation unit
controlling display of a virtual object whose degree of correlation with the object recognized by the acquisition unit satisfies a predetermined criterion as an image of the second virtual object;
The information processing device according to (2) above.
(4)
A virtual object whose degree of correlation meets a predetermined criterion is a virtual object whose degree of correlation is equal to or greater than a predetermined value,
The information processing device according to (3) above.
(5)
The video generation unit
controlling display of the image of the second virtual object in an area that does not interfere with other virtual objects placed in the virtual space;
The information processing apparatus according to any one of (2) to (4).
(6)
The acquisition unit
obtaining color information from a background video included in the content;
The video generation unit
controlling display of an image representing the virtual space based on the color information of the background image acquired by the acquisition unit;
The information processing apparatus according to any one of (1) to (5) above.
(7)
The video generation unit
controlling display of an image having a similar color to the color indicated by the color information of the background image acquired by the acquisition unit;
The information processing device according to (6) above.
(8)
The video generation unit
Based on the color information of the background image acquired by the acquisition unit, controlling display of an image of a third virtual object emitting light having a color similar to that indicated by the color information of the background image;
The information processing device according to (7) above.
(9)
The video generation unit
controlling the display of an image based on the color information of the background image acquired by the acquisition unit so as to hide boundaries with the virtual space at both ends of the first virtual object;
The information processing apparatus according to any one of (6) to (8).
(10)
the color information of the background video indicates a color near the boundary between the content and the virtual space in the content;
The information processing apparatus according to any one of (6) to (9).
(11)
The video generation unit
controlling display of an image based on color information near a left edge boundary between the content and the virtual space in the content so as to hide a boundary with the virtual space at the left edge of the first virtual object; controlling display of an image based on color information near the right edge boundary between the content and the virtual space so as to hide the boundary with the virtual space at the right edge of the first virtual object;
The information processing device according to (10) above.
(12)
The acquisition unit
Acquiring sound information included in the content output in the virtual space;
The video generation unit
controlling display of an image representing the virtual space based on the sound information acquired by the acquisition unit;
The information processing apparatus according to any one of (1) to (11).
(13)
The video generation unit
controlling display of a video of a fourth virtual object representing the virtual space based on the sound information acquired by the acquisition unit;
The information processing device according to (12) above.
(14)
The video generation unit
controlling video display that moves in response to the rhythm of the sound acquired by the acquisition unit;
The information processing apparatus according to (12) or (13).
(15)
The first virtual object is a screen having a curved shape,
The information processing apparatus according to any one of (1) to (14).
(16)
The curved shape has a radius of curvature corresponding to a predetermined area range in the virtual space,
The information processing device according to (15) above.
(17)
acquiring feature information from content included in a first virtual object placed in the virtual space;
controlling display of a second virtual object around at least the content arranged in the virtual space based on the acquired feature information;
A computer-implemented information processing method comprising:
(18)
A storage medium that non-temporarily stores a computer-executable program,
Said program
an acquisition function for acquiring feature information from content included in a first virtual object placed in the virtual space;
a display control function that controls display of a second virtual object around at least the content placed in the virtual space based on the feature information acquired by the acquisition function;
A storage medium that realizes

1 network 10 distribution system 20 camera 25 sensor 30 distribution server 310 control unit 320 communication unit 40 HMD
410 communication unit 420 storage unit 430 display unit 440 operation unit 450 control unit 451 color information detection unit 455 object/body recognition unit 459 correction processing unit 463 correlation processing unit 467 music analysis processing unit 471 space drawing processing unit

Claims

an acquisition unit that acquires feature information from content included in a first virtual object placed in the virtual space;
a video generation unit that controls display of a second virtual object around at least the content arranged in the virtual space based on the feature information acquired by the acquisition unit;
An information processing device.
The acquisition unit
obtaining the feature information from an object included in the content;
The information processing device according to claim 1 .
The acquisition unit
Recognizing an object based on the acquired feature information,
The video generation unit
controlling display of a virtual object whose degree of correlation with the object recognized by the acquisition unit satisfies a predetermined criterion as an image of the second virtual object;
The information processing apparatus according to claim 2.
A virtual object whose degree of correlation meets a predetermined criterion is a virtual object whose degree of correlation is equal to or greater than a predetermined value,
The information processing apparatus according to claim 3.
The video generation unit
controlling display of the image of the second virtual object in an area that does not interfere with other virtual objects placed in the virtual space;
The information processing apparatus according to claim 4.
The acquisition unit
obtaining color information from a background video included in the content;
The video generation unit
controlling display of an image representing the virtual space based on the color information of the background image acquired by the acquisition unit;
The information processing device according to claim 5 .
The video generation unit
controlling display of an image having a similar color to the color indicated by the color information of the background image acquired by the acquisition unit;
The information processing device according to claim 6 .
The video generation unit
Based on the color information of the background image acquired by the acquisition unit, controlling display of an image of a third virtual object emitting light having a color similar to that indicated by the color information of the background image;
The information processing apparatus according to claim 7.
The video generation unit
controlling the display of an image based on the color information of the background image acquired by the acquisition unit so as to hide boundaries with the virtual space at both ends of the first virtual object;
The information processing apparatus according to claim 8 .
the color information of the background video indicates a color near the boundary between the content and the virtual space in the content;
The information processing apparatus according to claim 9 .
The video generation unit
controlling display of an image based on color information near a left edge boundary between the content and the virtual space in the content so as to hide a boundary with the virtual space at the left edge of the first virtual object; controlling display of an image based on color information near the right edge boundary between the content and the virtual space so as to hide the boundary with the virtual space at the right edge of the first virtual object;
The information processing apparatus according to claim 10.
The acquisition unit
Acquiring sound information included in the content output in the virtual space;
The video generation unit
controlling display of an image representing the virtual space based on the sound information acquired by the acquisition unit;
The information processing device according to claim 11 .
The video generation unit
controlling display of a fourth virtual object representing the virtual space based on the sound information acquired by the acquisition unit;
The information processing apparatus according to claim 12.
The video generation unit
controlling the display of moving images in response to the rhythm of the sound acquired by the acquisition unit;
The information processing apparatus according to claim 13.
The first virtual object is a screen having a curved shape,
The information processing apparatus according to claim 14.
The curved shape has a radius of curvature corresponding to a predetermined area range in the virtual space,
The information processing device according to claim 15 .
acquiring feature information from content included in a first virtual object placed in the virtual space;
controlling display of a second virtual object around at least the content arranged in the virtual space based on the acquired feature information;
A computer-implemented information processing method comprising:
A storage medium that non-temporarily stores a computer-executable program,
Said program
an acquisition function for acquiring feature information from content included in a first virtual object placed in the virtual space;
a display control function that controls display of a second virtual object around at least the content placed in the virtual space based on the feature information acquired by the acquisition function;
A storage medium that realizes