JP7173249B2 - CLIENT DEVICE, DISPLAY SYSTEM, CLIENT DEVICE PROCESSING METHOD AND PROGRAM - Google Patents

Description

The present technology relates to a client device, a display system, a processing method of the client device, and a program .

A head-mounted display (HMD) is known, which is worn on the user's head and is capable of presenting an image to the individual user through a display or the like arranged in front of the user's eyes (see, for example, Patent Document 1). ). In recent years, individuals have begun to enjoy omnidirectional level images created for VR (virtual reality) using HMDs. It is expected that there will be a plurality of viewers enjoying themselves in their own space, and those viewers will be able to communicate by sharing not only their own VR space but also each other's VR space.

JP 2016-025633 A

The purpose of this technology is to enable multiple clients (viewers) to share each other's VR space and communicate with each other.

The concept of this technology is
Receiving a server-delivered stream including a video stream obtained by encoding image data of a background image from a server, and transmitting from another client device including substitute image meta information for displaying the substitute image of the other client. a receiver for receiving the stream;
decoding processing for decoding the video stream to obtain image data of a background image; substitute image data generation processing for generating image data for a substitute image based on the substitute image meta information; The client apparatus includes a control unit for controlling image data synthesis processing for synthesizing image data of images.

In the present technology, the receiving unit receives a server-delivered stream including a video stream obtained by encoding image data of a background image from a server, and displays a substitute image of another client from another client device. A client-sent stream containing substitute image meta information for is received. A substitute image is, for example, a symbol that can recognize an avatar or a character.

The control unit controls decoding processing, substitute image data generation processing, and image data synthesis processing. In the decoding process, the video stream is decoded to obtain image data of the background image. In the substitute image data generation process, image data of the substitute image is generated based on the substitute image meta information. In the image data synthesizing process, the image data of the background image is synthesized with the image data of the substitute image.

For example, information indicating the permissible composition range of the substitute image in the background image is inserted in the video stream layer and/or the server distribution stream layer, and the control unit determines whether the substitute image is the background image based on the information indicating the permissible composition range. The compositing process may be controlled so that the image is arranged within the allowable compositing range of the image.

In this case, the substitute image meta information includes compositing position information indicating a compositing position within the permissible compositing range of the substitute image. may be configured to control the above synthesis process. Further, in this case, for example, the substitute image meta information includes size information indicating the size of the substitute image, and the control unit synthesizes the substitute image with the background image in the size indicated by the size information. may be used to control the processing.

As described above, in the present technology, the image data of the substitute image is generated based on the image data of the background image and the meta information of the substitute image, and the image data of the substitute image is combined with the image data of the background image. Therefore, each of the clients can recognize that the substitute image of the other client is synthesized with the common background image, and can communicate well by sharing the VR space with each other.

Note that in the present technology, for example, the client transmission stream includes audio data corresponding to the substitute image meta information together with object metadata, and the control unit performs rendering processing on the audio data according to the object metadata. may further control the audio output processing for obtaining audio output data having the position of synthesizing the substitute image as the position of the sound image. This makes it possible for each client to perceive as if the sound from the client of the substitute image is coming from the composite position of each substitute image on the background image.

Further, in the present technology, for example, the client transmission stream includes caption data corresponding to the substitute image meta information together with the display position information, and the control unit determines whether the caption based on the caption data corresponds to the composition position of the substitute image. It may be configured such that a caption synthesizing process for synthesizing the display data of the caption with the image data of the background image is further controlled based on the display position information so that the caption is displayed at the position. This enables each client to recognize the caption from the client of the substitute image at the position corresponding to the composite position of each substitute image on the background image.

Further, in the present technology, for example, a transmitting unit that transmits a client transmission stream including substitute image meta information for displaying the substitute image of the user to another client device is provided. Image data of its own substitute image may be further generated based on the substitute image meta information for displaying the substitute image. As a result, it becomes possible to synthesize not only the substitute image of another client but also the substitute image of the client itself with the background image.

Further, in the present technology, for example, the image data of the background image is the image data of the wide-viewing-angle image, and the control unit performs image clipping processing for obtaining display image data by clipping a part of the image data of the background image. Further control may be performed. For example, an image based on display image data is displayed on an HMD, and the cutout range is determined according to, for example, the head posture detected by a sensor mounted on the HMD.

Another concept of this technology is
an imaging unit that captures an image of a subject and obtains image data of a background image;
a transmitting unit configured to transmit a server-delivered stream including a video stream obtained by encoding image data of the background image to a client device;
The server inserts information indicating the permissible composition range of the substitute image for the background image into the layer of the video stream and/or the layer of the server-delivered stream.

In the present technology, the imaging unit captures an image of a subject and obtains image data of a background image. For example, the image data of this background image may be image data of a wide viewing angle image. The transmitting unit transmits to the client device a server-delivered stream including a video stream obtained by encoding the image data of the background image. Here, information indicating the permissible composition range of the substitute image for the background image is inserted into the layer of the video stream and/or the layer of the server-delivered stream.

As described above, according to the present technology, information indicating the permissible composition range of the substitute image for the background image is inserted into the layer of the video stream and/or the layer of the server-delivered stream and delivered. Therefore, the client device can easily place the substitute image of each client in the background image in the range intended by the server based on the information indicating the allowable composition range.

According to this technology, it becomes possible for a plurality of clients to communicate with each other by sharing their VR space. Note that the effects described here are not necessarily limited, and may be any of the effects described in the present disclosure.

1 is a block diagram showing a configuration example of a space sharing display system as an embodiment; FIG. FIG. 2 is a diagram showing an example of the stream transmission/reception relationship between a server and a plurality of client devices; It is a block diagram which shows the structural example of a server. FIG. 10 is a diagram showing an example structure of a Video Attribute Information SEI message; FIG. 4 is a diagram showing the content of main information in a structural example of a Video Attribute Information SEI message; FIG. 4 is a diagram for explaining state information of a camera; FIG. FIG. 4 is a diagram showing an example of information stored in a video attribute information box; FIG. 4 is a block diagram showing a configuration example of a transmission system of a client device; FIG. FIG. 3 is a diagram showing an example structure of avatar rendering control information and the content of main information in the example structure; FIG. 3 is a diagram showing an example structure of avatar database selection information and the contents of main information in the example structure. FIG. 3 is a diagram showing a structural example of audio object rendering information as object metadata of each object, and the content of main information in the structural example. FIG. 4 is a diagram for explaining how to obtain the values of "Azimuth", "Radius", and "Elevation"; It is a figure for demonstrating the structure example of a TTML structure and metadata. 4 is a block diagram showing a configuration example of a reception system of a client device; FIG. 4 is a block diagram showing a configuration example of a receiving module; FIG. FIG. 4 is a block diagram showing a configuration example of an avatar database selection unit; FIG. 10 is a diagram showing an example of an avatar database list; FIG. 4 is a diagram showing an outline of rendering processing in a renderer; FIG. 4 is a diagram schematically showing sound pressure control by remapping in a renderer; It is a figure which shows an example of a background image. FIG. 10 is a diagram showing an example of a state in which an avatar and subtitles are synthesized within the allowable synthesis range (sy_window) of the background image;

DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, modes for carrying out the invention (hereinafter referred to as "embodiments") will be described. The description will be given in the following order.
1. Embodiment 2. Modification

<1. Embodiment>
[Spatial sharing display system]
FIG. 1 shows a configuration example of a spatial sharing display system 10 as an embodiment. This shared space display system 10 has a configuration in which a server 100 and a plurality of client devices 200 are connected via a network 300 such as the Internet.

The server 100 transmits a server delivery stream including a video stream obtained by encoding image data of a background image obtained by imaging a subject to each client device 200 via the network 300 . For example, the image data of the background image is the image data of the wide viewing angle image. Information indicating the permissible composition range of the substitute image for the background image is inserted in the video stream layer and/or the server delivery stream (container) layer. The substitute image is, for example, an avatar or a symbol by which a character can be recognized. In this embodiment, an avatar is assumed as the substitute image. Below, a substitute image is demonstrated as an avatar.

The client device 200 receives the server-delivered stream sent from the server 100 via the network 300, decodes the video stream included in this server-delivered stream, and obtains the image data of the background image. Also, the client device 200 receives a client transmission frame (container) including avatar meta information for displaying another client's avatar, which is sent from another client device 200 via the network 300 .

The client device 200 generates image data of the avatar based on the avatar meta information, and synthesizes the image data of the avatar with the image data of the background image data. In this case, the client device 200 determines whether the avatar is within the allowable synthesis range of the background image based on the information indicating the allowable synthesis range of the avatar in the background image inserted in the layer of the video stream and/or the layer of the server distribution stream. to be placed in

The avatar meta information includes synthesis position information indicating a synthesis position within the allowable synthesis range of the avatar, and the client device 200 allows the avatar to be synthesized at the synthesis position indicated by this synthesis position information. In addition, the avatar meta information includes size information indicating the size of the avatar, and the client device 200 allows the avatar to be combined with the background image in the size indicated by the size information.

Also, the client device 200 transmits a client transmission stream (container) including avatar meta information for displaying its own avatar to other client devices 200 via the network 300 . In this case, the client device 200 generates image data of its own avatar based on the avatar meta information for displaying its own avatar, and synthesizes this avatar image data with the image data of the background image data.

Note that some client devices 200 may not have a function of transmitting client transmission streams (containers) containing avatar meta information for displaying their own avatars to other client devices 200 .

When the image data of the background image is the image data of the normal viewing angle image, the client device 200 directly uses the image data of the background image combined with the image data of the avatar as the image data for display. send to On the other hand, when the image data of the background image is the image data of the wide-viewing-angle image, the client device 200 cuts out part of the image data of the background image to which the image data of the avatar is synthesized, and prepares the image data for display. Then, this display image data is sent to the HMD 400A as a display device. The cropping range in this case is determined, for example, according to the head posture detected by the HMD-equipped sensor.

Also, the client transmission stream sent from another client device 200 includes audio data corresponding to the avatar meta information together with the object meta data. Also, with respect to avatar meta information for displaying one's own avatar, corresponding audio data exists together with object meta data. The client device 200 performs rendering processing on the audio data according to the object metadata, obtains audio output data with the avatar synthesis position as the sound image position, and sends the audio output data to the headphone (HP) 400B as an audio output device.

Also, the client transmission stream sent from another client device 200 includes caption data corresponding to the avatar meta information together with the display position information. Also, subtitle data corresponding to avatar meta information for displaying one's own avatar exists together with display position information. The client device 200 synthesizes the display data of the caption with the image data of the background image based on the display position information so that the caption based on the caption data is displayed at a position corresponding to the avatar's synthesis position.

FIG. 2 shows an example of the stream transmission/reception relationship between the server 100 and a plurality of client devices 200 . In the illustrated example, there are three client devices 200, and client transmission frames are sent from all client devices 200 to other client devices 200. FIG. The client transmission frame includes avatar meta information, audio data and text data (caption data).

[Server configuration]
FIG. 3 shows a configuration example of the server 100. As shown in FIG. This server 100 includes a control unit 101, a locator 102, a video capture 103, a format conversion processing unit 104, a video encoder 105, an audio capture 106, an audio encoder 108, a container encoder 109, and a network interface 110. have. Each unit is connected by a bus 111 .

The control unit 101 controls operations of each unit of the server 100 . A user operation unit 101 a is connected to the control unit 101 . The locator 101 receives radio waves from GPS satellites and obtains position (longitude, latitude, altitude) information. A video capture 103 is a camera (imaging unit) that captures an image of a subject and obtains image data of a background image. The video capture 103 obtains wide-viewing-angle image data and a plurality of image data for obtaining wide-viewing-angle image data. A format conversion processing unit 104 performs mapping processing (deformation of a wide viewing angle image, synthesis of a plurality of images, etc.) on the image data obtained by the video capture 103 to obtain image data in the image format of the encoder input.

The video encoder 105 performs encoding such as HEVC on the image data obtained by the format conversion processing unit 104 to obtain encoded image data, and generates a video stream including this encoded image data. In this case, the video encoder 105 places a video attribute information SEI message (Video_attribute_information SEI message) in the SEI message group of "SEIs" of the access unit (AU).

In this SEI message, capture information indicating the imaging state of the camera (imaging unit), position information (GPS data) indicating the position of the camera (imaging position), and information indicating the allowable composition range of the avatar in the background image are inserted. It is

FIG. 4 shows an example structure (Syntax) of a Video Attribute Information SEI message, and FIG. 5 shows the contents (Semantics) of main information in the example structure. An 8-bit field of "message_id" indicates identification information for identifying a video attribute information SEI message. An 8-bit field of "byte_length" indicates the number of subsequent bytes as the size of this video attribute information SEI message.

An 8-bit field of "target_content_id" indicates identification information of the video content. A field of “capture_position( )” indicates an imaging position. A 16-bit field of “position_latitude” indicates the imaging position (latitude). A 16-bit field of “position_longitude” indicates the imaging position (longitude). A 16-bit field of "position_elevation" indicates an imaging position (elevation).

A 16-bit field of "camera_direction" indicates the direction in which the camera faces at the time of imaging. For example, as shown in FIG. 6(a), directions such as north, south, east and west are shown as directions in which the camera faces. A 16-bit field of "camera_V_angle" indicates the angle from the horizontal of the camera at the time of imaging, as shown in FIG. 6(b).

The "sy_window()" field indicates the allowable composition range of the avatar in the background image. A 16-bit field of "sy_window_x_start" indicates the start position (horizontal position) of the allowable synthesis range. A 16-bit field of "sy_window_y_start" indicates the start position (vertical position) of the allowable synthesis range. A 16-bit field of "sy_window_x_end" indicates the end position (horizontal position) of the allowable synthesis range. A 16-bit field of "sy_window_y_end" indicates the end position (vertical position) of the allowable synthesis range.

Returning to FIG. 3, the audio capture 106 collects audio (sound) corresponding to the subject imaged by the video capture 103, and converts audio data into two or more channels, such as 5.1 channels. It is a microphone that obtains The audio encoder 108 encodes the audio data obtained by the audio capture 106 according to MPEG-H Audio, AC4, etc., to generate an audio data stream.

The container encoder 109 generates a container including the video stream obtained by the video encoder 105 and the audio stream obtained by the audio encoder 108, here an MP4 stream, as a server distribution stream.

In this case, the container encoder 109 defines a video attribute information box ““vaib” box” in the initialization segment (IS) and the “udta” box defined in the “moof” box.

Similar to the video attribute information SEI message, this video attribute information box contains capture information indicating the imaging state of the camera (imaging unit) and location information (GPS) indicating the position of the camera (imaging position). data) and information indicating the allowable composition range of the avatar in the background image are inserted. Note that it is not always necessary to insert both the video attribute information box and the video attribute information SEI message, and only one of them may be inserted.

FIG. 7 shows an example of information stored in the "vaib" box. "position_latitude" is the imaging position (latitude), "position_longitude" is the imaging position (longitude), and "position_elevation" is the imaging position (elevation). "camera_direction" indicates the direction in which the camera faces at the time of imaging, and indicates the direction from the north direction. Also, "camera_V_angle" indicates the angle from the horizontal direction of the camera at the time of imaging. Also, "sy_window_x_start" is the start position of the allowable synthesis range (horizontal position), "sy_window_y_start" is the start position of the allowable synthesis range (vertical position), "sy_window_x_end" is the end position of the allowable synthesis range (horizontal position), "sy_window_y_end" is the end position (vertical position) of the allowable synthesis range.

Returning to FIG. 3, network interface 110 communicates with client device 200 via network 300 . Network interface 110 transmits the server delivery stream obtained by container encoder 109 to client device 200 via network 300 .

The operation of the server 100 shown in FIG. 3 will be briefly described. The video capture 103 captures an image of a subject and obtains wide-viewing-angle image data and a plurality of image data for obtaining wide-viewing-angle image data. Image data obtained by the video capture 103 is supplied to the format conversion processing unit 104 . In the format conversion processing unit 104, the image data supplied from the video capture 103 is subjected to mapping processing (deformation of a wide viewing angle image, synthesis of a plurality of images, etc.) to obtain image data in the image format of the encoder input. .

The image data obtained by the format conversion processing unit 104 is supplied to the video encoder 105 . In the video encoder 105, encoding such as HEVC is performed on the image data from the format conversion processing unit 104 to obtain encoded image data, and a video stream including this encoded image data is generated.

Also, in the video encoder 105, a video attribute information SEI message (see FIG. 4) is arranged in the SEI message group of "SEIs" of the access unit (AU). In this SEI message, capture information indicating the imaging state of the camera (imaging unit), position information (GPS data) indicating the position of the camera (imaging position), and information indicating the allowable composition range of the avatar in the background image are inserted. It is

The audio capture 106 collects the audio (sound) corresponding to the subject imaged by the video capture 103, and obtains multi-channel audio data of two or more channels. Audio data for each channel is supplied to audio encoder 108 . The audio encoder 108 encodes the audio data obtained by the audio capture 106 according to MPEG-H Audio, AC4, etc., to generate an audio data stream.

The video stream obtained by the video encoder 105 and the audio stream obtained by the audio encoder 108 are supplied to the container decoder 109 . A container encoder 109 generates a container including a video stream and an audio stream, here an MP4 stream, as a server delivery stream.

In the container encoder 109, a new video attribute information box (see FIG. 7) is defined in the initialization segment (IS) and the "udta" box defined in the "moof" box. This box contains capture information indicating the imaging state of the camera (imaging unit), position information (GPS data) indicating the position of the camera (imaging position), and information indicating the permissible composition range of the avatar in the background image. ing.

The server-delivered stream obtained by container encoder 109 is supplied to network interface 110 . The network interface 110 transmits the server-delivered stream to the client device 200 via the network 300 .

[Configuration of client device]
A configuration example of the client device 200 will be described. FIG. 8 shows a configuration example of the transmission system 200T of the client device 200. As shown in FIG. This transmission system 200T includes a control unit 201, a metadata generator 202, an audio capture unit 203, an object information generation unit 204, an audio encoder 205, a character generation unit 206, a caption encoder 207, a container encoder 208, It has a network interface 209 . Each unit is connected by a bus 210 .

The control unit 201 controls the operation of each unit of the client device 200, and thus the transmission system 200T. A user operation unit 201 a is connected to the control unit 201 . The metadata generator 202 generates avatar meta information according to user operations from the user operation unit 201a. Avatar meta information consists of avatar rendering control information (avator_rendering_control_information) and avatar database selection information (avator_database_selection).

The avatar rendering control information includes information indicating the composition position of the avatar within the allowable composition range of the background image and information indicating the size of the avatar. FIG. 9(a) shows an example structure (Syntax) of avatar rendering control information, and FIG. 9(b) shows the contents (Semantics) of main information in the example structure.

An 8-bit field of "message_id" indicates identification information identifying avatar rendering control information. An 8-bit field of "byte_length" indicates the number of subsequent bytes as the size of this avatar rendering control information.

An 8-bit field of "client_id" indicates identification information of the client (client device 200) that transmits this avatar meta information. An 8-bit field of "target_content_id" indicates identification information of the video content (background image) to be synthesized. An 8-bit field of "number_of_client_objects" indicates the number of objects, ie avatars, sent from the client.

The fields "client_object_id", "avator_center_position_x", "avator_center_position_y", and "avator_rendering_size" are repeated as many times as there are objects. An 8-bit field of "client_object_id" indicates identification information of an object (avatar) transmitted from the client.

A 16-bit field of "avator_center_position_x" indicates the x-coordinate (horizontal position) of the center coordinate of the avatar synthesis position within the allowable synthesis range (sy_window). A 16-bit field of "avator_center_position_y" indicates the y-coordinate (vertical position) of the central coordinate of the avatar synthesis position within the allowable synthesis range. A 16-bit field of "avator_rendering_size" indicates the size of the avatar to be synthesized. Note that the size is obtained from the diagonal line of the rectangle from the center coordinates of the avatar synthesis position. While maintaining the original aspect ratio of the avatar image in the database, size conversion is performed according to the ratio with the size of the avatar to be synthesized.

The avatar database selection information includes selection information for obtaining avatar image data from the avatar database. FIG. 10(a) shows an example structure (Syntax) of avatar database selection information, and FIG. 10(b) shows the contents (Semantics) of main information in the example structure.

An 8-bit field of "message_id" indicates identification information for identifying avatar database selection information. An 8-bit field of "byte_length" indicates the number of subsequent bytes as the size of this avatar database selection information. An 8-bit field of "client_id" indicates identification information of the client (client device 200) that transmits this avatar database selection information. An 8-bit field of "target_content_id" indicates identification information of the video content (background image) to be synthesized.

An 8-bit field of "number_of_client_objects" indicates the number of objects, ie avatars, sent from the client. The fields "client_object_id", "body_type", "body_angle", "emotional_type", and "face_angle" are repeated as many times as there are objects. An 8-bit field of "client_object_id" indicates identification information of an object (avatar) transmitted from the client.

A 16-bit field of "body_type" indicates the type of body system of the avatar. A 16-bit field of "body_angle" indicates the attribute of the orientation from the front of the avatar image. A 16-bit field of "emotional_type" indicates the type of facial expression/emotion of the avatar. A 16-bit field of "face_angle" indicates the orientation of the avatar's face.

Returning to FIG. 8, the voice capture 203 is a microphone that collects the voice (sound) of each object, that is, each avatar, to obtain voice data. The object information generation unit 204 generates object metadata for each object based on the avatar synthesis position information, and outputs object encoded data (encoded sample data, object metadata) of each object.

FIG. 11(a) shows an example structure of voice object rendering information (Voice_object_rendering_information) as object metadata of each object (avatar), and FIG. 11(b) shows the contents (semantics) of main information in the example structure. ing. An 8-bit field of "message_id" indicates identification information identifying audio object rendering information. An 8-bit field of "byte_length" indicates the number of subsequent bytes as the size of this avatar database selection information. An 8-bit field of "client_id" indicates identification information of the client (client device 200) that transmits this audio data. An 8-bit field of "target_content_id" indicates identification information of the video content (background image) to be synthesized.

An 8-bit field of "number_of_client_objects" indicates the number of objects, ie avatars, sent from the client. The "client_object_id", "Azimuth", "Radius", and "Elevation" fields are repeated as many times as there are objects. An 8-bit field of "client_object_id" indicates identification information of an object (avatar) transmitted from the client.

A 16-bit field of "Azimuth" indicates an azimuth as positional information of an avatar as an object. A 16-bit field of "Radius" indicates a radius as positional information of an avatar as an object. A 16-bit field of "Elevation" indicates an elevation as positional information of an avatar as an object.

Here, with reference to FIG. 12, how to obtain the values of "Azimuth", "Radius", and "Elevation" will be described. A point P represents the central coordinates of the avatar synthesis position on the image developed by the HMD 400A. As described above, the avatar synthesis position is within the avatar's allowable synthesis range in the background image, and is specified by the avatar synthesis position information (“avator_center_position_x”, “avator_center_position_y”).

In this embodiment, the allowable composition range of the avatar in the background image is set so as to correspond to the image range developed by the HMD 400A. Thereby, the coordinates of the point P on the image developed by the HMD 400A are specified based on the combined position information of the avatar. Also, in this embodiment, the image range developed by the HMD 400A corresponds to the allowable composition range of the avatar in the background image in the default display state.

LT, LB, RT, and RB indicate virtual speakers on an assumed display monitor. A point Q indicates the center of an assumed viewing position. Let the distance from point Q to point P be r, the angle between QA and QB be θ, and the angle between QB and QP be φ. It is required as follows.
Azimuth = θ
Elevation = φ
Radius = r

As described above, by transmitting the audio object rendering information (see FIG. 11) including the values of "Azimuth", "Radius", and "Elevation", which are synthesis position information of the object (avatar), the receiving side can can be used as it is by inputting it into the renderer as object metadata.

On the receiving side, the coordinates of the point P can be specified by the avatar synthesis position information (“avator_center_position_x”, “avator_center_position_y”) included in the avatar rendering control information (see FIG. 9). It is also possible to obtain the values of 'Azimuth', 'Radius' and 'Elevation' from the center point Q (see FIG. 12) and input them to the renderer as object metadata.

In that case, it is not necessary to transmit the values of "Azimuth", "Radius", and "Elevation", which are the synthesis position information of each object (avatar), using the audio object rendering information (see FIG. 11). number_of_client_objects”=0.

Even in that case, by sending the value of "Radius", it is possible for the server 100 to convey to the client device 200 an appropriate depth position as a synthesis position of each object (avatar). In this case, when inserting the values of "Azimuth", "Radius", and "Elevation", which are synthesis position information of each object (avatar), into the audio object rendering information (see Fig. 11), "Azimuth", "Elevation" is set to an invalid value, for example.

In addition, even if the value of "Radius" is not sent, the client device 200 side adjusts the size of the object (avatar) based on the information of "avator_rendering_size" included in the avatar rendering control information (see FIG. 9). By adjusting the value of "Radius" obtained accordingly, it is possible to set the depth position of the synthesis position of each object (avatar) to an appropriate position.

Returning to FIG. 8, the audio encoder 205 encodes the encoded object data (encoded sample data, object metadata) of each object obtained by the object information generation unit 107 to generate MPEG-H 3D Audio. is obtained. This encoded audio data constitutes audio data corresponding to the avatar meta information.

The character input unit 206 appropriately generates caption text data (character codes) DT corresponding to each object, that is, each avatar, based on a user operation from the user operation unit 201a. The caption encoder 207 receives the text data DT and obtains text information of a caption (subtitle) in a predetermined format, TTML (Timed Text Markup Language) in this embodiment. This TTML constitutes caption data corresponding to avatar meta information.

FIG. 13(a) shows the TTML structure. TTML is described based on XML. TTML consists of a header (head) and a body (body). The header includes elements such as metadata, styling, and layout. The metadata includes metadata title information, copyright information, and the like. Styling includes information such as color (color), font (fontFamily), size (fontSize), and alignment (textAlign) in addition to the identifier (id). The layout includes information such as the extent (extent), offset (padding), background color (backgroundColor), alignment (displayAlign), etc., in addition to the identifier (id) of the region where the subtitle is placed. The body contains text information for subtitles, etc.

In this embodiment, the TTML is populated with subtitle object rendering information. FIG. 13(b) shows an example of the structure of metadata (TTM: TTML Metadata), in which information of "target_content_id", "client_id", and "client_object_id" exists. “target_content_id” indicates identification information of the video content (background image) to be synthesized. "client_id" indicates the identification information of the client (client device 200) that transmits this caption data. "client_object_id" indicates the identification information of the object (avatar) transmitted from the client. Information on the display position of the caption is included in the body.

Returning to FIG. 8, the container encoder 208 converts the avatar meta information generated by the metadata generator 202, the encoded 3D audio data obtained by the audio encoder 205, and the caption text information obtained by the caption encoder 207 into A container containing a certain TTML, here an MP4 stream, is generated as a client transmission stream.

A network interface 209 communicates with other client devices 200 via the network 300 . Network interface 209 transmits the client transmission stream obtained by container encoder 208 to other client devices 200 via network 300 .

The operation of the transmission system 200T shown in FIG. 8 will be briefly described. In the metadata generator 202, avatar meta information is generated according to the user operation from the user operation unit 201a. This avatar meta information consists of avatar rendering control information (see FIG. 9) and avatar database selection information (see FIG. 10). The avatar rendering control information includes information indicating the composition position of the avatar within the allowable composition range of the background image and information indicating the size of the avatar. The avatar database selection information also includes selection information for obtaining image data of avatars from the avatar database.

The audio capture 203 collects the audio (sound) of each object, ie, each avatar, to obtain audio data. The audio data of each object (avatar) is supplied to the object information generating section 204 . The object information generation unit 204 is also supplied with information on the position of each object (avatar) in the background image to be combined.

The object information generation unit 204 generates object metadata for each object based on the object synthesis position information, and obtains object encoded data (encoded sample data, object metadata) of each object. Here, audio object rendering information (see FIG. 11) is included as object metadata of each object (avatar). This audio object rendering information includes position information (θ, φ, r) of each object (avatar).

The encoded object data (encoded sample data, object metadata) of each object obtained by the object information generator 204 is supplied to the audio encoder 205 . The audio encoder 205 encodes the encoded object data of each object to obtain MPEG-H 3D Audio encoded audio data.

In the character input unit 206, caption text data (character code) DT corresponding to each object, that is, each avatar, is appropriately generated based on the user's operation from the user operation unit 201a. This text data DT is supplied to the caption encoder 207 . Display position information of subtitles corresponding to each object (avatar) is supplied to the subtitle encoder 207 .

The caption encoder 207 obtains TTML as text information of captions (subtitles) based on the text data DT. Rendering information is inserted into, for example, metadata of this TTML (see FIG. 13). Information on the display position of the caption is included in the head. The avatar rendering information may be included in the layout under the head together with parts other than the metadata, such as "origin" and "extent".

The avatar meta information generated by the metadata generator 202, the encoded 3D audio data obtained by the audio encoder 205, and the subtitle text information TTML obtained by the subtitle encoder 207 are supplied to the container encoder 208. . At the container encoder 208, an MP4 stream containing avatar meta information, encoded audio data and TTML is generated as the client transmission stream.

The client-transmitted stream resulting from container encoder 208 is provided to network interface 209 . Network interface 209 is responsible for transmitting client delivery streams to other client devices 200 via network 300 .

FIG. 14 shows a configuration example of the receiving system 200R of the client device 200. As shown in FIG. The receiving system 200R includes a control unit 201, a network interface 211, a container decoder 212, a video decoder 213, a plane converter 214, receiving modules 215 and 215A, an audio decoder 216, a mixer 218, and a synthesizing unit 219. have. Each unit is connected by a bus 210 .

The control unit 201 controls the operation of each unit of the client device 200, and thus the receiving system 200R. A user operation unit 201 a is connected to the control unit 201 . Network interface 211 communicates with server 100 and other client devices 200 via network 300 . Network interface 211 receives the above-described server-delivered stream from server 100 . Also, the network interface 211 receives the client transmission stream described above from another client device 200 .

Container decoder 212 extracts the video and audio streams from the server-delivered stream (MP4 stream) received at network interface 211 . In this case, the container decoder 212 extracts the video attribute information box ““vaib” box” existing in the “udta” box defined in the initialization segment (IS) or the “moof” box, and controls the Send to section 201 . Thereby, the control unit 201 recognizes capture information indicating the imaging state of the camera, position information (GPS data) indicating the position of the camera (imaging position), and information indicating the permissible composition range of the avatar in the background image.

The video decoder 213 decodes the video stream extracted by the container decoder 212 to obtain image data of the background image. The video decoder 213 also extracts parameter sets and SEI messages inserted in the video stream and sends them to the control unit 201 .

This extracted information also includes the video attribute information SEI message (see FIG. 4) described above. Thereby, the control unit 201 recognizes capture information indicating the imaging state of the camera, position information (GPS data) indicating the position of the camera (imaging position), and information indicating the permissible composition range of the avatar in the background image.

A plane converter 214 converts the image data of the background image obtained by the video decoder 213 into linear image data when the image data is non-linear image data. Further, when the image data of the background image is the image data of the wide viewing angle image, the plane converter 214 extracts only the portion corresponding to the display viewing angle of the HMD 400A from the image data to obtain display image data.

For example, the size of the allowable composition range of the avatar in the background image is set corresponding to the display viewing angle of the HMD 400A, and the plane converter 214 cuts out image data corresponding to this allowable composition range in the default state. This is image data for display. After that, the plane converter 214 changes the cutout range according to the head posture detected by the HMD-equipped sensor, for example.

The audio decoder 216 decodes the audio stream extracted by the container decoder 212 to obtain 2-channel audio data for audio reproduction on the headphone (HP) 400B. Note that when multi-channel audio data such as 5.1-channel audio data is obtained in the decoding process, the audio decoder 216 further downmixes the audio data to 2-channel audio data to obtain 2-channel audio data.

The receiving module 215 processes the client-transmitted stream received by the network interface 214 to process the image data of the avatar, the synthesized position information of the avatar, the display data of the subtitles corresponding to the avatar and the display position information of the subtitles, and further the avatar. Acquire corresponding two-channel audio data.

In addition, the receiving module 215A processes the client transmission stream generated by the transmission system 200T (see FIG. 8) of its own client device 200, and generates image data of the avatar, combined position information of the avatar, and subtitles corresponding to the avatar. Display data, display position information of the subtitles, and 2-channel audio data corresponding to the avatar are obtained. The receiving module 215A is provided for compositing your avatar on the background image. If the client device 200 itself does not have the transmission system 200T (see FIG. 8), the reception module 215A in the reception system 200R (see FIG. 14) becomes unnecessary.

FIG. 15 shows a configuration example of the receiving module 215 (215A). This receiving module 215 (215A) includes a container decoder 221, a meta information analysis section 222, an avatar database selection section 223, an avatar database 224, a size conversion section 225, an audio decoder 226, a renderer 227, and a subtitle decoder. 228 and a font development unit 229 .

The container decoder 221 extracts avatar meta information, 3D audio encoded voice data, and TTML, which is text information of subtitles, from the client transmission stream. A meta information analysis unit 222 analyzes the avatar meta information obtained by the container decoder 221 .

The meta-information analysis unit 222 acquires selection information for obtaining avatar image data from the avatar database 224 based on the avatar database selection information (see FIG. 10). This selection information consists of each information of the avatar's whole body system type "body_type", direction from the front "body_angle", expression/emotion type "emotional_type", and face direction "face_angle".

In addition, the meta-information analysis unit 222, based on the avatar rendering control information (see FIG. 9), avatar composition position information “avator_center_position_x” and “avator_center_position_y” within the allowable composition range of the background image, and size information “avator_center_position_y” of the avatar. avator_rendering_size”.

The avatar database selection unit 223 refers to the selection information acquired by the meta information analysis unit 222 and obtains image data of the avatar based on the avatar configuration data acquired from the avatar database 224 .

FIG. 16 shows a configuration example of the avatar database selection unit 223. As shown in FIG. The avatar database selection unit 223 has a database mapping unit 223a. Avatar body system type "body_type", direction from the front "body_angle", expression/emotion type "emotional_type", and face direction "face_angle" are input to the database mapping unit 223a. Avatar configuration data is obtained from the avatar database 224 and mapped to obtain avatar image data.

FIG. 17 shows an example list of avatar database 224 . For example, three states of "upright", "sitting", and "lying" are held as configuration data for the type "body_type" of the avatar's whole body system. Also, for example, six states of "forward", "backward", "rightward", "leftward", "upward", and "downward" are held as configuration data of the orientation "body_angle" from the front. Further, for example, four states of "expressionless", "laughing", "crying", and "angry" are held as the "emotional_type" configuration data of the type of facial expression/emotion. In addition, as configuration data of the face orientation “face_angle”, two states of “front straight view” and “downcast eyes” are held.

Returning to FIG. 15, the size conversion unit 225 performs size conversion processing on the avatar image data obtained by the avatar database selection unit 223 based on the size information obtained by the meta information analysis unit 222. Obtain the size-converted avatar image data.

The audio decoder 226 decodes the encoded audio data obtained by the container decoder 221 to obtain encoded sample data as object encoded data and object metadata (audio object rendering information). The renderer 227 renders the encoded sample data and object metadata obtained by the audio decoder 226, and obtains channel data for each speaker so that the avatar synthesis position in the background image becomes the sound image position.

FIG. 18 shows an outline of rendering processing in the renderer 227. FIG. In FIG. 18, parts corresponding to those in FIG. 12 are denoted by the same reference numerals. Avatar position information (θ, φ, r) included in the object metadata corresponds to point P, which is the center coordinates of the avatar's synthesis position on the image developed by HMD 400A.

In the client device 200, as described above, the coordinates of the point P can be specified by the avatar synthesis position information (“avator_center_position_x”, “avator_center_position_y”) included in the avatar rendering control information (see FIG. 9). It is also possible to obtain the values of "Azimuth", "Radius", and "Elevation" from the point Q, which is the center of the assumed viewing position, and use them in the renderer 227 (see FIG. 12).

In that case, as for the value of "Radius", the value of "Radius" inserted in the audio object rendering information (see FIG. 11) sent from the server 100 is used, or the avatar rendering control information (see FIG. 9) is used. ), the depth position of the composite position of the avatar can be adjusted appropriately according to the size of the object (avatar). position can be set.

This point P is on vectors r_LT, r_LB, r_RT, and r-RB on axes Q-LT, Q-LB, Q-RT, and Q-RB extending from point Q, which is the central viewing position, to each speaker position. Projected. The sound pressure level of the channel data of each speaker is assumed to correspond to the vector quantity of these four vectors.

Note that the example of FIG. 18 shows a case where the image developed on the HMD 400A is in the default state, that is, the image developed on the HMD 400A corresponds to the allowable composition range of the avatar in the background image. As described above, the extraction range in the plane converter 214 is changed according to the head posture detected by the HMD-equipped sensor.

In this case, the position of point P on the image developed on HMD 400A also changes, and it is assumed that the position of point P on the image developed on HMD 400A may deviate depending on the amount of change. In this case, the renderer 227 sets the sound pressure level of the channel data of each speaker based on the changed position of the point P, not the position of the point P obtained from the avatar position information (θ, φ, r). be.

In addition, the renderer 227 applies sound pressure control by remapping to the channel data of each speaker as described above, converts the data into two-channel audio data to be reproduced by the headphones 400B, and outputs the data. Note that when sound output on the client side is performed by the speakers LT, LB, RT, and RB instead of the headphones 400B, the sound pressure control by this remapping is omitted.

FIG. 19 schematically shows sound pressure control by remapping in the renderer 227. FIG. D_LT, D_LB, D_RT, and D_RB indicate channel data to be output to speakers LT, LB, RT, and RB, respectively, and "Left ear" and "Right ear" indicate 2-channel audio data for reproduction with headphones 400B. there is Here, in the sound pressure control by remapping, the transfer characteristics from each speaker to the left and right ears, the so-called Head Related Transfer Function (HRTF), are convoluted with each channel data and then summed up into two channels. Downmixing is done.

Returning to FIG. 15 , the caption decoder 228 obtains caption text data and control codes from the TTML obtained by the container decoder 221 . Display position information is also obtained as one of the control codes. The font expansion unit 229 expands fonts based on the subtitle text data and control code obtained by the subtitle decoder 228 to obtain subtitle display data (bitmap data).

The operation of the receiving module 215 (215A) shown in FIG. 15 will be briefly described. The client sent stream is provided to container decoder 221 . The container decoder 221 extracts avatar meta information, 3D audio encoded voice data, and TTML, which is text information of subtitles, from the client transmission stream.

The avatar meta information extracted by the container decoder 221 is supplied to the meta information analysis section 222 . The meta-information analysis unit 222 acquires selection information for obtaining avatar image data from the avatar database 224 based on the avatar database selection information (see FIG. 10). This selection information consists of each information of the avatar's whole body system type "body_type", direction from the front "body_angle", expression/emotion type "emotional_type", and face direction "face_angle".

In addition, based on the avatar rendering control information (see FIG. 9), the meta-information analysis unit 222 performs avatar composition position information “avator_center_position_x” and “avator_center_position_y” within the background image allowable composition range, and avatar size information “avator_center_position_x” and “avator_center_position_y”. avator_rendering_size” is obtained.

The selection information acquired by the meta information analysis section 222 is supplied to the avatar database selection section 223 . The avatar database selection unit 223 acquires avatar configuration data based on the avatar configuration data acquired from the avatar database 224 based on the selection information and performs mapping to obtain avatar image data.

The avatar image data obtained by the avatar database selection unit 223 is supplied to the size conversion unit 225 . Also, the avatar size information acquired by the meta-information analysis unit 222 is supplied to the size conversion unit 225 . The size conversion unit 225 performs size conversion processing on the avatar image data supplied from the avatar database selection unit 223 based on the size information, and obtains size-converted avatar image data. The avatar image data obtained by the size conversion unit 225 in this way is output from the reception module 215 (215A) together with the avatar synthesis position information obtained by the meta-information analysis unit 222 .

Also, the encoded audio data extracted by the container decoder 221 is supplied to the audio decoder 226 . The audio decoder 226 decodes the encoded audio data to obtain encoded sample data as object encoded data and object metadata (audio object rendering information). This object encoded data is supplied to renderer 227 .

The renderer 227 renders the encoded object data (encoded sample data and object metadata) obtained by the audio decoder 226 so that the synthesized position of the avatar in the background image becomes the sound image position. Channel data of virtual speakers arranged on the left, right, top and bottom of the image developed by the HMD 400A is generated (see FIG. 18).

Further, the renderer 227 performs sound pressure control by remapping the four channel data using the head-related transfer function (HRTF) to generate two-channel audio data to be reproduced by the headphones 400B (Fig. 19). The two-channel audio data obtained by the renderer 227 in this way is output from the reception module 215 (215A).

Also, the TTML extracted by the container decoder 221 is supplied to the caption decoder 228 . The caption decoder 228 obtains caption text data and control codes from the TTML. Display position information is also obtained as one of the control codes.

The subtitle text data and control code obtained by the subtitle decoder 228 are supplied to the font development unit 229 . The font developing unit 229 develops the font based on the text data of the caption and the control code to obtain caption display data (bitmap data). The subtitle display data obtained by the font developing unit 229 in this way is output from the reception module 215 (215A) together with the subtitle display position information obtained by the subtitle decoder 228 .

Returning to FIG. 14, the mixer 218 synthesizes the two-channel audio data obtained by the audio decoder 216 and the two-channel audio data obtained by the receiving modules 215 and 215A (see FIG. 15), Acquire 2-channel audio data to be sent to HP) 400B.

Under the control of the control unit 201, the synthesizing unit 219 combines the image data for display obtained by the plane converter 214 with the avatar image data obtained by the receiving modules 215 and 215A, based on the synthesizing position information. A display image that is synthesized so that the avatar is placed at a specific position within the avatar allowable synthesis range of the image, and that caption display data obtained by the reception modules 215 and 215A are synthesized based on the display position information and sent to the HMD 400A. get the data.

Note that the configuration example of the receiving system 200R shown in FIG. 14 shows an example in which the receiving module 215A processes the client transmission stream generated by the transmitting system 200T (see FIG. 8) of the client device 200 itself. However, instead of this receiving module 215A, a module (receiving module 215A shown in FIG. 15) that processes avatar meta information, encoded audio data, and TTML generated by the transmitting system 200T (see FIG. 8) of its own client device 200 configuration excluding the container decoder 221), or a module that inputs avatar meta information, encoded audio data, other data and information corresponding to TTML and obtains the same output.

The operation of the receiving system 200R shown in FIG. 14 will be briefly described. The network interface 211 receives the server distribution stream from the server 100 via the network 300 . Also, the network interface 211 receives a client transmission stream from another client device 200 via the network 300 .

A server-delivered stream received at network interface 211 is provided to container decoder 212 . The container decoder 212 extracts a video stream and an audio stream from the server delivery stream (MP4 stream).

In the container decoder 212 , the video attribute information box existing in the initialization segment (IS) and the “udta” box defined in the “moof” box is extracted and sent to the control unit 201 . As a result, the control unit 201 recognizes the capture information indicating the imaging state of the camera, the position information (GPS data) indicating the position of the camera (imaging position), and the information indicating the allowable composition range of the avatar in the background image. .

Also, the video stream extracted by the container decoder 212 is supplied to the video decoder 213 . The video decoder 213 decodes the video stream to obtain image data of the background image.

Also, the video decoder 213 extracts the parameter set and SEI message inserted in the video stream and sends them to the control unit 201 . This extracted information also includes a video attribute information SEI message (see FIG. 4). As a result, the control unit 201 recognizes the capture information indicating the imaging state of the camera, the position information (GPS data) indicating the position of the camera (imaging position), and the information indicating the allowable composition range of the avatar in the background image. .

Image data of the background image obtained by the video decoder 213 is supplied to the plane converter 214 . In the plane converter 214, if the image data of the background image is non-linear image data, it is converted into linear image data. Also, in the plane converter 214, only a portion corresponding to the display viewing angle of the HMD 400A is cut out from the image data of the background image to obtain image data for display.

For example, the size of the permissible composition range of the avatar in the background image is set corresponding to the display viewing angle of the HMD 400A, and in the default state, the image data corresponding to this permissible composition range is extracted and displayed as image data. It is said that After that, the cropping range is changed according to the head posture detected by a sensor mounted on the HMD, for example.

Also, the audio stream extracted by the container decoder 212 is supplied to the audio decoder 216 . The audio decoder 216 decodes the audio stream to obtain 2-channel audio data for audio reproduction on the headphone (HP) 400B. Note that when multi-channel audio data such as 5.1-channel audio data is obtained in the decoding process, the audio decoder 216 further down-mixes the audio data into 2-channel audio data to obtain 2-channel audio data.

Also, client transmission streams from other client devices 200 received by the network interface 211 are supplied to the reception module 215 . In this receiving module 215, the client transmission stream is processed, and the image data of the avatar, the combined position information of the avatar, the display data of the caption corresponding to the avatar and the display position information of the caption, and the two-channel audio corresponding to the avatar are displayed. Data are obtained (see Figure 15).

Also, the client transmission stream generated by the transmission system 200T (see FIG. 8) of its own client device 200 is supplied to the reception module 215A. In this receiving module 215A, similarly to the receiving module 215, the client transmission stream is processed, and the image data of the avatar and the combined position information of the avatar, the display data of the caption corresponding to the avatar and the display position information of the caption, and further the avatar 2-channel audio data corresponding to are obtained (see FIG. 15).

The two-channel audio data obtained by the audio decoder 216 are supplied to the mixer 218 . Also, the mixer 218 is supplied with two-channel audio data obtained by the receiving modules 215 and 215A. In the mixer 218, the two-channel audio data obtained by the audio decoder 216 and the two-channel audio data obtained by the receiving modules 215 and 215A are combined to produce two-channel audio data to be sent to the headphone (HP) 400B. can get.

Also, the display image data obtained by the plane converter 214 is supplied to the synthesizing unit 219 . Also, the synthesizing unit 219 is supplied with avatar image data and avatar synthesizing position information obtained by the receiving modules 215 and 215A, caption display data and display position information. In the synthesizing unit 219, the avatar image data obtained by the receiving modules 215 and 215A are added to the display image data obtained by the plane converter 214, and the avatar image data obtained by the receiving modules 215 and 215A are specified based on the synthesizing position information within the avatar allowable synthesizing range of the background image. The avatars are synthesized so that the avatar is placed at the position, and the subtitle display data obtained by the reception modules 215 and 215A are synthesized based on the display position information to obtain the display image data to be sent to the HMD 400A.

FIG. 20 shows an example of a background image, and a rectangular dashed frame indicates the allowable composition range (sy_window) of the avatar. The center of this background image (indicated by the character "+") is the video attribute information SEI message (see Fig. 4) or the "camera_direction", " The position corresponds to the information of "camera_V_angle".

FIG. 21 shows an example of a state in which an avatar and subtitles are synthesized within the permissible synthesis range (sy_window) of the background image. In the illustrated example, three avatars A1, A2, and A3 are synthesized, and two subtitles are synthesized. Here, the avatar of A1 and the subtitles associated with it are from the client (client device 200) whose "clinent_id" is "0xA1". Also, the avatar of A2 belongs to a client whose "clinent_id" is "0xA2". Also, the avatar of A3 and the caption associated with it are those of the client (client device 200) whose "clinent_id" is "0xA3".

As described above, in the shared space display system 10 shown in FIG. 1, the client device 200 generates avatar image data based on the avatar meta information in the image data of the background image, and uses the avatar image data as the background image data. It is synthesized with the image data of the image. Therefore, each of the clients can recognize the avatar of the other client synthesized with the common background image, and can communicate well by sharing the VR space with each other.

In the shared space display system 10 shown in FIG. 1, the client transmission stream includes audio data corresponding to avatar meta information together with object metadata. Rendering processing can be performed to obtain audio output data having the avatar synthesis position as the sound image position. Therefore, it is possible for each client to perceive that the voice of the avatar's client is coming from the composite position of each avatar on the background image.

In the space sharing display system 10 shown in FIG. 1, the client transmission stream includes caption data corresponding to the avatar meta information together with the display position information. The display data of the caption can be combined with the image data of the background image based on the display position information so that the caption is displayed at a position corresponding to the position. Therefore, it is possible for each client to recognize the caption from the client of the avatar at the position corresponding to the composite position of each avatar on the background image.

In the spatially shared display system 10 shown in FIG. 1, the permissible synthesis of the avatar in the background image is performed on the layer of the video stream obtained by encoding the image data of the background image and/or the layer of the server distribution stream including the video stream. Information indicating the range is inserted and distributed. Therefore, the client device 200 can easily arrange each client's avatar in the background image in the range intended by the server 100 based on the information indicating the allowable composition range.

<2. Variation>
In the above-described embodiment, an example in which the client device 200 exists separately from the HMD 400A is shown, but an example in which the HMD 400A and the client device 200 are integrally configured is also conceivable. Also, although not described above, it is also possible to use a photographed image as an avatar.

Also, in the above-described embodiment, an example in which the container is MP4 (ISOBMFF) has been shown. However, the present technology is not limited to MP4 containers, and can be similarly applied to containers of other formats such as MPEG-2 TS and MMT.

Moreover, this technique can also take the following structures.
(1) Receiving a server-delivered stream containing a video stream obtained by encoding image data of a background image from a server, and providing substitute image meta information for displaying a substitute image of the other client from another client device. a receiver for receiving a client-transmitted stream containing
decoding processing for decoding the video stream to obtain image data of a background image; substitute image data generation processing for generating image data for a substitute image based on the substitute image meta information; A client device comprising a control unit for controlling image data synthesis processing for synthesizing image data of an image.
(2) information indicating an allowable composition range of the substitute image in the background image is inserted into the layer of the video stream and/or the layer of the server-delivered stream;
The client device according to (1), wherein the control unit controls the composition processing based on information indicating the allowable composition range so that the substitute image is arranged within the allowable composition range of the background image.
(3) the substitute image meta information includes composition position information indicating a composition position of the substitute image within the allowable composition range;
The client device according to (2), wherein the control unit controls the combining process such that the substitute image is combined at the combining position indicated by the combining position information.
(4) the substitute image meta information includes size information indicating the size of the substitute image;
The client apparatus according to (2) or (3), wherein the control unit controls the synthesis process such that the substitute image is synthesized with the background image in a size indicated by the size information.
(5) the client transmission stream includes audio data corresponding to the substitute image meta information together with object meta data;
The control unit performs rendering processing on the audio data according to the object metadata, and further controls audio output processing for obtaining audio output data having the synthetic position of the substitute image as a sound image position. client device.
(6) the client transmission stream includes caption data corresponding to the substitute image meta information together with display position information;
The control unit synthesizes the display data of the caption with the image data of the background image based on the display position information so that the caption based on the caption data is displayed at a position corresponding to the synthesis position of the substitute image. The client device according to (3) or (5), further controlling processing.
(7) further comprising a transmission unit that transmits a client transmission stream including substitute image meta information for displaying the substitute image of itself to another client device;
The substitute image data generation process further generates image data of the substitute image based on substitute image meta information for displaying the substitute image. client device.
(8) the image data of the background image is image data of a wide viewing angle image;
The client device according to any one of (1) to (7), wherein the control unit further controls image clipping processing for clipping a portion of the image data of the background image to obtain display image data.
(9) A receiving unit receives a server-delivered stream including a video stream obtained by encoding image data of a background image from a server, and substitutes for displaying a substitute image of the other client from another client device. a receiving step for receiving a client-transmitted stream containing image meta information;
A control unit decodes the video stream to obtain image data of a background image, a substitute image data generation process of generating image data of a substitute image based on the meta information of the substitute image, and an image of the background image. A processing method for a client device, comprising a control step of controlling image data synthesis processing for synthesizing image data of the substitute image with data.
(10) an imaging unit that captures an image of a subject and obtains image data of a background image;
a transmitting unit configured to transmit a server-delivered stream including a video stream obtained by encoding image data of the background image to a client device;
A server in which information indicating an allowable composition range of a substitute image for the background image is inserted into the layer of the video stream and/or the layer of the server-delivered stream.
(11) The server according to (10), wherein the image data of the background image is image data of a wide viewing angle image.
(12) an image capturing step in which the image capturing unit captures an image of a subject to obtain image data of a background image;
a transmitting step of transmitting a server-delivered stream including a video stream obtained by encoding image data of the background image to a client device;
A server processing method in which information indicating an allowable composition range of a substitute image for the background image is inserted into the video stream layer and/or the server delivery stream layer.

The main feature of this technology is that client transmission streams from other client devices contain avatar meta information, and the image data of the background image is combined with the avatar image data generated based on this avatar meta information. By doing so, each client can recognize the other client's avatar combined with a common background image, and it is possible to share each other's VR space and communicate well ( 2 and 21).

DESCRIPTION OF SYMBOLS 10... Space sharing display system 100... Server 101... Control part 101a... User operation part 102... Locator 103... Video capture 104... Format conversion part 105... Video encoder 106... Audio capture 108... Audio encoder 109... Container encoder 110... Network interface 111... Bus 200... Client device 200T... Transmission system 200R... Reception system 201... Control unit 201a User operation unit 202 Metadata generator 203 Audio capture 204 Object information generation unit 205 Audio encoder 206 Character generation unit 207 Subtitle encoder 208... Container encoder 209... Network interface 210... Bus 211... Network interface 212... Container decoder 213... Video decoder 214... Plane converter 215, 215A... Receiving module 216 Audio decoder 218 Mixer 219 Synthesis unit 221 Container encoder 222 Meta information analysis unit 223 Avatar database selection unit 223a Database mapping unit 224 Avatar Database 225 Size conversion unit 226 Audio decoder 227 Renderer 228 Subtitle decoder 229 Font development unit 300 Network 400A Head mount display (HMD)
400B Headphones (HP)

Claims (19)

Receiving a server-delivered stream containing a video stream obtained by encoding image data of a background image from a server, and a client-transmitted stream containing substitute image meta information for displaying a substitute image of another client from another client device a receiver for receiving
Decoding processing for decoding the video stream to obtain image data of a background image, substitute image data generation processing for generating image data for the substitute image based on the meta information for the substitute image, and image data for the background image. a controller for controlling image data synthesizing processing for synthesizing the image data of the substitute image, and image data output processing for outputting display image data based on the image generated by the image data synthesizing processing to the display unit ;
client device. The client device according to claim 1 , wherein the background image includes an image captured by a server . the client-transmitted stream includes text information corresponding to the substitute image meta information;
3. The client device according to claim 1 , wherein the control unit controls the image data synthesis process such that the text corresponding to the text information is synthesized at a position corresponding to the image data of the substitute image. . The substitute image meta information includes size information indicating the size of the substitute image,
4. The image data synthesizing process according to any one of claims 1 to 3, wherein the control unit controls the image data synthesizing process so that the substitute image is synthesized with the background image in a size indicated by the size information. client device. The image data of the background image is image data of a wide viewing angle image,
5. The client device according to any one of claims 1 to 4 , wherein the control unit further controls image clipping processing for clipping a portion of the image data of the background image to obtain display image data. Connected to an external device having the display unit
A client device according to any one of claims 1 to 5. further comprising the display unit
A client device according to any one of claims 1 to 5. The display unit is a head-mounted display.
A client device according to any one of claims 1 to 7. a client device according to any one of claims 1 to 8;
including the server;
The server is
an imaging unit that captures an image of a subject and obtains image data of the background image;
a transmitting unit configured to transmit a server-delivered stream including a video stream obtained by encoding image data of the background image to the client device;
display system . a client device according to claim 6;
including the external device;
The external device includes a display unit that displays the display image data.
display system. A receiving unit receives a server-delivered stream including a video stream obtained by encoding image data of a background image from a server, and receives substitute image meta information for displaying a substitute image of another client from another client device. a receiving step for receiving a client-transmitted stream containing
A control unit decodes the video stream to obtain image data of a background image, a substitute image data generation process of generating image data of a substitute image based on the substitute image meta information, and the background. Control for controlling image data synthesis processing for synthesizing the image data of the substitute image with the image data of the image, and image data output processing for outputting display image data based on the image generated by the image data synthesis processing to the display unit. A client device processing method comprising steps. The background image includes a captured image in the server
The processing method for a client device according to claim 11. the client-transmitted stream includes text information corresponding to the substitute image meta information;
In the control step, the control unit controls the image data synthesizing process so that the text corresponding to the text information is synthesized at a position corresponding to the image data of the substitute image.
13. The processing method for a client device according to claim 11 or 12. The substitute image meta information includes size information indicating the size of the substitute image,
In the control step, the control unit controls the image data synthesizing process so that the substitute image is synthesized with the background image in the size indicated by the size information.
A processing method for a client device according to any one of claims 11 to 13. The image data of the background image is image data of a wide viewing angle image,
In the control step, the control unit further controls image clipping processing for clipping a portion of the image data of the background image to obtain display image data.
A processing method for a client device according to any one of claims 11 to 14. The client device is connected to an external device having the display unit.
A processing method for a client device according to any one of claims 11 to 15. The client device includes the display unit
A processing method for a client device according to any one of claims 11 to 15. The display unit is a head-mounted display.
A processing method for a client device according to any one of claims 11 to 17. A program for causing a computer to perform the method according to any one of claims 11 to 18.

Images