JP7457525B2 - Receiving device, content transmission system, and program - Google Patents

Description

The present invention relates to a receiving device, a content transmission system, and a program that reproduce three-dimensional sound.

In recent years, with advances in AR (Augmented Reality)/VR (Virtual Reality) technology, AR/VR compatible terminals and AR/VR content have begun to spread. AR/VR compatible terminals include smartphones, tablet terminals, VR goggles, and AR glasses. For example, Patent Document 1 discloses a configuration example of an AR content viewing system.

When reproducing AR/VR content, visual information is rendered in real time using a GPU (Graphic Processing Unit) installed in the terminal, thereby realizing interactive graphical display according to the user's actions. An AR/VR compatible terminal is equipped with a plurality of sensors such as a gyro sensor and an acceleration sensor, and the self-position and orientation of the terminal are estimated using information acquired from these sensors. In the case of AR content that combines and displays objects given by CG data, etc. on images of real space captured by the device's camera, the images of real space captured by the camera are also subject to self-position/orientation estimation processing. It can be used as input information. Real-time graphic rendering processing is performed using the GPU in accordance with a viewport that reflects the viewpoint position and line-of-sight direction obtained as a result of this self-position/orientation estimation processing. The basic technologies necessary for playing AR/VR content, such as self-position and posture estimation processing on smartphones and tablet devices, are standard functions at the OS (Operating System) level, such as ARKit on iOS devices and ARCore on Android devices. Implementation is progressing, making it easier for general developers to develop and distribute AR/VR compatible applications.

The degree of freedom of the user's viewpoint position and line of sight direction when viewing AR/VR content is expressed in units called DoF (Degrees of Freedom). For example, in 360-degree VR video content, if the viewpoint position is fixed and there is interactivity only in the direction of the line of sight, the degree of freedom in the direction of the user's line of sight is rotation with three degrees of freedom (Role, Pitch, Yaw), so 3DoF It is called. On the other hand, in the case of AR/VR content where the viewpoint position can be freely moved in addition to the viewing direction, in addition to the 3 degrees of freedom in the viewing direction, there are also 3 degrees of freedom in moving the viewpoint position (X, Y, Z). ), so there are 6 degrees of freedom in total, so it is called 6DoF. In addition, while based on 3DoF, a system that adds a small degree of freedom to visual information by moving the viewpoint position within a limited range, such as moving the head while sitting on a fixed chair, is called 3DoF+. There are cases.

Multimodal stimulation that combines visual and auditory information can be expected to enhance the user's sense of immersion in the content when viewing AR/VR content. For example, Patent Document 2 discloses that in game content in which a user can freely move around in a virtual space, an application is applied depending on the relationship between the area where environmental sounds are generated and the position and direction of a virtual camera corresponding to the user's viewpoint. An example of the configuration of a system that generates environmental sounds is disclosed in . Further, Patent Document 3 discloses a model of a system that generates and presents a monaural sound or a stereo sound in which the sound of a sound source object is mixed according to the user's line of sight direction in a VR game.

In the techniques disclosed in the above-mentioned prior art documents, either the sound source object is assumed to be at the same height as the user's viewpoint position, or the sound source object is actually located at a different height from the user's viewpoint position. It is considered to be at a height. In other words, sounds from sound source objects and environmental sound generation areas located above and below the height of the viewpoint are localized on a two-dimensional plane horizontal to the ground at the height of the user's line of sight. Therefore, for example, when the user is facing straight ahead, even if the sound of an airplane flying overhead or the chirping of an insect near the ground near the feet of the user is presented, it is not possible to obtain a three-dimensional effect in the vertical direction.

A technology has been proposed that uses frequency domain acoustic processing using the frequency characteristics (head-related transfer function) of the transfer function of the sound wave from the position of the sound source object to the entrance of the user's ear canal to achieve three-dimensional sound image localization including directions above and below the height of the viewpoint, even if the final audio output is the same two-channel audio as stereo. In this way, three-dimensional audio by two-channel audio playback is generally called binaural audio. In addition to generating binaural audio by frequency domain calculation using the head-related transfer function, it can also be collected directly from the real space using a dummy head that mimics the shape of a human head and the ear canal. For this reason, in live-action 360-degree VR video content, wide-field shooting such as panoramic video using a 360-degree camera and binaural audio collection using a dummy head are sometimes performed simultaneously, and the content is provided as a packaged VR video content. However, the user can correctly obtain the stereoscopic effect of binaural audio only when the direction in which the dummy head was facing at the time of audio collection matches the user's line of sight.

Japanese Patent Application Publication No. 2019-008319 Japanese Patent Application Publication No. 2007-229241 JP2019-013765A

In 3DoF VR content where the user can freely change the line of sight, and 6DoF AR/VR content where the user can freely move within the 3D space of the content, the audio collected by the dummy head is used as is. I can't. That is, it is necessary to generate binaural audio in real time by performing calculations using head-related transfer functions in accordance with the relative relationship between the user's viewpoint position and the position of the sound source object, which changes from moment to moment. In order to achieve this, it is first necessary to transmit an independent audio stream for each sound source object and three-dimensional coordinates indicating the position of the content in three-dimensional space in a time-based manner. Next, in the receiving device, it is necessary to implement a function that receives data in which the audio stream of the sound source object and the three-dimensional coordinates are linked, and generates binaural audio in real time according to the sound source object and the user's viewpoint position. There is.

However, common mobile devices such as tablets and smartphones are not equipped with a DSP (Digital Signal Processor), which is dedicated hardware for audio processing that is equivalent to a GPU for graphic processing, and The lack of sufficient API (Application Programming Interface) for sound processing, which corresponds to OpenGL (Open Graphics Library) for processing, is an obstacle to implementing functions that support three-dimensional sound. In order to provide content to mobile terminals such as smartphones and tablet terminals, which are popular among a wide range of users, it is necessary to implement three-dimensional sound processing using software executed on a CPU (Central Processing Unit).

Furthermore, when the number of sound source objects streamed as components of content increases, the processing load on the receiving device increases in accordance with the number of sound sources, and an excessive load may occur. If an excessive processing load occurs due to the increase in the number of sound source objects, processing delays may cause out-of-synchronization of visual and auditory information, and sound skips may occur.

In other words, when providing AR/VR content that combines three-dimensional sound through streaming transmission, the movement of the sound source object in three-dimensional space, the movement of the user's viewpoint position (viewing position), and the rotation of the user's line of sight are dynamic. In this case, conventional binaural audio generation processing using head-related transfer functions requires a huge number of head-related transfer functions depending on the relative relationship between the sound source position and the listening position/direction, which requires a large number of memory and computational resources. The problem was that it was impractical to implement software on mobile terminals, which have limited capabilities. In addition, when the number of sound source objects placed in the three-dimensional space of the content increases, due to the limitations of computational resources such as CPU and memory, it is not possible to process the sounds of all sound source objects in real time, resulting in delays. This has caused issues such as deterioration in viewing quality, such as loss of synchronization with visual information and skipping of sound.

The purpose of the present invention, which was made in view of the above circumstances, is to suppress increases in the amount of calculations, program scale, and circuit scale, ensure real-time playback of AR/VR content that combines three-dimensional sound, and improve viewing quality. The purpose of the present invention is to provide a receiving device, a content transmission system, and a program capable of transmitting data at a reasonable implementation cost.

A receiving device according to an embodiment is a receiving device that receives audio chunks obtained by dividing an audio stream of a sound source object into blocks, and sound source metadata including three-dimensional coordinates of the sound source object in a world coordinate system, A coordinate conversion unit that converts the three-dimensional coordinates of the sound source object from the world coordinate system to the view coordinate system; and a sound source object selection area is defined in the view coordinate system based on the processing load of the receiving device, and the sound source object is selected. An object selection unit that selects a sound source object located within a region as a sound source object to be processed, and a three-dimensional sound rendering unit that generates binaural audio using audio chunks of the sound source object to be processed. .
In the receiving device according to one embodiment, in the above-described configuration, the object selection section defines that the sound source object selection area becomes smaller as the processing load becomes larger .
Further, in the receiving device according to one embodiment, in the above configuration, the sound source metadata includes a priority of the sound source object, and the object selection unit is configured to determine the distance from the origin of the sound source object in the view coordinate system. A sound source object that will be included in the sound source object selection area when the above priority is changed so that it becomes shorter as the priority increases is selected as the sound source object to be subjected to the sound processing.
Further, in the receiving device according to one embodiment, in the above configuration, the sound source metadata includes a maximum sound pressure level of the sound source object, and the object selection unit is configured to A sound source object that will be included in the sound source object selection area when the distance is changed such that it becomes shorter as the maximum sound pressure level increases is selected as the sound source object to be subjected to the sound processing.

Furthermore, in one embodiment, the position and orientation estimating unit estimates the direction of the user's line of sight, and the coordinate conversion unit converts the three-dimensional coordinates of the sound source object from the world coordinate system to the direction of the line of sight as an axial direction. The view coordinate system may be converted to the view coordinate system.

Furthermore, in one embodiment, the position and orientation estimation unit estimates a viewpoint position of the user, and the coordinate conversion unit converts the three-dimensional coordinates of the sound source object from the world coordinate system, with the viewpoint position as the origin. The view coordinate system may be converted to the view coordinate system.

Furthermore, in one embodiment, the three-dimensional sound rendering unit includes a mapping unit that allocates audio chunks of the sound source object to be processed for sound to a virtual multi-channel speaker arranged in the view coordinate system; and a downmix unit that generates the binaural audio using the audio chunks assigned to the audio chunks.

Further, a content transmission system according to an embodiment includes the receiving device, and a distribution device that associates the audio chunk with the sound source metadata and transmits the same to the receiving device.

Further, the program according to one embodiment causes a computer to function as the receiving device.

According to the present invention, in a receiving device that receives streaming transmission of AR/VR content, it is possible to suppress an increase in the amount of calculation and circuit scale, ensure real-time performance of content playback, and improve viewing quality. becomes. Furthermore, it is possible to use terminals of various performance with different CPU clocks and memory capacities as receiving devices, and to realize a service that can reproduce content according to the processing performance of each terminal.

FIG. 3 is a diagram showing an example of the viewing position of the receiving device and the arrangement of a plurality of sound source objects according to the first embodiment. FIG. 1 is a block diagram illustrating a configuration example of an AR content transmission system according to a first embodiment. FIG. 3 is a diagram showing blocking of an audio stream and a sound source metasequence according to the first embodiment. FIG. 2 is a block diagram illustrating a configuration example of a three-dimensional sound rendering unit according to the first embodiment. Example of world coordinate system (view from directly above) of 3D space content and placement of sound source objects FIG. 3 is a diagram illustrating an example of a view coordinate system (viewpoint from directly above) centered on an AR viewpoint. FIG. 3 is a diagram illustrating an example of a view coordinate system (viewpoint from directly above) arranged in a world coordinate system. FIG. 3 is a diagram illustrating an example of a view coordinate system (viewpoint from the side) arranged in a world coordinate system. FIG. 3 is a diagram illustrating processing of a three-dimensional sound rendering unit according to the first embodiment. FIG. 3 is a diagram illustrating processing of a three-dimensional sound rendering unit according to the first embodiment. A block diagram showing an example of the configuration of a 360-degree VR video content transmission system according to a second embodiment.

Embodiments of the present invention will be described in detail below with reference to the drawings.

(First embodiment)
In the first embodiment, an AR content transmission system that transmits AR content with a degree of freedom of 6 DoF will be described.

FIG. 1 is a diagram illustrating an example of a tablet-type receiving device 10 that receives multiple sound source objects arranged in a three-dimensional space and AR content. In addition, in the notation in the figure, the three-dimensional space is expressed as a right-handed Y-up coordinate system, but the coordinate system in each implementation is not limited to this. In the example shown in FIG. 1, the sound source object O ₁ is placed at (X ₁ , Y ₁ , Z ₁ ) in the three-dimensional space, the sound source object O ₂ is placed at (X ₂ , Y ₂ , Z ₂ ), Sound source object O ₃ is placed at (X ₃ , Y ₃ , Z ₃ ), sound source object O ₄ is placed at (X ₄ , Y ₄ , Z ₄ ), and sound source object O ₅ is placed at (X ₅ , Y ₅ , _Z5 ). The user can freely move within the space while holding the receiving device 10. The receiving device 10 generates binaural audio by processing the audio stream of the sound source object according to the user's viewpoint position and line of sight direction. The user listens to the audio through speakers (SP(L) and SP(R)) included in the receiving device 10, external stereo headphones, or the like.

FIG. 2 is a diagram illustrating a configuration example of an AR content transmission system according to the first embodiment. The AR content transmission system 1 includes a receiving device (AR receiving device) 10 and a distribution device 40. A time server (time server) 50 is provided to synchronize the receiving device 10 and the distribution device 40. In the example shown in FIG. 2, there is one time server 50, but the receiving device 10 and the distribution device 40 may refer to different time servers. The time server 50 may be provided on the Internet.

The distribution device 40 streams and transmits AR content via a transmission path 60 such as broadcasting or the Internet. The distribution device 40 associates the audio streams of multiple sound source objects arranged in the three-dimensional space of the AR content with sound source metadata including position information (three-dimensional coordinates in the world coordinate system) of the sound source objects, and transmits the audio streams to the receiving device. Send to 10. In the example shown in FIG. 2, the distribution device 40 includes a clock generation section 41 and a multiplexing section 42.

The clock generation section 41 generates a synchronous clock synchronized with the time input from the time server 50 and outputs it to the multiplexing section 42 .

The multiplexing unit 42 multiplexes a 3D model sequence (a model sequence of a 3D object), an audio stream (a sequence of audio chunks), and a sound source metasequence (a sequence of sound source metadata) to generate AR content and distribute it. It is transmitted to the outside of the device 40. For example, when MMT (MPEG Media Transport) is used as the multiplexing method of the multiplexer 42, audio chunks can be associated with MPUs (Media Processing Units). Further, the clock generation unit 41 outputs a presentation time stamp PTS (Presentation Time Stamp) based on UTC (Coordinated Universal Time), which is an absolute time, to the multiplexing unit 42, and the multiplexing unit 42 adds the PTS to each data. .

A three-dimensional model sequence is data in which, for example, geometry data representing the shape of a three-dimensional object and texture data representing the surface pattern of the three-dimensional object are sequenced at a constant frame rate and transmitted in real time.

A sound source metasequence is data obtained by sampling at a constant frame rate position coordinate information that indicates the position of a part of a 3D object that is a sound source (sound source object) in coordinates of the world coordinate system, and is processed in real time. Sent in For example, the point of the sound source is usually assumed to be the center of the mouth where the voice is emitted in the case of a human object, or the center point of the sound hole of a stringed instrument or the center of the striking surface of a percussion instrument in the case of a musical instrument. , which part is used as the sound generation source differs depending on the content. For example, in the case of content about a person tap dancing, the sole of the person object's shoe becomes the audio source. Furthermore, when two or more sound source objects are visually associated with one three-dimensional object, such as in the case of a person object who tap dances while singing (the center point of the mouth and the sole of the shoe are the two sound sources). There is also.

The audio stream is stream data of audio emitted from an audio source (sound source object) of a three-dimensional object, and is transmitted in real time.

FIG. 3 is a diagram illustrating blocking of an audio stream and a sound source metasequence. In order to link the audio stream and the audio source metasequence, the audio stream and the audio source metasequence are generally divided into blocks at regular intervals. Hereinafter, a block of an audio stream will be referred to as an "audio chunk" and a block of a sound source metasequence will be referred to as "sound source metadata." As control information when multiplexing various data, a sound source object ID (object_id) is assigned to each sound source object. In addition, a chronological sequence number (sequence_num) is assigned to the audio chunk and sound source metadata in order to correlate them on the time axis. Sound source metadata and audio chunks are linked by a sound source object ID and a sequence number. In addition to coordinates, the sound source metasequence may also include a priority, a maximum sound pressure level (maximum_level), etc., which will be described later.

The receiving device 10 is a mobile terminal such as a smartphone or a tablet terminal, AR glasses, video see-through type AR goggles, or the like. The receiving device 10 receives, from the distribution device 40, audio chunks in which the audio stream of the sound source object is divided into blocks, and sound source metadata including position information of the sound source object.

In the example shown in FIG. 2, the receiving device 10 includes a clock generator 11, a demultiplexer 12, a first buffer 13, a second buffer 14, a model decoder 15, a camera 16, and a frame memory 17. , the detection unit 18, the position and orientation estimation unit 19, the model rendering unit 20, the video synthesis unit 21, the display 22, the coordinate conversion unit 23, the processing load measurement unit 24, the object selection unit 25, and the audio decoding unit. 26, a three-dimensional sound rendering section 27, and a speaker 28.

The clock generation unit 11 generates a synchronized clock that is synchronized with the time input from the time server 50 and outputs it to the first buffer 13 and the second buffer 14.

The demultiplexer 12 receives AR content, in which a 3D model sequence, an audio stream, and a sound source metasequence are multiplexed, from the distribution device 40 via a transmission path 60 such as broadcasting or the Internet, and separates them. It then outputs the 3D model sequence to the first buffer 13, and outputs the audio stream and the sound source metasequence to the second buffer 14. There may be only one buffer, but in this embodiment, for the sake of convenience, the buffer is divided into the first buffer 13 and the second buffer 14.

After each data is stored in the first buffer 13 or the second buffer 14, subsequent processing is performed in synchronization with the synchronization clock input from the clock generation section 11. For example, when MMT is used as the multiplexing method, the first buffer 13 and the second buffer 14 each have a processing time so that the processing results of data assigned the same PTS are presented simultaneously at the final output. The data is output to the subsequent stage with an appropriate offset that takes into account the

The model decoding unit 15 decodes the 3D model sequence obtained from the first buffer 13 into a format that can be directly processed by the model rendering unit 20 using an existing method such as glTF (GL Transmission format) or H.265/HEVC (High Efficiency Video Coding), and outputs the decoded data to the model rendering unit 20. For example, when using functions defined in OpenGL for graphic rendering processing, the VBO (Vertex Buffer Object) format may be used.

The camera 16 photographs the surroundings of the receiving device 10 and outputs the photographed frame image to the frame memory 17.

The detection unit 18 includes one or more sensors such as a gyro sensor, an acceleration sensor, a geomagnetic sensor, and a gravity sensor. The detection unit 18 outputs sensor information detected by various sensors to the position and orientation estimation unit 19. A gyro sensor can detect the amount of rotation in three degrees of freedom (Role, Pitch, Yaw) by using the law of inertia, in which an object continues to move in the same direction. Further, the acceleration sensor can detect changes in moving speed in three degrees of freedom (X, Y, Z) by utilizing the law of inertia, in which an object continues to remain in the same place. Furthermore, the geomagnetic sensor can detect the north-south direction, and the gravity sensor can detect the direction perpendicular to the ground.

The position and orientation estimation unit 19 estimates the orientation of the receiving device 10 using the sensor information detected by the detection unit 18. The position and orientation estimating unit 19 may further use the frame images captured by the camera 16 to estimate the position and orientation of the receiving device 10. The position and orientation estimating unit 19 estimates the user's line-of-sight direction from the orientation of the receiving device 10. For example, the direction of the user's line of sight may be the direction in which the camera 16 is facing.

Further, the position and orientation estimating unit 19 estimates the user's viewpoint position based on the video captured by the camera 16. The user's viewpoint position may be the position of the camera 16. For example, the position and orientation estimating unit 19 detects a feature point from an image of one video frame captured in real space, and detects a point having the same feature amount as the feature point in the image of the next video frame. It is detected by neighborhood search and the amount of movement of one feature point is determined. Next, the distance between the receiving device 10 and the feature point can be determined by triangulation by combining the position of the feature point in the previous and previous two frames and the estimation result of the change in line of sight direction at the same time interval as the video frame. can. Similarly, the position and orientation estimating unit 19 detects a plane in real space by detecting the distance between a plurality of feature points and the receiving device 10 and determining whether the feature points exist on the same plane. be able to. Then, the position and orientation estimating unit 19 outputs viewpoint information indicating the user's viewpoint position and line-of-sight direction to the model rendering unit 20 and coordinate conversion unit 23.

Before actually starting to view AR content, the user may be made to perform a calibration operation in which the position and orientation estimation unit 19 of the terminal learns the state of the content viewing space, such as a plane in real space. When making the user perform pre-calibration, the camera 16 captures an image of the real space, centered on a plane (such as a floor or a table surface) on which the user intends to synthesize an object using AR, and plane detection is performed. In general, spatial learning through pre-calibration can make the synthesis of objects into real-space images using AR more stable, but there are cases in which pre-calibration is not required for viewing AR content.

The model rendering unit 20 inputs viewpoint information from the position and orientation estimation unit 19, and performs rendering of a viewport according to the viewpoint position and line of sight direction for the 3D model sequence decoded by the model decoding unit 15 to generate a rendering image, which is output to the video synthesis unit 21.

The video synthesis section 21 synthesizes the rendered image inputted from the model rendering section 20 and the frame image inputted from the frame memory 17 to generate a synthesized image, and causes the display 22 to display the synthesized image.

The coordinate conversion unit 23 inputs viewpoint information from the position and orientation estimation unit 19, and inputs sound source metadata (three-dimensional coordinates of the sound source object in the world coordinate system) from the second buffer 14. The coordinate conversion unit 23 converts the three-dimensional coordinates of the sound source object from the world coordinate system to a view coordinate system centered on the user's viewpoint position and with the line of sight as the axial direction, and outputs the sound source metadata after coordinate conversion to the object selection unit 25. For example, an affine transformation can be used for the coordinate conversion. If the viewpoint position is (Vx, Vy, Vz) and the unit vector of the line of sight is (Dx, Dy, Dz), then equation (1) holds for the rotation angle α around the Y axis and the rotation angle β around the X axis.

At this time, the rotation matrix A from the world coordinate system to the view coordinate system is expressed by equation (2).

Further, a coordinate movement matrix T whose origin is the viewpoint position is expressed by equation (3).

As described above, the coordinate transformation unit 23 coordinates transforms the three-dimensional coordinate vector P of the sound source object into the coordinate vector P' of the view coordinate system using the matrix operation formula (4). These coordinate transformation calculations are calculation processes that are generally performed even when drawing a three-dimensional model, and can use the functions of the GPU. The above transformation matrix is an example, and may vary depending on the difference between right-handed and left-handed systems in the world coordinate system and view coordinate system, the direction of axis polarity, and whether coordinate vectors are expressed as row vectors or column vectors. It may be different.

The processing load measurement unit 24 measures load information indicating the processing load of the receiving device 10, such as CPU usage rate and memory usage rate, and outputs it to the object selection unit 25.

The object selection unit 25 inputs audio chunks from the second buffer 14 at a timing controlled by a clock. The object selection unit 25 defines a sound source object selection area in the view coordinate system based on the processing load measured by the processing load measurement unit 24, and selects a sound source object located within the sound source object selection area as a sound processing target. Select as sound source object. In other words, the object selection unit 25, based on the processing load measured by the processing load measurement unit 24 and the sound source metadata (three-dimensional coordinates in the view coordinate system of the sound source object) input from the coordinate conversion unit 23, Select the sound source object for sound processing. Then, the object selection section 25 outputs the audio chunk of the selected sound source object to the audio decoding section 26, and outputs the three-dimensional coordinates of the sound source object to the three-dimensional sound rendering section 27.

Specifically, the object selection unit 25 calculates the processing load evaluation value L based on the load information input from the processing load measurement unit 24. For example, if the CPU usage rate is _R1 , the memory usage rate is _R2 , and the coefficients are _K1 and _K2 , the object selection unit 25 sets the evaluation value L= _K1 × _R1 + _K2 × _R2 . One of the coefficients K ₁ and K ₂ may be 0. The object selection unit 25 specifies that the larger the evaluation value L (processing load), the smaller the sound source object selection area. For example, when the evaluation value L exceeds the first threshold, the object selection unit 25 reduces the sound source object selection area so as to reduce the number of sound source objects to be processed, and the evaluation value L exceeds the second threshold. If the number of sound source objects is smaller than , the sound source object selection area is expanded to increase the number of sound source objects to be processed.

For example, the object selection unit 25 sets the sound source object selection area to a sphere with a radius R centered on the origin of the view coordinate system, which is the user's viewpoint position, and processes sound source objects for which the distance r from the origin satisfies r<R. It is possible to do so. That is, the object selection unit 25 sets the relationship between the processing load evaluation value L and the radius R to be R=f(L) (f(x) is a monotonically decreasing function), and when the processing load is larger than a predetermined threshold, the radius Control is performed such that the radius R is made small and the radius R is made large when the processing load is smaller than a predetermined threshold. In this embodiment, the sound source object selection area is defined as a sphere centered on the user's viewpoint position, but it can also be defined in other shapes, such as an ellipsoid with directivity in the user's viewing direction. .

The sound source metadata may include the priority p of the sound source object. In this case, the object selection unit 25 selects, as a sound source object to be acoustically processed, a sound source object that will be included in the sound source object selection area when the distance from the origin of the sound source object in the view coordinate system is changed so that the greater the priority p, the shorter it becomes. For example, for the actual distance r between the origin and the sound source object, r' = r * g(p) (g(x) is a monotonically decreasing function) may be used as the evaluation value r' when comparing with the radius R of the sound source object selection area. For example, when g(x) = 1/x, the object selection unit 25 does not select a sound source object with priority p = 1 as a sound source object to be acoustically processed unless r' = r ≦ R is satisfied, but selects a sound source object with priority p = 100 as a sound source object to be acoustically processed if r' = r/100 ≦ R is satisfied.

The sound source metadata may also include the maximum sound pressure level l of the sound source object. In this case, the object selection unit 25 selects a sound source that will be included in the sound source object selection area when the distance from the origin of the sound source object in the view coordinate system is changed so that the larger the maximum sound pressure level l, the shorter the distance from the origin of the sound source object. Select an object as a sound source object for acoustic processing. In other words, regarding the maximum sound pressure level l, the evaluation value is r' = r * h (l) (h (x) is a monotonically decreasing function) because the higher the sound pressure level, the more audible the sound even when it is farther from the listening position. . By transmitting the maximum sound pressure level as sound source metadata, it is possible to make a selection according to the sound pressure level without actually decoding the audio chunk in the receiving device 10, and it is possible to reduce the processing load. Considering both the priority p and the maximum sound pressure level l, the evaluation value r' of the origin distance of the sound source object is r' = r * g (p) * h (l) for the actual distance r, and the object The selection unit 25 selects a sound source object satisfying r'≦R as a sound source object to be subjected to sound processing.

The audio decoding unit 26 converts the audio chunk of the sound source object selected by the object selection unit 25 into a format that can be directly processed by the three-dimensional audio rendering unit 27. For example, the audio decoding unit 26 decodes a compressed stream such as MPEG-4 AAC (Advanced Audio Codec) or MPEG-H 3DA (3D Audio), and converts it into an uncompressed stream such as PCM (Pulse Code Modulation) data. . The audio decoding unit 26 outputs the decoded audio chunk (decoded audio chunk) to the three-dimensional audio rendering unit 27 .

The three-dimensional sound rendering unit 27 inputs the decoded audio chunk from the audio decoding unit 26 and receives the three-dimensional coordinates in the view coordinate system of the sound source object selected as a target for audio processing from the object selection unit 25. The three-dimensional sound rendering unit 27 generates binaural audio using audio chunks of the sound source object to be processed, and outputs the binaural audio from the speaker 28 or headphones (not shown).

Figure 4 is a block diagram showing an example configuration of the three-dimensional audio rendering unit 27. The three-dimensional audio rendering unit 27 includes a mapping unit 271 and a downmixing unit 272.

The mapping unit 271 inputs the decoded audio chunk and sound source metadata (three-dimensional coordinates of the sound source object) in a state that can be linked by the sound source object ID and sequence number. The mapping unit 271 then assigns (mixes) the decoded audio chunk to a predetermined number of virtual multi-channel speakers (virtual channel-based sound sources) arranged at predetermined positions centered on the viewpoint position in the view coordinate system.

FIG. 5 is a diagram showing an example of the world coordinate system (viewpoint from directly above) of three-dimensional space content and the arrangement of sound source objects O ₁ to O ₅ . FIG. 6 is a diagram showing an example of the view coordinate system (viewpoint from directly above) with the viewpoint position as the origin and the arrangement of virtual multichannel speakers. In the example shown in FIG. 6, eight virtual multi-channel speakers are arranged at equal intervals on a circle centered on the viewpoint position at the same height as the viewpoint position, and the same applies to the following FIGS. 7 to 10. It is. The number of virtual multi-channel speakers and their placement locations are not limited to these, and any number of virtual multi-channel speakers can be placed at any location. For example, a speaker arrangement for 5.1ch or 22.2ch multi-channel sound may be used.

Since the viewpoint position moves with the user's movements, the relative positional relationship between the world coordinate system and the view coordinate system changes depending on the user's movements. FIG. 7 is a diagram showing an example of a view coordinate system (viewpoint from directly above) arranged in the world coordinate system at a certain moment. FIG. 8 is a diagram illustrating an example of a view coordinate system (viewpoint from the side) arranged in the world coordinate system at a certain moment.

FIG. 9 is a diagram showing an example of mapping from a sound source object to a virtual multi-channel speaker in a view coordinate system as an example of processing by the mapping unit 271. In FIG. 9, the object selection unit 25 assumes that a sound source object located inside a spherical sound source object selection area centered on the viewpoint position is to be subjected to acoustic processing, and the sound source object selection area is indicated by a two-dot chain line. Note that the sound source object _O2 appears to be located inside the sound source object selection area in the projection view from above, but is located outside the spherical sound source object selection area centered on the origin when viewed spatially, and is therefore excluded from the mapping target. Also, FIG. 9 shows, as an example, that mapping is performed on a prescribed number of channels by using the distance between the coordinates of the sound source object converted into the view coordinate system and the virtual multi-channel speaker, and mapping to the virtual multi-channel speaker that is closest to the sound source object. Note that the mapping method from the sound source object to the virtual multi-channel speaker is not limited to this, and for example, one sound source object may be distributed to multiple virtual multi-channel speakers. In the mapping, for example, the sound pressure is attenuated according to the distance r between the origin of the view coordinate system and the sound source object. If the number of sound source objects to be subjected to acoustic processing is insufficient to the specified number of virtual multi-channel speakers, the object selection unit 25 may widen the sound source object selection area and perform the selection again.

The downmix unit 272 downmixes the decoded audio chunks assigned to the virtual multi-channel speakers into two-channel binaural audio corresponding to the left and right ears of the user. Specifically, the downmix unit 272 divides the audio signal into certain processing sections, converts the audio signal into the frequency domain, multiplies the frequency characteristics of a fixed number of head related transfer functions, and then converts the audio signal into time domain audio. By processing the signal back, it is downmixed to binaural audio.

FIG. 10 shows, as an example of the process of the downmix unit 272, a downmix process that generates binaural audio from a virtual multi-channel speaker in the view coordinate system. In the example shown in FIG. 10, the virtual multi-channel speakers in the front and rear direction are equally allocated to both the left and right speakers 28-1 and 28-2, and the other virtual multi-channel speakers are assigned to the nearest left or right speaker 28-2. 1 or 28-2. By calculating the frequency domain of the head-related transfer function for each of these mappings and mixing them, the user can experience a three-dimensional effect of the top, bottom, front, back, left, and right. According to the downmix shown in FIG. 10, the number of required head related transfer functions is ten. Note that the mapping shown in FIG. 9 and the downmix shown in FIG. 10 are merely examples, and the reproducibility of the three-dimensional sound field may be improved by a more advanced algorithm. For example, in FIG. 10, the head-related transfer function is considered for the virtual multi-channel speaker on the left in the viewing direction only for speaker 28-1, and for the virtual multi-channel speaker on the right for only speaker 28-2, but in the viewing direction The head-related transfer function may be increased so that the head-related transfer function from the virtual multi-channel speaker on the left side to the speaker 28-2 and the virtual multi-channel speaker on the right side to the speaker 28-1 is considered. By increasing the number of head-related transfer functions, more realistic binaural audio can be expected to be generated, but this comes at a trade-off with increased processing load.

Regarding streaming transmission of the sound source object, the distribution device 40 may multiplex all data including the audio coded data and sound source metadata of the audio chunk into UDP/IP packets, etc., and transmit the streaming transmission, or Instead of transmitting the actual data, location information and sound source metadata of audio chunks may be multiplexed into UDP/IP packets, etc., and then transmitted by streaming. As the location information, URL information for acquiring an audio chunk file using HTTP or the like, a multicast IP address and port number for receiving an additional audio chunk stream using IP multicasting, etc. can be specified. In this case, the receiving device 10 can reduce the load by acquiring the audio chunk file specified by the location information only for the sound source object selected by the object selection unit 25 as a target for acoustic processing. Alternatively, the distribution device 40 may include the location information in the sound source metadata and stream-transmit only the sound source metadata.

(Second embodiment)
Next, as a second embodiment, a 360-degree VR content transmission system that transmits VR content (360-degree VR video content) with a degree of freedom of 3DoF will be described.

Figure 11 is a diagram showing an example of the configuration of a 360-degree VR content transmission system 2 according to the second embodiment. The 360-degree VR content transmission system 2 includes a receiving device (VR receiving device) 10A and a distribution device 40A. A time server 50 is provided to synchronize the receiving device 10A and the distribution device 40A. Below, descriptions of the same configuration as the AR content transmission system 1 according to the first embodiment will be omitted as appropriate, and differences will be described.

The distribution device 40A transmits VR content by streaming via a transmission path 60 such as broadcasting or the Internet. The distribution device 40A transmits to the receiving device 10A an audio stream of multiple sound source objects placed in the three-dimensional space of the VR content, associated with sound source metadata including position information (three-dimensional coordinates) of the sound source objects. The distribution device 40A includes a clock generation unit 41 and a multiplexing unit 42.

The distribution device 40A is different from the distribution device 40 of the first embodiment in that it multiplexes and transmits a VR video sequence instead of a three-dimensional model sequence. That is, the multiplexing unit 42 multiplexes the VR video sequence, the audio stream (sequence of audio chunks), and the sound source metasequence (sequence of sound source metadata) to generate VR content, and transmits it to the outside of the distribution device 40. .

The receiving device 10A is a mobile terminal such as a smartphone or a tablet terminal, a terminal such as VR goggles, or a VR head mounted display. In the example shown in FIG. 11, the receiving device 10A includes a clock generation section 11, a demultiplexing section 12, a first buffer 13, a second buffer 14, a detection section 18, a position and orientation estimation section 19A, and a display 22. , coordinate conversion section 23A, processing load measurement section 24, object selection section 25, audio decoding section 26, three-dimensional sound rendering section 27, speaker 28, video decoding section 29, and video cutting section 30 and. Compared to the receiving device 10 of the first embodiment, the receiving device 10A does not include a model decoding section 15, a camera 16, a frame memory 17, a model rendering section 20, and a video composition section 21, and does not have a model decoding section 15, a camera 16, a frame memory 17, a model rendering section 20, and a video composition section 21. The difference is that it includes a section 29 and a video cutting section 30. Further, the processing of the position/orientation estimation section 19A and the coordinate transformation section 23A is different from the processing of the position/orientation estimation section 19 and the coordinate transformation section 23.

The demultiplexing unit 12 receives VR content, in which a VR video sequence, an audio stream, and an audio source metasequence are multiplexed, from the distribution device 40A via a transmission path 60 such as broadcast or the Internet, and separates them. It then outputs the VR video sequence to the first buffer 13, and outputs the audio source metasequence and audio stream to the second buffer 14.

The video decoding unit 29 decodes the VR video sequence obtained from the first buffer 13 using an existing method such as H.265/HEVC, and outputs it to the video extraction unit 30.

The position and orientation estimation unit 19A estimates the user's gaze direction using the sensor information input from the detection unit 18. For example, the position and orientation estimation unit 19A estimates the user's gaze direction from information from a gyro sensor. Since VR content with 3DoF degrees of freedom (fixed viewpoint position) is assumed here, there is no need to estimate the viewpoint position using an acceleration sensor or video captured by a camera, as in the position and orientation estimation unit 19 of the first embodiment. The position and orientation estimation unit 19A outputs viewpoint information indicating the user's gaze direction to the coordinate conversion unit 23A and the video cropping unit 30. Note that the position and orientation estimation unit 19A may output the user's viewpoint position information as a function equivalent to that of the position and orientation estimation unit 19, but the viewpoint position information is not used in 3DoF content.

The coordinate transformation unit 23A coordinates transforms the three-dimensional coordinate vector P of the sound source object into a coordinate vector P' of the view coordinate system using the rotation matrix A shown in equation (2) and the matrix calculation formula in equation (5). do. Here, we are assuming VR content with 3DoF degrees of freedom, so the sound source metadata is It is possible to create sound source location information. In this case, the conversion from the world coordinate system to the user-centered view coordinate system involves only rotational movement and no parallel movement.

The video cutout unit 30 cuts out the video of the viewport corresponding to the user's gaze direction estimated by the position and orientation estimation unit 19A from the VR video decoded by the video decoding unit 29, generates a cutout video, and generates a cutout video. The cutout image is displayed on the display 22.

As described above, the receiving device 10, 10A defines a sound source object selection area in the view coordinate system based on the processing load, and selects a sound source object located within the sound source object selection area as a sound source object to be subjected to acoustic processing. This configuration makes it possible to reduce the amount of calculation and circuit size of the audio decoding unit 26 and the three-dimensional acoustic rendering unit 27, ensure real-time content playback, and improve viewing quality. In addition, it is possible to realize a service in which terminals with various performances, such as different CPU clocks and memory capacity, can be used as receiving devices, and content can be played back according to the processing performance of each terminal.

The receiving device 10, 10A may also convert the position information of the sound source object transmitted in three-dimensional coordinates in the world coordinate system into three-dimensional coordinates in a view coordinate system with the user's viewpoint position as the origin. With this configuration, the user's viewpoint position can be regarded as a fixed center of the coordinate system, simplifying subsequent calculations.

Furthermore, the receiving devices 10 and 10A allocate audio chunks of the sound source object to be processed for acoustic processing to virtual multi-channel speakers arranged in the view coordinate system as a primary process, and then allocate audio chunks to the virtual multi-channel speakers as a secondary process. Binaural audio may be generated using the audio chunks. With this configuration, in the primary processing, it is possible to use low-load calculations such as simple sound pressure attenuation due to distance, without using the high-load frequency domain calculations required for binaural audio generation. The number of virtual channels and sound source positions are fixed through processing, and binaural audio can be generated using a finite number of head-related transfer functions in secondary processing. Therefore, the amount of calculation and circuit scale of the three-dimensional sound rendering section 27 can be further reduced.

Furthermore, the receiving devices 10 and 10A may enlarge or reduce the sound source object selection area according to the current processing load determined by load measurement. With this configuration, the processing load on the audio decoding section 26 and the three-dimensional audio rendering section 27 can be optimized.

The receiving device 10, 10A may also select, as a sound source object to be subjected to acoustic processing, a sound source object that would be included in the sound source object selection area when the distance from the origin of the sound source object in the view coordinate system is changed so that the greater the priority and/or the maximum sound pressure level, the shorter the distance. With such a configuration, the viewing quality of the content can be further improved.

<Program>
It is also possible to use a computer capable of executing program instructions to function as the above-mentioned receiving devices 10, 10A. The computer can be realized by storing a program that describes the processing contents for realizing the functions of the receiving devices 10 and 10A in the storage section of the computer, and having the processor of the computer read and execute this program. , at least a part of these processing contents may be realized by hardware. Here, the program instructions may be program codes, code segments, etc. for performing necessary tasks. The processor may be a CPU, GPU, DSP, ASIC (Application Specific Integrated Circuit), or the like.

Further, this program may be recorded on a computer-readable recording medium. Using such a recording medium, it is possible to install a program on a computer. Here, the recording medium on which the program is recorded may be a non-transitory recording medium. The non-transitory recording medium is not particularly limited, but may be a recording medium such as a CD-ROM or a DVD-ROM. Moreover, this program can also be provided by downloading via a network.

Although the embodiments described above have been described as representative examples, it will be apparent to those skilled in the art that many modifications and substitutions can be made within the spirit and scope of the invention. Therefore, the present invention should not be construed as being limited to the above-described embodiments, and various modifications and changes can be made without departing from the scope of the claims. For example, it is possible to combine a plurality of building blocks or processing steps described in the embodiments into one, or to divide one into a plurality of blocks or processing steps.

1 AR content transmission system 2 360 degree VR video content transmission system 10, 10A Receiving device 11 Clock generation section 12 Demultiplexing section 13 First buffer 14 Second buffer 15 Model decoding section 16 Camera 17 Frame memory 18 Detection section 19, 19A Position Posture estimation unit 20 Model rendering unit 21 Video synthesis unit 22 Display 23, 23A Coordinate conversion unit 24 Processing load measurement unit 25 Object selection unit 26 Audio decoding unit 27 Three-dimensional sound rendering unit 28 Speaker 29 Video decoding unit 30 Video cutting unit 40 , 40A distribution device 41 clock generation section 42 multiplexing section 50 time server 60 transmission line

Claims (8)

A receiving device that receives audio chunks obtained by blocking an audio stream of a sound source object and sound source metadata including three-dimensional coordinates of the sound source object in a world coordinate system, the receiving device comprising:
a coordinate conversion unit that converts the three-dimensional coordinates of the sound source object from a world coordinate system to a view coordinate system;
an object selection unit that defines a sound source object selection area in the view coordinate system based on the processing load of the receiving device, and selects a sound source object located within the sound source object selection area as a sound source object to be subjected to acoustic processing;
a three-dimensional sound rendering unit that generates binaural sound using the audio chunks of the sound source object to be processed;
Equipped with
The object selection unit defines the sound source object selection area to become smaller as the processing load becomes larger . A receiving device that receives audio chunks obtained by dividing an audio stream of a sound source object into blocks, and sound source metadata including three-dimensional coordinates of the sound source object in a world coordinate system, comprising:
a coordinate conversion unit that converts the three-dimensional coordinates of the sound source object from a world coordinate system to a view coordinate system;
an object selection unit that defines a sound source object selection area in the view coordinate system based on a processing load of the receiving device, and selects a sound source object located within the sound source object selection area as a sound source object to be subjected to sound processing;
a three-dimensional audio rendering unit that generates binaural audio using audio chunks of the sound source object to be subjected to audio processing;
Equipped with
the audio source metadata includes a priority of the audio source object;
The object selection unit selects, as the sound source object to be subjected to acoustic processing, a sound source object that would be included in the sound source object selection area when the distance from the origin of the sound source object in the view coordinate system is changed so that the greater the priority, the shorter the distance. A receiving device that receives audio chunks obtained by blocking an audio stream of a sound source object and sound source metadata including three-dimensional coordinates of the sound source object in a world coordinate system, the receiving device comprising:
a coordinate conversion unit that converts the three-dimensional coordinates of the sound source object from a world coordinate system to a view coordinate system;
an object selection unit that defines a sound source object selection area in the view coordinate system based on the processing load of the receiving device, and selects a sound source object located within the sound source object selection area as a sound source object to be subjected to acoustic processing;
a three-dimensional sound rendering unit that generates binaural sound using the audio chunks of the sound source object to be processed;
Equipped with
The sound source metadata includes a maximum sound pressure level of the sound source object,
The object selection unit selects a sound source object that will be included in the sound source object selection area when the distance from the origin of the sound source object in the view coordinate system is changed such that the larger the maximum sound pressure level, the shorter the distance from the origin. , a receiving device selected as the sound source object to be subjected to the acoustic processing. further comprising a position and orientation estimation unit that estimates the user's line of sight direction;
The coordinate conversion unit converts the three-dimensional coordinates of the sound source object from the world coordinate system to the view coordinate system with the line of sight as the axial direction. Receiving device. The position and orientation estimation unit estimates a viewpoint position of the user,
The receiving device according to claim 4 , wherein the coordinate conversion unit converts the three-dimensional coordinates of the sound source object from the world coordinate system to the view coordinate system having the viewpoint position as its origin. The three-dimensional sound rendering unit includes:
a mapping unit that allocates audio chunks of the sound source object to be processed for acoustic processing to virtual multi-channel speakers arranged in the view coordinate system;
a downmix unit that generates the binaural audio using the audio chunk assigned to the virtual multi-channel speaker;
The receiving device according to any one of claims 1 to 5 , comprising: A receiving device according to any one of claims 1 to 6 ,
a distribution device that associates the audio chunk with the sound source metadata and transmits the resulting association to the receiving device;
A content transmission system comprising: A program for causing a computer to function as the receiving device according to claim 1 .

Images