JP2018033107A - Video distribution device and distribution method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To realize compression of data capacity when enabling to view a space or a subject located therein from different direction, in distribution of video.SOLUTION: Based on one or more than one video data received from a photographic camera 210 capturing 3D space of distribution object or one or more than one subjects located therein, a distribution device 220 recognizes one or more than one subjects in 3D space, respectively, as 3D objects, and transmits to a user terminal 230. Thereafter, a 3D object with motion is detected, and motion data indicating the difference is generated. The distribution device 220 transmits the generated motion data to the user terminal 230, where the 3D object being held is moved based on the received motion data, and displayed on a display screen, as a video in the gaze direction, along with other 3D or 2D object with no motion in a period corresponding to the motion data.SELECTED DRAWING: Figure 2

Description

  The present invention relates to a moving image distribution apparatus and distribution method, and more particularly to a distribution apparatus and distribution method for VR moving images, AR moving images, and the like.

  With the increase in communication speed, the use of video is increasing in various situations. In particular, a great deal of attention has been focused on moving images using virtual reality (VR) technology that virtually creates a wide space such as 180 ° and 360 °.

  FIG. 1 shows an example of a VR moving image distribution system. The distribution system 100 is generated by equirectangular projection by stitching by the stitch server 120 and stitch server 120 for combining or stitching each video data shot by each camera of the 360 ° camera 110 and 360 ° camera 110. Encoding server 130 that encodes 360 ° video data having 360 ° information, and 360 ° video data received from encoding server 130, and displaying the video in the viewing direction of the user or viewer on the display screen. A portable terminal 140.

  The 360 ° camera 110 can be configured by, for example, combining a plurality of fisheye lens mounted cameras and fixing them to the rig. A 360-degree omnidirectional field of view, or at least a 360-degree field of view on a plane may be provided. In the figure, a 360 ° camera 110 is configured by arranging four single-lens mirrorless or action cameras at different angles by 90 ° on a predetermined plane. There are some cameras equipped with a plurality of cameras so that 360 degrees can be photographed in advance.

  The stitch server 120 coordinates the four video data received from the 360 ° camera 110 for each frame into an equirectangular projection to generate reproducible 360 ° video data.

  The encoding server 130 performs processing on 360 ° video data according to image quality, distribution bit rate, and the like.

  The mobile terminal 140 of the video viewer determines the viewer's line-of-sight direction, and displays the portion of the 360 ° video data corresponding to the line-of-sight direction on the display screen. A dedicated viewer for displaying a moving image can be installed on a mobile terminal such as a smartphone or a tablet, and the moving image can be played back by the viewer. The viewing direction of the viewer can be determined by detection by a gyro sensor included in the mobile terminal 140.

  For example, when a moving image is distributed in real time by using such VR technology, the viewer can appropriately change the direction of the line of sight to a desired direction and experience as if they were on the spot.

  However, the distribution of VR moving images has a problem that the processing load on the viewer is large because the data capacity is significantly increased as compared with the moving image distribution in which the conventional visual field is unidirectional. This problem occurs not only when the terminal is a mobile terminal but also when the user or viewer's terminal is a PC or the like. However, the problem is remarkable when the terminal is a mobile terminal such as a smartphone. It becomes.

  In addition to the VR technology that virtually creates the entire space for the viewer to experience, for example, the surroundings of the space such as the background are photographed by the mobile terminal 140 or a video is prepared in advance, and the subject in the space (for example, As for augmented reality (AR) technology that virtually creates and superimposes a person at a remote location, etc., in order to be able to view the subject from different angles such as 180 ° and 360 °, the same increase in data capacity is required. Problems arise.

  The present invention has been made in view of such problems, and an object of the present invention is to enable viewing of a space or a subject located therein from different directions in a moving image distribution apparatus and distribution method. It is to realize the compression of the data capacity.

  In order to achieve such an object, according to a first aspect of the present invention, there is provided a moving image distribution method from a shooting camera having one or more stereo cameras for shooting one or more subjects to be distributed. Receiving video data; recognizing one or more 3D objects respectively corresponding to the one or more subjects based on the video data; and transmitting the one or more 3D objects to a user terminal Detecting a moving 3D object based on video data from the photographing camera, generating motion data indicating the movement, and transmitting the motion data to the user terminal, Corresponding to the motion data in the user terminal after the motion data is received That updates the 3D object, characterized in that it is displayed on the display screen of the user terminal.

  According to a second aspect of the present invention, in the first aspect, the updated 3D object is displayed on the display screen together with other objects.

  According to a third aspect of the present invention, in the second aspect, the other object is a 3D object transmitted to the user terminal, and the 3D object has not moved during a period corresponding to the motion data. It is characterized by being.

  According to a fourth aspect of the present invention, in the second aspect, a subject whose depth measured from the video data is a predetermined threshold or more is recognized as a 2D object, and the other object is transmitted to the user terminal. It is the 2D object made.

  According to a fifth aspect of the present invention, in the second aspect, the other object is a 2D or 3D object stored in advance in the user terminal.

  According to a sixth aspect of the present invention, in the second aspect, the other object is data photographed in real time on the user terminal.

  According to a seventh aspect of the present invention, in any one of the first to sixth aspects, the one or more 3D objects are recognized based on video data of the first frame from the photographing camera. The motion data is generated based on a motion between a second frame after the first frame and the first frame.

  According to an eighth aspect of the present invention, in any one of the first to seventh aspects, the video data is live video data transmitted from the photographing camera.

  According to a ninth aspect of the present invention, in any one of the first to eighth aspects, the photographing camera includes a plurality of stereo cameras arranged in different directions.

  According to a tenth aspect of the present invention, in the ninth aspect, the method further includes a step of stitching each 3D object recognized from video data from each stereo camera.

  An eleventh aspect of the present invention is characterized in that, in any one of the first to tenth aspects, the photographing camera is capable of photographing the surrounding 360 °.

  According to a twelfth aspect of the present invention, in the eleventh aspect, the user terminal is a portable terminal, and the viewer's line-of-sight direction determined by detection by a gyro sensor included in the portable terminal is received. Limiting the motion data transmitted to the mobile terminal to motion data corresponding to the line-of-sight direction.

  According to a thirteenth aspect of the present invention, in any one of the first to tenth aspects, the photographing camera can photograph a subject from 360 °.

  According to a fourteenth aspect of the present invention, there is provided a program for causing a computer to execute a moving image distribution method, wherein the distribution method captures one or a plurality of subjects to be distributed. Receiving video data from a photographic camera having a camera; recognizing one or more 3D objects respectively corresponding to the one or more subjects based on the video data; and the one or more 3D objects Transmitting to the user terminal, detecting a moving 3D object based on the video data from the photographing camera, generating motion data indicating the movement, and transmitting the motion data to the user terminal And after the motion data is received, the user end Update the 3D object corresponding to the motion data, characterized in that it is displayed on the display screen of the user terminal in.

  According to a fifteenth aspect of the present invention, based on the video data, a receiving unit that captures one or a plurality of subjects to be distributed and receives video data from a photographing camera having one or more stereo cameras, A processing unit for recognizing one or a plurality of 3D objects respectively corresponding to the one or a plurality of subjects; and a transmission unit for transmitting the one or a plurality of 3D objects to a user terminal. 3D object with motion is detected based on the video data from, and motion data indicating the motion is generated, and the transmission unit transmits the motion data to the user terminal, and the motion data is received. Later, a 3D object corresponding to the motion data is updated in the user terminal to display a table of the user terminal. Characterized in that it is displayed on the screen.

  According to one aspect of the present invention, a 3D object of a subject is recognized based on video data from a photographing camera having one or a plurality of stereo cameras, and this is held in a user terminal, and the movement of the 3D object is determined. By reflecting the motion of the 3D object in the user terminal by the motion data shown, the data capacity can be compressed in moving image distribution.

It is a figure which shows an example of the delivery system of the conventional VR moving image. It is a figure for demonstrating the outline | summary of the moving image delivery apparatus concerning this invention. It is a figure which shows an example of the imaging | photography camera which makes the circumference | surroundings a subject in one Embodiment of this invention. It is a figure which shows the imaging | photography possible range in the case of having a stereo camera in several directions. It is a flowchart which shows the process in an imaging device in one Embodiment of this invention. It is a flowchart which shows the process in a user terminal in one Embodiment of this invention. It is a figure which shows an example which renders several received object data on a display screen in a user terminal. It is a figure which shows an example of the imaging | photography camera which image | photographs a to-be-photographed object from another around in another embodiment of this invention. It is a figure which shows an example of the imaging | photography camera which image | photographs a to-be-photographed object from the periphery in further embodiment of this invention.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

(Outline of the present invention)
FIG. 2 shows a moving image distribution apparatus according to the present invention. The distribution device 220 uses the one or more subjects in the 3D space based on one or more video data received from the imaging camera 210 that captures the 3D space to be delivered or one or more subjects located in the 3D space. Are recognized as 3D objects. Furthermore, one or a plurality of 3D objects recognized as necessary are cut into slices as data of different layers, and then a moving 3D object is detected and motion data indicating the difference is generated.

  The recognition of one or a plurality of 3D objects is performed based on the video data of the first frame from the photographing camera 210, and the generation of motion data is performed by using the second frame and the first frame after the first frame. Can be done based on the movement between. Alternatively, a 3D object can be recognized using video data of a plurality of frames.

  Each 3D object data can be stored in the storage unit 221 such as a memory or a hard disk of the distribution device 220, and is transmitted to the user terminal 230 including a mobile terminal such as a smartphone or a tablet. The received 3D object is stored in a storage unit (not shown) such as a memory or a hard disk in the user terminal 230 for a certain period.

  The distribution device 220 transmits the generated motion data to the user terminal 230, and the user terminal 230 moves the 3D object data held based on the received motion data, and moves in a period corresponding to the motion data. It is displayed on the display screen as an image in a certain line of sight along with other 3D objects or 2D objects that are not present.

  In this way, by recognizing a subject in the space to be imaged as a 3D object and holding the data in the user terminal 230 in advance, the transmission data from the distribution device 220 to the user terminal 230 is converted into the movement of the subject. It is possible to obtain motion data as shown, and compression with a large data capacity is realized. Especially in the case of live video distribution, it can deliver high-quality video with reduced clutter, improving the user experience.

  Here, the 3D object means 3D data representing a part of the 3D space. For example, a subject such as a person, a building, or a cloud standing in the space to be imaged is targeted. A fixed threshold value may be set, and a subject having a depth greater than that described below may be recognized as a 2D object represented by 2D data in a pseudo manner without being converted into a 3D object. For example, backgrounds such as sky and walls are targeted.

  Moreover, it is also possible to store data in advance in the distribution device 220 or the user terminal 230 as a 3D object or a 2D object for some subjects or backgrounds. Further, data such as a background photographed in real time in the user terminal 230 can also be used. It should be noted that the moving image distribution according to the present invention can be applied not only to the entire moving image but also to only a part thereof.

  In the above description and the description below, each process can be realized by causing a computer to execute a predetermined program, and the computer can be a single server or a plurality of servers. Similarly, a program can realize a required function by a plurality of programs. Examples of the computer include a computer including a communication unit such as a communication interface for data transmission / reception, a processing unit such as a CPU and a GPU, and a storage unit such as a memory and a hard disk.

(Embodiment in which the periphery of the shooting camera is the subject)
FIG. 3 shows a photographing camera for photographing a surrounding subject in one embodiment of the present invention. The photographing camera 310 includes a stereo camera having a right camera and a left camera in each of the first to sixth directions. Each stereo camera can measure the distance in the depth direction to each point of the subject 320 based on the parallax width of the left and right cameras, and the subject 320 can be converted into data as a 3D object that is three-dimensionally represented. it can. When the distance to the subject 320 is long, that is, when the depth is deep, it is often sufficient to simply recognize the object as a 2D object without performing a 3D object because the parallax width is small.

  As shown in FIG. 4, when a stereo camera is provided in a plurality of directions, the angle at which the subject 320 can be recognized as a 3D object is increased. A straight line along the front surface of each stereo camera indicates a shootable range when each stereo camera has an angle of view of 180 °. Here, the stereo cameras are arranged in the first to sixth directions, but various arrangements such as only the first direction and the first to fourth directions are conceivable.

  Based on the video data transmitted from each stereo camera of the photographing camera 310 (S501), the distribution apparatus 220 recognizes one or more 3D objects corresponding to one or more subjects (S502). Video data used for 3D object conversion may be single-frame video data or a plurality of frames of video data.

  If a plurality of stereo cameras are arranged, the 3D object data is stitched here (S503). A 3D object recognized from video data of each stereo camera is connected as a contact point to a point corresponding to the same part of the subject, and a 3D object used for reproduction on the user terminal 230 is generated. Stitching is performed for each 3D object, and when a subject is recognized as a 2D object, it can be connected by coordinate transformation to an equirectangular projection as in the prior art.

  Each recognized object is transmitted to the user terminal 230 as data of a different layer as needed (S504). Transmission data to the user terminal 230 can be optimized such as changing the image quality and bit rate according to the communication environment between the distribution device 220 and the user terminal 230. In the user terminal 230, each object can be stored for each layer.

  The distribution device 220 continuously or intermittently converts a 3D space to be imaged or a subject located in the 3D space into a 3D object, tracks each 3D object, and detects an object that moves beyond a certain threshold. Motion data indicating the motion is generated (S505) and transmitted to the user terminal 230 (S506). Examples of the movement include movement, rotation, tilt, and deformation. The 3D object can be expressed by, for example, vertex data, and the motion data can specify a calculation to be performed on each vertex.

  The user terminal 230 receives the data of each object (S601), moves the corresponding 3D object in accordance with the received motion data, and updates the object (S602). The updated object data may be overwritten or may be held as separate data. The moving image can be distributed by continuing the process of displaying both the updated object and the other object on the display screen of the user terminal 230 for each frame. It is also possible to simultaneously display 3D or 2D object data representing the background, the space around the subject, etc. stored in advance in the user terminal 230. At this time, the respective data may be combined and displayed (S603).

  Although a dedicated viewer can be installed and displayed, if the distribution method according to the present embodiment is used, the data capacity to be processed in the user terminal 230 can be increased by holding the 3D object in the user terminal 230 in advance. It is possible to suppress the content and reproduce it by a general-purpose viewer on a web browser having a low processing capacity (S604). In many cases, it is desirable to play on a web browser instead of playing a video on an application installed on a terminal such as a smartphone or tablet.

  FIG. 7 shows an example of rendering a plurality of received objects on the display screen in the user terminal 230. The first 3D object 741 and the second 3D object 742 are stored in the user terminal 730 in advance, and motion data 743 indicating the movement of the second object 742 is transmitted from the distribution server 720 to the user terminal 730. . In the user terminal 730, the second object 742 is updated according to the motion data 743, and each object is synthesized and reproduced. The distribution server 720 may be an instance on the cloud.

  When the video camera 310 shown in FIG. 3 is used, 360 ° video distribution is possible, and the user terminal 230 determines the viewing direction of the viewer and displays a portion corresponding to the viewing direction on the display screen. be able to. The viewing direction of the viewer can be determined by detection by a gyro sensor included in the user terminal 230, dragging by the viewer on the display screen, or the like.

  If data indicating the viewer's line-of-sight direction is transmitted continuously or intermittently from the user terminal 230 to the distribution device 220, motion data transmitted from the distribution device 220 to the user terminal 230 is necessary for moving image reproduction in the line-of-sight direction. Can be limited to the data, and further data capacity compression is possible. In addition, by predicting the viewing direction of the viewer at the user terminal 230 or the distribution device 220, data transmission from the distribution device 220 to the user terminal 230 can be accelerated.

  The user terminals 230 and 730 include general computers such as PCs. However, when the user terminals 230 and 730 are portable terminals such as smartphones, tablets, and twin-lens HMDs, the present invention significantly compresses the data capacity to provide VR videos, AR videos, and the like. Greatly improve the user experience for rich videos. Even in a situation where the user terminals 230 and 730 are remarkably reduced in size or weight, such as a contact lens having an IC, and the data processing capability that can be mounted is limited, it is possible to distribute moving images.

(Embodiment in which the photographing camera is arranged around the subject)
FIG. 8 shows a photographing camera arranged around a subject in another embodiment of the present invention. The photographing camera 810 includes a stereo camera having a right camera and a left camera in each of the first direction to the fourth direction. Each stereo camera can measure the distance in the depth direction to each point of the subject 820 based on the parallax width of the left and right cameras, and the subject 820 can be converted into data as a 3D object that is three-dimensionally represented. it can. When the distance to the subject 820 is long, that is, when the depth is deep, the width of the parallax is small, so that the object may be simply recognized as a 2D object without performing the 3D object.

  A straight line along the front surface of each stereo camera indicates a shootable range when each stereo camera has an angle of view of 180 °. Here, the stereo cameras are arranged in the first to fourth directions, but various arrangements such as only the first direction and the first to sixth directions are conceivable.

  The processing in the embodiment shown in FIG. 3 and the distribution device 220 are basically the same, but the processing may be different for the line-of-sight direction. When the user terminal 230 has a position sensor such as GPS, the viewpoint can be switched from a stereo camera directed in the first direction to a stereo camera directed in the second direction according to the position information of the user terminal 230. . Further, the viewer may manually select the stereo camera on the display screen of the user terminal 230. It is also possible to detect the direction of the line of sight by a gyro sensor of the user terminal 230 and determine which direction of moving image to be distributed from which stereo camera.

  In FIG. 8, one stereo camera is arranged in each of the four directions surrounding the subject 820. However, as shown in FIG. 9, a plurality of stereo cameras directed in each direction may be arranged side by side. Good.

  Moreover, each stereo camera can also detect the gaze direction with the gyro sensor which the user terminal 230 has, and can also be rotated toward the said gaze direction.

  At least one of the position and orientation of the user terminal 230 is received by the distribution device 220 continuously or intermittently, and the motion data transmitted from the distribution device 220 to the user terminal 230 is used for moving image reproduction at at least one of the position and orientation. The data can be limited to the necessary data, and further data capacity compression is possible. The same applies when a stereo camera is selected by the viewer.

(skin)
Up to this point, it has been described that video data from a shooting camera is analyzed to recognize a 3D object. More specifically, a 3D object includes a bone representing a shape and a skin drawn on the bone. And can be broadly divided. In either case, as described above, the object data is held in the user terminal in advance, and the motion data can be reproduced by representing the motion by the motion data and updating it. The skin may be corrected by CG.

  The motion data for the skin includes color (RGB), coordinate position (x axis, y axis, z axis), and the like.

(Other)
Characters such as user comments and live broadcast / commentary comments may be superimposed on one or more objects displayed on the user terminal as objects and reproduced.

  It is also possible to record the audio three-dimensionally in all directions or a part of the directions and distribute the acoustic data from the distribution device to the user terminal in time with the video.

DESCRIPTION OF SYMBOLS 100 Distribution system 110 Shooting camera 120 Stitch server 130 Encoding server 140 Portable terminal 210 Shooting camera 220 Distribution device 221 Storage device 230 User terminal 310 Shooting camera 320 Subject 720 Distribution server 730 Portable terminal 741 First object 742 Second object 743 Motion Data 810 Camera 820 Subject

Claims (15)

A video delivery method,
Receiving video data from a photographic camera having one or more stereo cameras that shoot one or more subjects to be distributed;
Recognizing one or more 3D objects respectively corresponding to the one or more subjects based on the video data;
Transmitting the one or more 3D objects to a user terminal;
Detecting a moving 3D object based on video data from the photographing camera and generating motion data indicating the movement;
Transmitting the motion data to the user terminal,
After the motion data is received, the 3D object corresponding to the motion data is updated on the user terminal and displayed on the display screen of the user terminal.   The distribution method according to claim 1, wherein the updated 3D object is displayed on the display screen together with other objects.   The delivery method according to claim 2, wherein the other object is a 3D object transmitted to the user terminal and has not moved during a period corresponding to the motion data. A subject whose depth measured from the video data is a predetermined threshold or more is recognized as a 2D object,
The delivery method according to claim 2, wherein the other object is the 2D object transmitted to the user terminal.   The delivery method according to claim 2, wherein the other object is a 2D or 3D object stored in advance in the user terminal.   The distribution method according to claim 2, wherein the other object is data captured in real time at the user terminal. The recognition of the one or more 3D objects is performed based on video data of a first frame from the photographing camera,
The generation of the motion data is performed based on a motion between a second frame after the first frame and the first frame. Delivery method.   The distribution method according to claim 1, wherein the video data is live video data transmitted from the photographing camera.   The distribution method according to claim 1, wherein the photographing camera has a plurality of stereo cameras arranged in different directions.   The distribution method according to claim 9, further comprising a step of stitching each 3D object recognized from video data from each stereo camera.   The distribution method according to claim 1, wherein the photographing camera is capable of photographing a surrounding 360 °. The user terminal is a mobile terminal;
Receiving a viewer's line-of-sight direction determined by detection by a gyro sensor of the mobile terminal;
The distribution method according to claim 11, further comprising: limiting motion data to be transmitted to the mobile terminal to motion data corresponding to the line-of-sight direction.   The distribution method according to claim 1, wherein the photographing camera can photograph a subject from 360 °. A program for causing a computer to execute a video distribution method, wherein the distribution method includes:
Receiving video data from a photographic camera having one or more stereo cameras that shoot one or more subjects to be distributed;
Recognizing one or more 3D objects respectively corresponding to the one or more subjects based on the video data;
Transmitting the one or more 3D objects to a user terminal;
Detecting a moving 3D object based on video data from the photographing camera and generating motion data indicating the movement;
Transmitting the motion data to the user terminal,
After the motion data is received, the 3D object corresponding to the motion data is updated on the user terminal and displayed on the display screen of the user terminal. A video distribution device,
A receiver that receives one or more subjects to be distributed and receives video data from a shooting camera having one or more stereo cameras;
A processing unit that recognizes one or more 3D objects respectively corresponding to the one or more subjects based on the video data;
A transmission unit that transmits the one or more 3D objects to a user terminal,
The processing unit detects a moving 3D object based on video data from the shooting camera, generates motion data indicating the movement,
The transmission unit transmits the motion data to the user terminal,
After the motion data is received, the 3D object corresponding to the motion data is updated on the user terminal and displayed on the display screen of the user terminal.

Images