JP6512575B2 - Method of distributing or broadcasting three-dimensional shape information - Google Patents

Description

  The present invention relates to a method of distributing or broadcasting three-dimensional shape information.

  In video distribution and television broadcasting using the Internet, two-dimensional video captured by a camera is distributed or broadcast.

  A 3D television that can be viewed stereoscopically from around 2010 is sold, and a program called left-eye and right-eye video that takes into consideration the parallax between the left and right eyes is a system called side-by-side that divides the screen horizontally It came to be produced at the department's broadcast station. Similarly, PC 3D compatible displays and projectors capable of stereoscopic viewing are also on the market, and side-by-side video distribution and streaming distribution using the Internet are also performed.

  In the field of virtual reality, low-cost head-mounted displays have been marketed with a function called head tracking that measures the position and direction of the viewer's head with a sensor or the like. In addition, a depth sensor capable of measuring the depth is also marketed as a peripheral device of a home-use game machine that can be used by a PC.

  Although the reproduced image of the object to be photographed pops out from the screen and appears to have depth by stereoscopically viewing the video delivered or broadcast by the side-by-side method, it is possible to accurately grasp the three-dimensional shape of the object to be photographed. Has not been reached.

  Currently, to create content that includes 3D shapes for virtual reality requires knowledge of 3D computer graphics, and it can be easily created like video distribution and streaming distribution performed on the Internet. There is no established method for delivering or broadcasting virtual reality content with real-time capabilities.

  The present invention is intended to solve such a problem, and three-dimensional shape information of an object to be photographed can be distributed as Internet or television broadcast as a two-dimensional video only by photographing with an easily available camera and a depth sensor. It is an object of the present invention to restore an accurate three-dimensional shape of an object to be shot from the moving image and reproduce it by computer graphics or use it as content for virtual reality.

  Create a single image including a color image and a depth image obtained by shooting with a camera and a depth sensor. At this time, the color image and the depth image are arranged in one image so as not to overlap and with a margin.

  In order to restore three-dimensional coordinates from depth information recorded as pixels of a depth image, information of the angle of view of the depth sensor used for photographing and the resolution of the depth image is required. The angle of view of the depth sensor, the resolution of the depth image, the two-dimensional coordinates representing the position of the pixel in the depth image, and the three-dimensional coordinates restored from depth information recorded as pixels represent the relative position from the depth sensor.

  In order to create point cloud data of a three-dimensional shape to be photographed, it is necessary to determine the color of the point in three-dimensional coordinates restored from each pixel of the depth image. In order to associate each pixel of the depth image with a pixel of the color image, information of the angle of view of the camera used for photographing and the resolution of the color image is required.

  In addition, when acquiring a color image and a depth image from one created image, information on which part of the one image the color image and the depth image are arranged is required.

  Information on these cameras, depth sensors, color images, and depth images is encoded into a two-dimensional code and placed in the margin of one image.

  When one image including a color image, a depth image, and a two-dimensional code necessary to create point cloud data of a three-dimensional shape to be photographed created by such a method is called a three-dimensional shape information image Do.

  When a subject to be photographed is photographed by a pair of cameras and a depth sensor, neither the color information nor the depth information can be recorded in the blind spot of the camera and the depth sensor, and the three-dimensional shape of the whole photographing target can not be restored. Therefore, three-dimensional shape of the part of the object to be photographed in a wider range compared to the case of photographing with one set of camera and depth sensor by photographing the object to be photographed with a plurality of sets of cameras and depth sensor and reducing the blind spot. Can be restored.

  As in the case of one set of camera and depth sensor, a plurality of sets of color images and depth images obtained by photographing with multiple sets of cameras and depth sensor are arranged to create one three-dimensional shape information image. An ID for uniquely identifying a set is assigned to each set of camera, depth sensor, color image, and depth image, and the ID of each set and information on the camera, depth sensor, color image, depth image are encoded. Create a two-dimensional code and place it in the margin of the three-dimensional shape information image.

  When photographing with a plurality of depth sensors, combining is performed in consideration of the positional relationship between point cloud data of a part of the photographing target represented by the relative position from each depth sensor, and one point of a wider part of the photographing target In order to create group data, it is necessary to know from which position each depth sensor is shooting a shooting target. Define a virtual coordinate system with one point in the real space being taken as the origin, measure the position from the origin of each depth sensor and the direction in which the depth sensor points, and respond as the installation position information of the depth sensor Included in the two-dimensional code. By considering the installation position of the depth sensor, it is possible to restore three-dimensional coordinates representing an absolute position in a virtual coordinate system from depth information recorded as each pixel of the depth image.

  In the case of a shooting target with motion, the points of the 3D shape of the shooting target at each time by creating a time-series three-dimensional shape information image by shooting continuously with one or more sets of cameras and the depth sensor Group data can be created. When one moving image is created from time-series three-dimensional shape information images, this moving image is called a three-dimensional shape information moving image.

  By creating a three-dimensional shape information moving image, it is possible to transfer time-series three-dimensional point cloud data of a shooting target having motion as one moving image data file.

  By distributing or broadcasting three-dimensional shape information moving images using the Internet video distribution or television broadcasting existing infrastructure, video distribution or television recording of time-series three-dimensional shape information of a moving object to be shot is performed on the Internet To broadcast.

  Alternatively, instead of distributing or broadcasting three-dimensional shape information moving pictures, three-dimensional shape information images are sequentially created by shooting continuously with a camera and a depth sensor, and using the existing streaming distribution or television broadcasting infrastructure of the Internet Live on the Internet or live on television.

  A time-series three-dimensional shape information image recorded in each frame of a three-dimensional shape information moving image received as a moving image data file or received from the Internet or a television broadcast is acquired.

  One or more two-dimensional codes are extracted from the three-dimensional shape information image. Information on a camera, a depth sensor, a color image, and a depth image is read from each two-dimensional code, and one or more sets of color images and depth images are acquired from the three-dimensional shape information image.

  For one or more sets of depth images and two-dimensional codes, restore the three-dimensional coordinates from the depth of field angle read from the two-dimensional code, depth image resolution, and depth information recorded as pixels of depth images Do.

  From the angle of view of the camera and depth sensor read from the two-dimensional code and the resolution of the color image and depth image for one set or plural sets of color image, depth image, and two-dimensional code, Correspondence with the pixels is performed, the color of the point of each three-dimensional coordinate restored from the depth image is determined, and point cloud data of the three-dimensional shape of the imaging target is created.

  When multiple sets of color image, depth image and two-dimensional code exist, point group data is created with absolute coordinates in consideration of the installation position of the depth sensor read from the two-dimensional code, color image, depth image and two-dimensional The point cloud data of the three-dimensional shape of a part of the imaging object created from each set of codes is synthesized to create one point cloud data of the three-dimensional shape of a wide area of the imaging object.

  When three-dimensional shape information is received from the Internet, point cloud data created using an apparatus such as a PC is drawn by computer graphics to reproduce a three-dimensional shape to be photographed. The motion of the object to be photographed is reproduced by continuously drawing time-series point cloud data.

  By moving, rotating and scaling the reproduced image of the object to be photographed on the computer graphics according to the operation of the viewer by the input device such as a mouse or joystick connected to a device such as a PC, the viewer The original shape can be accurately understood.

  By drawing left- and right-eye computer graphics images and right-eye computer graphics images in consideration of left-eye and right-eye parallax, and viewing stereoscopically using a 3D-compatible display for PC or a 3D-compatible projector, the viewer can Three-dimensional shape can be intuitively grasped.

  By drawing the point cloud data of the three-dimensional shape of the object to be photographed by computer graphics by a method adapted to a screen system such as a head mounted display or CAVE (registered trademark), the received three-dimensional shape information is a content for virtual reality It can be used as

  In order to receive three-dimensional shape information from a television broadcast, an apparatus and a program for executing a function of reproducing point cloud data of a three-dimensional shape to be photographed by computer graphics are implemented in a television receiver.

  As described above, the method of the present invention is a moving third order that can be used as virtual reality content only by photographing with an easily available camera and depth sensor without knowledge of three-dimensional computer graphics. It is possible to create a series of images or moving images including information of time-series point cloud data of the original shape.

  By treating time-series point cloud data as images or moving pictures, Internet distribution or television broadcasting can be performed using existing infrastructure, and when drawing received point group data by computer graphics, movement, rotation By performing an operation such as enlargement or reduction or stereoscopic viewing, it is possible to reproduce a three-dimensional shape of a shooting target that is more accurate than conventional 3D moving image distribution.

  In addition, by implementing the method of the present invention in a television receiver, it becomes possible to realize true 3D television broadcasting in which the viewer can freely change the viewpoint and view the broadcast contents.

Explanatory drawing showing an example of implementation of the method of the present invention A diagram showing an example of a three-dimensional shape information image created by a pair of cameras and a depth sensor A diagram showing an example of a three-dimensional shape information image created by two sets of cameras and a depth sensor

  Hereinafter, an embodiment of the present invention will be described based on FIGS. 1 to 3.

  As shown in FIG. 1, the transmitting side of the three-dimensional shape information captures an image of the imaging target 1 with a camera 2 and a depth sensor 3 connected to an apparatus 6 such as a PC for executing a program implementing the method of the present invention. , Get a set of color image and depth image.

  In order to simplify the correspondence between each pixel of the depth image and the pixel of the color image, the camera and the depth sensor are placed at the same position. FIG. 1 shows an example using a peripheral device of a home game machine in which a camera and a depth sensor are integrated.

  When using a pair of camera and depth sensor, a program implementing the method of the present invention operating on an apparatus 6 such as a PC, as shown in FIG. 2, one three-dimensional shape including color image 19 and depth image 20 An information image 17 is created.

  Two encoding information of angle of view of camera and depth sensor used for shooting, resolution of color image and depth image, two-dimensional coordinate information indicating where color image and depth image are arranged in the three-dimensional shape information image The dimensional code 18 is placed in the margin of the three-dimensional shape information image.

  The two-dimensional code utilizes existing coding technology such as QR Code (registered trademark). In order to simplify the process of extracting the two-dimensional code from the three-dimensional shape information image on the receiving side, the specification is determined in advance such that the two-dimensional code is disposed at the upper left of the three-dimensional shape information image. In addition, when a three-dimensional shape information image is encoded and decoded for distribution or broadcast, the two-dimensional code is used on the receiving side in consideration that not all pixels of the image are necessarily restored as it is. In order to read correctly, the two-dimensional code is included in the three-dimensional shape information image in a state of being expanded to some extent according to the execution environment.

  The coordinates of the two-dimensional coordinates representing the position of the color image and the depth image are determined in advance such that the upper left corner of the three-dimensional shape information image is the origin, the rightward direction is + X axis, and the lower direction is + Y axis. The system information is recorded in a two-dimensional code and notified to the receiver.

  When one set of camera and depth sensor is used, color information and depth information of the blind spot of the camera and depth sensor can not be obtained, so only three-dimensional shape information of a part of the object to be captured can be acquired. It is possible to acquire three-dimensional shape information of a wide area of an object to be photographed by photographing from different positions with a camera and a depth sensor. In FIG. 1, a pair of a camera 2 and a depth sensor 3 and a pair of a camera 4 and a depth sensor 5 capture images. When the device 6 is one PC, two sets of camera and depth sensor are connected to one PC to obtain two sets of color image and depth image. When the device 6 is a set of two PCs, the two PCs can communicate by network connection, etc., and each PC is connected to a set of camera and depth sensor to set a set of color image and depth Images are acquired, and a total of two sets of color images and depth images acquired by each PC are collected by one PC to perform processing of creating a three-dimensional shape information image.

  When using multiple sets of cameras and depth sensors, one three-dimensional shape information image including multiple sets of color images, depth images, and two-dimensional code is created. Each set is assigned an ID for uniquely identifying the set. The ID of each group, the angle of view of the camera and depth sensor forming the group, the resolution of the color image and the depth image, and the information of two-dimensional coordinates in the three-dimensional shape information image are encoded into a two-dimensional code.

  When two sets of cameras and depth sensors are used, as shown in FIG. 3, the three-dimensional shape information image 21 includes a first set of color image 24, a depth image 25, a two-dimensional code 22, and a second set of color images 26, It includes a depth image 27 and a two-dimensional code 23.

  When multiple sets of cameras and depth sensors are used, installation of each depth sensor to create point cloud data of a three-dimensional shape of a wide area of the shooting target by combining point cloud data created from each set of information Location information is required. Define a virtual coordinate system whose origin is a point in the real space being photographed, measure the position and orientation of each depth sensor, and encode it as the installation position information of the depth sensor into the corresponding two-dimensional code .

  The virtual coordinate system for representing the installation position of each depth sensor is specified in advance, such as the + Y axis is upward and the coordinate value is in meters, or the coordinate system information to be used is a two-dimensional code Record to and notify the receiver.

  In the case of a moving target, a series of three-dimensional shape information images are created by photographing continuously with one or more sets of cameras and a depth sensor, and one three-dimensional shape is generated from a series of three-dimensional shape information images. Create information videos. When reading a two-dimensional code from a three-dimensional shape information image recorded in each frame of a three-dimensional shape information moving image, a series of image processing for extracting a two-dimensional code may not be performed in every frame. It is desirable to predetermine that the position of the two-dimensional code of the three-dimensional shape information image is invariant.

  Although 3D shape information animation contains information necessary to create time-series point cloud data, it can be treated as a normal 2D animation created from a series of 2D images, so existing infrastructure and technology Can be used. When storing as a moving image data file, it is possible to select the MPEG-4 standard widely used as a format for moving picture distribution on the Internet, and the MPEG-2 standard used in digital television broadcasting. Using this video data file, video is distributed over the Internet or recorded on TV.

  Instead of creating a 3D shape information animation, 3D shape information images can be created sequentially by shooting continuously with one or more sets of cameras and a depth sensor, and live distribution on the Internet or live broadcast on a television . Although the three-dimensional shape information image contains the information necessary to create point cloud data, it can be handled as a normal two-dimensional image, so select the RTMP protocol widely used as a streaming distribution protocol on the Internet be able to.

  As shown in FIG. 1, the receiving side of the three-dimensional shape information receives the three-dimensional shape information moving image distributed by Internet or television broadcast by the device 9 such as a PC or the television receiver 8 and recorded in each frame of the moving image. Acquire a time-series three-dimensional shape information image.

  Hereinafter, a method for using a device such as a PC connected to the Internet on the receiving side will be described.

  One or more two-dimensional codes are extracted from the three-dimensional shape information image using image processing techniques. The resolution of the color image and the depth image and the position in the three-dimensional shape information image are read from each two-dimensional code, and a set of one or more color images, a depth image, and a two-dimensional code is acquired.

  When a plurality of two-dimensional codes exist, an ID capable of uniquely identifying a set is read from each two-dimensional code, and the ID, the camera forming the set, the depth sensor, the color image, and the depth image are associated.

  The angle of view of the depth sensor is read from the two-dimensional code, and the three-dimensional coordinates are restored from the resolution of the depth image and the depth information recorded as each pixel of the depth image. The restored three-dimensional coordinates are relative coordinates from the depth sensor.

  When a plurality of depth sensors are used, the installation position information of the depth sensor read from each two-dimensional code is applied to restore the three-dimensional coordinates of the absolute coordinates from the depth image.

  The pixel of the color image corresponding to each pixel of the depth image is determined from the angle of view of the depth sensor and the resolution of the depth image, the angle of view of the camera read from the two-dimensional code and the resolution of the color image, and restored from each pixel of the depth image Determine the color of the point in the 3D coordinates. If the angle of view between the camera and the depth sensor is different, there will be pixels that are only captured by either the camera or the depth sensor, but such pixels have either unknown three-dimensional coordinates or color Since the point cloud data can not be restored as a point that forms the point group data, it is excluded from the processing target as an invalid pixel. The three-dimensional coordinates and colors of all valid pixels included in the depth image are determined to create point cloud data of a three-dimensional shape to be photographed.

  Draw point cloud data with computer graphics. Depending on the ability of the device that performs computer graphics processing, point cloud data may be drawn as points, or surfaces called polygons may be created from a plurality of adjacent points so that no gap occurs.

  Even when a normal PC display is used as a display device, shooting is performed by drawing, moving, rotating, or scaling point cloud data in accordance with the operation of the viewer using an input device such as a mouse or joystick. The reproduced image of the object can be viewed from various directions, and the three-dimensional shape of the object to be photographed can be grasped more accurately than the video delivered by the conventional Internet distribution.

  When the 3D-compatible display 10 for PC or the 3D-compatible projector is used as a display device, stereoscopic viewing is performed by drawing left- and right-eye computer graphics images in consideration of left-eye and right-eye parallax. Stereoscopic viewing allows the viewer to intuitively grasp the three-dimensional shape of the object to be photographed.

  When using a virtual reality system such as a head mounted display 11 with a head tracking function implemented as the display device or the screen system 12 such as CAVE (registered trademark), point cloud data is generated from a three dimensional shape information image or a three dimensional shape information animation. By implementing the acquisition function in a virtual reality system, it is possible to use point cloud data of a three-dimensional shape to be captured as virtual reality content.

  Hereinafter, the method in the case of using the television receiver which receives television broadcasting is demonstrated.

  In order to use a television receiver as a display device for receiving a television broadcast and for displaying a reproduced image of an object to be photographed, an apparatus and a program for performing the method of the present invention are implemented on the television receiver and received three-dimensional shape information It is necessary to enable computer graphics to play back time-series point cloud data created from videos.

  In the case of 3D television, it is also necessary to implement the function of stereoscopic vision with computer graphics images for the left eye and computer graphics for the right eye.

  In addition, it is desirable that a function of moving, rotating, and scaling a reproduced image by a viewer's operation using a remote control device of a television or an input device such as a mouse or a joystick is implemented.

1 shooting target 2 first set of cameras 3 first set of depth sensors 4 second set of cameras 5 second set of depth sensors 6 control of camera and depth sensor, creation of 3D shape information image and 3D shape information animation , Equipment and devices such as PCs for Internet distribution or TV broadcasting, and programs and equipment 7 Internet or TV radio 8 Creation of 3D shape point cloud data for shooting from received 3D shape information animation, computer graphic A television receiver equipped with a function to play back point cloud data by computer 9 Internet connection, creation of point cloud data of 3D shape of shooting object from received 3D shape information animation, point cloud data by computer graphics A device or a group of devices such as a PC for performing reproduction and a program 10 PC 3D compatible display 11 Head mount display 12 CAVE (registered trademark) system 13 Reproduction image of the object to be photographed displayed on the television 14 Reproduction image for stereoscopic vision of the object to be photographed displayed on the 3D correspondence display for PC 15 Display on the head mount display Right-eye reconstructed image of the captured object 16 Reproduced image of the captured object displayed as virtual reality content on the CAVE® system 17 Three-dimensional shape information image created using a pair of cameras and a depth sensor 18 Camera and depth sensor and two-dimensional code encoding color and depth image information 19 Color image captured with camera 20 Depth image captured with depth sensor 21 3D created using two sets of camera and depth sensor Shape information image 22 First camera and depth Two-dimensional code encoding sensor and information of color image and depth image 23 Second camera and depth sensor and two-dimensional code encoding information of color image and depth image 24 color captured by first set camera Image 25 Depth image taken with the first set of depth sensors 26 Color image taken with the second set of cameras 27 Depth image taken with the second set of depth sensors

Claims (9)

Color image taken with a camera, depth image taken with a depth sensor, angle of view of the camera and the depth sensor, color image, resolution of the depth image and where in the three-dimensional shape information image A method of creating one three-dimensional shape information image in which information necessary for creating point group data of a three-dimensional shape to be imaged is recorded from a two-dimensional code obtained by encoding information of two-dimensional coordinates representing it. The plurality of sets of cameras and depth sensors are used to capture a portion of the object to be photographed which is not photographed as a blind spot when using the pair of cameras and the depth sensor. Acquiring a color image and a depth image, comprising a plurality of sets of the color image, the depth image, and the two-dimensional code, and recording positions of the camera and the depth sensor in the two-dimensional code ; A method of creating one three-dimensional shape information image.   The one or more sets of cameras and the depth sensor sequentially capture a moving imaging target to obtain the one or more sets of color images and depth images in time series, Item 3. Information necessary for creation of time-series three-dimensional shape point cloud data of the motion-captured object from the time-series three-dimensional shape information image of the motion-captured object created by the method according to item 2 How to create one recorded 3D shape information animation.   The set of color image, depth image and two-dimensional code are acquired from the three-dimensional shape information image created by the method according to claim 1, and the information encoded in the two-dimensional code and the depth image A method of creating point cloud data of the three-dimensional shape of the object to be photographed from three-dimensional coordinates restored from depth information recorded as pixels of and the color information recorded as pixels of the color image.   The plurality of sets of color image, depth image and two-dimensional code are acquired from the three-dimensional shape information image created by the method according to claim 2, and each set of color image, depth image and two-dimensional code is obtained. A method of performing the method according to claim 4 to create point cloud data of the three-dimensional shape of the object to be photographed.   The method according to claim 4 or claim 5 is performed on each of the three-dimensional shape information images recorded in each frame of the three-dimensional shape information moving image created by the method according to claim 3. To create time-series point cloud data of a shooting target.   The method according to claim 6, wherein the 3D shape information moving image created by the method according to claim 3 is video distributed over the Internet or recorded and broadcasted over the television, and the 3D shape information moving image received from the Internet or television broadcast A method of creating point cloud data of a series and reproducing the three-dimensional shape of the time series of the moving object to be photographed by computer graphics.   In Claim 7, instead of recording and broadcasting the three-dimensional shape information animation on the Internet or video recording on television, the method according to Claim 1 or 2 is continuously carried out, and A method of sequentially creating the three-dimensional shape information image and live broadcasting on the Internet or live broadcasting on television.   A method for reproducing the point cloud data created from the three-dimensional shape information moving picture received from the television broadcast according to claim 7 and claim 8 according to claim 4 and claim 5 and claim 6 using computer graphics. A television receiver with equipment and programs to run.

Images