EP2483750B1 - Selecting viewpoints for generating additional views in 3d video - Google Patents
Selecting viewpoints for generating additional views in 3d video Download PDFInfo
- Publication number
- EP2483750B1 EP2483750B1 EP10765509.4A EP10765509A EP2483750B1 EP 2483750 B1 EP2483750 B1 EP 2483750B1 EP 10765509 A EP10765509 A EP 10765509A EP 2483750 B1 EP2483750 B1 EP 2483750B1
- Authority
- EP
- European Patent Office
- Prior art keywords
- viewpoint
- rendering
- image
- video data
- data signal
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000009877 rendering Methods 0.000 claims description 71
- 238000000034 method Methods 0.000 claims description 31
- 238000004590 computer program Methods 0.000 claims description 8
- 230000001419 dependent effect Effects 0.000 claims description 2
- 238000010586 diagram Methods 0.000 description 8
- 238000013459 approach Methods 0.000 description 4
- 238000013461 design Methods 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 230000005540 biological transmission Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 230000021615 conjugation Effects 0.000 description 1
- 230000002349 favourable effect Effects 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
- H04N13/106—Processing image signals
- H04N13/111—Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
- H04N13/106—Processing image signals
- H04N13/161—Encoding, multiplexing or demultiplexing different image signal components
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
- H04N13/106—Processing image signals
- H04N13/128—Adjusting depth or disparity
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/20—Image signal generators
- H04N13/271—Image signal generators wherein the generated image signals comprise depth maps or disparity maps
Definitions
- This invention relates to a method of encoding a video data signal, the method comprising providing at least a first image of a scene as seen from a first viewpoint, providing rendering information for enabling the generation of at least one rendered image of the scene as seen from a rendering viewpoint and generating the video data signal comprising encoded data representing the first image and the rendering information.
- This invention further relates to a method of decoding the video data signal, an encoder, a decoder, computer program products for encoding or decoding, the video data signal and a digital data carrier.
- 3D video various methods exist for encoding a third dimension into the video data signal. Generally this is done by providing a viewer's eye with different views of the scene being watched.
- a popular approach for representing 3D video is to use one or more two-dimensional (2D) images plus a depth representation providing information of the third dimension. This approach also allows 2D images to be generated with different viewpoints and viewing angles than the 2D images which are included in the 3D video signal.
- Such an approach provides a number of advantages including allowing further views to be generated with relatively low complexity and providing an efficient data representation thereby reducing, e.g., storage and communication resource requirements for 3D video signals.
- the video data is extended with data that is not visible from the available viewpoints, but becomes visible from a slightly different viewpoint.
- This data is referred to as occlusion or background data.
- the occlusion data is generated from multiview data obtained by capturing a scene with multiple cameras at different viewpoints.
- JP 2006 041811A An example of a three dimensional video encoding system supporting view point shifting is provided in JP 2006 041811A .
- this object is achieved by providing a method of encoding a video data signal in accordance with claim 1.
- the quality of images generated from different viewing points is related to the availability of data needed for reconstruction of de-occluded objects.
- data is available for shifting the view point to the left, but not for shifting the view point to the right. Consequently, shifting the viewpoint to the left may lead to a generated image with a different quality than shifting the viewpoint to the right.
- a similar difference in quality may occur when insufficient de-occlusion information or no occlusion information is available for filling in de-occluded areas.
- the de-occluded areas maybe filled in using so-called hole-filling algorithms.
- hole-filling algorithms interpolate information from the direct vicinity of the de-occluded area. Consequently, shifting the viewpoint to the left may lead to a generated image with a different quality than shifting the viewpoint to the right.
- Such differences in quality are not only influenced by the availability of required data, but also by the size and the nature of the surface area being de-occluded when shifting the viewpoint.
- the quality of a 3D video or image may vary according to the newly selected viewpoint. It could, e.g., matter whether a new view is generated at the left side or at the right side of the viewpoint of the already available first image.
- the preferred direction indicator defines a preferred orientation of the rendering viewpoint for an additional view relative to the original viewpoint of the image already included in the video signal.
- the preferred direction indicator can be used for selecting the rendering viewpoint and generating the rendered image of the scene from the selected viewpoint.
- the video data signal may comprise a preferred direction indicator for every frame, every group of frames, every scene, or even for an entire video sequence. Encoding such information on a per frame basis rather than on a coarser granularity enables random access; e.g. for supporting trick play. As often the preferred rendering indicator is constant for various frames and the size of the encoded video signal is typically relevant, the amount of duplicate indicators may be reduced by instead encoding the information on group of frame basis. An even more efficient encoding is to encode the preferred direction on a per scene basis as preferably the preferred direction of rendering is kept the same throughout a scene, thereby warranting continuity within a scene.
- the size of the encoded signal is less critical information may be encoded on frame, group of frame and scene level alike, as long as all indicators are set in conformity with one another.
- the preferred direction indicator is preferably kept constant for several frames. It may e.g. be kept constant throughout a group of frames or alternatively throughout a scene. It is noted that even when the preferred direction indicator is kept constant for several frames, it may still be advantageous to encode the constant preferred direction indicator on a smaller granularity than strictly necessary to facilitate random access.
- the preferred orientation may be left, right, up, down or any combination of those directions.
- a preferred distance or preferred maximum distance may be provided together with the preferred direction indication. If sufficient information about occluded objects and/or depth values of objects in the first image is available, it may be possible to generate multiple high quality additional views from viewpoints which may be further away from the original viewpoint.
- the rendering information may, e.g., comprise occlusion data representing background objects being occluded by foreground objects in the first image, a depth map providing depth values of objects in the first image or transparency data for these objects.
- the preferred direction indicator indicates for which possible rendering viewpoint the best rendering information is available. For more information relating to the rendering of layered depth images, see e.g. International Application WO2007/063477 . For more information regarding hole filling algorithms for use in filling de-occluded regions when rendering layered depth images, see e.g. WO2007/099465 .
- a method of decoding a video data signal in accordance with claim 8 is provided.
- Figure 1 shows a block diagram of a system for encoding video data according to the invention.
- the system comprises two digital video cameras 11, 12 and an encoder 10.
- the first camera 11 and the second camera 12 record the same scene 100, but from a slightly different position and thus also from a slightly different angle.
- the recorded digital video signals from both video cameras 11, 12 are sent to the encoder 10.
- the encoder may, e.g., be part of a dedicated encoding box, a video card in a computer or a software implemented function to be executed by a general purpose microprocessor.
- the video cameras 11, 12 are analog video cameras and the analog video signals are converted to digital video signals before they are provided as input for the encoder 10. If the video cameras are coupled to the encoder 10, the encoding may take place during the recording of the scene 100. It's also possible to record the scene 100 first and to provide the recorded video data to the encoder 10 later.
- the encoder 10 receives the digital video data from the video cameras 11, 12, either directly or indirectly, and combines both digital video signals into one 3D video signal 15. It is to be noted that both video cameras 11, 12 may be combined in one 3D video camera. It is also possible to use more than two video cameras to capture the scene 100 from more than two viewpoints.
- FIG. 2 shows a flow diagram of an encoding method according to the invention.
- This encoding method may be performed by the encoder 10 of the system of figure 1 .
- the encoding method uses the recorded digital video data from the cameras 11, 12 and provides a video data signal 15 according to the invention.
- base image providing step 21 at least a first image of the scene is provided for inclusion in the video data signal 15.
- This base image may be standard 2D video data coming from one of the two cameras 11, 12.
- the encoder 10 may also use two base images; one from the first camera 11 and one from the second 12. From the base images, color values of all pixels in each frame of the recorded video can be derived.
- the base images represent the scene at a certain moment in time as seen from a specific viewpoint. In the following, this specific viewpoint will be called the base viewpoint.
- the incoming video data from the video cameras 11, 12 is used to add information to the base image.
- This added information must enable a decoder to generate a rendered image of the same scene from a different viewpoint.
- this added information is called rendering information.
- the rendering information may, e.g., comprise depth information or transparency values of objects in the base image.
- the rendering information may also describe objects that are blocked from being viewed from the base viewpoint by objects visible in the base image.
- the encoder uses known, preferably standardized, methods for deriving this rendering information from the recorded regular video data.
- the encoder 10 further adds a preferred direction indicator to the rendering information.
- the preferred direction indicator defines a preferred orientation of the rendering viewpoint for an additional view relative to the base viewpoint.
- the preferred direction indicator can be used for selecting the rendering viewpoint and generating the rendered image of the scene 100 from the selected viewpoint.
- the quality of a 3D video or image may vary according to the newly selected viewpoint. It could, e.g., matter whether a new view is generated at the left side or at the right side of the viewpoint of the already available first image.
- the preferred direction indicator which is added to the rendering information may, e.g., be a single bit indicating a left or right direction.
- a more advanced preferred direction indicator may also indicate an up or down direction and/or a preferred or maximum distance of the new viewpoint relative to the base viewpoint.
- the preferred direction indicator may provide a preferred orientation and/or distance of multiple rendering viewpoints relative to the first viewpoint. For example, it may be preferred to generate two rendering viewpoints at the same side of the first viewpoint or one at each side.
- the preferred position(s) of the rendering viewpoint(s) relative to the first viewpoint may depend on the distance between those two points. It may, e.g., be better to render an image from a viewpoint at the left side of the first viewpoint when both viewpoints are close together; while for larger distances between both viewpoints to the left side of the first viewpoint may be more suitable for generating the rendering viewpoint.
- the decision as to which particular direction is more favorable may be determined automatically; e.g. when encoding a stereo pair as a layered depth image, using an image, depth and occlusion representation, it is possible to use either the left or right image as the image of the layered depth image, and to reconstruct the other image based thereon. Subsequently a difference metric can be computed for both alternatives and the preferred encoding and therewith direction can be determined based thereon.
- the differences are weighted based on a model of the human visual perceptive system.
- the preferred direction could be selected based on user-interaction.
- the information provided in the previous steps 21, 22, 23 is used for generating a video data signal 15 according to the invention.
- the video data signal 15 at least represents the first image, the rendering information and the preferred direction indicator.
- a preferred direction indicator may be provided for each frame, for a group of frames or for a complete scene or even a complete video. Changing the position of the rendering viewpoint during a scene may have a negative influence on the perceived 3D image quality, but may on the other hand be necessary if the availability of rendering information for different directions changes considerably.
- FIG. 3 shows a block diagram of a system for decoding video data according to the invention.
- the system comprises a decoder 30 for receiving the video data signal 15 and converting the video data signal 15 into a display signal which is suitable for being displayed by a display 31.
- the video data signal 15 may reach the decoder 30 as part of a broadcasted signal, e.g., via cable or satellite transmission.
- the video data signal 15 may also be provided on request, e.g., via the Internet or via a video-on-demand service.
- the video data signal 15 is provided on a digital data carrier, such as a DVD or Blu-ray disc.
- the display 31 is capable of providing a 3D presentation of the scene 100 that was captured and encoded by the encoder 10 of the system of figure 1 .
- the display 31 may comprise the decoder 30 or may be coupled to the decoder 30.
- the decoder 30 may be part of a 3D video receiver which is to be coupled to one or more normal television or computer displays.
- the display is a dedicated 3D display 31 capable of providing different views to different eyes of a viewer.
- Figure 4 shows a flow diagram of a decoding method as it may be performed by the decoder 30 of figure 3 .
- the video data signal 15 encoded by the encoder 10 is received at an input of the decoder 30.
- the received video data signal 15 comprises encoded data representing at least the first image of the scene 100, rendering information and the preferred direction indicator.
- additional viewpoint selection step 42 the preferred direction indicator is used for selecting at least one additional viewpoint for a respective additional view.
- additional view rendering step 43 one or more additional views from the selected viewpoint or viewpoints are generated.
- display step 44 two or more views from different viewpoints may then be provided to the display 31 for showing the scene 100 in 3D.
- the invention also extends to computer programs, particularly computer programs on or in a carrier, adapted for putting the invention into practice.
- the program may be in the form of source code, object code, a code intermediate source and object code such as partially compiled form, or in any other form suitable for use in the implementation of the method according to the invention.
- a program may have many different architectural designs.
- a program code implementing the functionality of the method or system according to the invention may be subdivided into one or more subroutines. Many different ways to distribute the functionality among these subroutines will be apparent to the skilled person.
- the subroutines may be stored together in one executable file to form a self-contained program.
- Such an executable file may comprise computer executable instructions, for example processor instructions and/or interpreter instructions (e.g. Java interpreter instructions).
- one or more or all of the subroutines may be stored in at least one external library file and linked with a main program either statically or dynamically, e.g. at run-time.
- the main program contains at least one call to at least one of the subroutines.
- the subroutines may comprise function calls to each other.
- An embodiment relating to a computer program product comprises computer executable instructions corresponding to each of the processing steps of at least one of the methods set forth. These instructions may be subdivided into subroutines and/or be stored in one or more files that may be linked statically or dynamically.
- Another embodiment relating to a computer program product comprises computer executable instructions corresponding to each of the means of at least one of the systems and/or products set forth. These instructions may be subdivided into subroutines and/or be stored in one or more files that may be linked statically or dynamically.
- the carrier of a computer program may be any entity or device capable of carrying the program.
- the carrier may include a storage medium, such as a ROM, for example a CD ROM or a semiconductor ROM, or a magnetic recording medium, for example a floppy disc or hard disk.
- the carrier may be a transmissible carrier such as an electrical or optical signal, which may be conveyed via electrical or optical cable or by radio or other means.
- the carrier may be constituted by such cable or other device or means.
- the carrier may be an integrated circuit in which the program is embedded, the integrated circuit being adapted for performing, or for use in the performance of, the relevant method.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
EP10765509.4A EP2483750B1 (en) | 2009-10-02 | 2010-09-21 | Selecting viewpoints for generating additional views in 3d video |
Applications Claiming Priority (3)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
EP09172096 | 2009-10-02 | ||
EP10765509.4A EP2483750B1 (en) | 2009-10-02 | 2010-09-21 | Selecting viewpoints for generating additional views in 3d video |
PCT/IB2010/054251 WO2011039679A1 (en) | 2009-10-02 | 2010-09-21 | Selecting viewpoints for generating additional views in 3d video |
Publications (2)
Publication Number | Publication Date |
---|---|
EP2483750A1 EP2483750A1 (en) | 2012-08-08 |
EP2483750B1 true EP2483750B1 (en) | 2018-09-12 |
Family
ID=43129795
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
EP10765509.4A Active EP2483750B1 (en) | 2009-10-02 | 2010-09-21 | Selecting viewpoints for generating additional views in 3d video |
Country Status (8)
Country | Link |
---|---|
US (1) | US9167226B2 (ko) |
EP (1) | EP2483750B1 (ko) |
JP (1) | JP5859441B2 (ko) |
KR (1) | KR101727094B1 (ko) |
CN (1) | CN102549507B (ko) |
BR (1) | BR112012007115A2 (ko) |
RU (1) | RU2551789C2 (ko) |
WO (1) | WO2011039679A1 (ko) |
Families Citing this family (70)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
GB2438783B8 (en) * | 2005-03-16 | 2011-12-28 | Lucasfilm Entertainment Co Ltd | Three-dimensional motion capture |
US8866920B2 (en) | 2008-05-20 | 2014-10-21 | Pelican Imaging Corporation | Capturing and processing of images using monolithic camera array with heterogeneous imagers |
US11792538B2 (en) | 2008-05-20 | 2023-10-17 | Adeia Imaging Llc | Capturing and processing of images including occlusions focused on an image sensor by a lens stack array |
EP4336447A1 (en) | 2008-05-20 | 2024-03-13 | FotoNation Limited | Capturing and processing of images using monolithic camera array with heterogeneous imagers |
EP2502115A4 (en) | 2009-11-20 | 2013-11-06 | Pelican Imaging Corp | RECORDING AND PROCESSING IMAGES THROUGH A MONOLITHIC CAMERA ARRAY WITH HETEROGENIC IMAGE CONVERTER |
US20120012748A1 (en) | 2010-05-12 | 2012-01-19 | Pelican Imaging Corporation | Architectures for imager arrays and array cameras |
US8878950B2 (en) | 2010-12-14 | 2014-11-04 | Pelican Imaging Corporation | Systems and methods for synthesizing high resolution images using super-resolution processes |
CN107404609B (zh) | 2011-05-11 | 2020-02-11 | 快图有限公司 | 用于传送阵列照相机图像数据的方法 |
WO2013043751A1 (en) | 2011-09-19 | 2013-03-28 | Pelican Imaging Corporation | Systems and methods for controlling aliasing in images captured by an array camera for use in super resolution processing using pixel apertures |
CN104081414B (zh) | 2011-09-28 | 2017-08-01 | Fotonation开曼有限公司 | 用于编码和解码光场图像文件的系统及方法 |
EP2817955B1 (en) | 2012-02-21 | 2018-04-11 | FotoNation Cayman Limited | Systems and methods for the manipulation of captured light field image data |
US9210392B2 (en) | 2012-05-01 | 2015-12-08 | Pelican Imaging Coporation | Camera modules patterned with pi filter groups |
WO2014005123A1 (en) | 2012-06-28 | 2014-01-03 | Pelican Imaging Corporation | Systems and methods for detecting defective camera arrays, optic arrays, and sensors |
US20140002674A1 (en) | 2012-06-30 | 2014-01-02 | Pelican Imaging Corporation | Systems and Methods for Manufacturing Camera Modules Using Active Alignment of Lens Stack Arrays and Sensors |
EP3869797B1 (en) | 2012-08-21 | 2023-07-19 | Adeia Imaging LLC | Method for depth detection in images captured using array cameras |
WO2014032020A2 (en) | 2012-08-23 | 2014-02-27 | Pelican Imaging Corporation | Feature based high resolution motion estimation from low resolution images captured using an array source |
CZ308335B6 (cs) * | 2012-08-29 | 2020-05-27 | Awe Spol. S R.O. | Způsob popisu bodů předmětů předmětového prostoru a zapojení k jeho provádění |
CN104685860A (zh) | 2012-09-28 | 2015-06-03 | 派力肯影像公司 | 利用虚拟视点从光场生成图像 |
WO2014078443A1 (en) | 2012-11-13 | 2014-05-22 | Pelican Imaging Corporation | Systems and methods for array camera focal plane control |
WO2014122553A1 (en) | 2013-02-06 | 2014-08-14 | Koninklijke Philips N.V. | Method of encoding a video data signal for use with a multi-view stereoscopic display device |
US9462164B2 (en) | 2013-02-21 | 2016-10-04 | Pelican Imaging Corporation | Systems and methods for generating compressed light field representation data using captured light fields, array geometry, and parallax information |
WO2014133974A1 (en) | 2013-02-24 | 2014-09-04 | Pelican Imaging Corporation | Thin form computational and modular array cameras |
US9917998B2 (en) | 2013-03-08 | 2018-03-13 | Fotonation Cayman Limited | Systems and methods for measuring scene information while capturing images using array cameras |
US8866912B2 (en) | 2013-03-10 | 2014-10-21 | Pelican Imaging Corporation | System and methods for calibration of an array camera using a single captured image |
US9888194B2 (en) | 2013-03-13 | 2018-02-06 | Fotonation Cayman Limited | Array camera architecture implementing quantum film image sensors |
US9106784B2 (en) | 2013-03-13 | 2015-08-11 | Pelican Imaging Corporation | Systems and methods for controlling aliasing in images captured by an array camera for use in super-resolution processing |
WO2014164550A2 (en) | 2013-03-13 | 2014-10-09 | Pelican Imaging Corporation | System and methods for calibration of an array camera |
US9519972B2 (en) | 2013-03-13 | 2016-12-13 | Kip Peli P1 Lp | Systems and methods for synthesizing images from image data captured by an array camera using restricted depth of field depth maps in which depth estimation precision varies |
US9100586B2 (en) | 2013-03-14 | 2015-08-04 | Pelican Imaging Corporation | Systems and methods for photometric normalization in array cameras |
WO2014159779A1 (en) | 2013-03-14 | 2014-10-02 | Pelican Imaging Corporation | Systems and methods for reducing motion blur in images or video in ultra low light with array cameras |
WO2014145722A2 (en) * | 2013-03-15 | 2014-09-18 | Digimarc Corporation | Cooperative photography |
US9497429B2 (en) | 2013-03-15 | 2016-11-15 | Pelican Imaging Corporation | Extended color processing on pelican array cameras |
US9445003B1 (en) | 2013-03-15 | 2016-09-13 | Pelican Imaging Corporation | Systems and methods for synthesizing high resolution images using image deconvolution based on motion and depth information |
US10122993B2 (en) | 2013-03-15 | 2018-11-06 | Fotonation Limited | Autofocus system for a conventional camera that uses depth information from an array camera |
US9438888B2 (en) | 2013-03-15 | 2016-09-06 | Pelican Imaging Corporation | Systems and methods for stereo imaging with camera arrays |
ITRM20130244A1 (it) * | 2013-04-23 | 2014-10-25 | MAIOR Srl | Metodo per la riproduzione di un film |
RU2667605C2 (ru) | 2013-05-10 | 2018-09-21 | Конинклейке Филипс Н.В. | Способ кодирования сигнала видеоданных для использования с многовидовым устройством визуализации |
WO2015048694A2 (en) * | 2013-09-27 | 2015-04-02 | Pelican Imaging Corporation | Systems and methods for depth-assisted perspective distortion correction |
US9426343B2 (en) | 2013-11-07 | 2016-08-23 | Pelican Imaging Corporation | Array cameras incorporating independently aligned lens stacks |
US9578074B2 (en) | 2013-11-11 | 2017-02-21 | Amazon Technologies, Inc. | Adaptive content transmission |
US9634942B2 (en) | 2013-11-11 | 2017-04-25 | Amazon Technologies, Inc. | Adaptive scene complexity based on service quality |
US9641592B2 (en) | 2013-11-11 | 2017-05-02 | Amazon Technologies, Inc. | Location of actor resources |
US9596280B2 (en) | 2013-11-11 | 2017-03-14 | Amazon Technologies, Inc. | Multiple stream content presentation |
US9582904B2 (en) | 2013-11-11 | 2017-02-28 | Amazon Technologies, Inc. | Image composition based on remote object data |
US9805479B2 (en) | 2013-11-11 | 2017-10-31 | Amazon Technologies, Inc. | Session idle optimization for streaming server |
US9604139B2 (en) | 2013-11-11 | 2017-03-28 | Amazon Technologies, Inc. | Service for generating graphics object data |
US10119808B2 (en) | 2013-11-18 | 2018-11-06 | Fotonation Limited | Systems and methods for estimating depth from projected texture using camera arrays |
EP3075140B1 (en) | 2013-11-26 | 2018-06-13 | FotoNation Cayman Limited | Array camera configurations incorporating multiple constituent array cameras |
US10089740B2 (en) | 2014-03-07 | 2018-10-02 | Fotonation Limited | System and methods for depth regularization and semiautomatic interactive matting using RGB-D images |
EP2960864B1 (en) * | 2014-06-23 | 2018-12-05 | Harman Becker Automotive Systems GmbH | Device and method for processing a stream of video data |
CN113256730B (zh) | 2014-09-29 | 2023-09-05 | 快图有限公司 | 用于阵列相机的动态校准的系统和方法 |
US9942474B2 (en) | 2015-04-17 | 2018-04-10 | Fotonation Cayman Limited | Systems and methods for performing high speed video capture and depth estimation using array cameras |
US10204449B2 (en) * | 2015-09-01 | 2019-02-12 | Siemens Healthcare Gmbh | Video-based interactive viewing along a path in medical imaging |
EP3273686A1 (en) * | 2016-07-21 | 2018-01-24 | Thomson Licensing | A method for generating layered depth data of a scene |
US10353946B2 (en) * | 2017-01-18 | 2019-07-16 | Fyusion, Inc. | Client-server communication for live search using multi-view digital media representations |
CN110546688B (zh) * | 2017-05-30 | 2023-11-17 | 索尼公司 | 图像处理装置和方法、文件生成装置和方法及程序 |
US10482618B2 (en) | 2017-08-21 | 2019-11-19 | Fotonation Limited | Systems and methods for hybrid depth regularization |
ES2881320T3 (es) * | 2017-12-14 | 2021-11-29 | Canon Kk | Dispositivo de generación, procedimiento de generación y programa para modelo tridimensional |
KR102278848B1 (ko) * | 2018-07-31 | 2021-07-19 | 엘지전자 주식회사 | 다중 뷰포인트 기반 360 비디오 처리 방법 및 그 장치 |
CN112335258A (zh) * | 2018-11-12 | 2021-02-05 | 英特尔公司 | 从比赛参与者视角的自动视野估计 |
JP7273250B2 (ja) | 2019-09-17 | 2023-05-12 | ボストン ポーラリメトリックス,インコーポレイティド | 偏光キューを用いた面モデリングのためのシステム及び方法 |
DE112020004813B4 (de) | 2019-10-07 | 2023-02-09 | Boston Polarimetrics, Inc. | System zur Erweiterung von Sensorsystemen und Bildgebungssystemen mit Polarisation |
IT201900021645A1 (it) * | 2019-11-19 | 2021-05-19 | Sacmi | Apparato per l’ispezione ottica di sanitari |
KR20230116068A (ko) | 2019-11-30 | 2023-08-03 | 보스턴 폴라리메트릭스, 인크. | 편광 신호를 이용한 투명 물체 분할을 위한 시스템및 방법 |
US11195303B2 (en) | 2020-01-29 | 2021-12-07 | Boston Polarimetrics, Inc. | Systems and methods for characterizing object pose detection and measurement systems |
KR20220133973A (ko) | 2020-01-30 | 2022-10-05 | 인트린식 이노베이션 엘엘씨 | 편광된 이미지들을 포함하는 상이한 이미징 양식들에 대해 통계적 모델들을 훈련하기 위해 데이터를 합성하기 위한 시스템들 및 방법들 |
WO2021243088A1 (en) | 2020-05-27 | 2021-12-02 | Boston Polarimetrics, Inc. | Multi-aperture polarization optical systems using beam splitters |
US11954886B2 (en) | 2021-04-15 | 2024-04-09 | Intrinsic Innovation Llc | Systems and methods for six-degree of freedom pose estimation of deformable objects |
US11290658B1 (en) | 2021-04-15 | 2022-03-29 | Boston Polarimetrics, Inc. | Systems and methods for camera exposure control |
US11689813B2 (en) | 2021-07-01 | 2023-06-27 | Intrinsic Innovation Llc | Systems and methods for high dynamic range imaging using crossed polarizers |
Citations (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2006041811A (ja) * | 2004-07-26 | 2006-02-09 | Kddi Corp | 自由視点画像ストリーミング方式 |
Family Cites Families (24)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US6327381B1 (en) * | 1994-12-29 | 2001-12-04 | Worldscape, Llc | Image transformation and synthesis methods |
US6088006A (en) * | 1995-12-20 | 2000-07-11 | Olympus Optical Co., Ltd. | Stereoscopic image generating system for substantially matching visual range with vergence distance |
US6222551B1 (en) * | 1999-01-13 | 2001-04-24 | International Business Machines Corporation | Methods and apparatus for providing 3D viewpoint selection in a server/client arrangement |
RU2237283C2 (ru) * | 2001-11-27 | 2004-09-27 | Самсунг Электроникс Ко., Лтд. | Устройство и способ представления трехмерного объекта на основе изображений с глубиной |
JP4174001B2 (ja) * | 2002-09-27 | 2008-10-29 | シャープ株式会社 | 立体画像表示装置、記録方法、及び伝送方法 |
EP1431919B1 (en) * | 2002-12-05 | 2010-03-03 | Samsung Electronics Co., Ltd. | Method and apparatus for encoding and decoding three-dimensional object data by using octrees |
WO2004071102A1 (ja) * | 2003-01-20 | 2004-08-19 | Sanyo Electric Co,. Ltd. | 立体視用映像提供方法及び立体映像表示装置 |
JP4324435B2 (ja) * | 2003-04-18 | 2009-09-02 | 三洋電機株式会社 | 立体視用映像提供方法及び立体映像表示装置 |
US20060200744A1 (en) * | 2003-12-08 | 2006-09-07 | Adrian Bourke | Distributing and displaying still photos in a multimedia distribution system |
GB0415223D0 (en) | 2004-07-07 | 2004-08-11 | Sensornet Ltd | Intervention rod |
EP1686554A3 (en) * | 2005-01-31 | 2008-06-18 | Canon Kabushiki Kaisha | Virtual space generating system, image processing apparatus and information processing method |
JP5001286B2 (ja) * | 2005-10-11 | 2012-08-15 | プライム センス リミティド | 対象物再構成方法およびシステム |
KR100714672B1 (ko) * | 2005-11-09 | 2007-05-07 | 삼성전자주식회사 | 스플렛을 이용한 깊이 기반의 영상 렌더링 방법 및 그방법을 채용한 시스템 |
US8094928B2 (en) * | 2005-11-14 | 2012-01-10 | Microsoft Corporation | Stereo video for gaming |
US8073292B2 (en) | 2006-02-28 | 2011-12-06 | Koninklijke Philips Electronics N.V. | Directional hole filling in images |
US8594180B2 (en) | 2007-02-21 | 2013-11-26 | Qualcomm Incorporated | 3D video encoding |
GB0712690D0 (en) * | 2007-06-29 | 2007-08-08 | Imp Innovations Ltd | Imagee processing |
WO2009009465A1 (en) | 2007-07-06 | 2009-01-15 | Christopher William Heiss | Electrocoagulation reactor and water treatment system and method |
JP2009075869A (ja) * | 2007-09-20 | 2009-04-09 | Toshiba Corp | 多視点画像描画装置、方法及びプログラム |
MY162861A (en) | 2007-09-24 | 2017-07-31 | Koninl Philips Electronics Nv | Method and system for encoding a video data signal, encoded video data signal, method and system for decoding a video data signal |
EP2201784B1 (en) * | 2007-10-11 | 2012-12-12 | Koninklijke Philips Electronics N.V. | Method and device for processing a depth-map |
JP4703635B2 (ja) * | 2007-12-12 | 2011-06-15 | 株式会社日立製作所 | 立体視画像生成方法、その装置、及び立体視画像表示装置 |
CN101257641A (zh) * | 2008-03-14 | 2008-09-03 | 清华大学 | 基于人机交互的平面视频转立体视频的方法 |
JP4827881B2 (ja) | 2008-04-30 | 2011-11-30 | 三洋電機株式会社 | 映像ファイル処理方法及び映像送受信再生システム |
-
2010
- 2010-09-21 WO PCT/IB2010/054251 patent/WO2011039679A1/en active Application Filing
- 2010-09-21 CN CN201080044808.XA patent/CN102549507B/zh active Active
- 2010-09-21 EP EP10765509.4A patent/EP2483750B1/en active Active
- 2010-09-21 US US13/499,319 patent/US9167226B2/en active Active
- 2010-09-21 BR BR112012007115-7A patent/BR112012007115A2/pt not_active Application Discontinuation
- 2010-09-21 KR KR1020127011133A patent/KR101727094B1/ko active IP Right Grant
- 2010-09-21 JP JP2012531526A patent/JP5859441B2/ja active Active
- 2010-09-21 RU RU2012117572/08A patent/RU2551789C2/ru not_active IP Right Cessation
Patent Citations (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2006041811A (ja) * | 2004-07-26 | 2006-02-09 | Kddi Corp | 自由視点画像ストリーミング方式 |
Also Published As
Publication number | Publication date |
---|---|
RU2012117572A (ru) | 2013-11-10 |
CN102549507A (zh) | 2012-07-04 |
EP2483750A1 (en) | 2012-08-08 |
US20120188341A1 (en) | 2012-07-26 |
CN102549507B (zh) | 2014-08-20 |
BR112012007115A2 (pt) | 2020-02-27 |
US9167226B2 (en) | 2015-10-20 |
WO2011039679A1 (en) | 2011-04-07 |
JP5859441B2 (ja) | 2016-02-10 |
JP2013507026A (ja) | 2013-02-28 |
RU2551789C2 (ru) | 2015-05-27 |
KR101727094B1 (ko) | 2017-04-17 |
KR20120093243A (ko) | 2012-08-22 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
EP2483750B1 (en) | Selecting viewpoints for generating additional views in 3d video | |
KR101484487B1 (ko) | 깊이-맵을 프로세싱하는 방법 및 디바이스 | |
RU2554465C2 (ru) | Комбинирование 3d видео и вспомогательных данных | |
JP5317955B2 (ja) | 複数の視野の効率的な符号化 | |
US8488869B2 (en) | Image processing method and apparatus | |
EP2235685B1 (en) | Image processor for overlaying a graphics object | |
JP2015521407A (ja) | 3d映像を処理するための品質メトリク | |
US9596446B2 (en) | Method of encoding a video data signal for use with a multi-view stereoscopic display device | |
TW201010409A (en) | Versatile 3-D picture format | |
US20130063576A1 (en) | Stereoscopic intensity adjustment device, stereoscopic intensity adjustment method, program, integrated circuit and recording medium | |
US11528538B2 (en) | Streaming volumetric and non-volumetric video | |
US9584794B2 (en) | Depth helper data | |
US20150304640A1 (en) | Managing 3D Edge Effects On Autostereoscopic Displays | |
JPWO2012114975A1 (ja) | 画像処理装置および画像処理方法 | |
WO2014013405A1 (en) | Metadata for depth filtering |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PUAI | Public reference made under article 153(3) epc to a published international application that has entered the european phase |
Free format text: ORIGINAL CODE: 0009012 |
|
17P | Request for examination filed |
Effective date: 20120502 |
|
AK | Designated contracting states |
Kind code of ref document: A1 Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO SE SI SK SM TR |
|
DAX | Request for extension of the european patent (deleted) | ||
RAP1 | Party data changed (applicant data changed or rights of an application transferred) |
Owner name: KONINKLIJKE PHILIPS N.V. |
|
17Q | First examination report despatched |
Effective date: 20140407 |
|
REG | Reference to a national code |
Ref country code: DE Ref legal event code: R079 Ref document number: 602010053544 Country of ref document: DE Free format text: PREVIOUS MAIN CLASS: G04B0013000000 Ipc: H04N0013111000 |
|
GRAP | Despatch of communication of intention to grant a patent |
Free format text: ORIGINAL CODE: EPIDOSNIGR1 |
|
RIC1 | Information provided on ipc code assigned before grant |
Ipc: H04N 13/111 20180101AFI20180314BHEP Ipc: H04N 13/161 20180101ALI20180314BHEP |
|
INTG | Intention to grant announced |
Effective date: 20180403 |
|
GRAS | Grant fee paid |
Free format text: ORIGINAL CODE: EPIDOSNIGR3 |
|
GRAA | (expected) grant |
Free format text: ORIGINAL CODE: 0009210 |
|
AK | Designated contracting states |
Kind code of ref document: B1 Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO SE SI SK SM TR |
|
REG | Reference to a national code |
Ref country code: GB Ref legal event code: FG4D |
|
REG | Reference to a national code |
Ref country code: CH Ref legal event code: EP |
|
REG | Reference to a national code |
Ref country code: FR Ref legal event code: PLFP Year of fee payment: 9 |
|
REG | Reference to a national code |
Ref country code: IE Ref legal event code: FG4D |
|
REG | Reference to a national code |
Ref country code: DE Ref legal event code: R096 Ref document number: 602010053544 Country of ref document: DE |
|
REG | Reference to a national code |
Ref country code: AT Ref legal event code: REF Ref document number: 1041987 Country of ref document: AT Kind code of ref document: T Effective date: 20181015 |
|
REG | Reference to a national code |
Ref country code: NL Ref legal event code: MP Effective date: 20180912 |
|
REG | Reference to a national code |
Ref country code: LT Ref legal event code: MG4D |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: LT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20180912 Ref country code: NO Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20181212 Ref country code: GR Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20181213 Ref country code: SE Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20180912 Ref country code: FI Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20180912 Ref country code: BG Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20181212 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: HR Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20180912 Ref country code: AL Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20180912 Ref country code: LV Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20180912 |
|
REG | Reference to a national code |
Ref country code: AT Ref legal event code: MK05 Ref document number: 1041987 Country of ref document: AT Kind code of ref document: T Effective date: 20180912 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: CZ Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20180912 Ref country code: RO Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20180912 Ref country code: IT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20180912 Ref country code: ES Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20180912 Ref country code: AT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20180912 Ref country code: PL Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20180912 Ref country code: NL Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20180912 Ref country code: EE Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20180912 Ref country code: IS Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20190112 |
|
REG | Reference to a national code |
Ref country code: CH Ref legal event code: PL |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: PT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20190112 Ref country code: SM Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20180912 Ref country code: SK Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20180912 |
|
REG | Reference to a national code |
Ref country code: BE Ref legal event code: MM Effective date: 20180930 |
|
REG | Reference to a national code |
Ref country code: DE Ref legal event code: R097 Ref document number: 602010053544 Country of ref document: DE |
|
REG | Reference to a national code |
Ref country code: IE Ref legal event code: MM4A |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: LU Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20180921 |
|
PLBE | No opposition filed within time limit |
Free format text: ORIGINAL CODE: 0009261 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: MC Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20180912 Ref country code: IE Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20180921 Ref country code: DK Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20180912 |
|
26N | No opposition filed |
Effective date: 20190613 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: BE Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20180930 Ref country code: LI Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20180930 Ref country code: SI Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20180912 Ref country code: CH Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20180930 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: MT Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20180921 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: HU Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT; INVALID AB INITIO Effective date: 20100921 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: CY Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20180912 Ref country code: MK Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20180912 |
|
PGFP | Annual fee paid to national office [announced via postgrant information from national office to epo] |
Ref country code: TR Payment date: 20230908 Year of fee payment: 14 Ref country code: GB Payment date: 20230926 Year of fee payment: 14 |
|
PGFP | Annual fee paid to national office [announced via postgrant information from national office to epo] |
Ref country code: FR Payment date: 20230926 Year of fee payment: 14 Ref country code: DE Payment date: 20230928 Year of fee payment: 14 |