US11218687B2 - Apparatus and method for generating a representation of a scene - Google Patents

Apparatus and method for generating a representation of a scene Download PDF

Info

Publication number
US11218687B2
US11218687B2 US16/637,297 US201816637297A US11218687B2 US 11218687 B2 US11218687 B2 US 11218687B2 US 201816637297 A US201816637297 A US 201816637297A US 11218687 B2 US11218687 B2 US 11218687B2
Authority
US
United States
Prior art keywords
depth
image
placement
sensing
camera
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
US16/637,297
Other languages
English (en)
Other versions
US20210152802A1 (en
Inventor
Christiaan Varekamp
Bart Kroon
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Koninklijke Philips NV
Original Assignee
Koninklijke Philips NV
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Koninklijke Philips NV filed Critical Koninklijke Philips NV
Assigned to KONINKLIJKE PHILIPS N.V. reassignment KONINKLIJKE PHILIPS N.V. ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: VAREKAMP, CHRISTIAN, KROON, BART
Publication of US20210152802A1 publication Critical patent/US20210152802A1/en
Application granted granted Critical
Publication of US11218687B2 publication Critical patent/US11218687B2/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S17/00—Systems using the reflection or reradiation of electromagnetic waves other than radio waves, e.g. lidar systems
    • G01S17/87—Combinations of systems using electromagnetic waves other than radio waves
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S17/00—Systems using the reflection or reradiation of electromagnetic waves other than radio waves, e.g. lidar systems
    • G01S17/88—Lidar systems specially adapted for specific applications
    • G01S17/89—Lidar systems specially adapted for specific applications for mapping or imaging
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S7/00—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00
    • G01S7/48—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00 of systems according to group G01S17/00
    • G01S7/4808—Evaluating distance, position or velocity data
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
    • G06T7/33—Determination of transform parameters for the alignment of images, i.e. image registration using feature-based methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/80—Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/80—Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
    • G06T7/85—Stereo camera calibration
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
    • H04N13/106—Processing image signals
    • H04N13/111—Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
    • H04N13/106—Processing image signals
    • H04N13/156—Mixing image signals
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/20—Image signal generators
    • H04N13/204—Image signal generators using stereoscopic image cameras
    • H04N13/243—Image signal generators using stereoscopic image cameras using three or more 2D image sensors
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/20—Image signal generators
    • H04N13/204—Image signal generators using stereoscopic image cameras
    • H04N13/246—Calibration of cameras
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/20—Image signal generators
    • H04N13/204—Image signal generators using stereoscopic image cameras
    • H04N13/254—Image signal generators using stereoscopic image cameras in combination with electromagnetic radiation sources for illuminating objects
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/20—Image signal generators
    • H04N13/271—Image signal generators wherein the generated image signals comprise depth maps or disparity maps
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10028—Range image; Depth image; 3D point clouds
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20212—Image combination
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30168—Image quality inspection
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30204—Marker
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30244—Camera pose
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N2013/0074—Stereoscopic image analysis
    • H04N2013/0081—Depth or disparity estimation from stereoscopic image signals

Definitions

  • the invention relates to an apparatus and method for generating a representation of a scene from data captured by a plurality of depth-sensing cameras and in particular, but not exclusively, to generating an image for a virtual reality application from such a representation.
  • three-dimensional (3D) displays have been developed which add a third dimension to the viewing experience by providing a viewer's two eyes with different views of the scene being watched. This can be achieved by having the user wear glasses to separate two views that are displayed.
  • autostereoscopic displays that use means at the display (such as lenticular lenses, or barriers) to separate views, and to send them in different directions where they individually may reach the user's eyes.
  • two views are required whereas autostereoscopic displays typically require more views (such as e.g. nine views).
  • various algorithms are known for generating such new view images based on image and depth information, they tend to be highly dependent on the accuracy of the provided (or derived) depth information.
  • three-dimensional image information is often provided by a plurality of images corresponding to different view directions for a scene.
  • video content such as films or television programs, are increasingly generated to include some 3D information.
  • Such information can be captured using dedicated 3D or depth-sensing cameras, such as cameras that capture two simultaneous images from slightly offset camera positions.
  • the provided images may not directly correspond to the desired directions, or more images may be required.
  • more images are required, and indeed often 9-26 view images are used.
  • view point shifting processing may be employed. This is typically performed by a view shifting algorithm which uses an image for a single view direction together with associated depth information.
  • a view vector typically describes an orientation according to three degrees of freedom (3DoF), typically by providing yaw, pitch and roll values (or azimuth, elevation, and tilt) or a quaternion representation.
  • 3DoF degrees of freedom
  • a placement or pose vector may comprise six values/components with each value/component describing an individual property of the position/location or the orientation/direction of the corresponding object.
  • placement or pose vectors may have fewer components to represent the position and orientation, for example if one or more components are considered fixed (e.g. if all objects are considered to be at the same height and are level, four components may provide a full representation of the pose of an object).
  • One of the most difficult challenges to address is that of how to effectively capture a scene such that a representation of the scene can be generated which provides sufficient information to allow suitable viewpoint images of sufficiently high quality to be generated yet still allowing the representation to be easily communicated, stored, adapted, and/or processed.
  • calibration for multi-view camera capture systems includes different aspects:
  • accurate placement information can be ensured by a careful, accurate and highly controlled positioning and orientation of the cameras with known placement parameters.
  • a fixed rig of depth-sensing cameras where each camera is fixed at a known position and with a fixed orientation relative to the other cameras may be used.
  • such an approach is impractical or even impossible in many applications. For example, if a large number of cameras are used and/or if cameras are far from each other, it is typically not practical to manually measure or restrict positions and orientations to a high degree of accuracy.
  • the approach is not suitable for dynamic capture applications, such as when cameras may be dynamically added or removed from the capture system, or where one or more of the cameras may be a moving camera.
  • an improved approach for capturing scenes, and specifically for determining placement information for capturing depth-sensing cameras would be advantageous.
  • an approach that allows improved operation, increased flexibility, facilitated implementation, facilitated operation, reduced complexity, reduced resource demand, improved accuracy, better placement determination, and/or improved performance would be advantageous.
  • an apparatus for generating a representation of at least part of a scene from images of a scene captured by a plurality of distributed depth-sensing cameras, the apparatus comprising: a first receiver for receiving a first image captured by a first depth-sensing camera and first depth data captured by the first depth-sensing camera; a detector for detecting an image position property for a fiducial marker in the first image, the image position property being indicative of a position of the fiducial marker in the first image and the fiducial marker representing a placement of a second depth sensing image camera by having a predetermined placement difference with respect to the second depth sensing camera; a placement processor for determining a relative placement vector indicative of a placement of the second depth sensing image camera relative to the first depth-sensing camera in response to the image position property, and depth data of the first depth data for an image position of the fiducial marker, and the predetermined placement difference, the relative placement vector being at least three dimensional; a second receiver for receiving a first image captured by
  • the approach may allow a dynamically varying capture configuration to be supported with e.g. cameras being dynamically introduced and/or removed, and or with cameras moving in the environment.
  • the approach may in many embodiments and scenarios allow a more accurate and/or complete capture and representation of the scene. Often a substantially more accurate combination of information from different cameras with different placements can be achieved.
  • the representation may be a data representation describing visual properties of at least part of the scene.
  • the data representation may include three-dimensional data for the scene, such as e.g. depth information.
  • the representation of the scene may be at least one of a three dimensional model of the scene and a set of one or more images of the scene.
  • the set of images may correspond to one or more images corresponding to viewports from different viewpoints.
  • the representations may be generated e.g. as texture maps and associated maps, images and potentially associated depth maps, a three-dimensional structure and lighting data, etc.
  • Placement may refer to position and/or orientation.
  • the relative placement vector may comprise one or more position values and/or one or more orientation values.
  • a placement vector may provide a three dimensional position and/or a three dimensional direction indication.
  • the relative placement vector comprises at least three components, and/or may typically be indicative of both a relative position and a relative orientation between the first and second depth-sensing cameras.
  • An orientation may be given by one or more components selected from a pitch, yaw, roll components or from azimuth, elevation, and tilt components.
  • the relative placement vector may be indicative of a translation between a position of the first depth-sensing camera and a position of the second depth-sensing camera.
  • the relative placement vector may be indicative of a direction from the first depth-sensing camera to the second depth-sensing camera.
  • the placement processor may be arranged to generate the relative placement vector based on an assumption of a known relationship between the fiducial marker and the second depth-sensing camera. It may specifically generate the relative placement vector based on an assumed relationship between a placement of the fiducial marker and a placement of the second depth-sensing camera.
  • the placement processor may be arranged to generate the relative placement vector based on an assumed relationship between a position of the fiducial marker and a position of the second depth-sensing camera.
  • the placement processor may be arranged to generate the relative placement vector based on an assumed relationship between an orientation of the fiducial marker and an orientation of the second depth-sensing camera.
  • the first and second depth-sensing cameras may be arranged to have an overlap of the viewports corresponding to the first image and the second image.
  • the first and second depth-sensing cameras may be part of a capture system comprising a plurality of spatially distributed depth-sensing cameras, with the cameras having different placements, such as different positions and/or orientations.
  • the relative placement vector may be indicative of position and/or orientation offsets or differences between the first and second depth sensing camera.
  • the predetermined placement difference is indicative of a difference in the placements of the fiducial marker and the placement of the second depth sensing camera.
  • the generator is arranged to generate the representation as an image of the scene from a viewpoint different from viewpoints of the first depth-sensing camera and the second depth-sensing camera.
  • the approach may provide an improved generation of images of the scene, and may in particular often provide a higher quality and/or better coverage of the scene.
  • the different viewpoints may have a different position and/or orientation.
  • the generator is arranged to combine at least the first image and the second image based on the first depth data and the second depth data.
  • This may provide improved performance in many embodiments, and may for example allow, improve, and/or facilitate viewpoint shifting for generating images from different viewpoints.
  • the generator comprises a receiver for receiving relative placement vectors for a plurality of pairs of depth-sensing cameras; a coordinate processor for determining depth-sensing camera placements in a common coordinate system for each of the plurality of cameras in response to the relative placement vectors, and wherein the generator is arranged to generate the image in response to the determined camera placements in the same coordinate system.
  • the approach may allow a flexible yet accurate scene capturing system to be developed. In particular, it may allow many cameras to effectively be combined into a coherent capture system providing enhanced coverage and/or improved quality.
  • the approach may in particular facilitate or enable the interworking between a large number of cameras, and further may effectively support a changing configuration, e.g. due to cameras being added or removed, or due to moving cameras.
  • the coordinate processor is arranged to determine the depth-sensing camera placements in response to a minimization of a cost function, the cost function being dependent on a difference between the relative placement vectors and corresponding placement vectors determined from determined depth-sensing camera placements in the common coordinate system.
  • This may allow for a particularly efficient approach and may allow efficient combination of relative placement information from a large number of cameras to generate a coherent placement representation for the capture system.
  • initial estimated camera placements may be determined by placement data determined without relying on the first image, such as for example a GPS position functionality.
  • a visual property of the fiducial marker is indicative of an identity of the second depth-sensing camera, and the detector is arranged to determine the identity of the second depth-sensing camera in response to the visual property.
  • This may facilitate and/or improve the generation of the representation of the scene. It may in particular facilitate an automatic placement configuration determination for capturing systems comprising a large number of cameras.
  • the capture quality indication may specifically be indicative of a number of cameras that capture different parts of the scene.
  • An indication of an increasing number of capturing cameras may be indicative of an increasing quality.
  • the capture processor may specifically be arranged to determine at least one part of the scene for which the capture quality indication indicates that a number of cameras capturing the scene is below a threshold.
  • the user output may indicate a placement of the depth sensing image camera such that this will capture the part of the scene currently captured by less than the threshold of cameras.
  • the threshold may be one camera.
  • the generator may be arranged to generate the image in response to an evaluation of the three-dimensional model
  • FIG. 2 illustrates an example of elements of a capture configuration comprising a plurality of depth-sensing cameras
  • FIG. 3 illustrates an example of elements of a capture configuration comprising a plurality of depth-sensing cameras
  • FIG. 5 illustrates an example of an apparatus in accordance with some embodiments of the invention
  • FIG. 6 illustrates an example of elements of a capture configuration comprising a plurality of depth-sensing cameras
  • FIG. 7 illustrates an example of an apparatus in accordance with some embodiments of the invention.
  • FIG. 1 illustrates an example of a capturing system wherein a relatively large number of depth-sensing cameras capture a scene, which in the example is represented by three central objects 101 , 102 and 103 .
  • Each depth-sensing camera N 1 -N 7 is represented by a black triangle also indicating the viewport/view angle of the camera.
  • FIG. 1 illustrates the use of eight cameras, a substantially larger number may be used in many practical systems, and indeed the illustrated eight cameras may be seen as representing a substantially larger number of cameras.
  • Depth-sensing cameras in the capture configuration are also referred to as nodes of the configuration and the determination of placement information (typically with reference to a common coordinate system) is known as registration of the cameras/nodes.
  • the registration of nodes may be performed at a central server or registration unit which may gather the placement information for all nodes/cameras in the configuration.
  • placement and pose are used as a common term for position and/or direction/orientation.
  • the combination of the position and direction/orientation of e.g. an object, a camera, a head, or a view may be referred to as a pose or placement.
  • a placement or pose indication may comprise six values/components/degrees of freedom with each value/component typically describing an individual property of the position/location or the orientation/direction of the corresponding object.
  • a placement or pose may be represented by fewer components, for example if one or more components is considered fixed or irrelevant (e.g. if all objects are considered to be at the same height and have a horizontal orientation, four components may provide a full representation of the pose of an object).
  • the term pose is used to refer to a position and/or orientation which may be represented by one to six values (corresponding to the maximum possible degrees of freedom).
  • a possible solution may be to try out all pairs of nodes and see how successful the registration process is for each pair, and then only use pairs for which a successful registration is achieved. In practice, this will tend to result in a number of different problems, including it being very difficult to assess the quality of the registration process based on scene points alone.
  • FIG. 2 may represent a simple example of a range-sensing camera system with two camera nodes that observe a scene comprising a single marker. Both nodes observe a common scene including a marker, but not each other.
  • the cameras cannot be positioned in a common coordinate system, because (1) the rotations around the translation vectors are unknown, and (2) the angle between the translation vectors is unknown. It is thus impossible to merge captured 3D images of the two cameras.
  • a fiducial marker may be an object placed in the field of view of an imaging system such that it may appear in one or more of the images produced.
  • the fiducial marker may be used by the system as a point of reference.
  • the fiducial markers are not merely markers that may be detected and referenced in different images but rather are representative of the positions of the depth-sensing cameras.
  • the fiducial markers may be co-located with the depth-sensing cameras, and specifically each camera may be provided with a fiducial marker.
  • a fiducial marker may thus specifically be positioned at essentially the same position as a depth-sensing camera, and with an orientation that directly reflects the orientation of the depth-sensing camera.
  • the fiducial marker has a predetermined placement difference with respect to the corresponding depth sensing camera.
  • the predetermined placement difference is thus a known/predetermined difference between the placement of the fiducial marker and the placement of the depth sensing camera.
  • the predetermined placement difference may be a placement offset.
  • the predetermined placement difference may specifically be a predetermined position offset (typically in two or three dimensions but in some embodiments the position offset may be in just one dimension).
  • the predetermined placement difference may specifically be a predetermined orientation offset (typically in one, two or three dimensions).
  • the predetermined placement difference may be e.g. a standardized or generally accepted approach for positioning fiducial markers with respect to depth sensing cameras, for example it may prescribed that fiducial markers are positioned on the camera with the same orientation as the camera (e.g. affixed to the surfaces also having the lens of the depth sensing camera.
  • the predetermined placement difference may be determined during the setup and positioning of the fiducial markers. For example, in a given scenario, a user may decide that the fiducial markers are affixed to the tripod below the camera and may for example enter a corresponding value for the predetermined placement difference into the system for use in the subsequent calibration process (e.g. it may be entered that fiducial markers are affixed 20 cm below the depth sensing camera (lens)).
  • the fiducial marker may be in a rigid alignment with the corresponding depth sensing camera.
  • the fiducial marker may be comoving with the depth sensing camera, i.e. a change in the placement of the depth sensing camera will result in a corresponding change in the placement of the fiducial marker (and vice versa).
  • the fiducial marker may have a fixed position and/or orientation with respect to/relative to the corresponding depth sensing camera.
  • the predetermined placement difference may in some embodiments change between calibration sessions but will be known (or assumed to be known) prior to the start of the calibration process that determines the relative placement of the corresponding depth sensing camera based on the fiducial marker.
  • the predetermined placement difference thus reflects the correct placement difference between the fiducial marker and the depth sensing camera whereas the actual difference may be different. This may result in the calibration process determining positions of cameras (most likely) generating wrong, or at least inaccurate, estimated placements of the depth sensing cameras. However, it is of course not unexpected that erroneous results may result in a specific scenario wherein a mistake is made and the assumptions and premises on which the process is based are accordingly not met.
  • the predetermined placement difference may specifically in some embodiments be a placement difference vector indicative of an offset between the placement of the fiducial marker and of the corresponding depth sensing camera.
  • the predetermined placement difference may be represented by a six value vector indicating the offset in three positional dimensions and three orientation dimensions.
  • the determination of a placement of a given fiducial marker is accordingly also a determination of the placement of the corresponding depth-sensing camera.
  • the fiducial marker may in many embodiments be designed such that it can be used both for the identification of each depth sensing camera node and for the accurate registration of each node in a single common coordinate system.
  • a fiducial marker may specifically be positioned on or immediately adjacent to each camera to identify both the camera and the placement of the camera.
  • the fiducial marker is accordingly representative of the placement of the depth-sensing camera linked to the fiducial marker by having a predetermined placement difference with respect to the depth sensing camera.
  • fiducial markers with different properties may be used in different embodiments, and fiducial markers may have various shapes for placement detection and node identification, including for example:
  • the approach may specifically be used to find relative placements between pairs of depth-sensing cameras, and in embodiments with larger number of cameras such relative placements may be generated for multiple pairs with the resulting relative placements subsequently being used to place all depth-sensing cameras in a common coordinate system.
  • the first receiver 401 is coupled to a detector 405 which is arranged to detect an image position property for a fiducial marker in the first image.
  • the image position property is indicative of a position of the fiducial marker in the first image.
  • the detector 405 is coupled to a placement processor 407 which is arranged to determine a relative placement vector which is indicative of the placement of the second depth sensing image camera relative to the depth first depth-sensing camera.
  • the placement processor 407 further determines the relative placement vector based on the first depth data.
  • the relative placement vector accordingly provides at least a three-dimensional position difference between the first and second depth-sensing cameras.
  • it will typically comprise at least one orientation value for the second camera.
  • the apparatus may further be arranged to perform the reciprocal operation as well, i.e. it may determine a second relative placement vector which indicates the relative placement of the first depth-sensing camera as seen from the second depth-sensing camera.
  • the apparatus may accordingly comprise a second detector which repeats the operation of the detector 405 but instead seeks to detect a second fiducial marker that represents the position of the first depth-sensing camera in the second image. It may further comprise a second placement processor which repeats the operation of the placement processor 407 to determine the second relative placement vector from the images, depth data, a predetermined placement difference for the second fiducial marker with respect to the first depth sensing camera, a second predetermined and detected image position in the second image.
  • the second a detector and the second placement processor may be implemented by the detector 405 and placement processor 407 of FIG. 4 , respectively, repeating the same operation as when determining the first relative placement vector but swapping the first and second images as well as the first and second depth maps, and detecting the second fiducial marker rather than the first fiducial marker (in case individual fiducial marker searches are performed).
  • reciprocal operation is thus performed with the relative placements of the first and second depth-sensing camera being determined both from the point of view of the first depth-sensing camera and from the point of view of the second depth-sensing camera.
  • the generator 409 may in such an embodiment proceed to use both the first and the second determined relative placement vectors to generate the representation and specifically the image for a different viewpoint. As an example, the generator 409 may proceed to determine an average relative placement vector from the first and second relative placement vectors and the generator 409 may then use this average relative placement vector for the image generation.
  • the apparatus may for example receive images and depth maps from all the depth-sensing cameras. It may then proceed to perform the process described above for all pairs of cameras. It may specifically generate two relative placement vectors for each pair, one based on each camera of the pair (or equivalently the different directions may be treated as different pairs with each resulting in one relative placement vector).
  • the detector 405 may be arranged to perform the processing to address such uncertainty.
  • the received first image may be processed for each of the possible other cameras.
  • each camera uploading an image and depth map may also upload a description of the fiducial marker attached to this camera.
  • the detector 405 may then sequentially process the first image for all other cameras. It may specifically for a given camera search the image for a fiducial marker having properties corresponding to those reported by the given camera. If no match is found, the detector 405 proceeds to the next camera. However, if a match is found, the first camera proceeds to generate a relative placement vector as described above. After all potential cameras have been processed, the apparatus proceeds to the next camera and repeats the process.
  • the detector 405 may instead search the image to find any image objects that may correspond to a fiducial marker, e.g. based on generic visual (and/or depth) properties for fiducial markers. For each detected potential fiducial marker, the detector 405 may then proceed to evaluate specific properties of the fiducial marker which may identify the camera associated with the fiducial marker. For example, a specific pattern (e.g. a QR code) may uniquely identify the detected fiducial marker/camera. Based on this detection, the apparatus may proceed to identify the received image and depth maps for the detected camera.
  • a specific pattern e.g. a QR code
  • Identification may for instance be enabled by giving each fiducial marker a unique appearance (shape, color, pattern, etc.) or by using active (e.g. LED) markers that send identifying codes.
  • the relative placement vector may include a translation vector indicative of the difference in position, and in addition at least one orientation/direction parameter may be included.
  • the process accordingly results in a typically large number of relative placement vectors being generated which reflect the relative placements of the cameras.
  • the generator 409 may register the cameras in a common coordinate system.
  • the registration process may for each node specifically determine a rotation matrix and a translation vector that places the nodes in a common coordinate system.
  • the generator 409 may comprise a receiver 501 , which receives all the generated relative placement vectors.
  • the receiver 501 is coupled to a coordinate processor 503 , which is arranged to determine the camera placements in a common coordinate system based on the relative placement vectors.
  • the generator 409 may then generate the representation of the scene from this configuration.
  • the representation may be provided as the combination of the registration data, i.e. data describing the position of all the cameras in a single coordinate system, together with the received images and depth maps.
  • FIG. 6 illustrates an example of how a third camera may be registered.
  • node 2 and 3 (N 2 , N 3 ) may already be registered, i.e. their placements may be known.
  • the generator 409 may know the relative placement vectors of node 1 (N 1 ) relative to nodes 2 and 3 . Accordingly, the generator 409 knows the position of node 1 from either node 2 or node 3 , and the 3D orientation of node 1 because node 1 observed node 2 and 3 .
  • the example shows a configuration of three nodes where node 1 sees two other nodes 2 , 3 each having a fiducial marker for the purpose of identification and registration.
  • nodes 2 and 3 can both determine their placement relative to node 1 , the 3D data from all three nodes can be placed in a common coordinate system. Since node 1 observes the distance to both nodes 2 and 3 , an extra check/filter operation is possible on the relative position determination of nodes 2 and 3 .
  • a cost function may be defined which is dependent on the differences between the (measured) relative placement vectors and corresponding (adjusted) relative placement vectors that result from the placements of the nodes in the common coordinate system.
  • an adjusted relative placement vector between two nodes results from the allocated placements of these two nodes. As the placements change, so do the adjusted relative placement vectors that indicate the difference between the allocated placements.
  • the cost function may include a comparison of this adjusted relative placement vector to the corresponding measured relative placement vector determined by the measurements as described previously.
  • the adjusted relative placement vector resulting from the allocated placements of two cameras is compared to the measured relative placement vector for the two cameras, and the larger the difference, the larger the value of the cost function.
  • the cost function may include a comparison for each of the measured relative placement vectors for all camera pairs.
  • the coordinate processor 503 may then vary the allocated placements to find an optimized allocation for which the cost function is minimized and the corresponding configuration may be registered.
  • initial positions of the nodes/cameras may be allocated based on other functionality, such as for example based on position information from GPS position estimates included in each camera.
  • the placement processor 407 may be arranged to determine the relative placement vector based on a predetermined assumption that at least one orientation parameter is identical for the first depth sensing image camera and the second depth sensing image camera. For example, in many embodiments, the placement processor 407 may be based on assuming that the first and second cameras are both positioned such that they are level and fully horizontal.
  • Such an approach may allow the placement processor 407 to reduce the number of unknown variables and therefore may facilitate the determination of the relative placement vector. More significantly in many embodiments, it may facilitate the registration of the nodes in a common coordinate system. Specifically, it may allow for a more efficient optimization process where e.g. a simpler cost function can be used.
  • the system may generate a representation of part of the scene by one or more images that are specifically generated from the captured images and depth information.
  • the representation may simply be a single image of the scene generated from the captured images and depth maps, and based on combinations using the registration information.
  • a view shifting algorithm may be applied to one image with de-occluded areas being filled in from images from other cameras selected based on the placement information. In this way, an image from other viewpoints than the camera viewpoints can be generated.
  • the generator 409 may be arranged to generate a three-dimensional model of the scene based on the relative placement vectors and specifically based on the configuration of all the cameras in the common coordinate system.
  • each depth measurement of each camera can be used to fill a pre-defined 3D volume/array. After this filling operation, the collection of non-empty pixels form the 3D volumetric representation. The positions on the outer surface of the volume can converted to a mesh to represent the 3D model.
  • the distribution of the functionality of the described apparatus may be different in different embodiments depending on the preferences and requirements of the individual embodiment and application.
  • the apparatus may be implemented as a stand-alone central device or server to which all the cameras transmit captured image and depth information.
  • the functionality may be distributed in the capturing system.
  • a detector 405 may be implemented in each camera and the detected image position may be communicated to a central device which implements the placement processor 407 and the generator 409 , and which accordingly proceeds to generate the probability values and the representation.
  • a detector 405 and placement processor 407 may be implemented in each depth-sensing camera and the resulting relative placement vectors may be transmitted to generator 409 , which may be centrally located.
  • the capture processor 701 may specifically identify parts of the scene for which the capture quality indication indicates a reduced quality of the capture. For example, it may identify the parts of the scene for which none or only one camera has captured the scene. It may then proceed to display these parts and may indicate a placement for a new depth-sensing camera such that this quality can be improved for that part of the scene.
  • the approach may specifically be used to check the validity/performance of a given camera setup.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • General Physics & Mathematics (AREA)
  • Electromagnetism (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Radar, Positioning & Navigation (AREA)
  • Remote Sensing (AREA)
  • Theoretical Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Computer Graphics (AREA)
  • Geometry (AREA)
  • Software Systems (AREA)
  • Length Measuring Devices By Optical Means (AREA)
  • Image Analysis (AREA)
  • Stereoscopic And Panoramic Photography (AREA)
  • Studio Devices (AREA)
  • Testing, Inspecting, Measuring Of Stereoscopic Televisions And Televisions (AREA)
  • Image Processing (AREA)
  • Electrophonic Musical Instruments (AREA)
US16/637,297 2017-08-08 2018-08-07 Apparatus and method for generating a representation of a scene Active US11218687B2 (en)

Applications Claiming Priority (4)

Application Number Priority Date Filing Date Title
EP17185271 2017-08-08
EP17185271.8 2017-08-08
EP17185271.8A EP3441788A1 (en) 2017-08-08 2017-08-08 Apparatus and method for generating a representation of a scene
PCT/EP2018/071417 WO2019030235A1 (en) 2017-08-08 2018-08-07 APPARATUS AND METHOD FOR GENERATING A REPRESENTATION OF A SCENE

Publications (2)

Publication Number Publication Date
US20210152802A1 US20210152802A1 (en) 2021-05-20
US11218687B2 true US11218687B2 (en) 2022-01-04

Family

ID=59569191

Family Applications (1)

Application Number Title Priority Date Filing Date
US16/637,297 Active US11218687B2 (en) 2017-08-08 2018-08-07 Apparatus and method for generating a representation of a scene

Country Status (9)

Country Link
US (1) US11218687B2 (pt)
EP (2) EP3441788A1 (pt)
JP (1) JP7059355B6 (pt)
KR (1) KR102535945B1 (pt)
CN (1) CN111194416B (pt)
BR (1) BR112020002452B1 (pt)
RU (1) RU2769303C2 (pt)
TW (1) TWI795425B (pt)
WO (1) WO2019030235A1 (pt)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20230021345A1 (en) * 2019-12-26 2023-01-26 Nec Corporation Information processing device, control method, and storage medium

Families Citing this family (16)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108470323B (zh) * 2018-03-13 2020-07-31 京东方科技集团股份有限公司 一种图像拼接方法、计算机设备及显示装置
EP3703378A1 (en) * 2019-03-01 2020-09-02 Koninklijke Philips N.V. Apparatus and method of generating an image signal
EP3712843A1 (en) * 2019-03-19 2020-09-23 Koninklijke Philips N.V. Image signal representing a scene
GB2584122B (en) * 2019-05-22 2024-01-10 Sony Interactive Entertainment Inc Data processing
US10510155B1 (en) * 2019-06-11 2019-12-17 Mujin, Inc. Method and processing system for updating a first image generated by a first camera based on a second image generated by a second camera
EP3848899B1 (en) * 2020-01-09 2023-01-25 Stryker European Operations Limited Technique of determining a pose of a surgical registration device
US11943418B2 (en) * 2020-04-16 2024-03-26 Eys3D Microelectronics Co. Processing method and processing system for multiple depth information
JP7416231B2 (ja) * 2020-05-21 2024-01-17 日本電気株式会社 設置支援装置、設置支援方法、およびプログラム
EP4154222A1 (en) * 2020-05-22 2023-03-29 Google LLC Method and apparatus for combining an augmented reality object in a real-world image
US12002235B2 (en) * 2020-08-12 2024-06-04 Hong Kong Applied Science and Technology Research Institute Company Limited Apparatus and method for estimating camera orientation relative to ground surface
JP7508350B2 (ja) * 2020-12-04 2024-07-01 株式会社日立製作所 キャリブレーション装置およびキャリブレーション方法
DE102021119481A1 (de) 2021-07-27 2023-02-02 Karl Storz Se & Co. Kg Messverfahren und eine Messvorrichtung
US20230100989A1 (en) * 2021-09-29 2023-03-30 Cilag Gmbh International Surgical devices, systems, and methods using fiducial identification and tracking
CN116468206A (zh) * 2022-01-11 2023-07-21 大金工业株式会社 设备的维护方法、装置及系统
US20230419635A1 (en) * 2022-06-27 2023-12-28 Shopify Inc. Volumetric permissioning
TWI845192B (zh) * 2023-03-06 2024-06-11 國立臺灣大學 操作輔助系統

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7319479B1 (en) * 2000-09-22 2008-01-15 Brickstream Corporation System and method for multi-camera linking and analysis
US20090109280A1 (en) * 2007-10-31 2009-04-30 Technion Research And Development Foundation Ltd. Free viewpoint video
US20140028805A1 (en) * 2011-04-15 2014-01-30 Faro Technologies, Inc. System and method of acquiring three-dimensional coordinates using multiple coordinate measurment devices
US20170094251A1 (en) * 2015-09-30 2017-03-30 Faro Technologies, Inc. Three-dimensional imager that includes a dichroic camera
US20200064119A1 (en) * 2018-08-22 2020-02-27 Government Of The United States Of America, As Represented By The Secretary Of Commerce Non-contact coordinate measuring machine using a noncontact metrology probe

Family Cites Families (24)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
GB2372656A (en) * 2001-02-23 2002-08-28 Ind Control Systems Ltd Optical position determination
US20110304706A1 (en) * 2010-06-09 2011-12-15 Border John N Video camera providing videos with perceived depth
EP2405236B1 (de) * 2010-07-07 2012-10-31 Leica Geosystems AG Geodätisches Vermessungsgerät mit automatischer hochpräziser Zielpunkt-Anzielfunktionalität
KR101638028B1 (ko) * 2011-09-27 2016-07-11 라이카 게오시스템스 아게 좌표계에서 알려진 타겟 포인트를 마킹하기 위한 측정 시스템 및 방법
JP5804892B2 (ja) 2011-10-25 2015-11-04 セコム株式会社 カメラ姿勢算出装置
EP2587450B1 (en) * 2011-10-27 2016-08-31 Nordson Corporation Method and apparatus for generating a three-dimensional model of a region of interest using an imaging system
GB2496591B (en) * 2011-11-11 2017-12-27 Sony Corp Camera Movement Correction
GB2497119B (en) * 2011-12-01 2013-12-25 Sony Corp Image processing system and method
DE102012207931A1 (de) * 2012-01-07 2013-07-11 Johnson Controls Gmbh Kameraanordnung zur Distanzmessung
EP2615580B1 (en) * 2012-01-13 2016-08-17 Softkinetic Software Automatic scene calibration
US9083960B2 (en) * 2013-01-30 2015-07-14 Qualcomm Incorporated Real-time 3D reconstruction with power efficient depth sensor usage
US8773562B1 (en) * 2013-01-31 2014-07-08 Apple Inc. Vertically stacked image sensor
US10122993B2 (en) * 2013-03-15 2018-11-06 Fotonation Limited Autofocus system for a conventional camera that uses depth information from an array camera
JP6171671B2 (ja) * 2013-07-24 2017-08-02 富士通株式会社 情報処理装置、位置指定方法および位置指定プログラム
JP2015168936A (ja) * 2014-03-05 2015-09-28 キャタピラー エス エー アール エル 作業機械におけるキャブ保護構造
EP3120325B1 (en) 2014-03-21 2018-06-06 Omron Corporation Method and apparatus for detecting and mitigating optical impairments in an optical system
JP2016017913A (ja) * 2014-07-10 2016-02-01 国立大学法人鳥取大学 姿勢情報作成システム、姿勢情報作成方法及び姿勢情報作成プログラム
US20160044301A1 (en) * 2014-08-06 2016-02-11 Dejan JOVANOVICH 3d modeling of imaged objects using camera position and pose to obtain accuracy with reduced processing requirements
WO2016029939A1 (en) * 2014-08-27 2016-03-03 Metaio Gmbh Method and system for determining at least one image feature in at least one image
TWI564841B (zh) * 2014-10-20 2017-01-01 陳金仁 即時影像合成裝置、方法與電腦程式產品
CA2991811A1 (en) * 2015-07-13 2017-01-19 Koninklijke Philips N.V. Method and apparatus for determining a depth map for an image
KR101629610B1 (ko) * 2015-07-27 2016-06-13 주식회사 앤비젼 고해상도 영상 추출을 위한 광경로 변환 유닛, 광경로 변환 모듈, 광경로 변환 방법 및 이 방법을 수행하기 위한 프로그램을 기록한 기록매체, 반도체 기판 패턴 검사 장치
GB2541884A (en) * 2015-08-28 2017-03-08 Imp College Of Science Tech And Medicine Mapping a space using a multi-directional camera
CN106056664B (zh) * 2016-05-23 2018-09-21 武汉盈力科技有限公司 一种基于惯性和深度视觉的实时三维场景重构系统及方法

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7319479B1 (en) * 2000-09-22 2008-01-15 Brickstream Corporation System and method for multi-camera linking and analysis
US20090109280A1 (en) * 2007-10-31 2009-04-30 Technion Research And Development Foundation Ltd. Free viewpoint video
US20140028805A1 (en) * 2011-04-15 2014-01-30 Faro Technologies, Inc. System and method of acquiring three-dimensional coordinates using multiple coordinate measurment devices
US20170094251A1 (en) * 2015-09-30 2017-03-30 Faro Technologies, Inc. Three-dimensional imager that includes a dichroic camera
US20200064119A1 (en) * 2018-08-22 2020-02-27 Government Of The United States Of America, As Represented By The Secretary Of Commerce Non-contact coordinate measuring machine using a noncontact metrology probe

Non-Patent Citations (4)

* Cited by examiner, † Cited by third party
Title
International Search Report and Written Opinion From PCT/EP2018/071417 dated Oct. 9, 2018.
Katanacho et al "Surgical Navigation With QR Codes" Current Directions in Biomedical Engineering vol. 2, No. 1, Sep. 1, 2016 p. 355-358.
Svoboda et al A Convenient Multi-Camera Self-Calibration for Virtual Environments, Presence, 2005.
Zhang "Camera Calibration with One-Dimensional Objects" IEEE Transactions on Pattern Analysis and Machine Intelligence vol. 26, No. 7, Jul. 2004.

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20230021345A1 (en) * 2019-12-26 2023-01-26 Nec Corporation Information processing device, control method, and storage medium

Also Published As

Publication number Publication date
EP3441788A1 (en) 2019-02-13
RU2020109861A (ru) 2021-09-10
WO2019030235A1 (en) 2019-02-14
EP3665506B1 (en) 2022-03-09
CN111194416A (zh) 2020-05-22
JP7059355B2 (ja) 2022-04-25
EP3665506A1 (en) 2020-06-17
BR112020002452B1 (pt) 2024-03-05
RU2020109861A3 (pt) 2022-01-28
TWI795425B (zh) 2023-03-11
KR20200035297A (ko) 2020-04-02
CN111194416B (zh) 2024-01-23
JP7059355B6 (ja) 2022-06-03
JP2020529685A (ja) 2020-10-08
US20210152802A1 (en) 2021-05-20
BR112020002452A2 (pt) 2020-07-28
RU2769303C2 (ru) 2022-03-30
TW201920985A (zh) 2019-06-01
KR102535945B1 (ko) 2023-05-26

Similar Documents

Publication Publication Date Title
US11218687B2 (en) Apparatus and method for generating a representation of a scene
CN108765498B (zh) 单目视觉跟踪方法、装置及存储介质
US10699439B2 (en) Method and an apparatus for determining a gaze point on a three-dimensional object
US11568520B2 (en) Method and device for inpainting of colourised three-dimensional point clouds
KR20160003233A (ko) 컴퓨터 비전 애플리케이션 초기화를 용이하게 하는 방법들
KR20150050172A (ko) 관심 객체 추적을 위한 다중 카메라 동적 선택 장치 및 방법
JP2014529727A (ja) 自動シーン較正
US20130070094A1 (en) Automatic registration of multi-projector dome images
US20220148207A1 (en) Processing of depth maps for images
GB2477333A (en) Stereoscopic camera system with image processor to transform one of stereoscopic pair to give effect of increased stereo base
CN112700484B (zh) 一种基于单目深度相机的深度图彩色化方法
JP2023546739A (ja) シーンの３次元モデルを生成するための方法、装置、およびシステム
KR102298047B1 (ko) 디지털 콘텐츠를 녹화하여 3d 영상을 생성하는 방법 및 장치
US11854154B1 (en) Systems and methods for synchronizing the scanning of a scene with the viewing of the scan results relative to a live view of the scene
CN117710445A (zh) 一种应用于ar设备的目标定位方法、装置及电子设备
EP4374241A1 (en) Calibration method of a system comprising an eye tracking device and a computing device comprising one or multiple screens
CN117593497A (zh) 点云数据的拾取方法及装置、非易失性存储介质
CN116385537A (zh) 一种用于增强现实的定位方法及装置
CN118115586A (zh) 一种位姿标注方法、位姿估计方法、装置及电子设备

Legal Events

Date Code Title Description
AS Assignment

Owner name: KONINKLIJKE PHILIPS N.V., NETHERLANDS

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:VAREKAMP, CHRISTIAN;KROON, BART;SIGNING DATES FROM 20190102 TO 20190405;REEL/FRAME:051747/0914

FEPP Fee payment procedure

Free format text: ENTITY STATUS SET TO UNDISCOUNTED (ORIGINAL EVENT CODE: BIG.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY

STPP Information on status: patent application and granting procedure in general

Free format text: RESPONSE TO NON-FINAL OFFICE ACTION ENTERED AND FORWARDED TO EXAMINER

STPP Information on status: patent application and granting procedure in general

Free format text: NOTICE OF ALLOWANCE MAILED -- APPLICATION RECEIVED IN OFFICE OF PUBLICATIONS

STPP Information on status: patent application and granting procedure in general

Free format text: PUBLICATIONS -- ISSUE FEE PAYMENT VERIFIED

STCF Information on status: patent grant

Free format text: PATENTED CASE