WO2017007166A1 - Procédé et dispositif de génération d'image projetée et procédé de mappage de pixels d'image et de valeurs de profondeur - Google Patents

Procédé et dispositif de génération d'image projetée et procédé de mappage de pixels d'image et de valeurs de profondeur Download PDF

Info

Publication number
WO2017007166A1
WO2017007166A1 PCT/KR2016/006961 KR2016006961W WO2017007166A1 WO 2017007166 A1 WO2017007166 A1 WO 2017007166A1 KR 2016006961 W KR2016006961 W KR 2016006961W WO 2017007166 A1 WO2017007166 A1 WO 2017007166A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
acquisition
depth
pose
acquired
Prior art date
Application number
PCT/KR2016/006961
Other languages
English (en)
Korean (ko)
Inventor
도락주
Original Assignee
고려대학교 산학협력단
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority claimed from KR1020160056447A external-priority patent/KR101835434B1/ko
Application filed by 고려대학교 산학협력단 filed Critical 고려대학교 산학협력단
Priority to US15/515,315 priority Critical patent/US10602115B2/en
Priority to EP16821572.1A priority patent/EP3321888B1/fr
Priority to CN201680040327.9A priority patent/CN107836012B/zh
Priority to JP2018500685A priority patent/JP6687204B2/ja
Publication of WO2017007166A1 publication Critical patent/WO2017007166A1/fr

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
    • G06T17/05—Geographic models
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
    • G06T17/20—Finite element generation, e.g. wire-frame surface description, tesselation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T19/00—Manipulating 3D models or images for computer graphics
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis

Definitions

  • the present invention relates to image generation, and more particularly, to a method and apparatus for generating a projection image in three-dimensional space, and a mapping method between image pixels and depth values.
  • a 3D image can be displayed naturally according to a user's movement through a method of acquiring an image and a depth value for a plurality of continuous poses.
  • the binocular image is created by moving the field of view in the left and right directions based on the center point of the image, and displaying the image through the HMD (Head Mount Display) device. Due to the nature of the 360-degree rotation of the captured video can be viewed in 3D, but there is a limit that can not be observed outside the shooting path.
  • HMD Head Mount Display
  • Projection image generation method for achieving the above object is a plurality of images including a plurality of acquisition images acquired with respect to the actual three-dimensional space and the acquisition position and acquisition angle of each of the plurality of acquisition depth values Estimating the acquisition pose and the plurality of depth acquisition poses based on a reference coordinate system; Acquiring a user pose including a position and an angle of a user in a virtual three-dimensional space corresponding to the actual three-dimensional space based on a reference coordinate system; And generating a projection image in which the plurality of acquisition depth values are projected onto at least one of the plurality of acquired images based on the corresponding image acquisition pose corresponding to the user pose and at least one corresponding depth acquisition pose.
  • At least one of the plurality of acquisition images used for generating the projection image is an acquisition image acquired by the corresponding image acquisition pose and is set as an image representing an acquisition coordinate system which is a coordinate system whose origin is the corresponding image acquisition pose.
  • the plurality of acquisition depth values which are representative images and are used for generating the projection image, are a plurality of depth values obtained by the corresponding depth acquisition poses, or a plurality of depth values based on the origin of the reference coordinate system or the origin of the acquisition coordinate system. It may include a plurality of depth corresponding to the depth value included in the angle of view of the representative image from among a plurality of depth values based on.
  • the generating of the projection image may be performed by projecting the plurality of corresponding depth values to a plurality of corresponding pixels which are pixels included in the representative image corresponding to each of the plurality of corresponding depth values.
  • the user's pose is different from the corresponding image acquisition pose, and there is a play field area not included in the representative image among the virtual three-dimensional space regions corresponding to the viewing angle of the user in the user pose.
  • the generating of the projection image may include generating the projection image by using the reinforcement image and the representative image, the acquisition image including a plurality of missing pixels, which are pixels corresponding to the u-field area, among the plurality of acquisition images except for the representative image. Generating the projection image.
  • generating the projection image by using the reinforcement image and the representative image comprises: determining whether the idle field area exists; If it is determined that the idle field region exists, the plurality of acquisition images based on depth-image linkage information corresponding to the plurality of acquisition depth values corresponding to the plurality of pixels included in the plurality of acquisition images is matched. Selecting the reinforcement image; Generating a combined image by mutually combining the reinforcement image with a weight lower than the representative image; And generating a projection image in which the plurality of acquisition depth values are projected onto the combination image.
  • the step of selecting the reinforcement image from the plurality of acquired images is to perform a raycasting around the user pose to detect a plurality of user field depth values which are depth values included in the viewing angle of the user. Doing; Detecting a plurality of playing field depth values, which are depth values corresponding to the lost field of view, among the plurality of user field depth values; Detecting the plurality of missing pixels corresponding to the plurality of lost field depth values based on the depth-image linkage information; And selecting the reinforcement image from the plurality of acquired images based on the detected plurality of missing pixels.
  • the depth-image linkage calculating a plurality of first depth value distances which are distances from the plurality of acquisition depth values acquired in the depth acquisition pose to the depth acquisition pose; Using the image acquisition pose and the depth acquisition pose, the first depth value distances are a plurality of second depth values which are distances from the plurality of acquisition depth values acquired in the depth acquisition pose to the image acquisition pose Converting to distance; And mapping the plurality of acquisition depth values to a plurality of pixels included in the acquired image by using a camera matrix corresponding to the image sensor used to acquire the acquired image and the plurality of second depth value distances. Can be calculated.
  • the step of calculating a plurality of first depth value distances, which are distances from the plurality of acquisition depth values acquired by the depth acquisition pose, to the depth acquisition pose, is such that the acquisition time of each of the plurality of acquisition depth values is a predetermined time. It can be performed using only the plurality of acquisition depth values within the interval.
  • the user's pose is the same as the corresponding image acquisition pose, and a depth map in which the angle of view of the acquired image acquired in the corresponding image acquisition pose consists of the plurality of acquisition depth values acquired in the corresponding depth acquisition pose.
  • generating the projection image may include the plurality of depth maps and the representative image acquired from the plurality of corresponding depth acquisition poses falling within a predetermined distance from the corresponding image acquisition pose from which the representative image is acquired. It can be performed using.
  • the user's pose is the same as the corresponding image acquisition pose, and a depth map in which the angle of view of the acquired image acquired in the corresponding image acquisition pose consists of the plurality of acquisition depth values acquired in the corresponding depth acquisition pose.
  • generating the projection image may be performed by grouping the plurality of depth maps acquired from a plurality of corresponding depth acquisition poses within a predetermined distance from the corresponding image acquisition pose from which the representative image is acquired. The grouping depth map and the representative image may be performed.
  • the depth map composed of the plurality of acquisition depth values may be configured as an acquisition depth value of the mesh structure based on a preset parameter.
  • the generating of the projection image is performed using only the acquisition depth values acquired within a predetermined time from a time point at which each of the plurality of acquisition images to be used for generating the projection image among the plurality of acquisition depth values is acquired.
  • the acquisition angle included in each of the plurality of image acquisition poses is an omni-directional angle when the acquired image included in each of the plurality of image acquisition poses is acquired by an omni-directional camera. Can be.
  • the projection image generating apparatus for achieving the above object includes a plurality of acquisition images and the acquisition position and the acquisition angle of each of the plurality of acquisition images and a plurality of acquisition depth values acquired with respect to the actual three-dimensional space
  • a data pose estimator for estimating an image acquisition pose and a plurality of depth acquisition poses based on a reference coordinate system
  • a user pose acquisition unit for acquiring a user pose including a position and an angle of a user in a virtual three-dimensional space corresponding to the actual three-dimensional space based on a reference coordinate system
  • an image generation unit configured to generate a projection image in which the plurality of acquisition depth values are projected onto at least one of the plurality of acquired images based on a corresponding image acquisition pose corresponding to the user pose and at least one corresponding depth acquisition pose. can do.
  • At least one of the plurality of acquisition images used for generating the projection image is an acquisition image acquired by the corresponding image acquisition pose and is set as an image representing an acquisition coordinate system which is a coordinate system whose origin is the corresponding image acquisition pose.
  • the plurality of acquisition depth values which are representative images and are used for generating the projection image, are a plurality of depth values obtained by the corresponding depth acquisition poses, or a plurality of depth values based on the origin of the reference coordinate system or the origin of the acquisition coordinate system. It may include a plurality of depth corresponding to the depth value included in the angle of view of the representative image from among a plurality of depth values based on.
  • the corresponding field acquisition pose is different from the user pose, and there is a play field area not included in the representative image among the regions of the virtual three-dimensional space corresponding to the viewing angle of the user in the user pose.
  • the image generation unit may generate the projection image using the reinforcement image and the representative image, which is an acquisition image including a plurality of missing pixels, which are pixels corresponding to the u-field area, among the plurality of acquisition images excluding the representative image. Can be generated.
  • the image generating unit comprises a user field of view determining unit for determining whether or not the idle field area exists; If it is determined that the idle field region exists, the plurality of acquisition images based on depth-image linkage information corresponding to the plurality of acquisition depth values corresponding to the plurality of pixels included in the plurality of acquisition images is matched.
  • a reinforcement image selecting unit selecting the reinforcement image;
  • a combination image generator for combining the reinforcement image with a weight lower than the representative image to thereby combine the combined image;
  • a projection image generating unit projecting the projection image projecting the plurality of acquisition depth values onto the combination image.
  • the user's pose is the same as the corresponding image acquisition pose, and a depth map in which the angle of view of the acquired image acquired in the corresponding image acquisition pose consists of the plurality of acquisition depth values acquired in the corresponding depth acquisition pose.
  • the image generation unit uses the plurality of depth maps obtained from the plurality of corresponding depth acquisition poses within a predetermined distance from the corresponding image acquisition pose from which the representative image is acquired, and the projection by using the representative image. You can create an image.
  • the projection image generating apparatus for achieving the above object is a reference coordinate system a plurality of image acquisition poses including the acquisition position and the acquisition angle of the plurality of acquired images with respect to the actual three-dimensional space
  • a data pose estimator estimating a plurality of depth acquisition poses corresponding to a plurality of acquisition depth values based on an acquisition coordinate system based on each of the plurality of image acquisition poses
  • a user pose acquisition unit for acquiring a user pose including a position and an angle of the user in a virtual three-dimensional space based on a reference coordinate system
  • an image generation unit configured to generate a projection image in which the plurality of acquisition depth values are projected onto at least one of the plurality of acquired images based on a corresponding image acquisition pose corresponding to the user pose and at least one corresponding depth acquisition pose. do.
  • the mapping method between the image pixel and the depth value for achieving the above object is based on the image acquisition pose and the depth acquisition pose including the acquisition position and the acquisition angle of each of the acquired image and the acquired depth value for the actual three-dimensional space. Estimating based on a coordinate system; Calculating a plurality of first depth value distances which are distances from the plurality of acquisition depth values acquired in the depth acquisition pose to the depth acquisition pose; Using the image acquisition pose and the depth acquisition pose, the first depth value distances are a plurality of second depth values which are distances from the plurality of acquisition depth values acquired in the depth acquisition pose to the image acquisition pose Converting to distance; And mapping the plurality of acquisition depth values to a plurality of pixels included in the acquired image by using a camera matrix corresponding to the image sensor used to acquire the acquired image and the plurality of second depth value distances. do.
  • the step of converting the plurality of second depth value distances is performed by multiplying each of an inverse matrix of the matrix representing the image acquisition pose, the matrix representing the depth acquisition pose, and the first depth value distances.
  • an inverse matrix of the matrix representing the image acquisition pose the matrix representing the depth acquisition pose, and the first depth value distances.
  • mapping the plurality of acquisition depth values to a plurality of pixels included in the acquired image by selecting three or more initial connection depth values from among initial connection depth values which are a plurality of acquisition depth values mapped to the image pixel.
  • Constructing the mesh Calculating a number and positions of a plurality of unmapped pixels to be included in the polygon mesh from among a plurality of unmapped pixels that are image pixels having no corresponding acquisition depth value based on the resolution of the image sensor; Estimating a plurality of depth values corresponding to the plurality of unmapped pixels based on the calculated number and positions of the plurality of unmapped pixels; And mapping the estimated plurality of depth values to the plurality of unmapped pixels.
  • estimating a plurality of depth values corresponding to the plurality of unmapped pixels may be performed using interpolation.
  • image distortion generated in the process of converting the acquired image acquired on the basis of the acquisition coordinate system to the reference coordinate system reference can be removed.
  • image distortion generated in the process of converting the acquired image acquired on the basis of the acquisition coordinate system to the reference coordinate system reference can be removed.
  • the projection image is generated using only one representative image linked to each acquisition coordinate system of the image acquisition pose, the effect of generating a projection image with a much smaller amount of data than in the related art have.
  • the projection image is generated using a representative image linked to each acquisition coordinate system of the image acquisition pose, when the image corresponding to the actual three-dimensional space is changed, the image acquisition pose such as a signboard replacement. Only by acquiring a new acquired image and setting it as a representative image, it is possible to easily upgrade the entire image data to the latest image data.
  • the depth-image linkage information through the depth-image linkage information, it is possible to determine which pixels of the acquired image are included in the user's field of view through the depth value, and through this, it is possible to identify which acquired image should fall within the user's field of view. Therefore, even when the user is located in the user pose outside the image acquisition path, there is an effect of generating a projection image corresponding to an arbitrary view of the user in the user pose.
  • a distance sensor having an angle of view narrower than the angle of view of the image sensor while using it, there is an effect of generating a projection image having a wide angle of view corresponding to the wide angle of view of the image sensor.
  • the acquisition depth value to be used for generating the projection image of the plurality of acquisition depth values based on the acquisition time and the acquisition depth value of each of the acquired image, the image due to the ring closure situation, etc. This has the effect of minimizing distortion.
  • FIG. 1 is a flowchart illustrating a method of generating a projected image according to an embodiment of the present invention.
  • FIG. 2 is a diagram showing an example in which the depth value of the present invention is expressed as a basis vector.
  • FIG. 3 is a diagram illustrating an example of an integrated sensor for simultaneously acquiring an image and a depth value.
  • FIG. 4 is a diagram illustrating a 3D image generated based on a conventional reference coordinate system.
  • 5 is a diagram for explaining image distortion according to a position estimation error.
  • FIG. 6 is a diagram illustrating a depth acquisition pose according to a reference coordinate system.
  • FIG. 7 is a diagram for explaining terrain distortion and image distortion according to a position estimation error.
  • FIG. 8 is a view illustrating an angle of view of an image sensor.
  • FIG. 9 is a diagram illustrating an angle of view of a distance sensor.
  • FIG. 10 is a diagram for describing a method of generating a projection image by using one image acquisition pose and a plurality of depth acquisition poses.
  • FIG. 11 is a diagram illustrating an example in which depth values are grouped.
  • FIG. 12 is a view illustrating a ring closing situation.
  • FIG. 13 is a diagram for explaining the distortion of the depth grouping in the ring closing situation.
  • FIG. 14 is a flowchart illustrating a method of generating depth-associated image information according to an embodiment of the present invention.
  • 15A is a diagram illustrating depth-image linkage according to an embodiment of the present invention.
  • 15B is a diagram illustrating depth-image linkage according to another embodiment of the present invention.
  • 15C is a diagram illustrating depth-image linkage according to another embodiment of the present invention.
  • 16 is a flowchart illustrating a method of generating a projection image according to another exemplary embodiment of the present invention.
  • 17 is a flowchart illustrating a method of generating a projected image when there is an active field area according to an embodiment of the present invention.
  • 18 is a diagram illustrating an environment in which an angle of view of an image sensor includes a user's field of view.
  • 19 is a diagram illustrating an environment in which an angle of view of an image sensor does not include a user's field of view.
  • 20 is a flowchart illustrating a reinforcement image selection method according to an embodiment of the present invention.
  • 21 is a view illustrating a difference between the present invention and the prior art according to a user's movement.
  • FIG. 22 is a diagram illustrating a parameter-based depth mesh structure according to an embodiment of the present invention.
  • FIG. 23 is a diagram for describing a projection image generating device according to an embodiment of the present invention.
  • 24 is a diagram for explaining an image generating unit according to an embodiment of the present invention.
  • 25 is a view illustrating a polygon mesh structure of a depth value according to an embodiment of the present invention.
  • first, second, A, and B may be used to describe various components, but the components should not be limited by the terms. The terms are used only for the purpose of distinguishing one component from another.
  • the first component may be referred to as the second component, and similarly, the second component may also be referred to as the first component.
  • FIG. 1 is a flowchart illustrating a method of generating a projected image according to an embodiment of the present invention.
  • the projection image generating apparatus references the plurality of image acquisition poses and the plurality of depth acquisition poses including the acquisition positions and acquisition angles of each of the plurality of acquired images and the plurality of acquired depth values with respect to the actual three-dimensional space. Estimate based on the coordinate system.
  • the projection image generating apparatus includes a plurality of image acquisition poses and a plurality of acquisition depth values each including a position where each of the plurality of acquired images is acquired and an angle (including yaw, pitch, and rolling angle) from which the image is acquired.
  • the acquired image and image acquisition time Information on absolute acquisition time, relative acquisition time, and the posterior relationship of the acquired image, etc., the acquisition depth value and depth value acquisition time (absolute acquisition time, relative acquisition time, and acquisition depth value acquired at each depth acquisition pose) And so on).
  • the image acquisition pose may include only the position where the image is acquired, and the depth acquisition pose may include only the position where the depth value is obtained.
  • the acquisition angle included in the image acquisition pose may be an omnidirectional angle.
  • the acquired image may have a form that can be expressed as a basis vector of two degrees of freedom as a two-dimensional image of a three-dimensional space, and a form that expresses three-dimensional color values in two dimensions, such as a camera, or infrared rays to the camera. It may be a form in which the three-dimensional thermal information is expressed in two dimensions by mounting a filter.
  • the acquired image may be a panoramic image or a wide-angle image generated by using a plurality of cameras. Since the image in the present invention is defined as a form capable of representing two degrees of freedom as a basis vector, not only a general rectangular shape but also a camera characteristic It may have a geometric shape other than square.
  • the acquisition depth value has a point shape that can be expressed as a basis vector of three degrees of freedom, may be acquired using a distance sensor, or may be estimated by using two or more images photographed at different places.
  • the former include depth values obtained through LIDAR, SONAR, InfraRed, and Time of Flight (TOF) range detectors.
  • the latter include stereo cameras, multi-cameras, and omnidirectional stereos.
  • a device such as Kinect, JUMP, PrimeSense, Project Beyond or the like is used, the acquired image and the acquired depth value can be acquired simultaneously.
  • FIG. 2 is a diagram illustrating an example in which a depth value of the present invention is expressed as a basis vector, and a depth value is a polar coordinate system represented by ⁇ , ⁇ , or ⁇ as shown on the left side, or X, Y, Z as the right side.
  • the Cartesian Coordinate represented by the axis may be used as the basis vector of three degrees of freedom.
  • the depth value may be composed of a single point form or a plurality of point forms, and may also include a depth value set of a triangle-based mesh structure composed of points, and a depth value set of a specific geometry.
  • the depth value may be newly estimated and used through interpolation as well as the acquisition depth value acquired through the distance sensor. More specifically, three or more depth values may be selected from among a plurality of acquired depth values to form a polygonal (including triangle) mesh, and the new depth value may be estimated and added through interpolation within the polygon mesh. have.
  • the acquired image and the acquired depth value according to the exemplary embodiment of the present invention may be simultaneously acquired using the integrated sensor as shown in FIG. 3.
  • FIG. 3 is a diagram illustrating an example of an integrated sensor for simultaneously acquiring an image and a depth value
  • FIG. 3 illustrates an integrated sensor in which a wide-angle image sensor at the top and a distance sensor Lidar at the bottom are combined.
  • the wide-angle image sensor may have an omnidirectional angle of view vertically or horizontally, but a distance sensor such as Lidar generally has a narrow angle of view of a predetermined angle.
  • the pose in three-dimensional space may be estimated by further using an inertial sensor, an acceleration sensor, a geomagnetic sensor, and a wireless communication signal.
  • the projection image generating apparatus estimates a plurality of image acquisition poses and a plurality of depth acquisition poses previously stored based on the reference coordinate system based on the reference coordinate system, which may be referred to as an absolute coordinate system (W), All poses are displayed based on one origin.
  • a concept opposite to the reference coordinate system is an acquisition coordinate system, which may be referred to as a local coordinate system and has a plurality of origins, and displays a pose based on each of the plurality of origins.
  • an image acquisition pose from which an image is acquired is displayed based on a reference coordinate system
  • the pose of each of the plurality of cameras is displayed based on one specific origin, but the image acquisition pose Is displayed based on the acquisition coordinate system, the plurality of points where each of the plurality of cameras is located become the origin of the individual acquisition coordinate system.
  • the reference coordinate system when the reference coordinate system is applied to a building composed of a plurality of floors, the reference coordinate system may be set for each floor, and the origin may be set for each floor.
  • the projection image generating apparatus acquires a user pose including a position and an angle of the user in the virtual three-dimensional space corresponding to the actual three-dimensional space based on the reference coordinate system.
  • the user pose may be an angle indicating the position of the user's avatar displayed in the virtual three-dimensional space and the direction in which the user's view is directed.
  • the projection image generating apparatus generates a projection image in which the plurality of acquisition depth values are projected onto at least one of the plurality of acquired images based on the corresponding image acquisition pose corresponding to the user pose and the at least one corresponding depth acquisition pose. do.
  • the corresponding image acquisition pose refers to an image acquisition pose corresponding to the user pose
  • the corresponding depth acquisition pose refers to a depth acquisition pose corresponding to the user pose.
  • one image acquisition pose and at least one depth acquisition pose existing within a predetermined distance from the user pose are determined as the corresponding image acquisition pose and the corresponding depth acquisition pose, or one image acquisition pose located at the closest distance from the user pose.
  • one depth acquisition pose may be determined as a corresponding image acquisition pose and a corresponding depth acquisition pose.
  • the corresponding image acquisition pose may be determined as an image acquisition pose from which an acquired image having the highest overlap with the user's field of view in the user pose is acquired.
  • the user pose may be determined to be the same as the corresponding image acquisition pose or the depth acquisition pose, or may be determined to be different. For example, if the difference value between the user pose and the corresponding image acquisition pose and the difference value between the user pose and the depth acquisition pose are each within a predetermined threshold, it is determined that the user pose is the same as each of the corresponding image acquisition pose and the depth acquisition pose, otherwise The user pose may be determined to be different from the corresponding image acquisition pose and the depth acquisition pose, respectively.
  • the acquired image used for generating the projected image is a representative image set as an image representing the acquired coordinate system, which is a coordinate system having the corresponding image acquisition pose as the origin as the acquired image acquired from the corresponding image acquisition pose.
  • the plurality of acquisition depth values used for generating the projected image may be a plurality of depth values obtained by the corresponding depth acquisition poses, or a plurality of depth values based on the origin of the reference coordinate system or a plurality of reference values based on the origin of the acquisition coordinate system.
  • the depth value may be a plurality of corresponding depth values which are depth values included in the angle of view of the representative image.
  • a representative image for each corresponding image acquisition pose may be set based on the acquisition coordinate system. Accordingly, when the user enters a specific acquisition coordinate system, only one representative image is displayed in the acquisition coordinate system. . As a result, as shown in FIG. 4 to be described later, a natural image with minimal distortion is displayed to the user as compared to a conventional method of generating a projection image based on a reference coordinate system by combining a plurality of images having a predetermined size to generate a projection image corresponding to a user pose. You can do it.
  • the projection image generating apparatus may correspond to a plurality of corresponding pixels that are pixels included in the representative image corresponding to each of the plurality of corresponding depth values that are depth values included in the angle of view of the representative image. Project the depth value to generate the projected image.
  • the corresponding pixel corresponding to the corresponding depth value may be determined based on the depth-image linkage information.
  • a detailed operation of generating the depth-image linkage information will be described later with reference to FIGS. 14 and 15.
  • a projection image may be generated by projecting a plurality of acquisition depth values acquired in a plurality of corresponding depth acquisition poses onto one representative image acquired in one corresponding image acquisition pose, which is This is due to the difference in the angle of view of the depth map composed of the angle of view of the image and the plurality of depth values. This will be described later with reference to FIGS. 8 to 10.
  • the projection image when the user pose is different from the corresponding image acquisition pose and there is a non-field view area in the virtual three-dimensional space corresponding to the viewing angle of the user in the user pose that is not included in the representative image, the projection image
  • the generating apparatus may generate the projection image by using the representative image and the reinforcement image, which is an acquisition image including a plurality of missing pixels, which are pixels corresponding to the u-field area, among the plurality of acquisition images except for the representative image. It will be described later with reference to FIG.
  • FIG. 4 is a diagram illustrating a 3D image generated based on a conventional reference coordinate system.
  • a separate acquired image exists for each of a plurality of image acquisition poses based on a reference coordinate system, and a depth value is projected onto a combined image generated by combining a plurality of images existing for each image acquisition pose.
  • a projection image To generate a projection image, and finally generate a 3D image using two projection images corresponding to each of the left eye and the right eye.
  • the unnatural combined image is displayed to the user as shown in FIG. 4.
  • image distortion may occur due to a position estimation error of each of the image acquisition pose and the depth acquisition pose, which will be described with reference to FIG. 5.
  • 5 is a diagram for explaining image distortion according to a position estimation error.
  • an image distortion in which an image of a pillar is formed on a floor is illustrated.
  • the reason for such distortion may be caused by a data (image or depth value) acquisition point in a computer-implemented virtual 3D environment. While it is possible to know the absolute pose of, it is necessary to estimate the absolute coordinate values of the various poses obtained when the data is acquired in the real 3D environment, and an estimation error inevitably occurs in this estimation process.
  • FIG. 6 is a diagram illustrating a depth acquisition pose according to a reference coordinate system.
  • FIG. 6A ideal depth acquisition pose without error , , Is estimated based on W
  • FIG. 6B an estimated depth acquisition pose in which an error exists.
  • , 6 is indicated based on W. Referring to FIG. 6B, the farther from W, the larger the error of the estimated depth acquisition pose. That is, in FIG. 6B , , It can be seen that the error becomes larger as.
  • W represents the origin of the reference coordinate system
  • Li (L1, L2, L3) represents the point where the depth value is obtained
  • the estimated depth acquisition pose The process of estimating the ideal depth acquisition pose Is the sum of the errors , which is the square of the absolute sum of errors For example, this method is performed by minimizing indices such as Nearest Estimated Error Square considering the probability distribution.
  • Equation 1 represents a depth acquisition pose in a 3D environment and can be described in the 2D environment by lowering the dimension by one level, and the depth acquisition pose in the present invention can be defined in both a 3D environment and a 2D environment.
  • the depth acquisition pose is displayed using the H matrix, but the present invention is not limited thereto, but the combination of the position vector and the angle vector, Euler, Quaternion, and Dual for the angular part may be used. It is also possible to display them using quaternion and Rodriguez representations.
  • the image acquisition pose based on the reference coordinate system is for Ci which is the point where the image is acquired. , Which represents the 3D angle from the W to the Ci point. , Representing the 3D travel distance to the Ci point relative to W It can be configured by fusing.
  • the present invention is not limited thereto, and the image acquisition pose may be displayed using various expression methods other than the H matrix.
  • the pose relation between the image sensor and the distance sensor is generally known, the pose relation between the two Can be seen as a given constant value, according to which the depth acquisition And image acquisition If one value is estimated, the other value can be estimated without any measurement.
  • FIG. 7 is a diagram for explaining terrain distortion and image distortion according to a position estimation error.
  • FIG. 7A shows an ideal m-th data (image or depth value) pose of an error-free reference coordinate system.
  • the acquired image Im and the acquired depth value Dm are shown in the figure, and the ideal nth data pose is shown.
  • the acquired image In and the acquired depth value Dn acquired at are shown.
  • the portion corresponding to O represents a pillar and the portion corresponding to X represents a bottom.
  • FIG. 7B is an m-th estimated data pose of a reference coordinate system with an error present And nth estimated data poses Is shown, the mth estimated data pose Is the ideal m-th data pose of FIG. 7A Little error with, but with nth estimated data pose Is the ideal nth data pose of FIG. It can be seen from the right that the error biased to the right occurred.
  • the conventional method for generating a projected image based on a reference coordinate system has a problem in that image distortion occurs as shown in FIGS. 5 and 7B due to a position error.
  • a method of generating a projected image based on an acquired coordinate system but in this case, there is an advantage that there is no image distortion due to a position estimation error, but due to a difference between an angle of view of an image sensor and an angle of view of a distance sensor,
  • the projection image can be generated only for a narrow field of view where the angle of view and the angle of view of the distance sensor overlap.
  • FIG. 8 is a view illustrating an angle of view of an image sensor.
  • FIG. 9 is a diagram illustrating an angle of view of a distance sensor.
  • a person acquiring a depth value using the distance sensor looks at an obstacle indicated in gray at an Lj point, which is a depth value acquisition point, and shows a depth value.
  • Lj point which is a depth value acquisition point
  • only a part of the area indicated by the hatched line having a narrow range in the front and rear directions around Lj becomes the angle of view of the distance sensor. Therefore, only the projection image of the area where the angle of view of the image sensor of FIG. 8 and the distance sensor of FIG. 9 overlap may be manufactured.
  • This implementation uses the combination of two cameras to produce the same image angle and the corresponding depth map, but the accuracy of the depth value is significantly lowered, resulting in a large error in pose estimation based on the reference coordinate system of the image acquisition pose or depth acquisition pose. Will occur. In this case, not only distortion as shown in FIG. 4 or FIG. 7, but also in the space where the user is photographed in the order of A, B, C, D, ..., X, Y, and Z as shown in FIG. When the shift from C to X occurs, a sudden unnaturalness occurs at the moment of transition between the two points due to the pose estimation error at C and the pose estimation error at X.
  • the present invention generates a projection image using one image acquisition pose and a plurality of depth acquisition poses when the angle of view of the distance sensor is smaller than the angle of view of the image sensor, so that the projection image corresponding to the user may move out of the image acquisition path. It can be provided, which will be described with reference to FIG.
  • FIG. 10 is a diagram for describing a method of generating a projection image by using one image acquisition pose and a plurality of depth acquisition poses.
  • an angle of view of a wide-angle image sensor at a specific image acquisition pose is shown at the top, and an angle of view of a distance sensor at different depth acquisition poses is shown at the bottom.
  • the lower left side represents an angle of view of the distance sensor at the depth value acquisition point Lj-1
  • the lower center of the center represents an angle of view of the distance sensor at the depth value acquisition point Lj
  • the lower right side is at the depth value acquisition point Lj + 1.
  • the user pose is the same as the corresponding image acquisition pose
  • the angle of view of the acquired image acquired in the corresponding image acquisition pose consists of a plurality of acquisition depth values acquired in the corresponding depth acquisition pose. If the depth map is larger than the angle of view (different pose-angle of view), a plurality of depth maps and one representative image acquired from a plurality of corresponding depth acquisition poses within a predetermined distance from the corresponding image acquisition pose from which the representative image is acquired are used. To generate the projection image.
  • the projection image is generated using a distance sensor having an angle of view narrower than that of the image sensor as in the conventional technique, only a projection image having a narrow angle of view corresponding to the narrow angle of view of the distance sensor may be generated. Since a plurality of depth maps corresponding to all angles of view of the acquired image can be used as shown in 10, even through a distance sensor having an angle of view narrower than that of the image sensor, a projection image having a wide angle of view corresponding to a wide angle of view of the image sensor can be obtained. It can be created.
  • the image acquisition pose is represented based on the reference coordinate system, and the depth values acquired in the depth acquisition pose are also expressed based on the reference coordinate system, so that the image pixels acquired in the specific image acquisition pose have a certain depth. This is because it can be determined using a reference coordinate system. A more detailed description thereof will be described later with reference to FIG. 15B.
  • the same effect can be obtained by expressing a depth value based on an acquisition coordinate system having an image acquisition pose as a reference point, which will be described later with reference to FIG. 15C.
  • the depth values corresponding to all image pixels may not be found.
  • the depth values may not be mapped using the image pixels mapped to the depth values. Since the depth value of the image pixels can be easily estimated through interpolation or the like, there is no problem in implementing the present invention.
  • the representative image set in the corresponding image acquisition pose corresponding to the current user pose may be used, and corresponding to the current user pose. Even if the angle of view of the depth map acquired in one corresponding depth acquisition pose is narrower than the angle of view of the representative image, a depth map having the same angle of view as the representative image can be provided using the depth map acquired in a plurality of different depth acquisition poses. Even if the user is out of the image acquisition path, it is possible to produce a naturally realistic projection image.
  • all depth maps included in an angle of view of a representative image corresponding to a specific image acquisition pose must be found. Can be used. That is, when the corresponding image acquisition pose corresponding to the current user pose is determined, a plurality of depth maps included in the angle of view of the representative image acquired in the corresponding image acquisition pose are detected using the depth-image linkage information, and the detected The projection image may be generated using the plurality of depth maps and one representative image.
  • the depth values corresponding to the corresponding image acquisition pose are grouped only once in advance and then grouped.
  • the depth value may be used to generate a projection image. This will be described later with reference to FIG. 11.
  • FIG. 11 is a diagram illustrating an example in which depth values are grouped.
  • the depth values grouped in correspondence with the image acquisition pose are mapped, it is not necessary to calculate a new depth value each time, thereby increasing the computational efficiency.
  • FIG. 12 is a view illustrating a ring closing situation.
  • FIG. 12 illustrates a closed position situation.
  • a user visits a user's pose that is close to each other over two times (t1 and t100) with a long time interval, different shooting conditions due to the long time interval ( Lighting, exposure time, lens focal length, etc.) and the proximity of the distances, due to the proximity of the distances, despite the cumulative pose estimation errors during the second visit.
  • FIG. 13 is a diagram for explaining the distortion of the depth grouping in the ring closing situation.
  • the grouped depth values for Lj + 100 which are the j + 100th depth acquisition points, are displayed at an inclination by a predetermined angle due to a position error according to the ring closing situation, which is displayed on the acquired image acquired at Ci. Projection will result in severe image distortion
  • FIG. 14 is a flowchart illustrating a method of generating depth-associated image information according to an embodiment of the present invention.
  • the projection image generating apparatus estimates an image acquisition pose and a depth acquisition pose including an acquisition position and an acquisition angle of each of the acquired image and the acquired depth value with respect to the actual three-dimensional space, based on the reference coordinate system.
  • step 1420 the projection image generating apparatus calculates a plurality of first depth value distances which are distances from the plurality of acquisition depth values acquired in the depth acquisition pose to the depth acquisition pose.
  • the projection image generating apparatus includes a j-th depth acquisition pose. For the kth depth value obtained from, the jth depth acquisition pose from the kth depth value First depth value distance to Can be calculated.
  • the projection image generating apparatus uses the image acquisition pose and the depth acquisition pose to determine a plurality of first depth value distances, the plurality of which are distances from each of the plurality of acquisition depth values acquired at the depth acquisition pose to the image acquisition pose. Convert to the second depth value distance of.
  • the projection image generating apparatus may calculate the second depth value distance by Equation 2.
  • the second depth value distance I is a matrix representing the i-th image acquisition pose Inverse of , the matrix representing the jth depth acquisition pose And first depth value distance Can be calculated by multiplying each other.
  • the projection image generating apparatus maps the plurality of acquisition depth values to the plurality of pixels included in the acquired image by using a camera matrix corresponding to the image sensor used to acquire the acquired image and the plurality of second depth value distances. do.
  • the projection image generating apparatus may map a plurality of acquisition depth values to a plurality of image pixels included in the acquired image by Equation 3.
  • Equation 4 the camera matrix is a matrix for rotating and moving a pixel included in an image disposed on an actual three-dimensional space based on the position of the image sensor. Since the concept is obvious to those skilled in the art, a detailed description thereof will be omitted.
  • an unmapped pixel may occur in which no corresponding depth value exists among the image pixels.
  • the projection image generating apparatus estimates the unmapped pixel using interpolation. Mapped depth values.
  • the projection image generating apparatus selects three or more initial connection depth values from among initial connection depth values, which are a plurality of acquisition depth values mapped to the image pixels, to form a polygon mesh. And the number and positions of the plurality of unmapped pixels to be included in the polygon mesh among the plurality of unmapped pixels that are image pixels having no corresponding acquisition depth value based on the resolution of the image sensor. Calculate a plurality of depth values corresponding to the plurality of unmapped pixels based on the calculated number and positions of the unmapped pixels, and then map the estimated depth values to the plurality of unmapped pixels. Can be further performed.
  • FIG. 25 is a diagram illustrating a polygon mesh structure of a depth value according to an embodiment of the present invention.
  • a number of triangle meshes are displayed on an acquired image.
  • the vertices are shown in black as the acquisition depth, and there is no depth value mapped to the image pixels included in the triangle mesh.
  • the projection image generating apparatus may calculate the number and positions of the image pixels included in the triangle mesh and estimate and map depth values corresponding to the image pixels by using interpolation.
  • the depth value mapped to the plurality of unmapped pixels may be referred to as an additional connection depth value.
  • the projection image generating device finally generates the depth-image association information.
  • the depth-image linkage account thus generated may be used to estimate a depth value included in the acquired image.
  • the projection image generating apparatus performs raycasting around each of the plurality of image acquisition poses to obtain a plurality of connection depth values (initial depths) which are acquisition depth values included in the acquired images acquired in each of the plurality of image acquisition poses.
  • the linkage depth value and the additional linking depth value can be detected based on the reference coordinate system. do.
  • the projection image generating apparatus may detect a plurality of connection depth values by performing ray casting around each of the plurality of image acquisition poses.
  • depth acquisition pose are all displayed based on the reference coordinate system
  • the projection image generating apparatus Depth acquisition based on depth-image association information Depth acquisition by performing ray casting toward K depth values in Image acquisition pose It can be confirmed that it is included in the angle of view of the acquired image acquired from.
  • the linking depth value when detecting a plurality of linking depth values, may be detected only for a plurality of acquisition depth values for which the acquisition time of each of the plurality of acquisition depth values is within a predetermined time interval.
  • the projection image generating apparatus may detect the associated depth value using only the acquired depth value from the time t1 to t10.
  • the information matching the plurality of linking depth values based on the acquired coordinate system with the plurality of pixels included in the acquired image is depth-image linkage information, and the depth-image linked account is generated in each of the plurality of image acquisition poses. Can be.
  • the depth-image linkage information may be information obtained by matching a plurality of linking depth values based on the reference coordinate system with a plurality of pixels included in the acquired image.
  • the projection image generating apparatus performs the operation based on the reference coordinate system, but may perform the operation based on the acquisition coordinate system based on the origin of the image acquisition pose.
  • 15A is a diagram illustrating depth-image linkage according to an embodiment of the present invention.
  • Image acquisition pose And depth value acquisition points Depth acquisition pose at Means that it is estimated based on the reference coordinate system.
  • the axes indicated by are connected with the point representing the depth value and the arrow, It is indicated by the jth depth acquisition pose.
  • the jth depth acquisition pose For the kth depth value obtained from, the jth depth acquisition pose from the kth depth value First depth value distance to It is shown.
  • an image acquisition point is located at the top of FIG. 15A.
  • the image plane is mapped to The origin of the coordinate axis denoted by represents the center point of the image sensor (camera) and the image plane represents the two-dimensional image plane formed when the three-dimensional space is viewed from the CMOS sensor mounted on the image sensor.
  • the image acquisition point Distance to the depth value is the second depth value distance
  • Image acquisition point The image pixel p is displayed at the point where the straight line connected to the depth value from the image plane intersects the pixel p and the second depth value distance. Means that they are mapped to each other.
  • the image acquisition point Lines connected in a straight line to the depth value may be out of the image plane and there may not be an intersection point.
  • an unmapped pixel which is an image pixel having no corresponding depth value, is generated.
  • 15B is a diagram illustrating depth-image linkage according to another embodiment of the present invention.
  • Fig. 15B estimated i-th image acquisition point And the estimated j-1, j, j + 1th depth value acquisition points , , Is connected by an arrow to the axes labeled W, which is the image acquisition point.
  • Image acquisition pose and depth value acquisition point , , Depth acquisition poses at are estimated based on the reference coordinate system.
  • the points scattered and distributed at the bottom of FIG. 15B represent acquired depth values acquired in each of the depth acquisition poses.
  • a coordinate axis denoted by W is connected to a point and an arrow to indicate that the depth value is expressed based on a reference coordinate system. it means.
  • the acquisition depth value is the image acquisition point. It may be displayed based on the acquisition coordinate system whose origin is.
  • the image acquisition point A representative image is mapped to, which is the image acquisition point Indicates that the image is set to an image representing the acquisition coordinate system whose origin is.
  • the image acquisition pose and the depth acquisition pose are estimated based on a reference coordinate system, and in the image acquisition pose, one representative image is obtained from a plurality of acquired images acquired in the image acquisition pose.
  • the image acquisition pose is selected and linked as an image representing the acquisition coordinate system with the origin, and the acquisition depth value is estimated based on the reference coordinate system.
  • the angle of view of the image sensor in the embodiment of Figure 15b is larger than the angle of view of the distance sensor, the depth value acquisition point , , All the depth maps acquired from the depth acquisition poses If included in the angle of view of the representative image linked to the acquisition coordinate system of the projection image generating apparatus, the image acquisition point A representative image and a depth value acquisition point linked to the acquisition coordinate system of , ,
  • the projection image may be generated using the plurality of depth maps acquired from the depth acquisition poses in.
  • 15C is a diagram illustrating depth-image linkage according to another embodiment of the present invention.
  • Fig. 15C estimated i-th image acquisition point Is connected by an arrow to the axes labeled W, which is the image acquisition point. It means that the image acquisition pose at is estimated based on the reference coordinate system.
  • the estimated j-1, j, j + 1th depth value acquisition points , , Image acquisition point It is connected with the coordinate axis indicated by the arrow, which is the estimated j-1, j, j + 1th depth value acquisition point.
  • the depth acquisition pose of the image acquisition point It means that it is estimated on the basis of the acquired coordinate system with.
  • the image acquisition point in Fig. 15c The coordinate axis indicated by is connected to one of a plurality of points representing the depth value at the bottom with an arrow. It is displayed based on the acquisition coordinate system whose origin is. However, in another embodiment, the acquisition depth value may be displayed based on the reference coordinate system.
  • an image acquisition point A representative image is mapped to, which is the image acquisition point Indicates that the image is set to an image representing the acquisition coordinate system whose origin is.
  • the image acquisition pose is estimated based on a reference coordinate system, and the depth acquisition pose is estimated based on an acquisition coordinate system having the image acquisition pose as an origin, and the image acquisition pose is obtained by the image acquisition pose.
  • One representative image among a plurality of acquired images acquired in the pose is selected and linked as an image representing the acquisition coordinate system whose image acquisition pose is the origin, and the acquisition depth value is based on the acquisition coordinate system whose image acquisition pose is the origin. It is estimated.
  • 16 is a flowchart illustrating a method of generating a projection image according to another exemplary embodiment of the present invention.
  • the projection image generating apparatus estimates a plurality of image acquisition poses corresponding to a plurality of acquired images acquired in the actual three-dimensional space based on a reference coordinate system, and acquires a plurality of depths corresponding to the plurality of acquisition depth values.
  • the pose is estimated on the basis of an acquired coordinate system having each of a plurality of image acquisition poses as the origin.
  • the projection image generating apparatus acquires a user pose including a position and an angle of the user in the virtual three-dimensional space corresponding to the actual three-dimensional space based on the reference coordinate system.
  • the projection image generating apparatus generates a projection image in which the plurality of acquisition depth values are projected onto at least one of the plurality of acquired images based on the corresponding image acquisition pose corresponding to the user pose and the at least one corresponding depth acquisition pose. do.
  • the projection image generating apparatus generates a projection image by projecting onto the representative image using only the acquisition depth values acquired within a predetermined time from the time when each of the plurality of acquisition images to be used for generating the projection image among the plurality of acquisition depth values is acquired. It may be. This is to minimize distortion of the projected image due to occurrence of a ring closure situation or the like.
  • the projection image generating apparatus generates a projection image using the representative image linked to the corresponding depth acquisition pose, and may use depth-image association information when selecting the representative image to be linked to the corresponding depth acquisition pose.
  • the projection image generating apparatus may detect an overlapping region between the acquired image and the user's field of view through the depth-image linkage information, and then set the acquired image having the largest overlapping region as the representative image.
  • Such an embodiment may be easily implemented as a method of selecting an image linked to an image acquisition pose closest to a user pose as a representative image.
  • FIG. 17 is a flowchart illustrating a method of generating a projected image when there is an active field area according to an embodiment of the present invention
  • FIG. 18 is a view illustrating an environment in which an angle of view of an image sensor includes a user's field of view
  • FIG. 19 is a diagram for explaining an environment in which an angle of view of an image sensor does not include a user's field of view.
  • step 17 is an embodiment defining step 130 of FIG. 1 or step 1630 of FIG. 16, which is performed after steps 110 and 120 of the embodiment of FIG. 1, and after steps 1610 and 1620 of FIG. 16. It is a step.
  • the projection image generating apparatus determines whether there is an idle field region not included in the representative image among regions on the virtual three-dimensional space corresponding to the viewing angle of the user.
  • an obstacle having a gray square shape exists at the bottom of the i th image acquisition point Ci and the user's location point u, thereby limiting the angle of view of the image sensor located at the i th image acquisition point Ci, thereby preventing the image sensor.
  • a missing field of view represented by an x mark occurs.
  • the user looks at the lost viewing area in the virtual three-dimensional space, since the pixel corresponding to the missing viewing area does not exist in the representative image, it may be displayed in black or white.
  • the depth-image linkage information it is possible to determine which pixels of the acquired image are included in the user's field of view through the depth value, and through this, it is possible to identify which acquired image should fall within the user's field of view.
  • the displayed projection image can be generated such that the pixels corresponding to the lost field of view have the same color as in the actual three-dimensional space. That is, according to an embodiment of the present invention, even when there is an obstacle as shown in FIG. 19 or when the image acquisition pose and the user pose are different, it is possible to generate a natural projection image corresponding to an arbitrary field of view of the user at the corresponding position. It is effective.
  • the projection image generating apparatus is based on the depth-image linkage information, which is information on which a plurality of acquisition depth values corresponding to a plurality of pixels included in the plurality of acquired images are matched.
  • a reinforcement image is selected from a plurality of acquired images.
  • the projection image generating apparatus generates a combination image by combining the reinforcement image and the representative image with each other.
  • the projection image generating apparatus may generate the combined image by combining the reinforcement image with the same weight as the representative image or by combining the reinforcement image with the weight lower than the representative image.
  • step 1742 the projection image generating apparatus generates a projection image in which the plurality of acquired depth values are projected onto the combined image.
  • step 1744 if it is determined in step 1742 that the non-field-of-field area does not exist, the projection image generating apparatus generates a projection image in which a plurality of acquisition depth values are projected onto the representative image.
  • 20 is a flowchart illustrating a reinforcement image selection method according to an embodiment of the present invention.
  • the projection image generating apparatus detects a plurality of user view depth values, which are depth values included in the viewing angle of the user, based on the user pose.
  • the projection image generating apparatus may detect a plurality of user field depth values by performing ray casting or the like around the user pose.
  • the projection image generating apparatus detects a plurality of u-field depth values, which are depth values corresponding to the lost field of view, from the plurality of user-field depth values.
  • the lost field depth value when a user looks at the three-dimensional space while turning around in a specific user pose, there are a plurality of user field depth values corresponding to 360 degrees, but among them, a missing field of view exists at a specific angle.
  • the depth value corresponding to the lost field of view area is detected as the lost field depth value.
  • the projection image generating apparatus detects a plurality of missing pixels corresponding to the plurality of lost field depth values based on the depth-image linkage information.
  • the projection image generating apparatus may know what pixels correspond to each of the plurality of u-field depth values by using the depth-image linkage information, and thus, the pixels corresponding to each of the plurality of u-field depth values are lost. Detection as a pixel.
  • the projection image generating apparatus selects a reinforcement image which is an acquired image including the plurality of missing pixels from the plurality of acquired images, based on the detected plurality of missing pixels.
  • 21 is a view illustrating a difference between the present invention and the prior art according to a user's movement.
  • the image acquisition path is shown as ABCD ...- XYZ.
  • the projection image can be provided only when the user follows the path where such an image is acquired.
  • the present invention even when the user moves from C to X out of the image acquisition path, there is an advantage of providing a natural projection image.
  • FIG. 22 is a diagram illustrating a parameter-based depth mesh structure according to an embodiment of the present invention.
  • FIG. 22 illustrates a case in which a depth map is composed of acquired depth values of a mesh structure based on preset parameters, and the mesh on the left is a mesh structure approximated by a parameterized pattern defined by a box with a tilted ceiling.
  • the mesh structure is approximated by a bounding box containing all depth values, and the right side shows the geometric mesh structure approximated to include a large number of depth values while reducing the number of meshes.
  • Using a depth value of the mesh structure of this type has an advantage that a projection image can be generated in mesh units even when depth-image association is not performed for all image pixels.
  • FIG. 23 is a diagram for describing a projection image generating device according to an embodiment of the present invention.
  • a projection image generating apparatus includes a data pose estimating unit 2310, a user pose obtaining unit 2320, and an image generating unit 2330.
  • the data pose estimating unit 2310 uses a plurality of image acquisition poses and a plurality of depth acquisition poses including acquisition positions and acquisition angles of a plurality of acquired images and a plurality of acquired depth values for an actual three-dimensional space as a reference coordinate system. Estimates based on
  • the data pose estimator 2310 may estimate the image acquisition pose based on a reference coordinate system, and the depth acquisition pose may estimate based on an acquisition coordinate system based on each of the image acquisition poses.
  • the projection image generating apparatus Preferably, the projection image generating apparatus according to an embodiment of the present invention, the plurality of acquisition images, the acquisition time information from which the plurality of acquisition images were acquired, the acquisition depth values, and the acquisition time information from which the plurality of acquisition depth values were obtained
  • Receiving environmental sensing information from a plurality of sensors such as a plurality of image acquisition poses corresponding to a plurality of acquired images, depth acquisition poses corresponding to the plurality of acquisition depth values, inertia change information, marker observation information, and light wave information. It may further include a storage unit (not shown) for storing.
  • the sensor for sensing the environmental sensing information may be an integrated sensor in which the image sensor and the distance sensor are integrated as shown in FIG. 3, or may be an integrated sensor in which all other sensors are integrated.
  • the image sensor and the distance sensor are rigidly fastened.
  • the relative pose between the two sensors may be given in advance as a constant value.
  • the environmental sensing information may be information obtained when the integrated sensor mounted on the robot or the integrated sensor worn on the shoulder of a person is stopped or moved while moving the indoor or outdoor environment.
  • the data pose estimator 2310 reads the environmental sensing information stored in the storage unit and estimates all image acquisition poses and depth acquisition poses based on the reference coordinate system. However, in another embodiment, the data pose estimator 2310 estimates the depth acquisition pose based on the acquired coordinate system, and then estimates the depth acquisition pose based on the reference coordinate system using the relative pose between the depth acquisition pose and the image acquisition pose. It may be.
  • the data pose estimator 2310 sets a representative image representative of the acquired coordinate system whose image acquisition pose is the origin among a plurality of acquired images acquired by the image acquisition pose, stores the image in the storage unit after interworking with the acquired coordinate system. It may be. Accordingly, when the user enters the acquisition coordinate system of a specific image acquisition pose in a virtual three-dimensional space, even if the user moves within a distance within a predetermined threshold of the acquisition coordinate system, only the representative image representing the acquisition coordinate system is present to the user. Will be displayed.
  • the user pose acquisition unit 2320 acquires a user pose including a position and an angle of the user in the virtual three-dimensional space corresponding to the actual three-dimensional space based on the reference coordinate system.
  • the projection image generating apparatus may further include a user input unit (not shown) for moving the user in a virtual three-dimensional space, and can set the viewing angle of the user.
  • a user input unit (not shown) for moving the user in a virtual three-dimensional space, and can set the viewing angle of the user.
  • the user input unit may be in the form of a keyboard, a joystick, an input means based on an angular velocity sensor, an acceleration sensor, a geomagnetic sensor, etc. provided in a head mount display (HMD), or an input means through an optical flow of a camera image. It may be.
  • HMD head mount display
  • the image generator 2330 generates a projection image in which the plurality of acquisition depth values are projected onto at least one of the plurality of acquired images based on the corresponding image acquisition pose corresponding to the user pose and the at least one corresponding depth acquisition pose.
  • the image generating unit 2330 generates a projection image by projecting a plurality of depth values acquired by a plurality of corresponding depth acquisition poses onto one representative image (or combination image).
  • the projection image generating apparatus samples an image according to distance, movement time, and shape feature criteria of the acquired image from among the plurality of acquired images, thereby obtaining the acquired image used for generating the projected image.
  • It may further include an image management unit (not shown) for reducing the number. That is, the image manager may perform post-processing on the acquired image having a dynamic part of the acquired image among the acquired images stored in the storage unit or parts corresponding to personal information such as a human face.
  • the image manager performs correction to minimize the difference for the acquired images if a plurality of acquired images are acquired under different shooting conditions (lighting, exposure time, lens focal length, etc.) at the point where the ring closure condition occurs. You may.
  • the image manager can reduce the difference between the number of acquired images and the number of acquired images used when generating the projection image by storing the acquired images which have been subjected to such processing again in the storage unit.
  • an alarm may be generated, and then correction or deletion of the corresponding image may be reduced.
  • the projection image generating apparatus may further include a depth value management unit (not shown) that performs an operation of removing noise included in the acquired depth value acquired through the distance sensor.
  • a depth value management unit (not shown) that performs an operation of removing noise included in the acquired depth value acquired through the distance sensor.
  • the operation of removing the noise included in the acquisition depth value may be performed by referring to Korean Patent Application No. 10-2015-0062890 "Outer Space Feature Information Extraction Method" filed on May 6, 2015.
  • the acquisition depth value can contain a lot of noise for glass, mirrors or reflective objects.
  • one object may be observed many times, and a large number of depth values may be formed in a specific area. Since such noise or an unbalanced depth value distribution may significantly impair the realism of the projected image in the future, the depth value manager may perform an operation of removing noise included in the acquired depth value stored in the storage in advance. .
  • the image generator 2330 may generate a projection image by using the acquired image and the acquired depth value that have passed through the image manager and the depth value manager.
  • 24 is a diagram for explaining an image generating unit according to an embodiment of the present invention.
  • an image generator 2330 may include a user field determining unit 2332, a reinforcement image selecting unit 2334, a combination image generating unit 2336, and a projection image generating unit ( 2338).
  • the reinforcement image selecting unit 2334 and the combination image generating unit 2336 may be omitted.
  • the user visual field determination unit 2332 determines whether there is an idle field region not included in the representative image among the virtual three-dimensional space regions corresponding to the user's viewing angle in the user pose.
  • the user visual field determination unit 2332 may determine whether the user pose is the same as or different from the image acquisition pose and the depth acquisition pose.
  • the user pose and the image acquisition pose are the same when the field of view of the acquired image acquired by the image acquisition pose includes the user's field of view in the user pose more than a predetermined threshold, otherwise the user pose and the image acquisition pose are different. You might decide that your poses are different. For example, when the angle of view of the acquired image acquired by the image acquisition pose and the user's field of view in the user pose overlap by 90% or more, it may be determined that the user pose and the image acquisition pose are the same.
  • the reinforcement image selecting unit 2334 determines that the idle field region exists, a plurality of acquisition depth values corresponding to the plurality of pixels included in the plurality of acquired images are matched based on depth-image linkage information that is matched information.
  • the reinforcement image is selected from the acquired images.
  • the combined image generator 2336 may generate a combined image by combining the reinforcement image with the representative image.
  • the combination image generator 2336 may generate a combination image by combining the reinforcement image with the same weight as the representative image or by combining the reinforcement image with a weight lower than the representative image.
  • the combination image generator 2336 may generate a combination image having a low weight of the reinforcement image by combining the reinforcement image having a very high transparency with the representative image in the form of a layer.
  • the projection image generator 2338 generates a projection image in which a plurality of acquired depth values are projected onto the combined image.
  • the projection image generator 2338 may generate the projection image using the combination image, or generate the projection image using only the representative image.
  • the user image and the corresponding image acquisition pose corresponding to the user pose are provided. Depending on whether is the same or different, the operation is performed as follows.
  • the projection image generating unit 2338 When the user pose and the corresponding image acquisition pose are the same, and the angle of view of the depth map composed of a plurality of acquisition depth values acquired by the angle of view of the acquired image acquired in the corresponding image acquisition pose and the corresponding depth acquisition pose is the same.
  • the projection image generating unit 2338 In the same pose angle-of-view angle, the projection image generating unit 2338 generates the projection image using the acquired acquired image and the acquired depth value.
  • the projection image generator 2338 selects one representative image from among the plurality of acquired images acquired by the corresponding image acquisition pose, and generates a projection image by projecting an acquisition depth value to the selected representative image.
  • the projection image generator 2338 generates a projection image by projecting the plurality of depth maps acquired from the plurality of corresponding depth acquisition poses onto one representative image linked to the acquisition coordinate system of one corresponding image acquisition pose.
  • the projection image generator 2338 may combine a combination image and a plurality of combination images that combine the reinforcement image with one representative image linked to the acquisition coordinate system of one corresponding image acquisition pose.
  • the projection image is generated by using the plurality of depth maps acquired in the corresponding depth acquisition poses.
  • the weight of the representative image and the weight of the reinforcement image are the same when generating the combined image, a position estimate of at least a few pixels that is inevitably generated by representing one point in the actual three-dimensional space using a plurality of acquired images Because the error may significantly reduce the sensibility of the projected image, the weight of the representative image at the time of generating the combined image should be set higher than the weight of the reinforcement image. However, if the weight of the reinforcement image is very small, such as the weight of the representative image is 99.9 and the weight of the reinforcement image is 0.1, the reinforcement image has a minimal effect on the realisticity of the projection image. May not be used.
  • the representative image linked to the acquisition coordinate system of the image acquisition pose detects an overlapping region between the acquired image and the user's field of view through depth-image linkage information, and sets the acquired image having the largest overlapping region as the representative image,
  • the acquired image of the closest distance from the image acquisition pose may be set as a representative image.
  • the distance from the image acquisition pose to the acquired image is a value calculated based on the Euclidean metric. For example, an absolute difference value, a 2-norm of a vector, and the like may be used.
  • the present invention can be used in rooms, homes, shopping malls, hotels, terminals, airports of the indoor environment, the cultural heritage of the outdoor environment, the preservation of city-terrain information, and can be used in apartment complexes, school campuses, etc. However, by applying the present invention to such places, it is possible to create a 3D virtual map.
  • this 3D map information serves as a platform for creating various services, real estate linkage service, facility promotion service, game contents production service, historical site-tourist experience experience service, O2O service linking goods information in map, etc. Can be carried out.
  • key feature points are extracted from pre-fabricated 3D map information and stored in a database.
  • a user acquires data using camera or inertial sensor information in a corresponding real space
  • the feature points of the database and the feature points provided by the user are displayed.
  • the user's location can be found immediately, which enables the user to display the projection device by using the display device of the user device as the user's field of view, and provide a continuous projection image corresponding to the user's device operation.
  • the user can also check the details of a store's atmosphere or a particular item without having to enter the store.
  • the present invention can also be used for disaster safety management, in the case of a dangerous area such as a nuclear power plant where humans need to come in and out urgently, after simply acquiring data through a robot or the like in advance and constructing a virtual space according to the present invention, It is possible to make emergency response possible in the event of a disaster by allowing infiltration personnel to experience the space sufficiently through HMD.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Software Systems (AREA)
  • Computer Graphics (AREA)
  • Geometry (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Computer Hardware Design (AREA)
  • General Engineering & Computer Science (AREA)
  • Remote Sensing (AREA)
  • Processing Or Creating Images (AREA)

Abstract

L'invention concerne un procédé pour générer une image projetée d'un espace tridimensionnel. Le procédé de génération d'image projetée selon un mode de réalisation de la présente invention comprend les étapes consistant à: estimer, sur la base d'un système de coordonnées de référence, plusieurs poses obtenues de l'image et plusieurs poses obtenues de la profondeur, y compris la position obtenue et l'angle obtenu de chacune des multiples images obtenues et des multiples valeurs obtenues d'un espace tridimensionnel réel; obtenir, sur la base du système de coordonnées de référence, une pose de l'utilisateur comprenant la position et l'angle d'un utilisateur dans un espace tridimensionnel virtuel correspondant à l'espace tridimensionnel réel; et à générer une image projetée formée en projetant les multiples valeurs de profondeur obtenues sur au moins une des multiples images obtenues sur la base d'une pose obtenue d'image correspondante et d'au moins une pose obtenue de profondeur correspondante, correspondant à la pose de l'utilisateur.
PCT/KR2016/006961 2015-07-08 2016-06-29 Procédé et dispositif de génération d'image projetée et procédé de mappage de pixels d'image et de valeurs de profondeur WO2017007166A1 (fr)

Priority Applications (4)

Application Number Priority Date Filing Date Title
US15/515,315 US10602115B2 (en) 2015-07-08 2016-06-29 Method and apparatus for generating projection image, method for mapping between image pixel and depth value
EP16821572.1A EP3321888B1 (fr) 2015-07-08 2016-06-29 Procédé et dispositif de génération d'image projetée et procédé de mappage de pixels d'image et de valeurs de profondeur
CN201680040327.9A CN107836012B (zh) 2015-07-08 2016-06-29 投影图像生成方法及其装置、图像像素与深度值之间的映射方法
JP2018500685A JP6687204B2 (ja) 2015-07-08 2016-06-29 投影イメージ生成方法およびその装置、イメージピクセルと深度値との間のマッピング方法

Applications Claiming Priority (4)

Application Number Priority Date Filing Date Title
KR20150097442 2015-07-08
KR10-2015-0097442 2015-07-08
KR10-2016-0056447 2016-05-09
KR1020160056447A KR101835434B1 (ko) 2015-07-08 2016-05-09 투영 이미지 생성 방법 및 그 장치, 이미지 픽셀과 깊이값간의 매핑 방법

Publications (1)

Publication Number Publication Date
WO2017007166A1 true WO2017007166A1 (fr) 2017-01-12

Family

ID=57685744

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/KR2016/006961 WO2017007166A1 (fr) 2015-07-08 2016-06-29 Procédé et dispositif de génération d'image projetée et procédé de mappage de pixels d'image et de valeurs de profondeur

Country Status (1)

Country Link
WO (1) WO2017007166A1 (fr)

Cited By (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107704106A (zh) * 2017-10-17 2018-02-16 宁波视睿迪光电有限公司 姿态定位方法、装置及电子设备
CN109559271A (zh) * 2017-09-26 2019-04-02 富士通株式会社 对深度图像进行优化的方法和装置
CN111435535A (zh) * 2019-01-14 2020-07-21 株式会社日立制作所 一种关节点信息的获取方法及装置
CN111694429A (zh) * 2020-06-08 2020-09-22 北京百度网讯科技有限公司 虚拟对象驱动方法、装置、电子设备及可读存储
WO2020230921A1 (fr) * 2019-05-14 2020-11-19 엘지전자 주식회사 Procédé d'extraction de caractéristiques d'une image à l'aide d'un motif laser, et dispositif d'identification et robot l'utilisant
CN112913230A (zh) * 2018-10-23 2021-06-04 皇家飞利浦有限公司 图像生成装置及其方法
CN113965721A (zh) * 2020-07-21 2022-01-21 佐臻股份有限公司 图像与深度传输监视系统的对齐方法

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR20080085655A (ko) * 2007-03-20 2008-09-24 인천대학교 산학협력단 카메라 자세 추정 장치 및 방법
KR20140000315A (ko) * 2011-01-31 2014-01-02 마이크로소프트 코포레이션 깊이 맵을 이용하는 모바일 카메라 로컬라이제이션
KR20140001167A (ko) * 2012-06-26 2014-01-06 한국과학기술원 웨어러블 증강현실 환경에서 증강현실 서비스 제공 방법 및 장치
US20150145985A1 (en) * 2013-11-26 2015-05-28 Michael Jason Gourlay Large-Scale Surface Reconstruction That Is Robust Against Tracking And Mapping Errors

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR20080085655A (ko) * 2007-03-20 2008-09-24 인천대학교 산학협력단 카메라 자세 추정 장치 및 방법
KR20140000315A (ko) * 2011-01-31 2014-01-02 마이크로소프트 코포레이션 깊이 맵을 이용하는 모바일 카메라 로컬라이제이션
KR20140001167A (ko) * 2012-06-26 2014-01-06 한국과학기술원 웨어러블 증강현실 환경에서 증강현실 서비스 제공 방법 및 장치
US20150145985A1 (en) * 2013-11-26 2015-05-28 Michael Jason Gourlay Large-Scale Surface Reconstruction That Is Robust Against Tracking And Mapping Errors

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
See also references of EP3321888A4 *
YEON, SUYONG ET AL.: "Observability Analysis of 2D Geometric Features using the Condition Number for SLAM Applications", 13TH INTERNATIONAL CONFERENCE ON CONTROL, AUTOMATION AND SYSTEMS (ICCAS 2013, 23 October 2013 (2013-10-23), pages 1540 - 1543, XP032540056 *

Cited By (13)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109559271A (zh) * 2017-09-26 2019-04-02 富士通株式会社 对深度图像进行优化的方法和装置
CN109559271B (zh) * 2017-09-26 2023-02-28 富士通株式会社 对深度图像进行优化的方法和装置
CN107704106A (zh) * 2017-10-17 2018-02-16 宁波视睿迪光电有限公司 姿态定位方法、装置及电子设备
CN112913230A (zh) * 2018-10-23 2021-06-04 皇家飞利浦有限公司 图像生成装置及其方法
CN112913230B (zh) * 2018-10-23 2023-09-12 皇家飞利浦有限公司 图像生成装置及其方法
CN111435535A (zh) * 2019-01-14 2020-07-21 株式会社日立制作所 一种关节点信息的获取方法及装置
CN111435535B (zh) * 2019-01-14 2024-03-08 株式会社日立制作所 一种关节点信息的获取方法及装置
WO2020230921A1 (fr) * 2019-05-14 2020-11-19 엘지전자 주식회사 Procédé d'extraction de caractéristiques d'une image à l'aide d'un motif laser, et dispositif d'identification et robot l'utilisant
US11493931B2 (en) 2019-05-14 2022-11-08 Lg Electronics Inc. Method of extracting feature from image using laser pattern and device and robot of extracting feature thereof
CN111694429B (zh) * 2020-06-08 2023-06-02 北京百度网讯科技有限公司 虚拟对象驱动方法、装置、电子设备及可读存储
CN111694429A (zh) * 2020-06-08 2020-09-22 北京百度网讯科技有限公司 虚拟对象驱动方法、装置、电子设备及可读存储
CN113965721A (zh) * 2020-07-21 2022-01-21 佐臻股份有限公司 图像与深度传输监视系统的对齐方法
CN113965721B (zh) * 2020-07-21 2024-02-23 佐臻股份有限公司 图像与深度传输监视系统的对齐方法

Similar Documents

Publication Publication Date Title
WO2017007166A1 (fr) Procédé et dispositif de génération d'image projetée et procédé de mappage de pixels d'image et de valeurs de profondeur
WO2019132518A1 (fr) Dispositif d'acquisition d'image et son procédé de commande
WO2018124662A1 (fr) Procédé et dispositif électronique de commande de véhicule aérien sans pilote
WO2019093532A1 (fr) Procédé et système d'acquisition de coordonnées de position tridimensionnelle sans points de commande au sol à l'aide d'un drone de caméra stéréo
WO2015194867A1 (fr) Dispositif de reconnaissance de position de robot mobile utilisant le suivi direct, et son procédé
WO2018182192A1 (fr) Procédé et appareil d'affichage d'image sur la base d'informations de mouvement d'utilisateur
WO2015194864A1 (fr) Dispositif de mise à jour de carte de robot mobile et procédé associé
WO2015194866A1 (fr) Dispositif et procédé permettant de reconnaître un emplacement d'un robot mobile au moyen d'un réajustage basé sur les bords
WO2015194865A1 (fr) Dispositif et procede pour la reconnaissance d'emplacement de robot mobile au moyen d'appariement par correlation a base de recherche
WO2018199701A1 (fr) Procédé de fourniture de contenu et appareil associé
WO2020101420A1 (fr) Procédé et appareil de mesurer des caractéristiques optiques d'un dispositif de réalité augmentée
WO2019022509A1 (fr) Dispositif et procédé de fourniture de contenu
WO2015199502A1 (fr) Appareil et procédé permettant de fournir un service d'interaction de réalité augmentée
WO2021125395A1 (fr) Procédé pour déterminer une zone spécifique pour une navigation optique sur la base d'un réseau de neurones artificiels, dispositif de génération de carte embarquée et procédé pour déterminer la direction de module atterrisseur
EP3632119A1 (fr) Appareil d'affichage et serveur, et procédés de commande associés
WO2023008791A1 (fr) Procédé d'acquisition de distance à au moins un objet situé dans une direction quelconque d'un objet mobile par réalisation d'une détection de proximité, et dispositif de traitement d'image l'utilisant
WO2023048380A1 (fr) Procédé d'acquisition de distance par rapport à au moins un objet positionné devant un corps mobile en utilisant une carte de profondeur de vue d'appareil de prise de vues, et appareil de traitement d'image l'utilisant
WO2020189909A2 (fr) Système et procédé de mise en oeuvre d'une solution de gestion d'installation routière basée sur un système multi-capteurs 3d-vr
WO2021246758A1 (fr) Dispositif électronique et son procédé de fonctionnement
WO2020091347A1 (fr) Dispositif et procédé de mesure de profondeur tridimensionnelle
WO2023055033A1 (fr) Procédé et appareil pour l'amélioration de détails de texture d'images
WO2020251151A1 (fr) Procédé et appareil d'estimation de la pose d'un utilisateur en utilisant un modèle virtuel d'espace tridimensionnel
WO2013111938A1 (fr) Procédé et système d'édition en temps réel de cartes numériques, et serveur et support d'enregistrement associés
WO2021060636A1 (fr) Dispositif de détection de mouvement et procédé
WO2020149527A1 (fr) Appareil et procédé de codage dans un système d'appareil photo de profondeur structuré

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 16821572

Country of ref document: EP

Kind code of ref document: A1

REEP Request for entry into the european phase

Ref document number: 2016821572

Country of ref document: EP

ENP Entry into the national phase

Ref document number: 2018500685

Country of ref document: JP

Kind code of ref document: A

NENP Non-entry into the national phase

Ref country code: DE