WO2020048484A1 - Procédé et appareil de reconstruction d'image à super-résolution, terminal et support de stockage - Google Patents

Procédé et appareil de reconstruction d'image à super-résolution, terminal et support de stockage Download PDF

Info

Publication number
WO2020048484A1
WO2020048484A1 PCT/CN2019/104388 CN2019104388W WO2020048484A1 WO 2020048484 A1 WO2020048484 A1 WO 2020048484A1 CN 2019104388 W CN2019104388 W CN 2019104388W WO 2020048484 A1 WO2020048484 A1 WO 2020048484A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
dimensional
dimensional model
target object
target
Prior art date
Application number
PCT/CN2019/104388
Other languages
English (en)
Chinese (zh)
Inventor
方璐
戴琼海
李广涵
Original Assignee
清华-伯克利深圳学院筹备办公室
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 清华-伯克利深圳学院筹备办公室 filed Critical 清华-伯克利深圳学院筹备办公室
Publication of WO2020048484A1 publication Critical patent/WO2020048484A1/fr

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T15/00—3D [Three Dimensional] image rendering
    • G06T15/005—General purpose rendering architectures
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T15/00—3D [Three Dimensional] image rendering
    • G06T15/10—Geometric effects
    • G06T15/20—Perspective computation
    • G06T15/205—Image-based rendering
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
    • G06T3/4053—Scaling of whole images or parts thereof, e.g. expanding or contracting based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2215/00—Indexing scheme for image rendering
    • G06T2215/06—Curved planar reformation of 3D line structures

Definitions

  • the embodiments of the present application relate to the field of computational vision technology, for example, to a method, a device, a terminal, and a storage medium for super-resolution image reconstruction.
  • the accuracy of computer vision algorithms depends on the imaging quality of the input image or video. Therefore, the resolution of the input image or video needs to be improved.
  • the scene corresponding to an image or video contains two parts, static and dynamic, and the dynamic part contains rigid deformed objects and non-rigid deformed objects.
  • the shape and attitude of a rigidly deformed object will not change over time, you can directly use any frame of high-definition images to improve its resolution; for non-rigidly deformed objects, because their own shape and attitude will change over time, they cannot Increase the resolution of any frame of high-resolution images. Therefore, the difficulty in improving the accuracy of computer vision algorithms is to improve the resolution of non-rigidly deformed objects.
  • the related art there are mainly two methods for improving the resolution of a specific target object (ie, super-resolution reconstruction), one is a single-image super-resolution algorithm, and the other is a super-resolution algorithm based on a reference image.
  • super-resolution reconstruction when the input image and the training set are not similar, the single-image super-resolution algorithm cannot do a good super-resolution reconstruction of low-resolution input images with severe loss of detail, and all high-frequency details generated by this method are all low-frequency The information is generated and the authenticity is not high.
  • the reference image-based super-resolution algorithm needs to input the depth map of high-definition images. Although it can better complement high-frequency details, in practical applications, it is difficult to obtain high-definition depth images, and the algorithm is not universal.
  • the present application provides a super-resolution image reconstruction method, device, terminal, and storage medium to improve the resolution of non-rigid target objects in a low-resolution global image sequence.
  • an embodiment of the present application provides a super-resolution image reconstruction method.
  • the method includes: acquiring a first image of a target region in a current region at a first moment, and generating and the first image according to the first image.
  • At least one three-dimensional model corresponding to at least one first target object in the target region, wherein the first target object is a first non-rigid target object; obtaining a second image of the current region at a second time after the first time Extracting a third image corresponding to the target area from the second image, and updating the at least one three-dimensional model based on the third image; mapping the updated at least one three-dimensional model to at least one two Dimensional image, and stitching the at least one two-dimensional image into the second image to obtain a target super-resolution image.
  • an embodiment of the present application further provides a super-resolution image reconstruction apparatus, the apparatus includes: a three-dimensional model generating module configured to acquire a first image of a target region in a current region at a first moment, and The first image generates at least one three-dimensional model corresponding to at least one first target object in the target area, wherein the first target object is a first non-rigid target object; a three-dimensional model update module is configured to obtain the first target object; A second image of the current area at a second time after the first time, extracting a third image corresponding to the target area from the second image, and updating the at least one three-dimensional model based on the third image;
  • the super-resolution image acquisition module is configured to map the updated at least one three-dimensional model into at least one two-dimensional image, and stitch the at least one two-dimensional image into the second image to obtain a target super-resolution image.
  • an embodiment of the present application further provides a super-resolution image reconstruction terminal.
  • the terminal includes: one or more processors; and a storage device configured to store one or more programs.
  • the program is executed by the one or more processors, so that the one or more processors implement the super-resolution image reconstruction method according to any embodiment of the present application.
  • an embodiment of the present application further provides a computer-readable storage medium on which a computer program is stored.
  • the program is executed by a processor, the super-resolution image reconstruction method according to any one of the embodiments of the present application is implemented.
  • FIG. 1 is a flowchart of a super-resolution image reconstruction method according to an embodiment of the present application
  • FIG. 2 is a schematic structural diagram of a super-resolution image reconstruction device according to an embodiment of the present application
  • FIG. 3 is a schematic structural diagram of a super-resolution image reconstruction terminal in another embodiment of the present application.
  • FIG. 1 is a flowchart of a super-resolution image reconstruction method according to an embodiment of the present application. This embodiment is applicable to a case where the resolution of a non-rigid target object in a low-resolution global image sequence needs to be improved.
  • the image reconstruction apparatus performs the operation. As shown in FIG. 1, the method in this embodiment includes steps S110 to S130.
  • step S110 a first image of a target region in the current region at a first moment is obtained, and at least one three-dimensional model corresponding to at least one first target object in the target region is generated according to the first image, where the first target object Is the first non-rigid target object.
  • the target area may be an area containing at least one first target object.
  • the first target object may be a first non-rigid target object.
  • the non-rigid target object is an object whose shape and attitude can change with time.
  • a non-rigid target object may be a pedestrian.
  • the first image acquired at the first moment is a local image corresponding to the target region of the current region.
  • the first image can be acquired by a camera with a relatively small field of view.
  • the sharpness of the image is relatively high.
  • At least one two-dimensional image corresponding to the at least one first target object may be extracted from the first image. Based on the correspondence between the two-dimensional image and the three-dimensional model, the at least one two-dimensional image may be generated. At least one three-dimensional model corresponding to at least one first target object.
  • step S120 a second image of the current region at a second time after the first time is acquired, a third image corresponding to the target region is extracted from the second image, and at least one three-dimensional model is updated based on the third image.
  • the second image acquired at a second time after the first time is a global image corresponding to the current region, and the second image can use a relatively large field of view (compared to the camera that acquired the first image).
  • the camera acquires, correspondingly, the sharpness of the second image is relatively low.
  • the resolution of the camera that acquires the first image is the same as the resolution of the camera that acquires the second image, that is, the size of the first image and the second image are the same.
  • the second image Since the second image is acquired after the first image, the shape and posture of the first target object corresponding to the second image will be updated relative to the target object corresponding to the first image, so the second image can be used Update at least one three-dimensional model obtained by using the first image.
  • a third image corresponding to the target area may be extracted from the second image, and at least one may be updated with at least one two-dimensional image corresponding to at least one first target object extracted from the third image.
  • Three-dimensional model Three-dimensional model.
  • step S130 the updated at least one three-dimensional model is mapped into at least one two-dimensional image, and the at least one two-dimensional image is stitched into a second image to obtain a target super-resolution image.
  • super-resolution means increasing the resolution of the original image by means of hardware or software
  • super-resolution means the image after increasing the resolution.
  • mapping relationship between the three-dimensional model and the two-dimensional image. After obtaining the updated at least one three-dimensional model, the mapping relationship can be used to map the at least one three-dimensional model into at least one two-dimensional image.
  • the sharpness of the at least one two-dimensional image obtained by the mapping is equivalent to that of the first image, and is higher than the sharpness of the second image.
  • the at least one two-dimensional image obtained by the mapping is stitched into the second image by using an image stitching method , Using the at least one two-dimensional image with high definition to replace the corresponding low-resolution portion of the second image, and finally obtain the target super-resolution image.
  • the corresponding scenes are mainly static scenes and rigidly deformed objects.
  • rigidly deformed objects will move with time, but Since the shape and posture of the rigid deformation object will not change with time, in order to improve the overall resolution of the second image, the static scene of the second image and the corresponding part of the rigid deformation object in the first image can be directly stitched to The corresponding position of the second image to increase its resolution.
  • the at least one three-dimensional model generated using the first image includes both the shape and posture information of the at least one first target object and the texture information of the at least one first target object.
  • the system configured to obtain the first image and the second image may be a rotatable high-definition monitoring PTZ system.
  • the system may include a first-scale camera, a second-scale camera, and a rotatable PTZ There are three parts.
  • the first-scale camera is installed on the rotatable gimbal, and can rotate following the rotation of the gimbal.
  • the first scale camera may be a small field of view camera, configured to acquire a first image of the target area in the current area
  • the second scale camera may be a large field of view camera, configured to monitor the current area in real time and be able to continuously acquire the second area of the current area. image.
  • the resolution of the first-scale camera may be the same as that of the second-scale camera, and the size of the first image may be the same as that of the second image. Accordingly, the sharpness of the first image is higher than that of the second image. Clarity. Based on this, the resolution of the second image acquired at the second time may be improved by using the first image acquired at the first time based on the above scheme.
  • the super-resolution image reconstruction method obtains a first image of a target region in a current region at a first moment, and generates at least one three-dimensional model corresponding to at least one first target object in the target region according to the first image.
  • the first target object is a first non-rigid target object
  • a second image of the current region at a second time after the first time is obtained
  • a third image corresponding to the target region is extracted from the second image, and based on the first Three images update at least one three-dimensional model, map the updated at least one three-dimensional model to at least one two-dimensional image, and stitch at least one two-dimensional image into the second image to obtain the target super-resolution image, which improves the low-resolution global image Resolution of non-rigid target objects in the sequence.
  • generating at least one three-dimensional model corresponding to at least one first target object in the target area according to the first image includes: performing target object detection on the first image based on a preset target object detection method to obtain At least one first partial image corresponding to at least one first target object; using a preset two-dimensional pose point estimation method, two-dimensional pose point estimation is performed on each first partial image separately, and Each first two-dimensional pose point corresponding to the target object; For each first target object, the initial three-dimensional model is optimized using each first two-dimensional pose point to obtain a three-dimensional model corresponding to the first target object; for each three-dimensional model , Using the texture information in the corresponding first partial image to render the three-dimensional model to update the three-dimensional model.
  • At least one two-dimensional image corresponding to at least one target object may be acquired.
  • a first target object in the first image may be detected by using a preset target object detection method to obtain at least one first partial image corresponding to at least one first target object.
  • a first target object corresponds to a first partial image, and each first partial image can be labeled from the first image by using a square area.
  • the preset target object detection method may be a faster-rcnn detection algorithm, which has a high detection accuracy and a fast operation speed.
  • the faster-rcnn detection algorithm uses a deep learning method to propose an RPN network structure.
  • the convolutional neural network outputs two branches, one branch is the parameters corresponding to all candidate regions: the center coordinates x and y of the region, and the length and width of the region w. , H; The other branch is the probability that the candidate area is the first target object. Based on the two branches of the convolutional neural network output, the specific position of the at least one first target object in the first image can be determined to determine the position of the at least one first partial image.
  • the posture information of each first target object may be determined using each first partial image.
  • a preset two-dimensional pose point estimation method may be used to separately perform two-dimensional pose point estimation on each first partial image to obtain each first two-dimensional pose point corresponding to each first target object.
  • the preset two-dimensional pose point estimation method may be Openpose, which uses deep learning methods to predict each first partial image separately to obtain the two-dimensional pose points of all first target objects in each first partial image. , And then divide all the two-dimensional pose points according to the characteristics of the first target object, and finally determine the two-dimensional pose points corresponding to each first target object.
  • the first two-dimensional pose points obtained by predicting each first partial image by using Openpose are corresponding to each first target object. The first two-dimensional pose point.
  • an initial three-dimensional model can be constructed by using initialization parameters.
  • the decibels use each first two-dimensional attitude point to initialize the
  • the three-dimensional model is optimized to obtain a three-dimensional model corresponding to the first target object.
  • the three-dimensional model obtained by using the above method does not include the texture information of the first target object, and the two-dimensional image obtained by mapping the three-dimensional model does not include color information.
  • the three-dimensional model can be rendered using the texture information in the corresponding first partial image to update the three-dimensional model, so that the updated three-dimensional model contains both the shape and posture information of the first target object and the first Color information of the target object.
  • the initial three-dimensional model is optimized by using each first two-dimensional pose point to obtain a three-dimensional model corresponding to the first target object.
  • the method includes: The initial shape factor matrix ⁇ and the initial attitude angle vector ⁇ are used to construct an initial three-dimensional model.
  • the initial camera model parameter matrix K is used to perform a two-dimensional mapping on the initial three-dimensional model to obtain the initial two-dimensional attitude points corresponding to the initial three-dimensional model.
  • a target object Calculate a shape factor matrix ⁇ 1 and a first attitude angle vector ⁇ 1 that satisfy a preset condition, where the preset condition is a pair of matching points of each first two-dimensional pose point and each initial two-dimensional pose point.
  • the initial three-dimensional model is optimized by using the shape factor matrix ⁇ 1 and the first attitude angle vector ⁇ 1 to obtain a three-dimensional model corresponding to the first target object.
  • a 3D model consists of dense point clouds in a 3D space.
  • an initial three-dimensional model may be constructed based on a preset three-dimensional model construction method, an initial shape factor matrix ⁇ , and an initial attitude angle vector ⁇ .
  • an SMPLily algorithm may be used to construct the initial three-dimensional model. Taking the first target object as a human body as an example, the SMPLily algorithm constructs a three-dimensional model using a three-dimensional model of the SMPL human body, a shape factor matrix ⁇ , and an attitude angle vector ⁇ .
  • the three-dimensional human body model includes 6890 three-dimensional points and 24 three-dimensional joint points, of which 24 The three-dimensional joint points are used to control the position of the entire three-dimensional model point cloud, and then control the attitude of the three-dimensional model.
  • the shape factor matrix ⁇ controls the characteristics of the three-dimensional model such as height, fat, and thinness. The angle of rotation of the position of this point in the model.
  • Each of the 6,890 three-dimensional points in the three-dimensional model can be represented by a linear weighted average of 24 attitude angle vectors.
  • the initial camera model parameter matrix K can be used to perform two-dimensional mapping on the 24 three-dimensional joint points in the initial three-dimensional model. Each initial two-dimensional pose point corresponding to the three-dimensional model.
  • the shape factor matrix ⁇ 1 and The initial attitude angle vector ⁇ 1 is optimized by using the shape factor matrix ⁇ 1 and the initial attitude angle vector ⁇ 1 to obtain a three-dimensional model corresponding to the first target object.
  • a shape factor matrix ⁇ 1 and a first attitude angle vector ⁇ 1 that satisfy a preset condition may be calculated, where the preset conditions are each first two-dimensional pose point and each The sum of the differences between the pair of matching points of the initial two-dimensional pose points is the smallest, and the shape factor matrix ⁇ 1 is the smallest; the shape factor matrix ⁇ 1 and the first attitude angle vector ⁇ 1 are used to optimize the initial three-dimensional model to obtain A three-dimensional model corresponding to the first target object.
  • the method further includes: calculating a camera model parameter matrix that satisfies a preset condition.
  • Each three-dimensional model uses the texture information in the corresponding first partial image to render the three-dimensional model to update the three-dimensional model, including: for each three-dimensional model: using the camera model parameter matrix K1, converting the texture information in the corresponding first partial image Map to the 3D model to update the 3D model.
  • the camera model parameter matrix K may also be used to render texture information on the three-dimensional models.
  • a camera model parameter matrix K 1 that satisfies a preset condition is calculated, where the preset condition is a difference between each first two-dimensional pose point and each matching point pair of each initial two-dimensional pose point. The sum is minimal, and the form factor matrix ⁇ 1 is minimal.
  • the model using a camera parameter matrix K1 mapping texture information corresponding to a first partial image to the three-dimensional model, in order to update the three-dimensional model.
  • the method further includes: using a preset interpolation algorithm to map the three-dimensional model obtained.
  • the texture information is interpolated to obtain the texture information of the complete 3D model.
  • the first partial image that provides texture information for the three-dimensional model. Because the first partial image is a two-dimensional image, when the texture information in the first partial image is mapped to the three-dimensional model, the three-dimensional model is There must be some 3D points where texture information cannot be obtained, and these 3D points may include 3D points that can enter the field of view; in addition, when mapping a 3D model to a 2D image, it is only necessary to use the 3D model Able to enter three-dimensional points in the field of view. Therefore, texture information interpolation processing can be performed on the three-dimensional points that can enter the field of view and cannot obtain texture information, so that when the three-dimensional model is mapped to a two-dimensional image, complete texture information can be obtained. In an embodiment, a bilinear interpolation algorithm may be used to perform interpolation processing on the texture information of the mapped three-dimensional model to obtain the complete three-dimensional model texture information.
  • updating at least one three-dimensional model based on the third image includes: performing target object detection on the third image based on a preset target object detection method to obtain a one-to-one correspondence with at least one second target object in the target area.
  • At least one second partial image wherein the second target object is a second non-rigid target object; matching each first partial image with each second partial image to obtain a match between the at least one first partial image and the second partial image Pair to determine at least one three-dimensional model corresponding to a second target object in at least one second partial image; using a preset two-dimensional pose point estimation method to perform two-dimensional pose point estimation on each second partial image to obtain Each second two-dimensional pose point corresponding to each second target object; for each second target object, the second three-dimensional model corresponding to the second target object is updated using each second two-dimensional pose point.
  • the second target object may be the first target object after its shape and posture are changed.
  • the method for obtaining the second partial image is the same as the method for obtaining the first partial image, and also uses a faster-rcnn detection algorithm. After using the faster-rcnn detection algorithm to obtain at least one second partial image, the image matching algorithm is used to match each first partial image with each second partial image to obtain each second partial image that matches each first partial image. Since each first partial image corresponds to a three-dimensional model, based on each first partial image, each three-dimensional model corresponding to a second target object in each second partial image can be determined.
  • Each three-dimensional model corresponding to each second partial image determined by using the foregoing steps is determined using each first partial image. Therefore, the pose information of each three-dimensional model corresponds to the pose information of the first target object in each first partial image.
  • the posture information of each three-dimensional model may be updated using the posture information of each second target object.
  • a preset two-dimensional pose point estimation method may be used to separately perform two-dimensional pose point estimation on each second partial image to obtain each second two-dimensional pose point corresponding to each second target object. And for each second target object, using each second two-dimensional pose point to update the three-dimensional model corresponding to the second target object.
  • the preset two-dimensional pose point estimation method may be Openpose. The process of obtaining the second two-dimensional pose point by using Openpose is the same as the process of obtaining the first two-dimensional pose point by using Openpose.
  • updating the three-dimensional model corresponding to the second target object using each second two-dimensional pose point includes: for each second target object: using a preset deep learning algorithm , Each second two-dimensional attitude point is converted into a second attitude angle vector ⁇ 2 ; the shape factor matrix ⁇ 1 and the second attitude angle vector ⁇ 2 are used to update the three-dimensional model corresponding to the second target object to obtain the second target The 3D model corresponding to the object.
  • the three-dimensional model corresponding to the first partial image is obtained by using the shape factor matrix and the attitude angle vector optimization, the three-dimensional model can also be updated by using the updated two parameters, and because the target object is determined, the shape factor matrix There will be no change, so the 3D model can be updated with the updated attitude angle vector.
  • the second three-dimensional model corresponding to the second target object is updated by using each second two-dimensional pose point. After obtaining the second two-dimensional pose points corresponding to each second target object, the Set a deep learning algorithm to convert each second two-dimensional attitude point into a second attitude angle vector ⁇ 2 , and use the shape factor matrix ⁇ 1 and the second attitude angle vector ⁇ 2 to update the three-dimensional model corresponding to the second target object.
  • a three-dimensional model corresponding to the second target object is obtained.
  • the deep learning method is based on a deep residual network, and uses a combination of the most basic linear layer, RELU activation function, and reasonable network parameters to finally obtain the second attitude angle vector ⁇ 2 .
  • matching each first partial image with each second partial image to obtain at least one matching pair of the first partial image and the second partial image includes: determining each of the first partial image and each of the second partial image. The center point of the partial image; for each second partial image: Calculate the Euclidean distance between the center point of the second partial image and the center point of each first partial image; use the first partial image that minimizes the Euclidean distance as the second Matching pairs of local images.
  • the image matching algorithm that matches each first partial image with each second partial image may be a method of determining the center points of each of the first partial images and each of the second partial images, wherein the method of determining the center points is It can be the average value of the horizontal and vertical coordinates of the four vertices in the square area.
  • mapping the updated at least one three-dimensional model into at least one two-dimensional image includes: using the camera model parameter matrix K 1 to map the updated at least one three-dimensional model into at least one two-dimensional image.
  • FIG. 2 is a schematic structural diagram of a super-resolution image reconstruction device according to an embodiment of the present application.
  • the super-resolution image reconstruction device of this embodiment includes a three-dimensional model generation module 210, a three-dimensional model update module 220, and a super-resolution image acquisition module 230.
  • the three-dimensional model generating module 210 is configured to acquire a first image of a target region in the current region at a first moment, and generate at least one three-dimensional model corresponding to at least one first target object in the target region according to the first image.
  • a target object is a first non-rigid target object.
  • the three-dimensional model update module 220 is configured to obtain a second image of the current region at a second time after the first time, extract a third image corresponding to the target region from the second image, and update at least one three-dimensional model based on the third image. .
  • the super-resolution image acquisition module 230 is configured to map the updated at least one three-dimensional model into at least one two-dimensional image, and stitch the at least one two-dimensional image into a second image to obtain a target super-resolution image.
  • the super-resolution image reconstruction device obtains a first image of a target region in a current region at a first moment through a three-dimensional model generation module, and generates a first image corresponding to at least one first target object in the target region according to the first image.
  • At least one three-dimensional model wherein the first target object is a first non-rigid target object, and a three-dimensional model update module is used to obtain a second image of the current region at a second time after the first time, and the target region is extracted from the second image.
  • a corresponding third image and update at least one three-dimensional model based on the third image, and use the super-resolution image acquisition module to map the updated at least one three-dimensional model to at least one two-dimensional image, and stitch the at least one two-dimensional image to the first In the two images, the target super-resolution image is obtained, which improves the resolution of non-rigid target objects in the low-resolution global image sequence.
  • the three-dimensional model generation module 210 may include a first partial image acquisition sub-module, a first two-dimensional pose point acquisition sub-module, a three-dimensional model determination sub-module, and a texture information rendering sub-module.
  • the first partial image acquisition sub-module is configured to perform target object detection on the first image based on a preset target object detection method to obtain at least one first partial image corresponding to at least one first target object.
  • the first two-dimensional pose point acquisition sub-module is configured to use a preset two-dimensional pose point estimation method to perform two-dimensional pose point estimation on each first partial image respectively, and obtain each first partial object corresponding to each first target object. A two-dimensional pose point.
  • the three-dimensional model determination sub-module is set to optimize the initial three-dimensional model with each first two-dimensional pose point for each first target object to obtain a three-dimensional model corresponding to the first target object.
  • the texture information rendering sub-module is configured to, for each three-dimensional model, use the texture information in the corresponding first partial image to render the three-dimensional model to update the three-dimensional model.
  • the three-dimensional model determination sub-module may include an initial three-dimensional model construction unit, an initial two-dimensional pose point acquisition unit, a parameter acquisition unit, and a three-dimensional model acquisition unit.
  • the initial three-dimensional model construction unit is configured to construct an initial three-dimensional model based on a preset three-dimensional model construction method, an initial shape factor matrix ⁇ , and an initial attitude angle vector ⁇ .
  • the initial two-dimensional pose point acquisition unit is set to perform two-dimensional mapping on the initial three-dimensional model by using the initial camera model parameter matrix K to obtain each initial two-dimensional pose point corresponding to the initial three-dimensional model.
  • a parameter acquisition unit is set for each first target object: calculating a form factor matrix ⁇ 1 and a first attitude angle vector ⁇ 1 that satisfy a preset condition, where the preset conditions are each first two-dimensional pose point and each initial The sum of the differences between the pairs of matching points of the two-dimensional pose points is the smallest, and the form factor matrix ⁇ 1 is the smallest.
  • the three-dimensional model acquisition unit is configured to optimize the initial three-dimensional model by using the shape factor matrix ⁇ 1 and the first attitude angle vector ⁇ 1 to obtain a three-dimensional model corresponding to the first target object.
  • the parameter obtaining unit may be further configured to calculate a camera model parameter matrix K 1 that satisfies a preset condition, where the preset condition is each matching of each first two-dimensional pose point and each initial two-dimensional pose point. The sum of the differences between the point pairs is minimal, and the form factor matrix ⁇ 1 is minimal.
  • the texture information rendering sub-module may be configured to: for each three-dimensional model: use the camera model parameter matrix K1 to map the texture information in the corresponding first partial image to the three-dimensional model to update the three-dimensional model.
  • the texture information rendering sub-module may be further configured to: for each three-dimensional model: after using the camera model parameter matrix K1 to map the texture information in the corresponding first partial image onto the three-dimensional model, a preset is adopted The interpolation algorithm interpolates the texture information of the mapped 3D model to obtain the texture coordinates of the complete 3D model.
  • the three-dimensional model update module 220 may include a second local image acquisition sub-module, a local image matching sub-module, a second two-dimensional pose point acquisition sub-module, and a three-dimensional model update sub-module.
  • the second partial image acquisition sub-module is configured to perform target object detection on the third image based on a preset target object detection method to obtain at least one second partial image corresponding to at least one second target object in the target area.
  • the second target object is a second non-rigid target object.
  • the partial image matching sub-module is configured to match each first partial image with each second partial image to obtain a matching pair of at least one first partial image and a second partial image, so as to determine a first pair of at least one second partial image.
  • the second two-dimensional pose point acquisition sub-module is configured to use a preset two-dimensional pose point estimation method to perform two-dimensional pose point estimation on each second partial image respectively, and obtain each first partial object corresponding to each second target object. Two-dimensional and two-dimensional pose points.
  • the three-dimensional model update sub-module is configured to update the three-dimensional model corresponding to the second target object with each second two-dimensional pose point for each second target object.
  • the local image matching sub-module may include an image center point determination unit, a European-style distance calculation unit, and a local image matching pair determination unit.
  • the image center point determination unit is configured to determine a center point of each of the first partial images and each of the second partial images.
  • the Euclidean distance calculation unit is set for each second partial image: respectively calculate the Euclidean distance between the center point of the second partial image and the center point of each first partial image.
  • the local image matching pair determination unit is configured to use the first partial image with the smallest Euclidean distance as the matching pair of the second partial image.
  • the three-dimensional model updating sub-module may include: for each second target object: a second attitude angle vector determination unit, configured to use a preset deep learning algorithm to convert each second two-dimensional attitude point into a first Two attitude angle vectors ⁇ 2 ; a three-dimensional model update unit configured to update the three-dimensional model corresponding to the second target object by using the shape factor matrix ⁇ 1 and the second attitude angle vector ⁇ 2 to obtain the three-dimensional model corresponding to the second target object .
  • the super-resolution image acquisition module 230 is configured to: use the camera model parameter matrix K 1 to map the updated at least one three-dimensional model into at least one two-dimensional image.
  • the super-resolution image reconstruction apparatus provided in the embodiment of the present application can execute the super-resolution image reconstruction method provided in any embodiment of the present application, and has corresponding function modules for executing the method.
  • FIG. 3 is a schematic structural diagram of a super-resolution image reconstruction terminal according to another embodiment of the present application.
  • FIG. 3 shows a block diagram of an exemplary super-resolution image reconstruction terminal 312 suitable for implementing the embodiments of the present application.
  • the super-resolution image reconstruction terminal 312 shown in FIG. 3 is merely an example, and should not impose any limitation on the functions and scope of use of the embodiments of the present application.
  • the super-resolution image reconstruction terminal 312 is expressed in the form of a general-purpose computing device.
  • the components of the super-resolution image reconstruction terminal 312 may include, but are not limited to, one or more processors 316, a memory 328, and a bus 318 connecting different system components (including the memory 328 and the processor 316).
  • the bus 318 represents one or more of several types of bus structures, including a memory bus or a memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any of a variety of bus structures.
  • these architectures include, but are not limited to, the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MAC) bus, the enhanced ISA bus, and the Video Electronics Standards Association (Vedio Electronics Standard). Association (VESA) local area bus and Peripheral Component Interconnect (PCI) bus.
  • the super-resolution image reconstruction terminal 312 typically includes a variety of computer system-readable media. These media can be any available media that can be accessed by the super-resolution image reconstruction terminal 312, including volatile and non-volatile media, removable and non-removable media.
  • the memory 328 may include a computer system readable medium in the form of volatile memory, such as Random Access Memory (RAM) 330 and / or cache memory 332.
  • the super-resolution image reconstruction terminal 312 may include other removable / non-removable, volatile / nonvolatile computer system storage media.
  • the storage device 334 may be configured to read and write non-removable, non-volatile magnetic media (not shown in FIG. 3 and is commonly referred to as a “hard drive”).
  • a disk drive for reading and writing to a removable non-volatile disk such as a “floppy disk”
  • a read-only storage for a removable non-volatile optical disk such as a compact optical disk
  • each drive may be connected to the bus 318 through one or more data medium interfaces.
  • the memory 328 may include at least one program product having a set (eg, at least one) of program modules configured to perform the functions of the embodiments of the present application.
  • a program / utility tool 340 having a set (at least one) of program modules 342 may be stored in, for example, the memory 328.
  • Such program modules 342 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data Each of these examples, or some combination, may include an implementation of a network environment.
  • the program module 342 generally performs functions and / or methods in the embodiments described in this application.
  • the super-resolution image reconstruction terminal 312 may also communicate with one or more external devices 314 (such as a keyboard, a pointing device, a display 324, etc., where the display 324 can decide whether to configure it according to actual needs), and may also communicate with one or more users
  • external devices 314 such as a keyboard, a pointing device, a display 324, etc., where the display 324 can decide whether to configure it according to actual needs
  • This communication can be performed through an input / output (I / O) interface 322.
  • the super-resolution image reconstruction terminal 312 may also communicate with one or more networks (such as a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through the network adapter 320. As shown, the network adapter 320 communicates with other modules of the super-resolution image reconstruction terminal 312 through the bus 318. It should be understood that although not shown in FIG. 3, other hardware and / or software modules may be used in combination with the super-resolution image reconstruction terminal 312, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID Systems, tape drives, and data backup storage devices.
  • the processor 316 executes various functional applications and data processing by running a program stored in the memory 328, for example, implementing a super-resolution image reconstruction method provided by any embodiment of the present application.
  • An embodiment of the present application further provides a computer-readable storage medium on which a computer program is stored.
  • the program is executed by a processor, the super-resolution image reconstruction method provided by the embodiment of the present application is implemented.
  • the method includes: A first image of a target region in the region at a first moment, and at least one three-dimensional model corresponding to at least one first target object in the target region is generated according to the first image, where the first target object is a first non-rigid target object ; Obtain a second image of the current area at a second time after the first time, extract a third image corresponding to the target area from the second image, and update at least one three-dimensional model based on the third image; The three-dimensional model is mapped into at least one two-dimensional image, and the at least one two-dimensional image is stitched into a second image to obtain a target super-resolution image.
  • the computer-readable storage medium provided in the embodiment of the present application is not limited to the method operations described above, and the computer program stored on the computer program may also be implemented in the super-resolution image reconstruction method provided by any embodiment of the present application. Related operations.
  • the computer storage medium in the embodiments of the present application may adopt any combination of one or more computer-readable media.
  • the computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium.
  • the computer-readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof.
  • a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in combination with an instruction execution system, apparatus, or device.
  • the computer-readable signal medium may include a data signal in baseband or propagated as part of a carrier wave, which carries a computer-readable program code. Such a propagated data signal may take many forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing.
  • the computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, and the computer-readable medium may send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device .
  • the program code contained on the computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wire, optical fiber cable, radio frequency (RF), etc., or any suitable combination of the foregoing.
  • any appropriate medium including but not limited to wireless, wire, optical fiber cable, radio frequency (RF), etc., or any suitable combination of the foregoing.
  • Computer program code for performing the operations of this application may be written in one or more programming languages, or a combination thereof, including programming languages such as Java, Smalltalk, C ++, and also conventional Procedural programming language—such as "C" or similar programming language.
  • the program code can be executed entirely on the user's computer, partly on the user's computer, as an independent software package, partly on the user's computer, partly on a remote computer, or entirely on a remote computer or server.
  • the remote computer can be connected to the user's computer through any kind of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (such as through an Internet service provider) Internet connection).
  • LAN local area network
  • WAN wide area network
  • Internet service provider Internet service provider

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Graphics (AREA)
  • Computing Systems (AREA)
  • Geometry (AREA)
  • Image Analysis (AREA)
  • Image Processing (AREA)

Abstract

La présente invention concerne, dans des modes de réalisation, un procédé et un dispositif de reconstruction d'image à super-résolution, un terminal et un support de stockage. Le procédé de reconstruction d'image à super-résolution comprend les étapes consistant à : obtenir une première image d'une région cible dans une région actuelle à un premier instant, et générer au moins un modèle tridimensionnel correspondant à au moins un premier objet cible dans la région cible selon la première image, le premier objet cible étant un premier objet cible non rigide ; obtenir une deuxième image de la région actuelle à un second instant après le premier instant, extraire une troisième image correspondant à la région cible à partir de la deuxième image, et à mettre à jour au moins un modèle tridimensionnel sur la base de la troisième image ; et mapper au moins un modèle tridimensionnel mis à jour dans au moins une image bidimensionnelle, et assembler l'au moins une image bidimensionnelle dans la deuxième image pour obtenir une image à super-résolution cible.
PCT/CN2019/104388 2018-09-04 2019-09-04 Procédé et appareil de reconstruction d'image à super-résolution, terminal et support de stockage WO2020048484A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201811027057.5A CN109191554B (zh) 2018-09-04 2018-09-04 一种超分辨图像重建方法、装置、终端和存储介质
CN201811027057.5 2018-09-04

Publications (1)

Publication Number Publication Date
WO2020048484A1 true WO2020048484A1 (fr) 2020-03-12

Family

ID=64914431

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2019/104388 WO2020048484A1 (fr) 2018-09-04 2019-09-04 Procédé et appareil de reconstruction d'image à super-résolution, terminal et support de stockage

Country Status (2)

Country Link
CN (1) CN109191554B (fr)
WO (1) WO2020048484A1 (fr)

Families Citing this family (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109191554B (zh) * 2018-09-04 2021-01-01 清华-伯克利深圳学院筹备办公室 一种超分辨图像重建方法、装置、终端和存储介质
CN109859296B (zh) * 2019-02-01 2022-11-29 腾讯科技（深圳）有限公司 Smpl参数预测模型的训练方法、服务器及存储介质
WO2021063271A1 (fr) * 2019-09-30 2021-04-08 Oppo广东移动通信有限公司 Procédé et système de reconstruction de modèle de corps humain, et support de stockage associé
CN113643433A (zh) * 2020-04-27 2021-11-12 成都术通科技有限公司 形态和姿态预估方法、装置、设备及存储介质
CN112634139B (zh) * 2021-02-25 2021-05-28 杭州海康威视数字技术股份有限公司 光场超分辨成像方法、装置及设备
CN113112402A (zh) * 2021-03-22 2021-07-13 深兰科技（上海）有限公司 模型获取方法、装置、电子设备及存储介质
CN113538649B (zh) * 2021-07-14 2022-09-16 深圳信息职业技术学院 一种超分辨率的三维纹理重建方法、装置及其设备
CN113610713B (zh) * 2021-08-13 2023-11-28 北京达佳互联信息技术有限公司 视频超分辨模型的训练方法、视频超分辨方法及装置

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20100124383A1 (en) * 2008-11-19 2010-05-20 Nec Laboratories America, Inc. Systems and methods for resolution-invariant image representation
CN102682442A (zh) * 2012-04-28 2012-09-19 东南大学 基于光流场的运动目标超分辨率图像重建方法
CN103810685A (zh) * 2014-02-25 2014-05-21 清华大学深圳研究生院 一种深度图的超分辨率处理方法
CN105741252A (zh) * 2015-11-17 2016-07-06 西安电子科技大学 基于稀疏表示与字典学习的视频图像分级重建方法
CN109191554A (zh) * 2018-09-04 2019-01-11 清华-伯克利深圳学院筹备办公室 一种超分辨图像重建方法、装置、终端和存储介质

Family Cites Families (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104091364B (zh) * 2014-07-10 2017-01-11 西北工业大学 单幅图像超分辨重建方法
CN105006018B (zh) * 2015-06-30 2017-11-14 四川大学 三维ct岩心图像超分辨率重建方法
CN105023275B (zh) * 2015-07-14 2018-08-28 清华大学 超分辨率光场采集装置及其的三维重建方法
US9918792B1 (en) * 2016-05-31 2018-03-20 American Medical Technologies, Llc Methods and system for atrial fibrillation ablation using a fluoroscopy and/or medical images based cardiac mapping system with optional esophageal temperature monitoring
CN107888707B (zh) * 2017-12-08 2021-04-20 北京奇艺世纪科技有限公司 一种图片传输方法、装置及电子设备
CN107944428B (zh) * 2017-12-15 2021-07-30 北京工业大学 一种基于超像素集的室内场景语义标注方法
CN108038905B (zh) * 2017-12-25 2018-12-07 北京航空航天大学 一种基于超像素的目标重建方法
CN108416821B (zh) * 2018-03-08 2019-08-02 山东财经大学 一种深度神经网络的ct图像超分辨重建方法
CN108447020A (zh) * 2018-03-12 2018-08-24 南京信息工程大学 一种基于极深卷积神经网络的人脸超分辨率重建方法
CN108376386A (zh) * 2018-03-23 2018-08-07 深圳天琴医疗科技有限公司 一种图像的超分辨率模型的构建方法及装置

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20100124383A1 (en) * 2008-11-19 2010-05-20 Nec Laboratories America, Inc. Systems and methods for resolution-invariant image representation
CN102682442A (zh) * 2012-04-28 2012-09-19 东南大学 基于光流场的运动目标超分辨率图像重建方法
CN103810685A (zh) * 2014-02-25 2014-05-21 清华大学深圳研究生院 一种深度图的超分辨率处理方法
CN105741252A (zh) * 2015-11-17 2016-07-06 西安电子科技大学 基于稀疏表示与字典学习的视频图像分级重建方法
CN109191554A (zh) * 2018-09-04 2019-01-11 清华-伯克利深圳学院筹备办公室 一种超分辨图像重建方法、装置、终端和存储介质

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
ZHENG, HAITIAN ET AL.: "Combining Exemplar-based Approach and Learning-Based Approach for Light Field Super-resolution Using a Hybrid Imaging System", 2017 IEEE INTERNATIONAL CONFERENCE ON COMPUTER VISION WORKSHOPS, 31 December 2017 (2017-12-31), XP033303718 *

Also Published As

Publication number Publication date
CN109191554B (zh) 2021-01-01
CN109191554A (zh) 2019-01-11

Similar Documents

Publication Publication Date Title
WO2020048484A1 (fr) Procédé et appareil de reconstruction d'image à super-résolution, terminal et support de stockage
EP3786890B1 (fr) Procédé et appareil de détermination de pose de dispositif de capture d'image, et support d'enregistrement correspondant
WO2020001168A1 (fr) Procédé, appareil et dispositif de reconstruction tridimensionnelle, et support d'informations
CN108335353B (zh) 动态场景的三维重建方法、装置和系统、服务器、介质
JP7373554B2 (ja) クロスドメイン画像変換
WO2019238114A1 (fr) Procédé, appareil et dispositif de reconstruction de modèle dynamique tridimensionnel, et support de stockage
CN112927362A (zh) 地图重建方法及装置、计算机可读介质和电子设备
US20220358675A1 (en) Method for training model, method for processing video, device and storage medium
CN112598780B (zh) 实例对象模型构建方法及装置、可读介质和电子设备
WO2021237743A1 (fr) Procédé et appareil d'interpolation de trame vidéo et support de stockage lisible par ordinateur
US11961266B2 (en) Multiview neural human prediction using implicit differentiable renderer for facial expression, body pose shape and clothes performance capture
CN115578515B (zh) 三维重建模型的训练方法、三维场景渲染方法及装置
US11741671B2 (en) Three-dimensional scene recreation using depth fusion
EP4292059A1 (fr) Prédiction humaine neuronale multivue à l?aide d?un moteur de rendu différentiable implicite pour l?expression faciale, la forme et la pose du corps, et la capture de performance de vêtements
CN116863078A (zh) 三维人体模型重建方法、装置、电子设备和可读介质
CN115272575B (zh) 图像生成方法及装置、存储介质和电子设备
CN112085842A (zh) 深度值确定方法及装置、电子设备和存储介质
CN113920282B (zh) 图像处理方法和装置、计算机可读存储介质、电子设备
KR20230078502A (ko) 이미지 처리 장치 및 방법
CN116797713A (zh) 一种三维重建方法和终端设备
CN116246026B (zh) 三维重建模型的训练方法、三维场景渲染方法及装置
CN115100360B (zh) 图像生成方法及装置、存储介质和电子设备
US20240071009A1 (en) Visually coherent lighting for mobile augmented reality
CN117294956A (zh) 基于动目标生成图像和光流的方法、相关方法及相关产品
CN116978102A (zh) 一种人脸特征建模识别方法、芯片及终端

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19856744

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 19856744

Country of ref document: EP

Kind code of ref document: A1