WO2019140688A1 - Procédé et appareil de traitement d'image, et support d'informations lisible par ordinateur - Google Patents

Procédé et appareil de traitement d'image, et support d'informations lisible par ordinateur Download PDF

Info

Publication number
WO2019140688A1
WO2019140688A1 PCT/CN2018/073632 CN2018073632W WO2019140688A1 WO 2019140688 A1 WO2019140688 A1 WO 2019140688A1 CN 2018073632 W CN2018073632 W CN 2018073632W WO 2019140688 A1 WO2019140688 A1 WO 2019140688A1
Authority
WO
WIPO (PCT)
Prior art keywords
point
deformation
feature
folding
feature point
Prior art date
Application number
PCT/CN2018/073632
Other languages
English (en)
Chinese (zh)
Inventor
周游
刘洁
唐克坦
刘昂
Original Assignee
深圳市大疆创新科技有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 深圳市大疆创新科技有限公司 filed Critical 深圳市大疆创新科技有限公司
Priority to CN201880012242.9A priority Critical patent/CN110313020A/zh
Priority to PCT/CN2018/073632 priority patent/WO2019140688A1/fr
Publication of WO2019140688A1 publication Critical patent/WO2019140688A1/fr

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects

Definitions

  • the present invention relates to the field of image processing technologies, and in particular, to an image processing method, device, and computer readable storage medium.
  • Computer animation is a technique for making animations by means of a computer, including two-dimensional animation (2D) and three-dimensional animation (3D).
  • Computer animation can be done by CG (Computer Graphics).
  • CG Computer Graphics
  • the field of visual design and production using computer technology is called CG
  • CG is a general term for all graphics drawn by computer, such as web design, film and television special effects, multimedia technology, and the like.
  • the invention provides an image processing method, a device and a computer readable storage medium, which can reduce the acquisition complexity of an image having a folding effect (such as a folding city special effect) and improve the user experience.
  • a folding effect such as a folding city special effect
  • a first aspect of the embodiments of the present invention provides an image processing method, where the method includes:
  • a second aspect of embodiments of the present invention provides an image processing apparatus including a memory and a processor
  • the memory is configured to store program code
  • the processor is configured to invoke the program code, when the program code is executed, to perform the following operations: acquiring a plurality of 2D original images including a target object;
  • a third aspect of the embodiments of the present invention provides a computer readable storage medium, where the computer readable storage medium stores computer instructions, and when the computer instructions are executed, the image processing method is implemented, that is, the embodiment of the present invention The image processing method proposed in the first aspect.
  • the mobile platform since the mobile platform has good trajectory planning and has functions such as automatic flight and intelligent following, after the mobile platform acquires multiple 2D original images including the target object, the mobile platform can acquire according to the 2D original image. Folding the effect of 2D virtual images, reducing the acquisition complexity of images with folding effects (such as folding city effects), the above process is automatically completed, the user can shoot images with folding effect without intervention, and improve the user experience.
  • Figure 1 is a schematic structural view of a drone
  • FIG. 2 is a schematic diagram of an embodiment of an image processing method
  • Figure 3 is a schematic diagram of a deformation function
  • FIG. 4 is a schematic diagram of another embodiment of an image processing method
  • 5A-5F are schematic views of conversion of feature points
  • FIG. 6 is a schematic diagram of another embodiment of an image processing method
  • FIG. 7 is a schematic diagram of an embodiment of another image processing method
  • Figure 8 is a block diagram of one embodiment of an image processing apparatus.
  • first, second, third, etc. may be used to describe various information in the present invention, such information should not be limited to these terms. These terms are used to distinguish the same type of information from each other.
  • first information may also be referred to as the second information without departing from the scope of the invention.
  • second information may also be referred to as the first information.
  • word "if” may be interpreted as "when", or "when", or "in response to determination.”
  • An embodiment of the present invention provides an image processing method for acquiring an image having a folding effect (such as a folding city effect), and the method can be applied to an image processing device.
  • the image processing device may acquire a plurality of original images including the target object for the mobile platform, and use the original images to obtain an image having a folding effect.
  • the image processing device may also be a control device. After the mobile platform acquires a plurality of original images including the target object, the original image is sent to the control device, and after the control device acquires the plurality of original images including the target object, the original image is utilized. Get an image with a folding effect.
  • the mobile platform may include, but is not limited to, a robot, a drone, an unmanned vehicle, a VR glasses, and an AR glasses.
  • control device may include, but is not limited to: a remote control, a smart phone/mobile phone, a tablet, a personal digital assistant (PDA), a laptop computer, a desktop computer, a media content player, a video game station/system, a virtual reality system, Augmented reality systems, wearable devices (eg, watches, glasses, gloves, headwear (eg, hats, helmets, virtual reality headsets, augmented reality headsets, head mounted devices (HMD), headbands), pendants , armbands, leg loops, shoes, vests, gesture recognition devices, microphones, any electronic device capable of providing or rendering image data, without limitation.
  • PDA personal digital assistant
  • the image processing device is a mobile platform, and the mobile platform is an unmanned aerial vehicle.
  • the unmanned aerial vehicle is installed with a cloud platform, and the shooting device (such as a camera, a camera, etc.) is fixed on the cloud platform, and the drone has Good trajectory planning, with automatic flight and intelligent following functions, can obtain multiple original images containing the target object through the shooting device, and obtain the image with folding effect according to the original image.
  • FIG. 1 for a schematic diagram of the structure of the drone.
  • 10 indicates the nose of the drone
  • 11 indicates the propeller of the drone
  • 12 indicates the fuselage of the drone
  • 13 indicates the tripod of the drone
  • 14 indicates the gimbal on the drone
  • 15 indicates the mount of the gimbal 14
  • the photographing apparatus 15 is connected to the body 12 of the drone through the pan/tilt head 14, 16 is a photographing lens of the photographing apparatus, and 17 is a target object.
  • the pan/tilt head 14 may be a three-axis pan/tilt head, that is, the pan/tilt head 14 rotates on the axis of the pan/tilt (Roll axis), the pitch axis (Pitch axis), and the yaw axis (Yaw axis). As shown in Fig. 1, 1 indicates the Roll axis of the pan/tilt head, 2 indicates the Pitch axis of the pan/tilt head, and 3 indicates the Yaw axis of the pan/tilt head.
  • the roll angle of the pan/tilt changes; when the pan/tilt is rotated with the Pitch axis as the axis, the pitch angle of the pan/tilt changes; when the pan/tilt is rotated with the Yaw axis as the axis, The yaw angle of the gimbal has changed.
  • the photographing device 15 rotates following the rotation of the pan-tilt 14 so that the photographing device 15 can be taken from different photographing directions and photographing angles.
  • the target object 17 is photographed.
  • the fuselage 12 of the drone can also be rotated by the Roll axis, the Pitch axis, and the Yaw axis of the fuselage.
  • the roll angle of the fuselage changes;
  • the body of the drone rotates with the Pitch axis as the axis, the pitch angle of the fuselage changes;
  • the body of the drone rotates with the Yaw axis as the axis, the yaw angle of the fuselage changes.
  • FIG. 2 is a flowchart of an image processing method according to an embodiment of the present invention.
  • the method may be applied to an image processing device, and the method may include:
  • Step 201 Acquire a plurality of 2D original images including the target object.
  • the acquiring a plurality of 2D original images including the target object includes: planning a flight trajectory of the mobile platform (such as a drone) according to the location of the target object, and controlling the mobile platform to fly according to the flight trajectory, and the flight process on the mobile platform A plurality of 2D original images containing the target object are acquired.
  • the drone it has a good trajectory planning, with automatic flight and intelligent following functions. Therefore, the flight path of the drone can be planned according to the position of the target object, and the drone automatically flies according to the flight trajectory. During the automatic flight, the drone maintains the orientation of the shooting device in real time, ensuring that the target object is always at the center of the screen, so that multiple images containing the target object can be acquired by the shooting device, and the image containing the target object is called for convenience of distinction. 2D original image.
  • Step 202 Acquire a first type of point cloud of a 3D scene of the target object according to the plurality of 2D original images.
  • the first type of point cloud for acquiring a 3D scene of the target object according to the plurality of 2D original images includes: processing a plurality of 2D original images by an image processing algorithm to obtain a first type of point cloud of the 3D scene of the target object, where
  • the first type of point cloud may include a plurality of feature points having three-dimensional information.
  • the image processing algorithm may include, but is not limited to, a SfM (Structure from Motion) algorithm; the SfM algorithm may include: a sparse SfM algorithm; or a dense SfM algorithm.
  • SfM Structure from Motion
  • the image processing algorithm is used to process a plurality of 2D original images to obtain a first type of point cloud, and the process is not limited.
  • the input data is a plurality of 2D original images
  • the output data can be obtained after initializing the image sequence and camera calibration, extracting feature points and calculating feature descriptions, image matching calculations, solving motion estimation structure problems, and optional optimization processing.
  • the output data is the first type of cloud point of the 3D scene of the target object.
  • the first type of point cloud is a collection of multiple feature points, each of which has three-dimensional information, that is, three-dimensional coordinates (X, Y, Z).
  • each feature point can also have contents such as laser reflection intensity and color information, and there is no limitation thereto.
  • the sparse SfM algorithm can be used, and the first type of point cloud obtained by the sparse SfM algorithm is a sparse point cloud, that is, the number of feature points is relatively small, and the distance between the feature points and the feature points is relatively large.
  • a dense SfM algorithm can be used.
  • the first type of point cloud obtained by the dense SfM algorithm is a dense point cloud, that is, the number of feature points is relatively large, and the distance between the feature points and the feature points is relatively small.
  • Step 203 Perform a folding process on the first type of point cloud by using a deformation function to obtain a second type of point cloud.
  • the deformation function may include, but is not limited to: an exponential function; or a parabolic function; or an Archimedes spiral function, the deformation function is not limited, and may be selected according to experience, as long as the first type of point cloud can be performed Folding can be done.
  • the first type of point cloud is folded by using a deformation function to obtain a second type of point cloud (which may also include multiple feature points having three-dimensional information), which may include:
  • N deformation points are selected on the curve corresponding to the deformation function, and N is a positive integer greater than or equal to 1.
  • the value of N can be selected according to experience, such as 5, 10, etc., which is not limited.
  • selecting N deformation points on the curve corresponding to the deformation function may include: selecting a deformation point every first distance in the direction of the abscissa of the curve corresponding to the deformation function; or, in the ordinate direction of the curve corresponding to the deformation function, Select a deformation point every second distance.
  • the first distance can be configured according to experience, and there is no limitation on this.
  • the second distance can be configured according to experience, and there is no limitation on this.
  • FIG. 5A is a schematic diagram of a curve corresponding to a deformation function
  • FIG. 5A is a side view of a plane as an example. It is assumed that the flight direction is the x direction, the direction is the z direction, and the y direction is confirmed by the right hand rule.
  • N deformation points can be selected from the curve corresponding to the deformation function, and the adjacent two deformation points form a line segment, and the more the number of deformation points, the closer the line segment composed of the deformation points is to the curve, so that the multi-line segment is used. Approximate approximation curve.
  • the deformation point O, the deformation point A, the deformation point B, the deformation point C, and the deformation point D may be selected from the curve corresponding to the deformation function.
  • the number of deformation points can be more.
  • other methods may be used to select the deformation point, such as the distance between the deformation point O and the deformation point A, and the deformation point A and The distance of the deformation point B is different, and there is no limitation on this.
  • the line segment between the deformation point O and the deformation point A, the deformation point A and the deformation point B can be used.
  • the line segment between the line segment, the deformation point B and the deformation point C, the line segment between the deformation point C and the deformation point D approximates the curve corresponding to the deformation function, that is, approximates the curve shown in FIG. 5A.
  • an RDP (Ramer Douglas Peucker, Douglas-Pucker) algorithm can also be used.
  • the RDP algorithm is only an example, and no limitation is imposed thereon.
  • the RDP algorithm is an algorithm that approximates a curve as a series of points and reduces the number of points. As the number of points increases, the more the number of line segments is fitted, the closer it is to the original curve, but the higher the complexity; as the number of points decreases, the smaller the number of line segments fit, the greater the difference from the original curve, but The lower the complexity. Therefore, it is possible to select an appropriate number of points based on experience, and there is no limitation thereto.
  • Step 2032 Determine, for each feature point in the first type of point cloud, at least one deformation point corresponding to the feature point.
  • the deformation point corresponding to the feature point may be determined according to the abscissa value of the feature point.
  • the determining the at least one deformation point corresponding to the feature point may include: selecting an abscissa value (ie, an abscissa value of the deformation point) that is smaller than the feature point based on an abscissa value of each of the N deformation points.
  • the deformation point of the abscissa value, the selected deformation point is used as the deformation point corresponding to the feature point.
  • the abscissa value of the deformation point O is X O
  • the abscissa value of the deformation point A is X A
  • the abscissa value of the deformation point B is X B
  • the abscissa value of the deformation point C is X.
  • the horizontal coordinate value of the deformation point D is X D
  • the feature point 1 in the first type of point cloud is described (hereinafter, the feature point 1 is taken as an example, and the processing manner of each feature point in the first type of point cloud is The feature point 1 is similar, and will not be described later.
  • the abscissa value of the feature point 1 is located between X C and X D , and the abscissa value of the deformation point O, the deformation point A, the deformation point B, and the deformation point C are both It is smaller than the abscissa value of the feature point 1, and therefore, the deformation point corresponding to the feature point 1 is determined to be the deformation point O, the deformation point A, the deformation point B, and the deformation point C.
  • Step 2033 the feature points are folded by using the determined slope of each deformation point, and the folded feature points are obtained, that is, each feature point in the first type of point cloud corresponds to a folded feature point.
  • the feature point is folded by using the determined slope of each deformation point to obtain the folded feature point, including: if the feature point corresponds to the M deformation points, then the first deformation point to the Mth deformation point The order is obtained by using M slopes corresponding to the slopes of the M deformation points, and the feature points are subjected to M folding processing to obtain the folded feature points; M is a positive integer greater than or equal to 1, and M is less than or equal to N.
  • the feature point 1 is subjected to folding processing using the slope corresponding to the deformation point O (that is, the slope of the line segment between the deformation point O and the deformation point A) to obtain the feature point 1a.
  • the feature point 1a is subjected to folding processing using the slope corresponding to the deformation point A (that is, the slope of the line segment between the deformation point A and the deformation point B) to obtain the feature point 1b.
  • the feature point 1b is subjected to folding processing using the slope corresponding to the deformation point B (that is, the slope of the line segment between the deformation point B and the deformation point C) to obtain the feature point 1c.
  • the feature point 1c is folded to obtain the feature point 1d, and the folded feature point of the feature point 1 is the feature point 1d. .
  • the feature point is folded by using the determined slope of each deformation point to obtain the folded feature point, including: if the feature point corresponds to the M deformation points, the slope corresponding to the i-th deformation point is used.
  • the feature point is subjected to the ith folding process to obtain the i-th post-folding feature point; wherein, the value of i is 1, 2, ..., M, M is a positive integer greater than or equal to 1, and M is less than or equal to N.
  • the feature point 1 is subjected to the first folding process by the slope corresponding to the first deformation point O, and the first post-folding feature point 1a is obtained.
  • the feature point 1a is subjected to the second folding process by the slope corresponding to the second deformation point A, and the second post-folding feature point 1b is obtained.
  • the feature point 1b is subjected to the third folding process by the slope corresponding to the third deformation point B, and the third post-folding feature point 1c is obtained.
  • the feature point 1c is subjected to the fourth folding process using the slope corresponding to the fourth deformation point C to obtain the fourth post-folding feature point 1d. Therefore, the post-folding feature point of the feature point 1 is the feature point 1d.
  • the process of performing the i-th folding process on the feature point by using the slope corresponding to the i-th deformation point to obtain the i-th post-folding feature point may include the following steps:
  • Step 20331 Acquire a rotation parameter of the coordinate system by using a slope corresponding to the i-th deformation point.
  • the obtaining the rotation parameter of the coordinate system by using the slope corresponding to the i-th deformation point may include: obtaining a first angle of the i-th deformation point and the abscissa by using a slope corresponding to the i-th deformation point. Obtaining a target angle corresponding to the i-th deformation point by using the first angle and the second angle; wherein the second angle may be an angle between the i-1th deformation point and the abscissa.
  • the rotation parameter of the coordinate system is acquired by using the target angle corresponding to the i-th deformation point (the rotation parameter may also be referred to as a rotation matrix, that is, a Rotation Matrix).
  • the obtaining the first angle between the i-th deformation point and the abscissa by using the slope corresponding to the i-th deformation point may include: using the coordinate value of the i-th deformation point and the coordinate value of the i+1th deformation point, Obtain a slope corresponding to the i-th deformation point, and obtain a first angle by using a slope corresponding to the i-th deformation point.
  • the coordinate value of the deformation point C and the coordinate value of the deformation point D can be used to obtain the slope corresponding to the deformation point C, and the slope corresponding to the deformation point C is obtained.
  • the first angle (angle 3) of the deformation point B, the first angle (angle 2) of the deformation point A, and the first angle (angle 1) of the deformation point O can be obtained.
  • the obtaining the target angle corresponding to the i-th deformation point by using the first angle and the second angle may include: determining a difference between the first angle and the second angle as the target angle corresponding to the i-th deformation point .
  • the target angle is the difference between the angle 4 corresponding to the deformation point C and the angle 3 corresponding to the deformation point B.
  • the acquiring the rotation parameter of the coordinate system by using the target angle corresponding to the i-th deformation point may include: converting the target angle into a rotation matrix; and determining the rotation matrix as a rotation parameter of the coordinate system. Further, converting the target angle into a rotation matrix may include: converting the target angle into a rotation matrix by using a formula: ⁇ is the target angle.
  • Step 20332 Acquire a displacement parameter of the coordinate system by using an abscissa value of the i-th deformation point.
  • Obtaining the displacement parameter of the coordinate system by using the abscissa value of the i-th deformation point may include: constructing the displacement matrix by using the abscissa value of the i-th deformation point; determining the displacement matrix as the displacement parameter of the coordinate system.
  • constructing the displacement matrix by using the abscissa value of the i-th deformation point may include: constructing the displacement matrix by using the following formula: [x, 0, 0] T ; wherein x is the abscissa value of the i-th deformation point.
  • Step 20333 Perform the ith folding process on the feature point by using the rotation parameter and the displacement parameter to obtain the i-th post-folding feature point. For example, using the rotation parameter and the displacement parameter, the i-th folding feature point corresponding to the i-1th feature point corresponding to the feature point is subjected to the ith folding process to obtain the i-th post-folding feature point.
  • the i-th folding feature point corresponding to the i-1th feature point of the feature point is subjected to the ith folding process, and the ith post-folding feature point corresponding to the feature point is obtained, including:
  • step 2033 the feature point 1 can be subjected to the first folding process using the slope corresponding to the deformation point O to obtain the feature point 1a, and the feature point 1a is subjected to the second folding process using the slope corresponding to the deformation point A to obtain the feature point.
  • step 2033 the feature point 1b is subjected to the third folding process using the slope corresponding to the deformation point B to obtain the feature point 1c, and the feature point 1c is subjected to the fourth folding process by the slope corresponding to the deformation point C to obtain the feature point 1d.
  • the above folding process may be a transformation problem between different coordinate systems (such as F 0 , F 1 , F 2 , F 3 , etc.), so as long as the rotation parameter R between the coordinate systems is known (such as Rotation)
  • the folding process can be achieved by the Matrix and the displacement parameter T (eg Translation Matrix).
  • the adjacent coordinate systems are only the target angles, for example, the target angle ⁇ 1 of the deformation point O is the difference between the angles 1 and 0, and the deformation point
  • the target angle ⁇ 2 of A is the difference between the angle 2 and the angle 1
  • the target angle ⁇ 3 of the deformation point B is the difference between the angle 3 and the angle 2
  • the target angle ⁇ 4 of the deformation point C is the difference between the angle 4 and the angle 3.
  • the target angle can be converted into a rotation matrix, see the above conversion formula.
  • the rotation parameter R1 of the deformation point O can be obtained.
  • the rotation parameter R2 of the deformation point A can be obtained, and when ⁇ 3 is substituted into the above formula, the deformation point B can be obtained.
  • the rotation parameter R4 of the deformation point C can be obtained.
  • the displacement is the distance of each deformation point on the horizontal axis, such as the abscissa value X O of the deformation point O , and the abscissa of the deformation point A.
  • the value of X a, point B strain X-abscissa value B, the strain point of the abscissa value C X-C and the like can be configured using a strain point X O O displacement parameters Tl, X a displacement parameters configured using strain point a T2, using X B to construct the displacement parameter T3 of the deformation point B, the displacement parameter T4 of the deformation point C is constructed by X C .
  • three-dimensional information of the feature point 1d can be obtained.
  • multiple feature points may be folded together.
  • all the feature points corresponding to the deformation point O are folded.
  • the specific folding formula refer to feature point 1, and details are not described herein.
  • all the feature points corresponding to the deformation point O can be converted from the original coordinate system F 0 to the folded coordinate system F 1 .
  • all the feature points corresponding to the deformation point A are folded.
  • No further details are provided here.
  • all the feature points corresponding to the deformation point A can be converted from the original coordinate system F 1 to The folded coordinate system F 2 , and so on.
  • Step 2034 Determine the folded feature points corresponding to each feature point in the first type of point cloud as the second type of point cloud. That is to say, after folding each feature point in the first type of point cloud to obtain the folded feature point, all the folded feature points can be determined as the second type point cloud.
  • the second type of point cloud may include the folded feature point 1d corresponding to the feature point 1, the folded feature point 2d corresponding to the feature point 2, the folded feature point 3c corresponding to the feature point 3, and so on.
  • Step 204 projecting the second type of point cloud onto the image plane to obtain a 2D virtual image having a folding effect. For example, each feature point in the second type of point cloud (ie, the folded feature point) is projected onto the image plane.
  • the second type of point cloud is projected onto the image plane to obtain a 2D virtual image (ie, a final target image) having a folding effect (such as a folded city effect), which may include:
  • Step 2041 Obtain location information and posture information corresponding to the mobile platform.
  • the obtaining the location information and the posture information corresponding to the mobile platform may include: acquiring the shooting position and the shooting posture corresponding to the mobile platform according to the plurality of 2D original images; or acquiring the shooting position and shooting corresponding to the mobile platform according to the plurality of 2D original images. a gesture, and acquiring a virtual position and a virtual posture according to the shooting position and the shooting attitude.
  • the shooting position and the shooting attitude corresponding to the mobile platform can be obtained by other methods.
  • the shooting position and the shooting posture corresponding to the mobile platform can be directly measured by the sensor of the mobile platform, and no limitation is imposed thereon.
  • the acquiring the shooting position and the shooting posture corresponding to the mobile platform according to the plurality of 2D original images may include: after obtaining the plurality of 2D original images, the shooting positions of the mobile platform may be obtained by using the 2D original images (ie, the actual position and Attitude), thereby obtaining a photographing position (Rotation) and a photographing posture (Rotation), and the manner in which the photographing position and the photographing posture are acquired is not limited.
  • the mobile platform has a multi-sensor fusion positioning system (such as VO, IMU, GPS, etc.), and the multi-sensor fusion positioning system can give the position and attitude relationship of each 2D original image, thereby reducing positional uncertainty.
  • the multi-sensor fusion positioning system can give the position and attitude relationship of each 2D original image, thereby reducing positional uncertainty.
  • the acquiring the virtual position and the virtual posture according to the shooting position and the shooting attitude may include: virtualizing a virtual machine position, that is, a virtual position and a virtual posture, according to the actual shooting position and the shooting posture. For example, a curve is virtualized according to a plurality of actual shooting positions and shooting positions, and a viewing angle is selected from the curve as a virtual position and a virtual posture.
  • a virtual machine position that is, a virtual position and a virtual posture
  • a curve is virtualized according to a plurality of actual shooting positions and shooting positions, and a viewing angle is selected from the curve as a virtual position and a virtual posture.
  • Step 2042 Projecting each feature point in the second type of point cloud to the image plane of the mobile platform according to the location information and the posture information corresponding to the mobile platform, to obtain a 2D virtual image having a folding effect.
  • the feature image and the posture information corresponding to the mobile platform are used to project each feature point in the second type of point cloud to the image plane of the mobile platform to obtain a 2D virtual image having a folding effect, which may include: using a mobile platform corresponding to the mobile platform The internal parameter, the position information, and the posture information, projecting each feature point in the second type of point cloud to the image plane of the mobile platform, and obtaining a 2D pixel point corresponding to each feature point; The 2D pixel points corresponding to each feature point constitute a 2D virtual image.
  • each feature point in the second type of point cloud is projected onto the image plane of the mobile platform by using the internal parameter corresponding to the mobile platform, the position information, and the posture information, to obtain a 2D pixel point corresponding to each feature point.
  • the method may include: acquiring a 2D pixel corresponding to the feature point in the second type of point cloud by using the following formula:
  • K is an internal parameter corresponding to the mobile platform
  • R is the position information
  • T is the posture information
  • (x w , y w , z w ) is a three-dimensional corresponding to the feature points in the second type of point cloud.
  • the information, (u, v) is two-dimensional information corresponding to 2D pixel points corresponding to the feature points.
  • the second type of point cloud obtained in step 203 may include a plurality of feature points (ie, a plurality of post-folded feature points), each feature point having three-dimensional information, such as (x w , y w , z w ).
  • the second type of point cloud includes the above-mentioned feature point 1d, and the feature point 1d is taken as an example for description.
  • the processing of the other feature points in the second type of point cloud is similar to the processing of the feature point 1d, and will not be described again.
  • the three-dimensional information (x w , y w , z w ) corresponding to the feature point 1d can be obtained, and then the three-dimensional information (x w , y w , z w ) corresponding to the feature point 1d is substituted into the above formula.
  • a 2D pixel point corresponding to the feature point 1d can be obtained.
  • the 2D pixel points corresponding to each feature point can be obtained, and then the 2D pixel points can be composed into a 2D virtual image, and the 2D virtual image is also It is a 2D virtual image with a folding effect, which is a 2D image.
  • the internal reference K corresponding to the mobile platform is a matrix
  • the internal parameter K is a known parameter, which is a camera internal parameter, and no limitation is imposed thereon.
  • the position information R may be a rotation matrix
  • the attitude information T may be a displacement matrix
  • the rotation matrix R and the displacement matrix T are camera external parameters
  • the rotation and displacement transformation of the world coordinate system to the camera coordinate system are expressed in a three-dimensional space. There is no limit to this.
  • each feature point in the second type of point cloud is a 3D feature point
  • the 3D feature point can be converted into a 2D pixel point.
  • all 2D pixel points are composed into a 2D virtual image, so that the folded three-dimensional model can be reconverted into a 2D virtual image, that is, a 2D virtual image with a folding effect is obtained.
  • the mobile platform since the mobile platform has good trajectory planning and has functions such as automatic flight and intelligent following, after the mobile platform acquires multiple 2D original images including the target object, the mobile platform can acquire according to the 2D original image. Folding the effect of 2D virtual images, reducing the acquisition complexity of images with folding effects (such as folding city effects), the above process is automatically completed, the user can shoot images with folding effect without intervention, and improve the user experience.
  • the first type of point cloud when the first type of point cloud is obtained by using the dense SfM algorithm, most or all 2D points of the 2D original image correspond to 3D feature points in the first type of point cloud, so that in the second type of point cloud There are corresponding 3D feature points in the middle.
  • the first type of point cloud when the first type of point cloud is obtained by using the sparse SfM algorithm, only part of the 2D point in the 2D original image (for convenience of distinction, this part of the 2D point is called a 2D feature point) corresponds to the first type of point cloud.
  • the method may further include:
  • Step 701 Determine a 2D feature point corresponding to each feature point in the second type of point cloud from the 2D original image.
  • the feature points in the second type of point cloud correspond to the feature points in the first type of point cloud, and the feature points in the first type of point cloud are acquired based on the 2D feature points in the 2D original image, therefore, the second Each feature point in the class point cloud has a corresponding 2D feature point in the 2D original image.
  • Step 702 The 2D original image is divided into a plurality of triangular regions by using the 2D feature points; wherein, for each triangular region, three 2D feature points and a plurality of 2D original points may be included.
  • the dividing the 2D original image into the plurality of triangular regions by using the 2D feature points may include: connecting the 2D feature points in the 2D original image into a plurality of triangular regions by using a triangulation algorithm.
  • the 2D feature points may be triangulated by a triangulation algorithm (such as the Delaunay Triangulation algorithm), thereby dividing a plurality of triangular regions, and the triangle is divided.
  • a triangulation algorithm such as the Delaunay Triangulation algorithm
  • the three vertices of each triangle area are three 2D feature points, and the other points in the triangle area are 2D original points.
  • Step 703 For each triangular region, use the three 2D feature points included in the triangular region to acquire projection parameters corresponding to the triangular region (for projecting the 2D original point to the 2D virtual image).
  • the obtaining the projection parameters corresponding to the triangular region by using the three 2D feature points included in the triangular region includes: using the two 2D feature points in the 2D original image, the two 2D feature points in the The two-dimensional information in the 2D virtual image acquires projection parameters corresponding to the triangular region.
  • obtaining the projection parameters corresponding to the triangular region by using the two-dimensional information of the two 2D feature points in the 2D original image and the two-dimensional information of the three 2D feature points in the 2D virtual image may include: Obtain the projection parameters corresponding to the triangle region by using the following formula:
  • p' is two-dimensional information of the 2D feature point in the 2D virtual image
  • p is the two-dimensional information of the 2D feature point in the 2D original image
  • a and t are projection parameters corresponding to the triangular region.
  • the triangular region includes 3 2D feature points, which are assumed to be 2D feature points 1, 2D feature points 2 and 2D feature points 3, because 2D feature points 1 are in the 2D original image.
  • the dimension information p 1 is known, the 2D information p 1 ' of the 2D feature point 1 in the 2D virtual image is known, the 2D information p 2 of the 2D feature point 2 in the 2D original image is known, and the 2D feature point 2 is in 2D
  • the two-dimensional information p 2 ' in the virtual image is known, the two-dimensional information p 3 of the 2D feature point 3 in the 2D original image is known, and the two-dimensional information p 3 ' of the 2D feature point 3 in the 2D virtual image is known, Therefore, by the above p 1 , p 1 ', p 2 , p 2 ', p 3 , p 3 ', six unknown parameters can be calculated, so that the projection parameters A and t corresponding to the triangular regions can be calculated.
  • Step 704 Perform projection processing on all 2D original points in the triangular area by using projection parameters corresponding to the triangular area, that is, project all 2D original points in the triangular area to the 2D virtual image.
  • the projecting the all 2D original points in the triangular region by using the projection parameters corresponding to the triangular regions may include: projecting the 2D original points in the triangular region to the 2D virtual image by using the following formula: Where p is the two-dimensional information of the 2D original point in the 2D original image, A and t are the projection parameters corresponding to the triangular region, and p' is the two-dimensional information of the 2D original point in the 2D virtual image. Since p, A, and t are known, p' can be calculated by the above formula, and p' is two-dimensional information of the 2D original point in the 2D virtual image, that is, the 2D original point is projected to the 2D virtual image.
  • all 2D original points in the triangular region can be projected to the 2D virtual image. Furthermore, after performing the above processing for each of the triangular regions, all 2D original points of all the triangular regions can be projected to the 2D virtual image. Through the above processing, all 2D original points in the 2D original image can be projected to the 2D virtual image.
  • an embodiment of the present invention further provides an image processing apparatus 80, which includes a memory 801 and a processor 802 (such as one or more processors).
  • a processor 802 such as one or more processors.
  • the memory is configured to store program code
  • the processor is configured to invoke the program code, when the program code is executed, to perform the following operations: acquiring a plurality of 2D original images including a target object Obtaining a first type of point cloud of the 3D scene of the target object according to the plurality of 2D original images; performing a folding process on the first type of point cloud by using a deformation function to obtain a second type of point cloud; The second type of point cloud is projected onto the image plane to obtain a 2D virtual image with a folding effect.
  • the processor is configured to: when planning a plurality of 2D original images including the target object, plan a flight trajectory of the mobile platform according to the location of the target object, control the mobile platform to fly according to the flight trajectory, and collect during the flight. Contains multiple 2D raw images of the target object.
  • the processor is configured to: when the first type of point cloud of the 3D scene of the target object is acquired according to the plurality of 2D original images, process the plurality of 2D original images by using an image processing algorithm, Obtaining a first type of point cloud of the 3D scene of the target object; the first type of point cloud includes a plurality of feature points having three-dimensional information.
  • the processor is configured to perform folding processing on the first type of point cloud by using a deformation function, and when the second type of point cloud is obtained, the processor is specifically configured to: select N deformation points on the curve corresponding to the deformation function, and N is greater than a positive integer equal to 1; for each feature point in the first type of point cloud, determining at least one deformation point corresponding to the feature point; and folding the feature point by using the determined slope of each deformation point to obtain a folded Feature points; the folded feature points corresponding to each feature point in the first type of point cloud are determined as the second type of point cloud.
  • the processor when the processor selects N deformation points on the curve corresponding to the deformation function, the processor specifically selects: in the abscissa direction of the curve corresponding to the deformation function, selects a deformation point every first distance; or, corresponds to the deformation function In the ordinate direction of the curve, a deformation point is selected every second distance.
  • the processor when determining the at least one deformation point corresponding to the feature point, is specifically configured to: select an abscissa value of the feature point based on an abscissa value of each of the N deformation points The deformation point of the value is taken as the deformation point corresponding to the feature point.
  • the processor performs folding processing on the feature point by using the determined slope of each deformation point to obtain a folded feature point, and is specifically used to: if the feature point corresponds to M deformation points, follow the first The order of the deformation point to the Mth deformation point is performed by using the slope corresponding to the M deformation points, and the feature points are subjected to M folding processing to obtain a folded feature point; M is a positive integer greater than or equal to 1, and M is smaller than Equal to N.
  • the processor performs folding processing on the feature point by using the determined slope of each deformation point, and obtains the post-folded feature point, and is specifically used to: if the feature point corresponds to M deformation points, use the ith The slope corresponding to the deformation point is subjected to the ith folding process to obtain the i-th post-folding feature point; wherein the value of i is 1, 2, ..., M, and the M is greater than or equal to A positive integer of 1, M is less than or equal to N.
  • the processor performs the i-th folding process on the feature point by using the slope corresponding to the i-th deformation point, and obtains the i-th post-folding feature point, and is specifically used to: use the i-th deformation point correspondingly Obtaining the rotation parameter of the coordinate system, obtaining the displacement parameter of the coordinate system by using the abscissa value of the i-th deformation point, and performing the ith folding process on the feature point by using the rotation parameter and the displacement parameter to obtain the i-th Subfolded feature points after the fold.
  • the processor is configured to acquire the first clip of the i-th deformation point and the abscissa by using the slope corresponding to the i-th deformation point when acquiring the rotation parameter of the coordinate system by using the slope corresponding to the i-th deformation point.
  • An angle is obtained by using a target angle corresponding to the i-th deformation point.
  • the processor is configured to: when using the abscissa value of the i-th deformation point to obtain the displacement parameter of the coordinate system, construct the displacement matrix by using the abscissa value of the i-th deformation point; and determine the displacement matrix as The displacement parameter of the coordinate system.
  • the processor performs the ith folding process on the feature point by using the rotation parameter and the displacement parameter to obtain the i-th post-folding feature point, and is specifically used to: use the rotation parameter and the The displacement parameter is subjected to the i-th folding process of the i-th post-folding feature point corresponding to the feature point, and the i-th post-folding feature point corresponding to the feature point is obtained.
  • the processor is configured to: acquire the location information and the posture information corresponding to the mobile platform when the second type of point cloud is projected to the image plane to obtain the 2D virtual image with the folding effect; according to the mobile platform Corresponding position information and posture information, each feature point in the second type of point cloud is projected onto the image plane of the mobile platform to obtain a 2D virtual image having a folding effect.
  • the processor is configured to: acquire the shooting position and the shooting posture corresponding to the mobile platform according to the plurality of 2D original images when acquiring the position information and the posture information corresponding to the mobile platform; or, according to the multiple 2D
  • the original image acquires a shooting position and a shooting attitude corresponding to the mobile platform, and acquires a virtual position and a virtual posture according to the shooting position and the shooting posture.
  • the processor projects each feature point in the second type of point cloud to the image plane of the mobile platform according to the location information and the posture information corresponding to the mobile platform, to obtain a 2D virtual image with a folding effect.
  • the image is specifically used for: projecting each feature point in the second type of point cloud to the image plane by using the internal parameter corresponding to the mobile platform, the position information, and the posture information, and obtaining 2D corresponding to each feature point.
  • a pixel point; a 2D pixel corresponding to each feature point in the second type of point cloud is composed of the 2D virtual image.
  • the processor is further configured to: determine each feature in the second type of point cloud from the 2D original image. Pointing a corresponding 2D feature point; dividing the 2D original image into a plurality of triangular regions by using the 2D feature point; wherein, for each triangular region, including 3 2D feature points, a plurality of 2D original points; The three 2D feature points acquire projection parameters corresponding to the triangular regions; and use the projection parameters to perform projection processing on all 2D original points in the triangular region.
  • the processor when the processor divides the 2D original image into a plurality of triangular regions by using the 2D feature points, the processor is specifically configured to: connect the 2D feature points in the 2D original image into multiple triangles by using a triangulation algorithm. region.
  • the processor When the processor acquires the projection parameters corresponding to the triangular region by using the three 2D feature points, the processor specifically uses: two-dimensional information in the 2D original image using the three 2D feature points, and the three 2D feature points.
  • the two-dimensional information in the 2D virtual image acquires projection parameters corresponding to the triangular region.
  • the embodiment of the present invention further provides a computer readable storage medium, where the computer readable storage medium stores computer instructions, and when the computer instructions are executed, the image processing method is implemented. .
  • the system, apparatus, module or unit set forth in the above embodiments may be implemented by a computer chip or an entity, or by a product having a certain function.
  • a typical implementation device is a computer, and the specific form of the computer may be a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email transceiver, and a game control.
  • embodiments of the invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or a combination of software and hardware. Moreover, embodiments of the invention may take the form of a computer program product embodied on one or more computer usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) including computer usable program code.
  • computer usable storage media including but not limited to disk storage, CD-ROM, optical storage, etc.
  • these computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture comprising the instruction device.
  • the instruction means implements the functions specified in one or more blocks of the flowchart or in a flow or block diagram of the flowchart.

Landscapes

  • Physics & Mathematics (AREA)
  • Engineering & Computer Science (AREA)
  • Computer Graphics (AREA)
  • Geometry (AREA)
  • Software Systems (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Processing Or Creating Images (AREA)
  • Image Processing (AREA)

Abstract

L'invention porte sur un procédé et sur un dispositif de traitement d'image, ainsi que sur un support d'informations lisible par ordinateur. Le procédé comprend les étapes consistant : à acquérir de multiples images originales en 2D contenant un objet cible ; à acquérir, en fonction des multiples images originales en 2D, un nuage de points de premier type d'un scénario en 3D de l'objet cible ; à plier le nuage de points de premier type à l'aide d'une fonction de déformation afin d'acquérir un nuage de points de second type ; et à projeter le nuage de points de second type sur un plan d'image afin d'acquérir une image virtuelle en 2D présentant un effet de pliage. Des modes de réalisation de la présente invention réduisent la complexité d'acquisition d'images présentant un effet de pliage (tel qu'un effet spécial de ville pliable) et permettent à un utilisateur de capturer des images à effet de pliage, ce qui permet d'améliorer l'expérience de l'utilisateur.
PCT/CN2018/073632 2018-01-22 2018-01-22 Procédé et appareil de traitement d'image, et support d'informations lisible par ordinateur WO2019140688A1 (fr)

Priority Applications (2)

Application Number Priority Date Filing Date Title
CN201880012242.9A CN110313020A (zh) 2018-01-22 2018-01-22 图像处理方法、设备及计算机可读存储介质
PCT/CN2018/073632 WO2019140688A1 (fr) 2018-01-22 2018-01-22 Procédé et appareil de traitement d'image, et support d'informations lisible par ordinateur

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/CN2018/073632 WO2019140688A1 (fr) 2018-01-22 2018-01-22 Procédé et appareil de traitement d'image, et support d'informations lisible par ordinateur

Publications (1)

Publication Number Publication Date
WO2019140688A1 true WO2019140688A1 (fr) 2019-07-25

Family

ID=67301242

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2018/073632 WO2019140688A1 (fr) 2018-01-22 2018-01-22 Procédé et appareil de traitement d'image, et support d'informations lisible par ordinateur

Country Status (2)

Country Link
CN (1) CN110313020A (fr)
WO (1) WO2019140688A1 (fr)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20220084415A1 (en) * 2019-05-27 2022-03-17 SZ DJI Technology Co., Ltd. Flight planning method and related apparatus

Families Citing this family (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR101935820B1 (ko) 2017-08-28 2019-01-07 세종화학 (주) 인산암모늄 폐액을 이용한 고순도 인산이수소암모늄의 제조방법
CN113362236B (zh) * 2020-03-05 2024-03-05 北京京东乾石科技有限公司 点云增强方法、点云增强装置、存储介质与电子设备
CN113628322B (zh) * 2021-07-26 2023-12-05 阿里巴巴（中国）有限公司 图像处理、ar显示与直播方法、设备及存储介质

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN1928492A (zh) * 2006-09-15 2007-03-14 东南大学 三维扫描系统中基于格雷码的相位展开方法
CN101425183A (zh) * 2008-11-13 2009-05-06 上海交通大学 基于二阶锥规划的变形体三维跟踪方法
US9087381B2 (en) * 2013-11-13 2015-07-21 Thomas Tsao Method and apparatus for building surface representations of 3D objects from stereo images
CN104915986A (zh) * 2015-06-26 2015-09-16 北京航空航天大学 一种实体三维模型自动建模方法
CN106973569A (zh) * 2014-05-13 2017-07-21 Pcp虚拟现实股份有限公司 生成和回放虚拟现实多媒体的方法、系统和装置
CN107576275A (zh) * 2017-08-11 2018-01-12 哈尔滨工业大学 一种利用摄影测量技术对充气结构进行应变场测量的方法

Family Cites Families (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP5790345B2 (ja) * 2011-09-07 2015-10-07 株式会社リコー 画像処理装置、画像処理方法、プログラムおよび画像処理システム
US9466143B1 (en) * 2013-05-03 2016-10-11 Exelis, Inc. Geoaccurate three-dimensional reconstruction via image-based geometry
GB2559157A (en) * 2017-01-27 2018-08-01 Ucl Business Plc Apparatus, method and system for alignment of 3D datasets

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN1928492A (zh) * 2006-09-15 2007-03-14 东南大学 三维扫描系统中基于格雷码的相位展开方法
CN101425183A (zh) * 2008-11-13 2009-05-06 上海交通大学 基于二阶锥规划的变形体三维跟踪方法
US9087381B2 (en) * 2013-11-13 2015-07-21 Thomas Tsao Method and apparatus for building surface representations of 3D objects from stereo images
CN106973569A (zh) * 2014-05-13 2017-07-21 Pcp虚拟现实股份有限公司 生成和回放虚拟现实多媒体的方法、系统和装置
CN104915986A (zh) * 2015-06-26 2015-09-16 北京航空航天大学 一种实体三维模型自动建模方法
CN107576275A (zh) * 2017-08-11 2018-01-12 哈尔滨工业大学 一种利用摄影测量技术对充气结构进行应变场测量的方法

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20220084415A1 (en) * 2019-05-27 2022-03-17 SZ DJI Technology Co., Ltd. Flight planning method and related apparatus

Also Published As

Publication number Publication date
CN110313020A (zh) 2019-10-08

Similar Documents

Publication Publication Date Title
US10958891B2 (en) Visual annotation using tagging sessions
US11967162B2 (en) Method and apparatus for 3-D auto tagging
US20240290359A1 (en) Cinematic space-time view synthesis for enhanced viewing experiences in computing environments
US11288857B2 (en) Neural rerendering from 3D models
JP6775776B2 (ja) 自由視点移動表示装置
WO2018214078A1 (fr) Procédé et dispositif de commande de photographie
JP6643357B2 (ja) 全球状取込方法
US10127722B2 (en) Mobile capture visualization incorporating three-dimensional and two-dimensional imagery
WO2019140688A1 (fr) Procédé et appareil de traitement d'image, et support d'informations lisible par ordinateur
WO2019118877A1 (fr) Montage de vidéo sphérique
US11776142B2 (en) Structuring visual data
US11788845B2 (en) Systems and methods for robust self-relocalization in a visual map
CN106716497B (zh) 在表面重构中使用自由形式变形
WO2023280038A1 (fr) Procédé de construction d'un modèle tridimensionnel de scène réelle et appareil associé
CN105765631A (zh) 对跟踪和映射误差稳健的大规模表面重构
WO2019084719A1 (fr) Procédé de traitement d'images et véhicule aérien sans équipage
WO2022022449A1 (fr) Procédé et appareil de positionnement spatial
WO2019144286A1 (fr) Procédé de détection d'obstacles, plateforme mobile et support de stockage lisible par ordinateur
WO2022040868A1 (fr) Procédé de photographie panoramique, dispositif électronique et support de stockage
WO2023239955A1 (fr) Service de traitement de localisation et service de reconstruction de scène observée
CN111222586A (zh) 一种基于三维倾斜模型视角的倾斜影像匹配方法及装置
JP2010049346A (ja) 画像表示装置
WO2022246608A1 (fr) Procédé de génération de vidéo panoramique, appareil et plateforme mobile
US20210092306A1 (en) Movable body, image generation method, program, and recording medium
WO2020119572A1 (fr) Dispositif de déduction de forme, procédé de déduction de forme, programme et support d'enregistrement

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 18901473

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 18901473

Country of ref document: EP

Kind code of ref document: A1