WO2019140688A1 - Image processing method and apparatus and computer readable storage medium - Google Patents

Abstract

An image processing method and apparatus and a computer readable storage medium. The method comprises: acquiring multiple 2D original images containing a target object; acquiring, according to the multiple 2D original images, a first-type point cloud of a 3D scenario of the target object; folding the first-type point cloud by using a deformation function to acquire a second-type point cloud; and projecting the second-type point cloud onto an image plane to acquire a 2D virtual image having a folding effect. Embodiments of the present invention reduce acquisition complexity of images having a folding effect (such as a folding city special effect) and enable a user to capture folding effect images, thereby improving the user experience.

Description

Image processing method, device and computer readable storage medium Technical field

The present invention relates to the field of image processing technologies, and in particular, to an image processing method, device, and computer readable storage medium.

Background technique

Computer animation is a technique for making animations by means of a computer, including two-dimensional animation (2D) and three-dimensional animation (3D). Computer animation can be done by CG (Computer Graphics). Among them, the field of visual design and production using computer technology is called CG, and CG is a general term for all graphics drawn by computer, such as web design, film and television special effects, multimedia technology, and the like.

In the movie special effects of computer animation, folding urban special effects is a typical application. However, currently, CG technology is usually used to produce images with folding city special effects, which has high complexity and poor experience.

Summary of the invention

The invention provides an image processing method, a device and a computer readable storage medium, which can reduce the acquisition complexity of an image having a folding effect (such as a folding city special effect) and improve the user experience.

A first aspect of the embodiments of the present invention provides an image processing method, where the method includes:

Obtaining a plurality of 2D original images containing the target object;

Acquiring a first type of point cloud of the 3D scene of the target object according to the plurality of 2D original images;

Folding the first type of point cloud by using a deformation function to obtain a second type of point cloud;

Projecting the second type of point cloud onto the image plane yields a 2D virtual image with a folding effect.

A second aspect of embodiments of the present invention provides an image processing apparatus including a memory and a processor;

The memory is configured to store program code;

The processor is configured to invoke the program code, when the program code is executed, to perform the following operations: acquiring a plurality of 2D original images including a target object;

Acquiring a first type of point cloud of the 3D scene of the target object according to the plurality of 2D original images;

Folding the first type of point cloud by using a deformation function to obtain a second type of point cloud;

Projecting the second type of point cloud onto the image plane yields a 2D virtual image with a folding effect.

A third aspect of the embodiments of the present invention provides a computer readable storage medium, where the computer readable storage medium stores computer instructions, and when the computer instructions are executed, the image processing method is implemented, that is, the embodiment of the present invention The image processing method proposed in the first aspect.

Based on the foregoing technical solution, in the embodiment of the present invention, since the mobile platform has good trajectory planning and has functions such as automatic flight and intelligent following, after the mobile platform acquires multiple 2D original images including the target object, the mobile platform can acquire according to the 2D original image. Folding the effect of 2D virtual images, reducing the acquisition complexity of images with folding effects (such as folding city effects), the above process is automatically completed, the user can shoot images with folding effect without intervention, and improve the user experience.

DRAWINGS

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings to be used in the embodiments of the present invention or in the description of the prior art will be briefly described below. Obviously, the drawings in the following description It is merely some of the embodiments described in the present invention, and those skilled in the art can also obtain other drawings according to the drawings of the embodiments of the present invention.

Figure 1 is a schematic structural view of a drone;

2 is a schematic diagram of an embodiment of an image processing method;

Figure 3 is a schematic diagram of a deformation function;

4 is a schematic diagram of another embodiment of an image processing method;

5A-5F are schematic views of conversion of feature points;

6 is a schematic diagram of another embodiment of an image processing method;

7 is a schematic diagram of an embodiment of another image processing method;

Figure 8 is a block diagram of one embodiment of an image processing apparatus.

Detailed ways

The technical solutions in the embodiments of the present invention are clearly and completely described in the following with reference to the accompanying drawings in the embodiments of the present invention. It is obvious that the described embodiments are only a part of the embodiments of the present invention, but not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative efforts are within the scope of the present invention. Further, the features of the following embodiments and examples may be combined with each other without conflict.

The terminology used herein is for the purpose of describing particular embodiments, The singular forms "a", "the" and "the" It will be understood that the term "and/or" as used herein refers to any and all possible combinations of one or more of the associated listed items.

Although the terms first, second, third, etc. may be used to describe various information in the present invention, such information should not be limited to these terms. These terms are used to distinguish the same type of information from each other. For example, the first information may also be referred to as the second information without departing from the scope of the invention. Similarly, the second information may also be referred to as the first information. Depending on the context, in addition, the word "if" may be interpreted as "when", or "when", or "in response to determination."

An embodiment of the present invention provides an image processing method for acquiring an image having a folding effect (such as a folding city effect), and the method can be applied to an image processing device. The image processing device may acquire a plurality of original images including the target object for the mobile platform, and use the original images to obtain an image having a folding effect. The image processing device may also be a control device. After the mobile platform acquires a plurality of original images including the target object, the original image is sent to the control device, and after the control device acquires the plurality of original images including the target object, the original image is utilized. Get an image with a folding effect. The mobile platform may include, but is not limited to, a robot, a drone, an unmanned vehicle, a VR glasses, and an AR glasses. This is not limited, as long as it is a vehicle with multiple cameras. In addition, the control device may include, but is not limited to: a remote control, a smart phone/mobile phone, a tablet, a personal digital assistant (PDA), a laptop computer, a desktop computer, a media content player, a video game station/system, a virtual reality system, Augmented reality systems, wearable devices (eg, watches, glasses, gloves, headwear (eg, hats, helmets, virtual reality headsets, augmented reality headsets, head mounted devices (HMD), headbands), pendants , armbands, leg loops, shoes, vests, gesture recognition devices, microphones, any electronic device capable of providing or rendering image data, without limitation.

For convenience of description, the image processing device is a mobile platform, and the mobile platform is an unmanned aerial vehicle. The unmanned aerial vehicle is installed with a cloud platform, and the shooting device (such as a camera, a camera, etc.) is fixed on the cloud platform, and the drone has Good trajectory planning, with automatic flight and intelligent following functions, can obtain multiple original images containing the target object through the shooting device, and obtain the image with folding effect according to the original image.

See Figure 1 for a schematic diagram of the structure of the drone. 10 indicates the nose of the drone, 11 indicates the propeller of the drone, 12 indicates the fuselage of the drone, 13 indicates the tripod of the drone, 14 indicates the gimbal on the drone, and 15 indicates the mount of the gimbal 14 The photographing apparatus 15 is connected to the body 12 of the drone through the pan/ tilt head 14, 16 is a photographing lens of the photographing apparatus, and 17 is a target object.

The pan/tilt head 14 may be a three-axis pan/tilt head, that is, the pan/tilt head 14 rotates on the axis of the pan/tilt (Roll axis), the pitch axis (Pitch axis), and the yaw axis (Yaw axis). As shown in Fig. 1, 1 indicates the Roll axis of the pan/tilt head, 2 indicates the Pitch axis of the pan/tilt head, and 3 indicates the Yaw axis of the pan/tilt head. When the pan/tilt is rotated with the Roll axis as the axis, the roll angle of the pan/tilt changes; when the pan/tilt is rotated with the Pitch axis as the axis, the pitch angle of the pan/tilt changes; when the pan/tilt is rotated with the Yaw axis as the axis, The yaw angle of the gimbal has changed. Moreover, when the pan/tilt is rotated by one or more of the Yaw axis, the Pitch axis, and the Yaw axis, the photographing device 15 rotates following the rotation of the pan-tilt 14 so that the photographing device 15 can be taken from different photographing directions and photographing angles. The target object 17 is photographed.

Similar to the pan/tilt head 14, the fuselage 12 of the drone can also be rotated by the Roll axis, the Pitch axis, and the Yaw axis of the fuselage. When the fuselage of the drone rotates with the Roll axis as the axis, the roll angle of the fuselage changes; when the body of the drone rotates with the Pitch axis as the axis, the pitch angle of the fuselage changes; When the body of the drone rotates with the Yaw axis as the axis, the yaw angle of the fuselage changes.

Based on the foregoing application scenario, as shown in FIG. 2, which is a flowchart of an image processing method according to an embodiment of the present invention, the method may be applied to an image processing device, and the method may include:

Step 201: Acquire a plurality of 2D original images including the target object.

The acquiring a plurality of 2D original images including the target object includes: planning a flight trajectory of the mobile platform (such as a drone) according to the location of the target object, and controlling the mobile platform to fly according to the flight trajectory, and the flight process on the mobile platform A plurality of 2D original images containing the target object are acquired.

Among them, for the drone, it has a good trajectory planning, with automatic flight and intelligent following functions. Therefore, the flight path of the drone can be planned according to the position of the target object, and the drone automatically flies according to the flight trajectory. During the automatic flight, the drone maintains the orientation of the shooting device in real time, ensuring that the target object is always at the center of the screen, so that multiple images containing the target object can be acquired by the shooting device, and the image containing the target object is called for convenience of distinction. 2D original image.

Step 202: Acquire a first type of point cloud of a 3D scene of the target object according to the plurality of 2D original images.

The first type of point cloud for acquiring a 3D scene of the target object according to the plurality of 2D original images includes: processing a plurality of 2D original images by an image processing algorithm to obtain a first type of point cloud of the 3D scene of the target object, where The first type of point cloud may include a plurality of feature points having three-dimensional information.

The image processing algorithm may include, but is not limited to, a SfM (Structure from Motion) algorithm; the SfM algorithm may include: a sparse SfM algorithm; or a dense SfM algorithm.

In one example, the image processing algorithm is used to process a plurality of 2D original images to obtain a first type of point cloud, and the process is not limited. For example, the input data is a plurality of 2D original images, and the output data can be obtained after initializing the image sequence and camera calibration, extracting feature points and calculating feature descriptions, image matching calculations, solving motion estimation structure problems, and optional optimization processing. The output data is the first type of cloud point of the 3D scene of the target object. The first type of point cloud is a collection of multiple feature points, each of which has three-dimensional information, that is, three-dimensional coordinates (X, Y, Z). Of course, each feature point can also have contents such as laser reflection intensity and color information, and there is no limitation thereto.

In order to obtain the first type of point cloud, the sparse SfM algorithm can be used, and the first type of point cloud obtained by the sparse SfM algorithm is a sparse point cloud, that is, the number of feature points is relatively small, and the distance between the feature points and the feature points is relatively large. In addition, a dense SfM algorithm can be used. The first type of point cloud obtained by the dense SfM algorithm is a dense point cloud, that is, the number of feature points is relatively large, and the distance between the feature points and the feature points is relatively small.

Step 203: Perform a folding process on the first type of point cloud by using a deformation function to obtain a second type of point cloud.

Wherein, the deformation function may include, but is not limited to: an exponential function; or a parabolic function; or an Archimedes spiral function, the deformation function is not limited, and may be selected according to experience, as long as the first type of point cloud can be performed Folding can be done. For convenience of description, the following is an example of the deformation function being an exponential function. Referring to FIG. 3, the lower plane is the original ground. For the convenience of viewing, there is a certain inclination, and by constructing a deformation function, that is, an exponential function y=e ^x , Can be folded into a curved surface.

As shown in FIG. 4, the first type of point cloud is folded by using a deformation function to obtain a second type of point cloud (which may also include multiple feature points having three-dimensional information), which may include:

In step 2031, N deformation points are selected on the curve corresponding to the deformation function, and N is a positive integer greater than or equal to 1. Among them, the value of N can be selected according to experience, such as 5, 10, etc., which is not limited.

Wherein, selecting N deformation points on the curve corresponding to the deformation function may include: selecting a deformation point every first distance in the direction of the abscissa of the curve corresponding to the deformation function; or, in the ordinate direction of the curve corresponding to the deformation function, Select a deformation point every second distance. The first distance can be configured according to experience, and there is no limitation on this. The second distance can be configured according to experience, and there is no limitation on this.

Referring to FIG. 5A, which is a schematic diagram of a curve corresponding to a deformation function, FIG. 5A is a side view of a plane as an example. It is assumed that the flight direction is the x direction, the direction is the z direction, and the y direction is confirmed by the right hand rule. To point to the paper. Through multi-segment folding, you can gradually approach the curve of the target. The more segments you have, the more you can approach the real curve infinitely. Therefore, N deformation points can be selected from the curve corresponding to the deformation function, and the adjacent two deformation points form a line segment, and the more the number of deformation points, the closer the line segment composed of the deformation points is to the curve, so that the multi-line segment is used. Approximate approximation curve.

Referring to FIG. 5B, the deformation point O, the deformation point A, the deformation point B, the deformation point C, and the deformation point D may be selected from the curve corresponding to the deformation function. In FIG. 5B, taking five deformation points as an example, in practical applications, the number of deformation points can be more. In FIG. 5B, taking a deformation point every first distance in the direction of the abscissa as an example, in practical applications, other methods may be used to select the deformation point, such as the distance between the deformation point O and the deformation point A, and the deformation point A and The distance of the deformation point B is different, and there is no limitation on this.

Referring to FIG. 5C, after selecting the deformation point O, the deformation point A, the deformation point B, the deformation point C, and the deformation point D, the line segment between the deformation point O and the deformation point A, the deformation point A and the deformation point B can be used. The line segment between the line segment, the deformation point B and the deformation point C, the line segment between the deformation point C and the deformation point D, approximates the curve corresponding to the deformation function, that is, approximates the curve shown in FIG. 5A.

In practical applications, in order to fit the line segment into a curve, an RDP (Ramer Douglas Peucker, Douglas-Pucker) algorithm can also be used. Of course, the RDP algorithm is only an example, and no limitation is imposed thereon. Among them, the RDP algorithm is an algorithm that approximates a curve as a series of points and reduces the number of points. As the number of points increases, the more the number of line segments is fitted, the closer it is to the original curve, but the higher the complexity; as the number of points decreases, the smaller the number of line segments fit, the greater the difference from the original curve, but The lower the complexity. Therefore, it is possible to select an appropriate number of points based on experience, and there is no limitation thereto.

Step 2032: Determine, for each feature point in the first type of point cloud, at least one deformation point corresponding to the feature point. For example, the deformation point corresponding to the feature point may be determined according to the abscissa value of the feature point.

The determining the at least one deformation point corresponding to the feature point may include: selecting an abscissa value (ie, an abscissa value of the deformation point) that is smaller than the feature point based on an abscissa value of each of the N deformation points. The deformation point of the abscissa value, the selected deformation point is used as the deformation point corresponding to the feature point.

For example, referring to FIG. 5D, the abscissa value of the deformation point O is X _O , the abscissa value of the deformation point A is X _A , the abscissa value of the deformation point B is X _B , and the abscissa value of the deformation point C is X. _C , the horizontal coordinate value of the deformation point D is X _D , and the feature point 1 in the first type of point cloud is described (hereinafter, the feature point 1 is taken as an example, and the processing manner of each feature point in the first type of point cloud is The feature point 1 is similar, and will not be described later. It is assumed that the abscissa value of the feature point 1 is located between X _C and X _D , and the abscissa value of the deformation point O, the deformation point A, the deformation point B, and the deformation point C are both It is smaller than the abscissa value of the feature point 1, and therefore, the deformation point corresponding to the feature point 1 is determined to be the deformation point O, the deformation point A, the deformation point B, and the deformation point C.

Step 2033, the feature points are folded by using the determined slope of each deformation point, and the folded feature points are obtained, that is, each feature point in the first type of point cloud corresponds to a folded feature point.

The feature point is folded by using the determined slope of each deformation point to obtain the folded feature point, including: if the feature point corresponds to the M deformation points, then the first deformation point to the Mth deformation point The order is obtained by using M slopes corresponding to the slopes of the M deformation points, and the feature points are subjected to M folding processing to obtain the folded feature points; M is a positive integer greater than or equal to 1, and M is less than or equal to N.

For example, the feature point 1 is subjected to folding processing using the slope corresponding to the deformation point O (that is, the slope of the line segment between the deformation point O and the deformation point A) to obtain the feature point 1a. Then, the feature point 1a is subjected to folding processing using the slope corresponding to the deformation point A (that is, the slope of the line segment between the deformation point A and the deformation point B) to obtain the feature point 1b. Then, the feature point 1b is subjected to folding processing using the slope corresponding to the deformation point B (that is, the slope of the line segment between the deformation point B and the deformation point C) to obtain the feature point 1c. Then, using the slope corresponding to the deformation point C (ie, the slope of the line segment between the deformation point C and the deformation point D), the feature point 1c is folded to obtain the feature point 1d, and the folded feature point of the feature point 1 is the feature point 1d. .

The feature point is folded by using the determined slope of each deformation point to obtain the folded feature point, including: if the feature point corresponds to the M deformation points, the slope corresponding to the i-th deformation point is used. The feature point is subjected to the ith folding process to obtain the i-th post-folding feature point; wherein, the value of i is 1, 2, ..., M, M is a positive integer greater than or equal to 1, and M is less than or equal to N.

For example, the feature point 1 is subjected to the first folding process by the slope corresponding to the first deformation point O, and the first post-folding feature point 1a is obtained. The feature point 1a is subjected to the second folding process by the slope corresponding to the second deformation point A, and the second post-folding feature point 1b is obtained. The feature point 1b is subjected to the third folding process by the slope corresponding to the third deformation point B, and the third post-folding feature point 1c is obtained. The feature point 1c is subjected to the fourth folding process using the slope corresponding to the fourth deformation point C to obtain the fourth post-folding feature point 1d. Therefore, the post-folding feature point of the feature point 1 is the feature point 1d.

In an example, the process of performing the i-th folding process on the feature point by using the slope corresponding to the i-th deformation point to obtain the i-th post-folding feature point may include the following steps:

Step 20331: Acquire a rotation parameter of the coordinate system by using a slope corresponding to the i-th deformation point.

The obtaining the rotation parameter of the coordinate system by using the slope corresponding to the i-th deformation point may include: obtaining a first angle of the i-th deformation point and the abscissa by using a slope corresponding to the i-th deformation point. Obtaining a target angle corresponding to the i-th deformation point by using the first angle and the second angle; wherein the second angle may be an angle between the i-1th deformation point and the abscissa. The rotation parameter of the coordinate system is acquired by using the target angle corresponding to the i-th deformation point (the rotation parameter may also be referred to as a rotation matrix, that is, a Rotation Matrix).

The obtaining the first angle between the i-th deformation point and the abscissa by using the slope corresponding to the i-th deformation point may include: using the coordinate value of the i-th deformation point and the coordinate value of the i+1th deformation point, Obtain a slope corresponding to the i-th deformation point, and obtain a first angle by using a slope corresponding to the i-th deformation point.

Referring to FIG. 5E, assuming that the i-th deformation point is the deformation point C, the coordinate value of the deformation point C and the coordinate value of the deformation point D can be used to obtain the slope corresponding to the deformation point C, and the slope corresponding to the deformation point C is obtained. First angle (angle 4). Similarly, the first angle (angle 3) of the deformation point B, the first angle (angle 2) of the deformation point A, and the first angle (angle 1) of the deformation point O can be obtained.

The obtaining the target angle corresponding to the i-th deformation point by using the first angle and the second angle may include: determining a difference between the first angle and the second angle as the target angle corresponding to the i-th deformation point .

For example, if the i-th deformation point is the deformation point C, the i-1st deformation point is the deformation point B. Therefore, the target angle is the difference between the angle 4 corresponding to the deformation point C and the angle 3 corresponding to the deformation point B.

The acquiring the rotation parameter of the coordinate system by using the target angle corresponding to the i-th deformation point may include: converting the target angle into a rotation matrix; and determining the rotation matrix as a rotation parameter of the coordinate system. Further, converting the target angle into a rotation matrix may include: converting the target angle into a rotation matrix by using a formula:

θ is the target angle.

Step 20332: Acquire a displacement parameter of the coordinate system by using an abscissa value of the i-th deformation point.

Obtaining the displacement parameter of the coordinate system by using the abscissa value of the i-th deformation point may include: constructing the displacement matrix by using the abscissa value of the i-th deformation point; determining the displacement matrix as the displacement parameter of the coordinate system.

Further, constructing the displacement matrix by using the abscissa value of the i-th deformation point may include: constructing the displacement matrix by using the following formula: [x, 0, 0] ^T ; wherein x is the abscissa value of the i-th deformation point.

Step 20333: Perform the ith folding process on the feature point by using the rotation parameter and the displacement parameter to obtain the i-th post-folding feature point. For example, using the rotation parameter and the displacement parameter, the i-th folding feature point corresponding to the i-1th feature point corresponding to the feature point is subjected to the ith folding process to obtain the i-th post-folding feature point.

Wherein, using the rotation parameter and the displacement parameter, the i-th folding feature point corresponding to the i-1th feature point of the feature point is subjected to the ith folding process, and the ith post-folding feature point corresponding to the feature point is obtained, including:

The three-dimensional information of the i-th post-folding feature point can be obtained by using the following formula: P _i =R*(P _i-1 -T)+T; wherein P _i represents the three-dimensional information of the i-th post-folding feature point, P _i-1 represents three-dimensional information of the i-1th post-folding feature point, R represents the rotation parameter, and T represents the displacement parameter.

The process of the above step 2033 is described in detail below in conjunction with a specific application scenario. In step 2033, the feature point 1 can be subjected to the first folding process using the slope corresponding to the deformation point O to obtain the feature point 1a, and the feature point 1a is subjected to the second folding process using the slope corresponding to the deformation point A to obtain the feature point. 1b, the feature point 1b is subjected to the third folding process using the slope corresponding to the deformation point B to obtain the feature point 1c, and the feature point 1c is subjected to the fourth folding process by the slope corresponding to the deformation point C to obtain the feature point 1d. Referring to FIG. 5F, the above folding process may be a transformation problem between different coordinate systems (such as F ₀ , F ₁ , F ₂ , F _{3 ,} etc.), so as long as the rotation parameter R between the coordinate systems is known (such as Rotation) The folding process can be achieved by the Matrix and the displacement parameter T (eg Translation Matrix).

In the process of solving the rotation parameter R (ie, the rotation relationship between adjacent coordinate systems), the adjacent coordinate systems are only the target angles, for example, the target angle θ1 of the deformation point O is the difference between the angles 1 and 0, and the deformation point The target angle θ2 of A is the difference between the angle 2 and the angle 1, the target angle θ3 of the deformation point B is the difference between the angle 3 and the angle 2, and the target angle θ4 of the deformation point C is the difference between the angle 4 and the angle 3. Further, the target angle can be converted into a rotation matrix, see the above conversion formula. When θ1 is substituted into the above formula, the rotation parameter R1 of the deformation point O can be obtained. When θ2 is substituted into the above formula, the rotation parameter R2 of the deformation point A can be obtained, and when θ3 is substituted into the above formula, the deformation point B can be obtained. When the parameter R3 is rotated, when the θ4 is substituted into the above formula, the rotation parameter R4 of the deformation point C can be obtained.

In the solution process of the displacement parameter T (ie, the displacement relationship between adjacent coordinate systems), the displacement is the distance of each deformation point on the horizontal axis, such as the abscissa value X _O of the deformation point _O , and the abscissa of the deformation point A. the value of X _a, point B strain X-abscissa value _B, the strain point of the abscissa value C _X-C and the like, and therefore, can be configured using a strain point X _O O displacement parameters Tl, X _a displacement parameters configured using strain point a T2, using X _{B to} construct the displacement parameter T3 of the deformation point B, the displacement parameter T4 of the deformation point C is constructed by X _C .

Then, the first folding process is performed on the feature point 1 by using the following formula: P ₁ = R1 * (P ₀ - T1) + T1; wherein P ₀ is the three-dimensional information of the feature point 1 (ie, the coordinate point of the coordinate system F ₀ ) P ₁ is the three-dimensional information of the feature point 1a (ie, the coordinate point of the coordinate system F ₁ ). Then, the second folding process is performed on the feature point 1 by using the following formula: P ₂ = R2 * (P ₁ - T2) + T2; wherein P ₂ is the three-dimensional information of the feature point 1b (ie, the coordinate point of the coordinate system F ₂ ). And so on. Finally, three-dimensional information of the feature point 1d can be obtained.

In practical applications, when there are multiple feature points, multiple feature points may be folded together. For example, all the feature points corresponding to the deformation point O are folded. For the specific folding formula, refer to feature point 1, and details are not described herein. In this way, all the feature points corresponding to the deformation point O can be converted from the original coordinate system F ₀ to the folded coordinate system F ₁ . Then, all the feature points corresponding to the deformation point A are folded. For the specific folding formula, refer to the feature point 1. No further details are provided here. Thus, all the feature points corresponding to the deformation point A can be converted from the original coordinate system F ₁ to The folded coordinate system F ₂ , and so on.

Step 2034: Determine the folded feature points corresponding to each feature point in the first type of point cloud as the second type of point cloud. That is to say, after folding each feature point in the first type of point cloud to obtain the folded feature point, all the folded feature points can be determined as the second type point cloud.

For example, the second type of point cloud may include the folded feature point 1d corresponding to the feature point 1, the folded feature point 2d corresponding to the feature point 2, the folded feature point 3c corresponding to the feature point 3, and so on.

Step 204, projecting the second type of point cloud onto the image plane to obtain a 2D virtual image having a folding effect. For example, each feature point in the second type of point cloud (ie, the folded feature point) is projected onto the image plane.

For example, as shown in FIG. 6, the second type of point cloud is projected onto the image plane to obtain a 2D virtual image (ie, a final target image) having a folding effect (such as a folded city effect), which may include:

Step 2041: Obtain location information and posture information corresponding to the mobile platform.

The obtaining the location information and the posture information corresponding to the mobile platform may include: acquiring the shooting position and the shooting posture corresponding to the mobile platform according to the plurality of 2D original images; or acquiring the shooting position and shooting corresponding to the mobile platform according to the plurality of 2D original images. a gesture, and acquiring a virtual position and a virtual posture according to the shooting position and the shooting attitude. Of course, in the actual application, the shooting position and the shooting attitude corresponding to the mobile platform can be obtained by other methods. For example, the shooting position and the shooting posture corresponding to the mobile platform can be directly measured by the sensor of the mobile platform, and no limitation is imposed thereon.

The acquiring the shooting position and the shooting posture corresponding to the mobile platform according to the plurality of 2D original images may include: after obtaining the plurality of 2D original images, the shooting positions of the mobile platform may be obtained by using the 2D original images (ie, the actual position and Attitude), thereby obtaining a photographing position (Rotation) and a photographing posture (Rotation), and the manner in which the photographing position and the photographing posture are acquired is not limited.

Further, the mobile platform has a multi-sensor fusion positioning system (such as VO, IMU, GPS, etc.), and the multi-sensor fusion positioning system can give the position and attitude relationship of each 2D original image, thereby reducing positional uncertainty. By taking the position and attitude relationship as initial values, when acquiring the shooting position and shooting posture of the mobile platform, the calculation speed can be accelerated, and the consumption of computing resources can be reduced.

The acquiring the virtual position and the virtual posture according to the shooting position and the shooting attitude may include: virtualizing a virtual machine position, that is, a virtual position and a virtual posture, according to the actual shooting position and the shooting posture. For example, a curve is virtualized according to a plurality of actual shooting positions and shooting positions, and a viewing angle is selected from the curve as a virtual position and a virtual posture. Alternatively, it is also possible to rotate the 3D scene according to the user drag and drop, and manually create a virtual machine position to obtain a virtual position and a virtual posture.

Step 2042: Projecting each feature point in the second type of point cloud to the image plane of the mobile platform according to the location information and the posture information corresponding to the mobile platform, to obtain a 2D virtual image having a folding effect.

The feature image and the posture information corresponding to the mobile platform are used to project each feature point in the second type of point cloud to the image plane of the mobile platform to obtain a 2D virtual image having a folding effect, which may include: using a mobile platform corresponding to the mobile platform The internal parameter, the position information, and the posture information, projecting each feature point in the second type of point cloud to the image plane of the mobile platform, and obtaining a 2D pixel point corresponding to each feature point; The 2D pixel points corresponding to each feature point constitute a 2D virtual image.

Further, each feature point in the second type of point cloud is projected onto the image plane of the mobile platform by using the internal parameter corresponding to the mobile platform, the position information, and the posture information, to obtain a 2D pixel point corresponding to each feature point. The method may include: acquiring a 2D pixel corresponding to the feature point in the second type of point cloud by using the following formula:

In the above formula, K is an internal parameter corresponding to the mobile platform, R is the position information, T is the posture information, and (x _w , y _w , z _w ) is a three-dimensional corresponding to the feature points in the second type of point cloud. The information, (u, v) is two-dimensional information corresponding to 2D pixel points corresponding to the feature points.

For example, the second type of point cloud obtained in step 203 may include a plurality of feature points (ie, a plurality of post-folded feature points), each feature point having three-dimensional information, such as (x _w , y _w , z _w ). For example, the second type of point cloud includes the above-mentioned feature point 1d, and the feature point 1d is taken as an example for description. The processing of the other feature points in the second type of point cloud is similar to the processing of the feature point 1d, and will not be described again. For the feature point 1d, the three-dimensional information (x _w , y _w , z _w ) corresponding to the feature point 1d can be obtained, and then the three-dimensional information (x _w , y _w , z _w ) corresponding to the feature point 1d is substituted into the above formula. A 2D pixel point corresponding to the feature point 1d can be obtained.

Further, after performing the above processing on each feature point in the second type of point cloud, the 2D pixel points corresponding to each feature point can be obtained, and then the 2D pixel points can be composed into a 2D virtual image, and the 2D virtual image is also It is a 2D virtual image with a folding effect, which is a 2D image.

In the above formula, the internal reference K corresponding to the mobile platform is a matrix, and the internal parameter K is a known parameter, which is a camera internal parameter, and no limitation is imposed thereon. In addition, the position information R may be a rotation matrix, the attitude information T may be a displacement matrix, the rotation matrix R and the displacement matrix T are camera external parameters, and the rotation and displacement transformation of the world coordinate system to the camera coordinate system are expressed in a three-dimensional space. There is no limit to this.

In one example, since the folding process is operated on the three-dimensional model, that is, each feature point in the second type of point cloud is a 3D feature point, therefore, after the above processing, the 3D feature point can be converted into a 2D pixel point. And all 2D pixel points are composed into a 2D virtual image, so that the folded three-dimensional model can be reconverted into a 2D virtual image, that is, a 2D virtual image with a folding effect is obtained.

Based on the foregoing technical solution, in the embodiment of the present invention, since the mobile platform has good trajectory planning and has functions such as automatic flight and intelligent following, after the mobile platform acquires multiple 2D original images including the target object, the mobile platform can acquire according to the 2D original image. Folding the effect of 2D virtual images, reducing the acquisition complexity of images with folding effects (such as folding city effects), the above process is automatically completed, the user can shoot images with folding effect without intervention, and improve the user experience.

In an example, when the first type of point cloud is obtained by using the dense SfM algorithm, most or all 2D points of the 2D original image correspond to 3D feature points in the first type of point cloud, so that in the second type of point cloud There are corresponding 3D feature points in the middle. In another example, when the first type of point cloud is obtained by using the sparse SfM algorithm, only part of the 2D point in the 2D original image (for convenience of distinction, this part of the 2D point is called a 2D feature point) corresponds to the first type of point cloud. There are 3D feature points, so that there are 3D feature points in the second type of point cloud, so that the 2D virtual image only includes the 2D pixel points corresponding to the 2D feature points, and the remaining 2D points in the 2D original image (for convenience of distinction) The remaining 2D points are referred to as 2D original points, and can also be mapped to 2D virtual images. The process will be described below.

As shown in FIG. 7 , after the second type of point cloud is projected onto the image plane, after the 2D virtual image is obtained, in order to map the remaining 2D original points in the 2D original image to the 2D virtual image, the method may further include:

Step 701: Determine a 2D feature point corresponding to each feature point in the second type of point cloud from the 2D original image. Wherein, the feature points in the second type of point cloud correspond to the feature points in the first type of point cloud, and the feature points in the first type of point cloud are acquired based on the 2D feature points in the 2D original image, therefore, the second Each feature point in the class point cloud has a corresponding 2D feature point in the 2D original image.

Step 702: The 2D original image is divided into a plurality of triangular regions by using the 2D feature points; wherein, for each triangular region, three 2D feature points and a plurality of 2D original points may be included.

The dividing the 2D original image into the plurality of triangular regions by using the 2D feature points may include: connecting the 2D feature points in the 2D original image into a plurality of triangular regions by using a triangulation algorithm.

Specifically, after all the 2D feature points are acquired from the 2D original image, the 2D feature points may be triangulated by a triangulation algorithm (such as the Delaunay Triangulation algorithm), thereby dividing a plurality of triangular regions, and the triangle is divided. There is no restriction on the way to split. The three vertices of each triangle area are three 2D feature points, and the other points in the triangle area are 2D original points.

Step 703: For each triangular region, use the three 2D feature points included in the triangular region to acquire projection parameters corresponding to the triangular region (for projecting the 2D original point to the 2D virtual image).

The obtaining the projection parameters corresponding to the triangular region by using the three 2D feature points included in the triangular region includes: using the two 2D feature points in the 2D original image, the two 2D feature points in the The two-dimensional information in the 2D virtual image acquires projection parameters corresponding to the triangular region.

Further, obtaining the projection parameters corresponding to the triangular region by using the two-dimensional information of the two 2D feature points in the 2D original image and the two-dimensional information of the three 2D feature points in the 2D virtual image may include: Obtain the projection parameters corresponding to the triangle region by using the following formula:

Wherein p' is two-dimensional information of the 2D feature point in the 2D virtual image, p is the two-dimensional information of the 2D feature point in the 2D original image, and A and t are projection parameters corresponding to the triangular region.

In the above formula,

That is to say, there are 6 unknown parameters in total, and the triangular region includes 3 2D feature points, which are assumed to be 2D feature points 1, 2D feature points 2 and 2D feature points 3, because 2D feature points 1 are in the 2D original image. The dimension information p _{1 is} known, the 2D information p ₁ ' of the 2D feature point 1 in the 2D virtual image is known, the 2D information p _{2 of the} 2D feature point 2 in the 2D original image is known, and the 2D feature point 2 is in 2D The two-dimensional information p ₂ ' in the virtual image is known, the two-dimensional information p _{3 of the} 2D feature point 3 in the 2D original image is known, and the two-dimensional information p ₃ ' of the 2D feature point 3 in the 2D virtual image is known, Therefore, by the above p ₁ , p ₁ ', p ₂ , p ₂ ', p ₃ , p ₃ ', six unknown parameters can be calculated, so that the projection parameters A and t corresponding to the triangular regions can be calculated.

Step 704: Perform projection processing on all 2D original points in the triangular area by using projection parameters corresponding to the triangular area, that is, project all 2D original points in the triangular area to the 2D virtual image.

The projecting the all 2D original points in the triangular region by using the projection parameters corresponding to the triangular regions may include: projecting the 2D original points in the triangular region to the 2D virtual image by using the following formula:

Where p is the two-dimensional information of the 2D original point in the 2D original image, A and t are the projection parameters corresponding to the triangular region, and p' is the two-dimensional information of the 2D original point in the 2D virtual image. Since p, A, and t are known, p' can be calculated by the above formula, and p' is two-dimensional information of the 2D original point in the 2D virtual image, that is, the 2D original point is projected to the 2D virtual image.

After performing the above processing for all 2D original points in the triangular region, all 2D original points in the triangular region can be projected to the 2D virtual image. Furthermore, after performing the above processing for each of the triangular regions, all 2D original points of all the triangular regions can be projected to the 2D virtual image. Through the above processing, all 2D original points in the 2D original image can be projected to the 2D virtual image.

Based on the same concept as the above method, as shown in FIG. 8, an embodiment of the present invention further provides an image processing apparatus 80, which includes a memory 801 and a processor 802 (such as one or more processors).

The memory is configured to store program code, and the processor is configured to invoke the program code, when the program code is executed, to perform the following operations: acquiring a plurality of 2D original images including a target object Obtaining a first type of point cloud of the 3D scene of the target object according to the plurality of 2D original images; performing a folding process on the first type of point cloud by using a deformation function to obtain a second type of point cloud; The second type of point cloud is projected onto the image plane to obtain a 2D virtual image with a folding effect.

Preferably, the processor is configured to: when planning a plurality of 2D original images including the target object, plan a flight trajectory of the mobile platform according to the location of the target object, control the mobile platform to fly according to the flight trajectory, and collect during the flight. Contains multiple 2D raw images of the target object.

Preferably, the processor is configured to: when the first type of point cloud of the 3D scene of the target object is acquired according to the plurality of 2D original images, process the plurality of 2D original images by using an image processing algorithm, Obtaining a first type of point cloud of the 3D scene of the target object; the first type of point cloud includes a plurality of feature points having three-dimensional information.

Preferably, the processor is configured to perform folding processing on the first type of point cloud by using a deformation function, and when the second type of point cloud is obtained, the processor is specifically configured to: select N deformation points on the curve corresponding to the deformation function, and N is greater than a positive integer equal to 1; for each feature point in the first type of point cloud, determining at least one deformation point corresponding to the feature point; and folding the feature point by using the determined slope of each deformation point to obtain a folded Feature points; the folded feature points corresponding to each feature point in the first type of point cloud are determined as the second type of point cloud.

Preferably, when the processor selects N deformation points on the curve corresponding to the deformation function, the processor specifically selects: in the abscissa direction of the curve corresponding to the deformation function, selects a deformation point every first distance; or, corresponds to the deformation function In the ordinate direction of the curve, a deformation point is selected every second distance.

Preferably, when determining the at least one deformation point corresponding to the feature point, the processor is specifically configured to: select an abscissa value of the feature point based on an abscissa value of each of the N deformation points The deformation point of the value is taken as the deformation point corresponding to the feature point.

Preferably, the processor performs folding processing on the feature point by using the determined slope of each deformation point to obtain a folded feature point, and is specifically used to: if the feature point corresponds to M deformation points, follow the first The order of the deformation point to the Mth deformation point is performed by using the slope corresponding to the M deformation points, and the feature points are subjected to M folding processing to obtain a folded feature point; M is a positive integer greater than or equal to 1, and M is smaller than Equal to N.

Preferably, the processor performs folding processing on the feature point by using the determined slope of each deformation point, and obtains the post-folded feature point, and is specifically used to: if the feature point corresponds to M deformation points, use the ith The slope corresponding to the deformation point is subjected to the ith folding process to obtain the i-th post-folding feature point; wherein the value of i is 1, 2, ..., M, and the M is greater than or equal to A positive integer of 1, M is less than or equal to N.

Preferably, the processor performs the i-th folding process on the feature point by using the slope corresponding to the i-th deformation point, and obtains the i-th post-folding feature point, and is specifically used to: use the i-th deformation point correspondingly Obtaining the rotation parameter of the coordinate system, obtaining the displacement parameter of the coordinate system by using the abscissa value of the i-th deformation point, and performing the ith folding process on the feature point by using the rotation parameter and the displacement parameter to obtain the i-th Subfolded feature points after the fold.

Preferably, the processor is configured to acquire the first clip of the i-th deformation point and the abscissa by using the slope corresponding to the i-th deformation point when acquiring the rotation parameter of the coordinate system by using the slope corresponding to the i-th deformation point. Obtaining a target angle corresponding to the i-th deformation point by using the first angle and the second angle; wherein the second angle is a clip between the i-1th deformation point and the abscissa An angle is obtained by using a target angle corresponding to the i-th deformation point.

Preferably, the processor is configured to: when using the abscissa value of the i-th deformation point to obtain the displacement parameter of the coordinate system, construct the displacement matrix by using the abscissa value of the i-th deformation point; and determine the displacement matrix as The displacement parameter of the coordinate system.

Preferably, the processor performs the ith folding process on the feature point by using the rotation parameter and the displacement parameter to obtain the i-th post-folding feature point, and is specifically used to: use the rotation parameter and the The displacement parameter is subjected to the i-th folding process of the i-th post-folding feature point corresponding to the feature point, and the i-th post-folding feature point corresponding to the feature point is obtained.

Preferably, the processor is configured to: acquire the location information and the posture information corresponding to the mobile platform when the second type of point cloud is projected to the image plane to obtain the 2D virtual image with the folding effect; according to the mobile platform Corresponding position information and posture information, each feature point in the second type of point cloud is projected onto the image plane of the mobile platform to obtain a 2D virtual image having a folding effect.

Preferably, the processor is configured to: acquire the shooting position and the shooting posture corresponding to the mobile platform according to the plurality of 2D original images when acquiring the position information and the posture information corresponding to the mobile platform; or, according to the multiple 2D The original image acquires a shooting position and a shooting attitude corresponding to the mobile platform, and acquires a virtual position and a virtual posture according to the shooting position and the shooting posture.

Preferably, the processor projects each feature point in the second type of point cloud to the image plane of the mobile platform according to the location information and the posture information corresponding to the mobile platform, to obtain a 2D virtual image with a folding effect. The image is specifically used for: projecting each feature point in the second type of point cloud to the image plane by using the internal parameter corresponding to the mobile platform, the position information, and the posture information, and obtaining 2D corresponding to each feature point. a pixel point; a 2D pixel corresponding to each feature point in the second type of point cloud is composed of the 2D virtual image.

Preferably, after the projector projects the second type of point cloud onto the image plane to obtain a 2D virtual image with a folding effect, the processor is further configured to: determine each feature in the second type of point cloud from the 2D original image. Pointing a corresponding 2D feature point; dividing the 2D original image into a plurality of triangular regions by using the 2D feature point; wherein, for each triangular region, including 3 2D feature points, a plurality of 2D original points; The three 2D feature points acquire projection parameters corresponding to the triangular regions; and use the projection parameters to perform projection processing on all 2D original points in the triangular region.

Preferably, when the processor divides the 2D original image into a plurality of triangular regions by using the 2D feature points, the processor is specifically configured to: connect the 2D feature points in the 2D original image into multiple triangles by using a triangulation algorithm. region.

When the processor acquires the projection parameters corresponding to the triangular region by using the three 2D feature points, the processor specifically uses: two-dimensional information in the 2D original image using the three 2D feature points, and the three 2D feature points. The two-dimensional information in the 2D virtual image acquires projection parameters corresponding to the triangular region.

Based on the same concept as the above method, the embodiment of the present invention further provides a computer readable storage medium, where the computer readable storage medium stores computer instructions, and when the computer instructions are executed, the image processing method is implemented. .

The system, apparatus, module or unit set forth in the above embodiments may be implemented by a computer chip or an entity, or by a product having a certain function. A typical implementation device is a computer, and the specific form of the computer may be a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email transceiver, and a game control. A combination of a tablet, a tablet, a wearable device, or any of these devices.

For the convenience of description, the above devices are described separately by function into various units. Of course, the functions of the various units may be implemented in one or more software and/or hardware in the practice of the invention.

Those skilled in the art will appreciate that embodiments of the invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or a combination of software and hardware. Moreover, embodiments of the invention may take the form of a computer program product embodied on one or more computer usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) including computer usable program code.

The present invention has been described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (system), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flowchart illustrations and/or FIG. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing device to produce a machine for the execution of instructions for execution by a processor of a computer or other programmable data processing device. Means for implementing the functions specified in one or more of the flow or in a block or blocks of the flow chart.

Moreover, these computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture comprising the instruction device. The instruction means implements the functions specified in one or more blocks of the flowchart or in a flow or block diagram of the flowchart.

These computer program instructions can also be loaded onto a computer or other programmable data processing device such that a series of operational steps are performed on a computer or other programmable device to produce computer-implemented processing for execution on a computer or other programmable device. The steps are provided to implement the functions specified in one or more blocks of the flowchart or in a block or blocks of the flowchart.

The above is only the embodiments of the present invention and is not intended to limit the present invention. It will be apparent to those skilled in the art that various modifications and changes can be made in the present invention. Any modifications, equivalents, and improvements made within the spirit and scope of the invention are intended to be included within the scope of the appended claims.

Claims (39)

1. An image processing method, the method comprising:

   Obtaining a plurality of 2D original images containing the target object;

   Acquiring a first type of point cloud of the 3D scene of the target object according to the plurality of 2D original images;

   Folding the first type of point cloud by using a deformation function to obtain a second type of point cloud;

   Projecting the second type of point cloud onto the image plane yields a 2D virtual image with a folding effect.
2. The method of claim 1 wherein

   The acquiring a plurality of 2D original images including the target object includes:

   The flight path of the mobile platform is planned according to the position of the target object, and the mobile platform is controlled to fly according to the flight path, and a plurality of 2D original images including the target object are collected during the flight.
3. The method according to claim 1, wherein the acquiring a first type of point cloud of the 3D scene of the target object according to the plurality of 2D original images comprises:

   The plurality of 2D original images are processed by an image processing algorithm to obtain a first type of point cloud of the 3D scene of the target object; and the first type of point cloud includes a plurality of feature points having three-dimensional information.
4. The method of claim 3 wherein:

   The image processing algorithm includes: a structural reconstruction SfM algorithm;

   The SfM algorithm includes: a sparse SfM algorithm; or a dense SfM algorithm.
5. The method of claim 1 wherein

   The first type of point cloud is folded by using a deformation function to obtain a second type of point cloud, including:

   N deformation points are selected on the curve corresponding to the deformation function, and N is a positive integer greater than or equal to 1;

   Determining at least one deformation point corresponding to the feature point for each feature point in the first type of point cloud; folding the feature point by using the determined slope of each deformation point to obtain the folded feature point;

   The folded feature points corresponding to each feature point in the first type of point cloud are determined as the second type point cloud.
6. The method according to claim 1 or 5, wherein the deformation function comprises:

   Exponential function; or, a parabolic function; or, an Archimedes spiral function.
7. The method of claim 5 wherein:

   The N deformation points are selected on the curve corresponding to the deformation function, including:

   In the abscissa direction of the curve corresponding to the deformation function, one deformation point is selected every first distance; or, in the longitudinal direction of the curve corresponding to the deformation function, a deformation point is selected every second distance.
8. The method of claim 5 wherein:

   Determining at least one deformation point corresponding to the feature point, including:

   Based on the abscissa value of each of the N deformation points, a deformation point whose abscissa value is smaller than the abscissa value of the feature point is selected as the deformation point corresponding to the feature point.
9. The method according to claim 5, wherein the folding of the feature points by using the determined slope of each deformation point to obtain the folded feature points comprises:

   If the feature points correspond to the M deformation points, the feature points are M-folded by the slope corresponding to the M deformation points according to the order of the first deformation point to the Mth deformation point, and the folding is obtained. Post-feature point; M is a positive integer greater than or equal to 1, and M is less than or equal to N.
10. The method according to claim 5, wherein the folding of the feature points by using the determined slope of each deformation point to obtain the folded feature points comprises:

    If the feature point corresponds to the M deformation points, the i-th folding process is performed on the feature point by using the slope corresponding to the i-th deformation point to obtain the i-th post-folding feature point; wherein the value of the i is In order of 1, 2, ..., M, the M is a positive integer greater than or equal to 1, and M is less than or equal to N.
11. The method according to claim 10, wherein the i-th folding process is performed on the feature point by using a slope corresponding to the i-th deformation point, and the i-th post-folding feature point is obtained, including:

    Obtaining a rotation parameter of the coordinate system by using a slope corresponding to the i-th deformation point, acquiring a displacement parameter of the coordinate system by using an abscissa value of the i-th deformation point, and performing the feature point on the feature point by using the rotation parameter and the displacement parameter The i-folding process is performed to obtain the i-th post-folding feature point.
12. The method of claim 11 wherein

    The acquiring the rotation parameter of the coordinate system by using the slope corresponding to the i-th deformation point, including:

    Obtaining a first angle of the i-th deformation point and the abscissa by using a slope corresponding to the i-th deformation point;

    Obtaining, by the first angle and the second angle, a target angle corresponding to the i-th deformation point; wherein the second angle is an angle between the i-1th deformation point and the abscissa;

    The rotation parameter of the coordinate system is acquired by the target angle corresponding to the i-th deformation point.
13. The method of claim 11 wherein

    And acquiring the displacement parameter of the coordinate system by using the abscissa value of the i-th deformation point, including:

    Constructing a displacement matrix by using an abscissa value of the i-th deformation point;

    The displacement matrix is determined as a displacement parameter of the coordinate system.
14. The method of claim 11 wherein

    And performing the ith folding process on the feature point by using the rotation parameter and the displacement parameter to obtain the i-th post-folding feature point, including:

    Using the rotation parameter and the displacement parameter, the i-th folding feature point corresponding to the feature point is subjected to an ith folding process to obtain an i-th post-folding feature point corresponding to the feature point.
15. The method according to claim 1, wherein the projecting the second type of point cloud to the image plane to obtain a 2D virtual image having a folding effect comprises:

    Obtaining location information and posture information corresponding to the mobile platform;

    And projecting each feature point in the second type of point cloud to an image plane of the mobile platform according to position information and posture information corresponding to the mobile platform, to obtain a 2D virtual image having a folding effect.
16. The method of claim 15 wherein:

    And obtaining the location information and the posture information corresponding to the mobile platform, including:

    Obtaining a shooting position and a shooting attitude corresponding to the mobile platform according to the plurality of 2D original images;

    Alternatively, the shooting position and the shooting attitude corresponding to the mobile platform are acquired according to the plurality of 2D original images, and the virtual position and the virtual posture are acquired according to the shooting position and the shooting attitude.
17. The method according to claim 15, wherein each feature point in the second type of point cloud is projected to an image plane of the mobile platform according to position information and posture information corresponding to the mobile platform. Get a 2D virtual image with a folding effect, including:

    Projecting each feature point in the second type of point cloud to the image plane by using the internal parameter corresponding to the mobile platform, the position information, and the posture information, to obtain a 2D pixel point corresponding to each feature point;

    The 2D pixel points corresponding to each feature point in the second type of point cloud are composed of the 2D virtual image.
18. The method according to claim 1, wherein after the projecting the second type of point cloud to the image plane to obtain a 2D virtual image having a folding effect, the method further comprises:

    Determining a 2D feature point corresponding to each feature point in the second type of point cloud from the 2D original image;

    The 2D original image is divided into a plurality of triangular regions by using the 2D feature points; wherein, for each triangular region, three 2D feature points and a plurality of 2D original points are included;

    Obtaining, by using the three 2D feature points, a projection parameter corresponding to the triangular region;

    Projecting processing of all 2D original points in the triangular region using the projection parameters.
19. The method of claim 18, wherein

    Dividing the 2D original image into a plurality of triangular regions by using the 2D feature points includes:

    The triangulation algorithm is used to join 2D feature points in the 2D original image into a plurality of triangular regions.
20. The method of claim 18, wherein

    Obtaining the projection parameters corresponding to the triangle region by using the three 2D feature points, including:

    The two-dimensional information of the three 2D feature points in the 2D original image and the two-dimensional information of the three 2D feature points in the 2D virtual image are used to acquire projection parameters corresponding to the triangular region.
21. An image processing device, comprising: a memory and a processor;

    The memory is configured to store program code;

    The processor is configured to invoke the program code, when the program code is executed, to perform the following operations: acquiring a plurality of 2D original images including a target object;

    Acquiring a first type of point cloud of the 3D scene of the target object according to the plurality of 2D original images;

    Folding the first type of point cloud by using a deformation function to obtain a second type of point cloud;

    Projecting the second type of point cloud onto the image plane yields a 2D virtual image with a folding effect.
22. The device according to claim 21, wherein

    The processor is specifically configured to: when planning a plurality of 2D original images including the target object, plan a flight trajectory of the mobile platform according to the location of the target object, control the mobile platform to fly according to the flight trajectory, and collect the target object during the flight. Multiple 2D original images.
23. The device according to claim 21, wherein the processor is specifically configured to: through an image processing algorithm, when acquiring a first type of point cloud of a 3D scene of the target object according to the plurality of 2D original images The plurality of 2D original images are processed to obtain a first type of point cloud of the 3D scene of the target object; the first type of point cloud includes a plurality of feature points having three-dimensional information.
24. The device according to claim 21, wherein the processor performs a folding process on the first type of point cloud by using a deformation function to obtain a second type of point cloud, specifically for:

    N deformation points are selected on the curve corresponding to the deformation function, and N is a positive integer greater than or equal to 1;

    Determining at least one deformation point corresponding to the feature point for each feature point in the first type of point cloud; folding the feature point by using the determined slope of each deformation point to obtain the folded feature point;

    The folded feature points corresponding to each feature point in the first type of point cloud are determined as the second type point cloud.
25. The device according to claim 24, wherein

    When the processor selects N deformation points on the curve corresponding to the deformation function, it is specifically used to: select a deformation point every first distance in the abscissa direction of the curve corresponding to the deformation function; or, in the longitudinal direction of the curve corresponding to the deformation function In the coordinate direction, select a deformation point every second distance.
26. The device according to claim 24, wherein

    The determining, when determining the at least one deformation point corresponding to the feature point, is specifically: selecting, based on an abscissa value of each of the N deformation points, a deformation whose abscissa value is smaller than an abscissa value of the feature point Point as the deformation point corresponding to the feature point.
27. The device according to claim 24, wherein the processor performs folding processing on the feature point by using the determined slope of each deformation point to obtain a folded feature point, which is specifically used if the feature point corresponds to M deformation points, according to the order of the first deformation point to the Mth deformation point, using the slope corresponding to the M deformation points, the feature points are subjected to M folding processing to obtain the folded feature points; For a positive integer greater than or equal to 1, M is less than or equal to N.
28. The device according to claim 24, wherein the processor performs folding processing on the feature point by using the determined slope of each deformation point to obtain a folded feature point, which is specifically used if the feature point corresponds to For the M deformation points, the i-th folding process is performed on the feature points by using the slope corresponding to the i-th deformation point, and the i-th post-folding feature points are obtained; wherein the values of the i are sequentially 1, 2 , M, the M is a positive integer greater than or equal to 1, and M is less than or equal to N.
29. The device according to claim 28, wherein the processor performs the ith folding process on the feature point by using the slope corresponding to the i-th deformation point to obtain the i-th folded feature point. And a method for acquiring a rotation parameter of a coordinate system by using a slope corresponding to the i-th deformation point, acquiring a displacement parameter of the coordinate system by using an abscissa value of the i-th deformation point, and using the rotation parameter and the displacement parameter to the feature point The i-th folding process is performed to obtain the i-th post-folding feature point.
30. The device according to claim 29, characterized in that

    When the processor obtains the rotation parameter of the coordinate system by using the slope corresponding to the i-th deformation point, the processor is specifically configured to: obtain the first angle of the i-th deformation point and the abscissa by using the slope corresponding to the i-th deformation point; The first angle and the second angle acquire a target angle corresponding to the i-th deformation point; wherein the second angle is an angle between the i-1th deformation point and the abscissa; The target angle corresponding to the i-th deformation point acquires a rotation parameter of the coordinate system.
31. The device according to claim 29, wherein the processor is configured to use the abscissa value of the i-th deformation point when acquiring the displacement parameter of the coordinate system by using the abscissa value of the i-th deformation point. a displacement matrix; the displacement matrix is determined as a displacement parameter of the coordinate system.
32. The device according to claim 29, characterized in that

    The processor performs the ith folding process on the feature point by using the rotation parameter and the displacement parameter to obtain the i-th post-folding feature point, and is specifically used to: use the rotation parameter and the displacement parameter And performing the i-th folding process on the i-th post-folding feature point corresponding to the feature point, and obtaining the i-th post-folding feature point corresponding to the feature point.
33. The device according to claim 21, wherein

    When the processor projects the second type of point cloud to the image plane to obtain a 2D virtual image with a folding effect, the processor is specifically configured to: acquire location information and posture information corresponding to the mobile platform; and according to the location corresponding to the mobile platform Information and attitude information, projecting each feature point in the second type of point cloud to the image plane of the mobile platform to obtain a 2D virtual image with a folding effect.
34. The device according to claim 33, wherein

    When acquiring the location information and the posture information corresponding to the mobile platform, the processor is configured to: acquire a shooting position and a shooting posture corresponding to the mobile platform according to the plurality of 2D original images; or acquire according to the multiple 2D original images. The shooting position and the shooting attitude corresponding to the mobile platform are acquired, and the virtual position and the virtual posture are acquired according to the shooting position and the shooting attitude.
35. The device according to claim 33, wherein the processor projects each feature point in the second type of point cloud to the mobile platform according to location information and posture information corresponding to the mobile platform. The image plane is used to obtain a 2D virtual image with a folding effect.

    Projecting each feature point in the second type of point cloud to the image plane by using the internal parameter corresponding to the mobile platform, the position information, and the posture information, to obtain a 2D pixel point corresponding to each feature point;

    The 2D pixel points corresponding to each feature point in the second type of point cloud are composed of the 2D virtual image.
36. The device according to claim 21, wherein the processor is further configured to: after projecting the second type of point cloud to the image plane to obtain a 2D virtual image having a folding effect:

    Determining a 2D feature point corresponding to each feature point in the second type of point cloud from the 2D original image;

    The 2D original image is divided into a plurality of triangular regions by using the 2D feature points; wherein, for each triangular region, three 2D feature points and a plurality of 2D original points are included;

    Obtaining, by using the three 2D feature points, a projection parameter corresponding to the triangular region;

    Projecting processing of all 2D original points in the triangular region using the projection parameters.
37. The device according to claim 36, wherein the processor is configured to: when the 2D original image is divided into a plurality of triangular regions by using the 2D feature points, use a triangulation algorithm to convert the 2D original image The 2D feature points in the connection are connected into a plurality of triangular regions.
38. The device according to claim 36, wherein the processor is configured to use the 3 2D feature points in 2D when acquiring the projection parameters corresponding to the triangular regions by using the 3 2D feature points. The two-dimensional information in the original image, the two-dimensional information of the three 2D feature points in the 2D virtual image, and the projection parameters corresponding to the triangular region are acquired.
39. A computer readable storage medium, wherein the computer readable storage medium stores computer instructions that, when executed, implement the image processing method of any one of claims 1-20.

Images