WO2020198963A1

WO2020198963A1 - Data processing method and apparatus related to photographing device, and image processing device

Info

Publication number: WO2020198963A1
Application number: PCT/CN2019/080500
Authority: WO
Inventors: 黄胜; 薛唐立; 梁家斌; 马东东
Original assignee: 深圳市大疆创新科技有限公司
Priority date: 2019-03-29
Filing date: 2019-03-29
Publication date: 2020-10-08
Also published as: CN111247389B; CN111247389A

Abstract

A data processing method related to a photographing device, the method being applied to an image processing device. The image processing device may obtain attitude data of the photographing device. The photographing device is disposed on a mobile platform, and the motion state of the mobile platform is a linear motion state. The method comprises: obtaining first attitude data when the photographing device collects a first image (S401); according to the first attitude data, determining a first rotation relationship between a device coordinate system of the photographing device and a reference coordinate system when the first image is collected (S402); and determining location information of a pixel point in the first image in the reference coordinate system according to the first rotation relationship (S403). Embodiments of the present invention further provide a data processing apparatus related to the photographing device and the image processing device. In the linear motion state, accuracy in converting a pixel point to three-dimensional coordinates may be ensured, thereby better implementing tasks such as environment monitoring and drawing.

Description

Data processing method, device and image processing equipment for photographing equipment

Technical field

The present invention relates to the field of image processing technology, in particular to a data processing method, device and image processing equipment related to photographing equipment.

Background technique

With the development of technology, mobile platforms such as unmanned aerial vehicles have been greatly utilized. Cameras can be mounted on the mobile platform through PTZ and other equipment, so that when the mobile platform is moving, the camera will take pictures of the target environment. Based on the image, and perform three-dimensional reconstruction based on the image to realize the functions of environmental mapping and mapping.

At present, the steps of 3D reconstruction using images taken by shooting equipment include: Based on the image, using Structure from Motion (SFM) technology to recover the correct spatial posture of each shooting device, that is, the shooting device is in a certain reference coordinate system The posture of the camera includes the position and angle information of the shooting. The angle information can be, for example, the pitch angle, roll angle, and yaw angle. Then, starting from the posture of the shooting device in the reference coordinate system, the image on the The coordinates of the three-dimensional points in the actual space corresponding to the pixel points complete the three-dimensional reconstruction of the environmental space, and then complete tasks such as instantaneous localization and mapping (Simultaneous Localization and Mapping, SLAM).

The reference coordinate system related to surveying and mapping is generally the Earth-centered (Earth-fixed, ECEF) or the Northeast Sky (ENU) coordinate system established by a known point on the ground or the North East (NED) ) Geographical coordinate systems such as coordinate systems. The current mainstream 3D modeling software can correctly restore the correct pose of the shooting device in a reference coordinate system such as a geodetic coordinate system or a geographic coordinate system based on the images in multiple directions, so as to determine the device coordinate system and the reference coordinate system The relationship between.

Research has found that when the mobile platform is moving in a straight line or approximately a straight line, such as the case of a UAV flying with a single belt, the final device coordinate system and the reference coordinate system are obtained based on the images taken during the linear motion and the SFM. The relationship between them is not accurate enough, so that the coordinate systems cannot be accurately aligned, and the coordinate position conversion of the pixel points has a large error.

Summary of the invention

The embodiments of the present invention provide a data processing method, device, and image processing equipment for shooting equipment, which can more accurately determine the rotation between the equipment coordinate system of the shooting equipment and the reference coordinate system when the mobile platform is in a linear motion state. Relationship, and more accurately complete the coordinate system position conversion of pixels.

On the one hand, an embodiment of the present invention provides a data processing method related to a photographing device, characterized in that the method is applied to an image processing device, and the image processing device can obtain posture data of the photographing device, and the photographing device is set at On a mobile platform, and the motion state of the mobile platform is a linear motion state, the method includes:

Acquiring first posture data when the photographing device collects the first image;

Determine, according to the first posture data, a first rotation relationship between the device coordinate system of the photographing device and a reference coordinate system when the first image is collected;

According to the first rotation relationship, the position information of the pixel on the first image in the reference coordinate system is determined.

On the other hand, an embodiment of the present invention also provides a data processing device related to a photographing device. The device is applied to an image processing device that can obtain posture data of the photographing device, and the photographing device is set on a mobile platform. And the motion state of the mobile platform is a linear motion state, and the device includes:

An acquiring module, configured to acquire first posture data when the photographing device acquires the first image;

A determining module, configured to determine, according to the first posture data, a first rotation relationship between the device coordinate system of the shooting device and a reference coordinate system when the first image is collected;

The processing module is configured to determine the position information of the pixel on the first image in the reference coordinate system according to the first rotation relationship.

In yet another aspect, an embodiment of the present invention also provides an image processing device that can obtain posture data of the shooting device, the shooting device is set on a mobile platform, and the motion state of the mobile platform is linear motion State, the image processing equipment includes: a communication interface unit and a processing unit;

The communication interface unit is used to communicate with an external device and obtain data of the external device;

The processing unit is configured to obtain, through the communication interface unit, the first posture data of the shooting device when the first image is collected; according to the first posture data, determine the status of the shooting device when the first image is collected A first rotation relationship between the device coordinate system and a reference coordinate system; according to the first rotation relationship, the position information of a pixel on the first image in the reference coordinate system is determined.

In the embodiment of the present invention, when the mobile platform is in a linear motion state, the rotation relationship between the device coordinate system where the photographing device is located and the reference coordinate system is determined, and the posture data of the photographing device when the image is collected is referred to. , The rotation relationship between the equipment coordinate system and the reference coordinate system can be determined more accurately, and the accuracy of the conversion from pixel points to three-dimensional coordinates in the state of linear motion is better ensured, so as to better realize environmental monitoring, Tasks such as drawing.

Description of the drawings

In order to explain the embodiments of the present invention or the technical solutions in the prior art more clearly, the following will briefly introduce the drawings needed in the embodiments. Obviously, the drawings in the following description are only some of the present invention. Embodiments, for those of ordinary skill in the art, without creative work, other drawings can be obtained from these drawings.

FIG. 1 is a schematic diagram of a task scene of a drone equipped with a shooting device according to an embodiment of the present invention;

Figure 2 is a schematic diagram of the North-East coordinate system of an embodiment of the present invention;

FIG. 3 is a schematic diagram of the relationship between an image coordinate system and a device coordinate system according to an embodiment of the present invention;

4 is a schematic flowchart of a data processing method related to a photographing device according to an embodiment of the present invention;

FIG. 5 is a schematic diagram of a process for determining a rotation relationship according to an embodiment of the present invention;

FIG. 6 is a schematic diagram of a multi-belt movement and a single belt movement in a straight state of a mobile platform according to an embodiment of the present invention;

FIG. 7 is a schematic flowchart of an optimized rotation relationship obtained according to an embodiment of the present invention;

FIG. 8 is a schematic diagram of a flow of coordinate position conversion according to an embodiment of the present invention;

FIG. 9 is a schematic diagram of a flow of performing rotation relationship optimization and coordinate position conversion according to an embodiment of the present invention;

FIG. 10 is a schematic structural diagram of a data processing device for photographing equipment according to an embodiment of the present invention;

Fig. 11 is a schematic structural diagram of an image processing device according to an embodiment of the present invention.

detailed description

In the embodiment of the present invention, the shooting device used to shoot images can be directly mounted on the mobile platform, or can be mounted on the mobile platform through a pan-tilt. During the movement of the mobile platform, the shooting device can capture multiple frames of images of the current environment, and for each frame of image, it can read the posture data when the image was taken. These posture data may include The pitch angle, roll angle and yaw Yaw angle. When it is determined that the mobile platform is currently in linear motion, the corresponding image processing device can be triggered to use the image posture data as the intermediate data to convert between the device coordinate system, the pan/tilt coordinate system, and the reference coordinate system to convert the image captured by the camera The above points are converted to the reference coordinate system, and the three-dimensional points of these points in the reference coordinate system are determined, so as to realize tasks such as SLAM environmental drawing.

Please refer to FIG. 1, which shows a scene diagram in which an unmanned aerial vehicle is equipped with a shooting device to acquire images and then perform corresponding image processing to achieve tasks such as SLAM. The scene diagram includes a drone 101 and a photographing device 102 provided at the bottom of the drone 101. In one embodiment, the drone 101 may be provided with a pan-tilt, and the photographing device 102 may be provided on the pan-tilt. The drone may be a rotary-wing drone as shown in FIG. 1, such as a quadrotor, a hexarotor, or an octorotor, etc. In some embodiments, it may also be a fixed-wing aircraft. The pan/tilt head mainly refers to a three-axis pan/tilt head, which can rotate in the pitch direction, roll direction, and yaw direction, so as to drive the camera to shoot images in different directions. In addition to drones, in other embodiments, the mobile platform can also be an unmanned car, an intelligent mobile robot, etc. running on land. It can also be an unmanned car, a pan/tilt on an intelligent robot, and a photographing device. Take pictures of different orientations as needed. Based on the images taken by the camera on the drone and the posture data when the camera is taking the images, the image processing device 100 on the ground can perform a series of data processing in the embodiment of the present invention. In FIG. 1, the drone 101 and the image processing device 100 are designed separately. In other embodiments, the image processing device 100 can also be mounted on the drone to receive and read the image taken by the camera 102. Attitude data; or the image processing device 100 itself, as a component of the UAV, can be directly or indirectly connected to the camera 102 and the pan/tilt to obtain image and attitude data.

Please refer to FIGS. 2 to 4, which are schematic diagrams of the coordinate system involved in the embodiment of the present invention. Among them, Figure 2 shows the North-East coordinate system. The North-East coordinate system belongs to the geographic coordinate system and is one of the reference coordinate systems in the embodiment of the present invention. The positive direction of the X-axis of the North-East coordinate system points to the geographic The direction north is North, the positive direction of the Y axis points to the geographic direction East, and the positive direction of the Z axis points to Down.

Figure 3 shows the relationship between the device coordinate system and the image coordinate system, where the XY plane in the device coordinate system is parallel to the image plane, the Z axis of the device coordinate system is the main axis of the camera, and the origin O is the projection center (optical center). The coordinate system is a three-dimensional coordinate system. At the same time, in Figure 3, the origin O ₁ of the image coordinate system where the image plane is located is the intersection point of the camera's principal axis and the image plane, also called the principal point. The distance between point O and point O1 is the focal length f, the imaging plane coordinate The system is a two-dimensional coordinate. It can be understood that FIG. 3 is only for illustration. For example, in some embodiments, due to the manufacturing of the camera, the principal point of the image is usually not in the exact center of the imaging plane. It can be seen from Figure 3 that the point (x, y) on the image coordinate system and the point Q on the device coordinate system are (X, Y, Z).

Based on the above-mentioned schematic and description about the coordinate system, please refer to FIG. 4 again, which is a schematic flowchart of a data processing method for a photographing device according to an embodiment of the present invention. It is executed by an image processing device, which may be, for example, a smart mobile terminal, tablet computer, personal computer, notebook computer, etc. The image processing device can obtain the posture data of the shooting device, and the shooting device is set on a mobile platform, and the image processing device can obtain the posture data of the shooting device through the mobile platform. In one embodiment, if the motion state of the mobile platform is a linear motion state (that is, the multiple displacement points of the mobile platform are basically on a straight line), the image processing device acquires the first image when the shooting device collects the first image in S401. A posture data. During the movement of the mobile platform, the photographing device may photograph multiple frames of images to form an image set, and the first image may be any one of the multiple frames of images.

When the shooting device shoots the first image and other images, it can record the posture data of the corresponding shooting device. These posture data are obtained through the sensing data processing of sensors such as gyroscopes, and can include the horizontal position of the shooting device when the image is taken. Roll angle, pitch Pitch angle and Yaw angle. In one embodiment, if the camera is mounted on the mobile platform via a pan-tilt, the pan-tilt angle information determined based on the sensor data of the sensor set on the pan-tilt can be used as First posture data, and record first posture data for the first image. PTZ angle information includes: Roll angle, Pitch angle, and Yaw angle. If the PTZ is directly fixed on a mobile platform such as a drone, the angle data of the mobile platform determined based on the sensing data of the sensors on the mobile platform can be used as the first attitude data, and the first attitude data can be recorded for the first image , The first posture data includes: Roll angle, Pitch angle, and Yaw angle of the mobile platform. The conversion process from the PTZ coordinate system where the sensor on the pan/tilt is located to the reference coordinate system (geographic coordinate system or geodetic coordinate system, etc.), and the conversion from the coordinate system where the sensor of the mobile platform is located to the reference coordinate system (geographic coordinate system) Or the geodetic coordinate system, etc.) have the same conversion process. On this basis, the following uses the pan-tilt coordinate system as an example to illustrate the conversion process between the device coordinate system of the shooting device and the reference coordinate system.

The pan/tilt angle information can be recorded in the extension field of the corresponding environmental image, so that the image processing device can directly obtain it when needed. The photographing device can automatically acquire the pan/tilt angle information from the pan/tilt, or the pan/tilt actively sends the pan/tilt angle information to the photographing device, so that the photographing device can record the corresponding pan/tilt angle information for the image after the image is captured. In the embodiment of the present invention, the first posture data includes: the angle information of the pan/tilt corresponding to the pan/tilt when the first image is collected. The pan/tilt angle information can be read directly from the extended field of the first image.

The embodiment of the present invention specifically triggers the execution of S401 when the motion state of the mobile platform is the linear motion state. In the embodiment of the present invention, there are multiple ways to detect whether the motion state of the mobile platform is the linear motion state. In one embodiment, it can be determined whether the movement route planned by the mobile platform is a straight line. For example, when the drone is flying based on the set navigation trajectory, it detects whether the navigation trajectory in the current period of time is a straight line, and if so, it is determined to move The platform is in linear motion. For another example, when the user plans to manually control the mobile platform to move along a straight line or approximately a straight line, an instruction can be manually triggered and sent to the image processing device. The instruction is used to indicate that the mobile platform is in a linear motion state. That is to say, the state of motion of the mobile platform is a linear motion state, which mainly means that the mobile platform will have a linear motion state. It does not require that the mobile platform must move along a straight line. In the actual processing process, the mobile platform is in a straight line or Approximately linear motion, or even if it is a curved motion but receives an instruction from the user to indicate that the mobile platform is in a linear motion state, it can be considered that the motion state of the mobile platform is a linear motion state.

The pan/tilt angle information of the first image included in the first attitude data can be considered to be the clockwise rotation angle of the pan/tilt relative to the X-axis, Y-axis, and Z-axis of the northeast coordinate system. The rotation of these three axes can be Obtain the rotation relationship between the PTZ coordinate system and the northeast coordinate system NED, that is, the image processing device determines the device coordinate system of the shooting device when the first image is collected according to the first posture data in S402 The first rotational relationship with the reference coordinate system. The reference coordinate system includes a geographic coordinate system or a geodetic coordinate system. In the embodiment of the present invention, the North-East coordinate system shown in FIG. 2 is taken as an example for description. It is understandable that some of the currently known coordinate systems can determine the relative relationship between the coordinate systems through some technical means, such as rotation relationship, translation relationship, etc. Based on these relative relationships, the alignment and alignment of different coordinate systems can be achieved. For the conversion of position coordinates, in the embodiment of the present invention, when aligning the device coordinate system of the photographing device with the reference coordinate system, the first posture data, that is, the pan/tilt angle information of the pan/tilt is referred to. One of the specific implementation manners of S402 is shown in FIG. 5, which may specifically include the following steps.

In S501, the second rotation relationship between the pan-tilt coordinate system of the pan-tilt and the reference coordinate system is acquired according to the first posture data, and the device coordinate system of the photographing device to the pan-tilt coordinate is acquired The third rotational relationship between the lines.

Assuming that the angles corresponding to Roll, Pitch, and Yaw angles in the PTZ angle information are θ _r, θ _p, θ _y , then the rotation matrix from the PTZ coordinate system to the Northeast coordinate system (reference coordinate system) is the first The two rotation relations are:

The third rotation relationship can be determined based on the assembly relationship between the camera and the pan/tilt. The Roll angle, Pitch angle, and Yaw angle in the pan/tilt angle information correspond to the X axis, Y axis, and Z axis of the pan/tilt coordinate system, respectively. According to the orientation of each axis of the device coordinate system and the pan/tilt coordinate system, the rotation matrix from the device coordinate system to the pan/tilt coordinate system, that is, the third rotation relationship, is described in the following formula 2. In one embodiment, the third rotation relationship is directly configured according to the assembly relationship between the photographing device and the pan/tilt, and is directly read and used when determining the first rotation relationship.

According to the above formula 1 and formula 2, in S502 according to the second rotation relationship and the third rotation relationship, the first rotation relationship between the device coordinate system and the reference coordinate system when the first image is acquired is determined , That is, the rotation matrix from the equipment coordinate system to the NED coordinate system, that is, the first rotation relationship:

R _{camera_to_ned} = R _{gimbal_to_ned} *R _{camera_to_gimbal} formula 3;

If the PTZ angle information recorded in the image is completely accurate, each device coordinate system can be correctly aligned to the NED coordinate system. After obtaining the accurate rotation matrix R _{camera_to_ned} , the image processing device determines the position information of the pixels on the first image in the reference coordinate system according to the first rotation relationship in S403. For a certain two-dimensional point on the first image, based on the above-mentioned relationship between the image coordinate system and the device coordinate system, the two-dimensional point of the image can be converted from the image coordinate system of the first image to the device coordinate system , And then based on R _{camera_to_ned} combining the translation matrix t between the device coordinate system and the northeast coordinate system and the scaling relationship that is the scale s, and then transform from the device coordinate system to the northeast coordinate system again. Among them, the translation matrix t can be calculated based on the overlap between multiple frames of images, and the scale s represents the scaling relationship between the device coordinate system and the reference coordinate system. The device coordinate system needs to be enlarged to facilitate reference with the Northeast coordinate system, etc. The coordinate system is aligned. For example, after the two-dimensional point (x, y) on the first image is converted to the device coordinate system, it _{becomes the} three-dimensional point Q(X, Y, Z), s*R _{camera_to_ned} *Q+t can obtain the three-dimensional The three-dimensional point coordinates of point Q in the northeast coordinate system.

After the first rotation relationship of the first image is obtained, other images can also be obtained based on the first rotation relationship (especially the second image adjacent to the first image in the acquisition time, for example, the first image is taken immediately after the first image is taken). The position information from the point on the image to the reference coordinate system. Specifically, based on the image overlapping part of the second image and the first image and/or the pan/tilt angle information corresponding to the two images, the relative rotation relationship between the first image and the second image can be calculated, and then based on the first rotation Relationship and relative rotation relationship, the rotation relationship between the device coordinate system and the reference coordinate system of the second image can be obtained, and the second image can be obtained based on the rotation relationship between the device coordinate system and the reference coordinate system of the second image The position information of the pixel on the above reference coordinate system. In an embodiment, the translation relationship between the device coordinate system and the reference coordinate system of the second image may also be obtained based on the translation relationship corresponding to the first image and the relative translation relationship between the first image and the second image. The scaling relationship s of the image is the same as the scaling relationship of the first image.

In the embodiment of the present invention, the north-east coordinate system is taken as an example for description, and other types of reference coordinate systems perform the same processing. In other words, other types of reference coordinate systems can have a known conversion relationship with the northeast coordinate system. Based on the ability to determine R _{camera_to_ned} , it can be further based on other coordinate systems such as the northeast sky coordinate system (another A conversion relationship between a geographic coordinate system) and a north-east coordinate system to obtain a rotation matrix between the device coordinate system and these coordinate systems, and the rotation matrix is used to characterize the rotation relationship between the coordinate systems.

In one embodiment, if the motion state of the mobile platform is not a linear motion state, for example, when it is determined that the drone is moving in a multi-aircraft belt, the first image corresponding to the first image can be determined through the existing image-based Rotation relationship and related rotation relationship of other images. In one embodiment, as shown in Figure 6, the movement routes of the UAV's single and multiple belts are shown, and the black dots indicate the waypoints that are automatically planned when performing a certain task or the user manually clicks to set Connecting these waypoints constitutes the flight route of the drone.

Among them, in the case of multiple flight belts, the rotation relationship, translation relationship and scale corresponding to the image can be directly calculated based on the vision algorithm. Specifically, using multiple flight belts for SFM or SLAM, since the calculated visual coordinate center and the GPS center corresponding to the shooting device when each frame of image is taken are on multiple straight lines (refer to Figure 6), the following formula can be used to calculate A correct similarity transformation matrix is solved to obtain the rotation matrix R, the translation matrix t, and the scale s. In the following formulas, n is defined as frames with an image in a multi-flight movement during shooting, y _i refers to the photographing apparatus in the reference coordinate system of the actual center coordinates, x _i refers to the center of the imaging device coordinate system apparatus coordinate.

After the similar transformation matrix is obtained, the device coordinate system of the shooting device can be aligned to a reference coordinate system such as the North-East coordinate system using this similar transformation matrix. Due to the lack of a single degree of freedom, an exception occurs when solving the similar transformation matrix problem, because if a given rotation matrix R satisfies the condition, the new rotation is obtained after rotating around the straight line where the single air belt is located. The matrix also meets the condition, and each R corresponds to a t, so there is a solution without an array that satisfies the condition, that is, it is impossible to accurately calculate a correct similar transformation matrix theoretically. Therefore, in the case of a single belt movement as shown in FIG. 6, it is necessary to calculate the first rotation based on the above-mentioned method with the help of the posture data of the shooting device mentioned in the above-mentioned embodiment of the present invention, or the pan/tilt angle information. Relationship, and then correctly align the device coordinate system to a reference coordinate system such as the North-East coordinate system.

As mentioned above, R _{camera_to_ned} is calculated based on the accurate values of Roll angle, Pitch angle and Yaw angle. In some scenarios, the Roll and Pitch angles in the PTZ angle information are more accurate and can be used directly, but the Yaw angle may have a large deviation. At this time, when performing S403, the image processing equipment is specific The first rotation relationship may be optimized according to a preset optimization algorithm, so as to determine the position information of the pixel points on the first image in the reference coordinate system based on the optimized first rotation relationship.

The optimization algorithm may be some minimization algorithm. In one embodiment, the above S403 may include: performing optimization processing on the first rotation relationship according to a preset first algorithm; and according to the optimized first rotation Relationship, determining the position information of the pixel on the first image in the reference coordinate system. The idea of the first algorithm is to make the center coordinates corresponding to each image determined based on the optimized first rotation relationship be in the second coordinate in the reference coordinate system, and the camera device determined based on the sensor data is in the reference coordinate system The difference between the first coordinates below and the preset first minimization condition are satisfied, and the center coordinates refer to the center coordinates of the device coordinate system. As described above, the first attitude data includes pitch angle, roll angle, and yaw angle. Since the Roll angle and pitch angle in the first attitude data are considered accurate, the optimization of the first rotation relationship The processing includes optimizing the yaw angle.

In one embodiment, as shown in FIG. 7, the optimizing processing of the first rotation relationship according to the preset first algorithm includes: S701: acquiring each image in the first image set collected by the photographing device When a frame of image is collected, the center coordinates of the device coordinate system and the first coordinates in the reference coordinate system of the center coordinates determined using sensor data; S702: based on the center coordinates and the first coordinates, The first rotation relationship is optimized, so that the difference between the second coordinate in the reference coordinate system and the corresponding first coordinate of each of the center coordinates determined based on the optimized first rotation relationship is sum Meet the preset first minimization condition; wherein, the first image set includes the first image and at least one frame of second image. The first image set includes multiple frames of images including the first image, for example, three frames of images, five frames of images or even more images taken continuously within a period of time, except for the first image Other images can be understood as second images. The second image is an image adjacent to the first image in the acquisition time. For example, the second image refers to the second image that is captured immediately after the first image is captured. , Or the third image, etc. After the center coordinates of the device coordinate system of each frame of image are converted through the coordinate system of the rotation relationship, translation relationship and zoom relationship corresponding to each frame of image, a position coordinate in the reference coordinate system will be obtained, that is, the second coordinate Satisfying the first minimization condition is that the sum of the difference between all the second coordinates and the actual coordinates corresponding to each frame image sensed by the sensor, that is, the first coordinates, is the smallest. The first coordinates are the three-dimensional coordinates sensed by the GPS sensor and the altimeter when the shooting device shoots the corresponding image.

In an embodiment, the specific expression form of the first algorithm can refer to the following formula 4, where s*R _{camera_to_ned} *C _v +t is the conversion position coordinate, where C _v refers to the device coordinate system of the shooting device The center coordinate of, that is, the above-mentioned second coordinate, and _Cw corresponds to the above-mentioned first coordinate.

Among them, R _{camera_to_ned} can be calculated based on the Pitch angle, the Roll angle, and the Yaw angle using the above formula 1, formula 2, and formula 3. The value of n is 3, 4, 5 or other values, which refers to the 3 frames, 4 frames, and 5 frames included in the first image set. The first coordinate is based on the spatial position information of the shooting device when the image is taken, and is calculated by using sensor data. The sensor data includes the data collected by the global positioning sensor and the data collected by the height sensor, that is, The first coordinate is mainly determined based on the data sensed by GPS positioning devices and altimeters installed on mobile platforms such as drones or shooting equipment. Using GPS and altimeters, the center of the camera can be obtained. The position coordinates (GPS coordinates plus altitude) in a reference coordinate system such as the North East coordinate system.

In an embodiment, C _v and C _w in formula 4 can be directly obtained, R _{camera_to_ned} can directly use the value in the pan/tilt angle information based on Pitch angle and Roll angle, and Yaw angle can be used as a parameter to be optimized. Of course, The value of Yaw angle in the pan/tilt angle information can be used as an initial value. The translation relationship, that is, the translation matrix t and the scaling relationship, that is, the scale s can be calculated by other existing methods. In one embodiment, t and s can also be considered as a parameter to be optimized, so that only an initial value is provided to be added to the calculation in Formula 4. In the process of minimizing equation 4, the value of Yaw angle is constantly updated, and the values of t and s will also be updated and optimized. Finally, the updated values of Yaw angle, t, and s can minimize the result of equation 4 above. That is, the optimizing the first rotation relationship based on the center coordinates and the first coordinates may include: performing the first rotation based on the center coordinates and the first coordinates The relationship, the translation relationship between the device coordinate system and the reference coordinate system, and the scaling relationship between the device coordinate system and the reference coordinate system are optimized.

It is further explained that for the R _{camera_to_ned} in the above formula, it can be calculated based on the pan/tilt angle information of the first image as described above. Therefore, for the first image, based on the pan/tilt angle information of the first image and t, s, _Cv , and _Cw can calculate the difference between the second coordinate corresponding to the first image and the first coordinate determined by the sensor data when the first image is taken;

For the second image, the relative rotation relationship of the second image with respect to the first image can be calculated according to the pan/tilt angle information of the second image, and the relative rotation relationship can be calculated based on the difference of the pan/tilt angle information corresponding to the two frames of images Therefore, the rotation relationship between the device coordinate system corresponding to the second image and the reference coordinate system can be determined on the basis of the R _{camera_to_ned of the} first image and based on the relative rotation relationship. Through the same calculation method for the first image described above, the same The difference between the second coordinate corresponding to the second image and the first coordinate determined by the sensor data when the second image is taken can be calculated; and so on, the respective second coordinates and first coordinates of n frames of images can be obtained The difference between, and the summation and minimization of n differences can be calculated to obtain the R _{camera_to_ned of the} first image. That is to say, based on the rotation angle of the pan-tilt, the relative rotation relationship between each second image and the first image can be obtained. Therefore, in the above formula 4, only the R _{camera_to_ned of the} first image can be used instead of device coordinates corresponding to the second image directly tied to the relationship between the rotating reference frame, and further in the process optimization can be solved in _R camera_to_ned, or angle values therein Yaw angle. In the same way, when t is the parameter to be optimized and needs to be optimized, the value of t can also be calculated based on the difference between the pan/tilt angles of each second image and the first image to obtain the relative translation relationship, and then only the first image is optimized. Translation relationship. For the scale s, the first image and all the second images correspond to the same s.

After the first rotation relationship, translation relationship, and zoom relationship are optimized based on the above formula, the pixels on the first image can be determined according to the optimized first rotation relationship, optimized translation relationship, and optimized zoom relationship. The position information of the point in the reference coordinate system. Specifically, the two-dimensional point (x, y) on the first image is first converted to the three-dimensional point Q(X, Y, Z) in the device coordinate system, based on (optimized s)*(optimized R _{camera_to_ned} )*Q+(optimized t), you can calculate the three-dimensional point coordinates of the three-dimensional point Q in the northeast coordinate system, and then determine the position coordinates of the pixel points on the first image in the reference coordinate system.

In one embodiment, the first image is the first frame image in the first image set, and after optimization processing can obtain the optimized Yaw angle (or the first rotation relationship), t and s, The Yaw angle (or the first rotation relationship), the translation matrix t, and the scale s are all associated with the first frame of image. In this way, it can be determined according to the first rotation relationship, the optimized first translation matrix, and the optimized scale The position coordinates of the pixel points on each image in the first image set in the reference coordinate system; the foregoing describes the position coordinates of the pixel points on the first image in the reference coordinate system. When calculating the subsequent second image, it is only necessary to calculate the relative rotation relationship and relative translation relationship between the second image and the first image, and then the rotation relationship from the device coordinate system to the reference coordinate system corresponding to the second image and The translation relationship, from which the position coordinates of the pixel points on each second image to the reference coordinate system can be calculated. In one embodiment, the PNP (Perspecitve-n-Point) method (a computer vision algorithm) can be used to calculate the rotation matrix (relative rotation relationship) and translation matrix (relative translation relationship) of each second image relative to the first image. ).

In simple terms, the rotation matrix and the translation matrix can align the origin of the two coordinate systems with the coordinate axis, and the scale can be reduced or enlarged on one of the coordinate systems, so that the scales of the two coordinate systems are aligned. In an embodiment, the first image may also be any frame of image in the image set.

Through the above-mentioned optimization method, the rotation relationship between the device coordinate system where the shooting device is located and the reference coordinate system is made more accurate, so that in the state of linear motion, the conversion of pixels to three-dimensional coordinates is more accurate.

In an embodiment, after the first rotation relationship is obtained, the position of the pixel points on the images in the second image set may be converted based on the first rotation relationship. The first rotation relationship in the embodiment of the present invention may refer to the rotation relationship obtained through the above formula 1, formula 2, and formula 3, or it may be based on the above formula 1, formula 2 and formula 3, through the first An algorithm such as formula 4 optimizes the rotation relationship. The second image set may include the first image and an image captured again after obtaining the first rotation relationship, or the images included in the second image set are actually consistent with the images of the first image set mentioned above. In the embodiment of the present invention, according to the first rotation relationship, the position information in the reference coordinate system of the pixel on each frame of the second image set collected by the shooting device can be determined; and The second image set includes the first image and at least one frame of the third image. In one embodiment, the first image may refer to an image captured by a photographing device in the first frame of the second image set. .

When using the first rotation relationship to calculate the position information of the pixel points of the other images in the second image set in the reference coordinate system, the relative transformation relationship between the other images and the first image is mainly used. For details, please refer to FIG. 8, which is a schematic diagram of a flow of position transformation according to an embodiment of the present invention. The third image in the second image set is used for description. According to the first rotation relationship, the first image captured by the photographing device is determined. The position information of the pixel on each frame of the image in the second image set in the reference coordinate system may include: S801: using a visual algorithm to determine the relative transformation relationship between the third image and the first image; S802 : Based on the first rotation relationship and the relative transformation relationship, determine a fourth rotation relationship between the device coordinate system and the reference coordinate system when acquiring the third image; S803: according to the first rotation Relationship, determine the position information of the pixel on the first image in the reference coordinate system, and determine the position of the pixel on the third image in the reference coordinate system according to the fourth rotation relationship location information.

The PNP method mentioned above can be used to calculate the relative transformation relationship between the first image and the third image. The relative transformation relationship includes the relative rotation relationship between the first image and the third image, and can also include the first image and the third image. The relative translation relationship between the third images. On the basis of the first rotation relationship and the relative transformation relationship, the fourth rotation relationship from the device coordinate system to the reference coordinate system corresponding to the third image can be obtained. After the fourth rotation relationship is obtained, combined with the translation relationship and scale from the device coordinate system corresponding to the third image to the reference coordinate system, the pixels on the third image can be converted to the reference coordinate system, and the pixels in the reference The position coordinates on the coordinate system. The translation relationship and scale from the device coordinate system to the reference coordinate system corresponding to the third image can be directly obtained through a visual algorithm; or it can be based on the combination of the relative translation relationship in the relative transformation relationship between the third image and the first image The first image calculates the translation relationship and the scale from the device coordinate system to the reference coordinate system corresponding to the third image through the translation relationship obtained by the optimization and the optimized scale.

Further, in the process of determining the position information of the pixels on the first image in the reference coordinate system based on the first rotation relationship, and determining the position information of the pixels on the third image in the reference coordinate system based on the fourth rotation relationship , The first rotation relationship and the fourth rotation relationship may be optimized first, and the position of the pixel point on the corresponding image in the reference coordinate system is determined according to the optimized first rotation relationship and the fourth rotation relationship. In one embodiment, please refer to FIG. 9, when the image processing device executes the S803, it may specifically include: S901: optimizing the first rotation relationship and the fourth rotation relationship according to a preset second algorithm Processing; S902: Determine the position information of the pixel on the first image in the reference coordinate system according to the optimized first rotation relationship, and determine the third image according to the optimized fourth rotation relationship The position information of the pixel on the above reference coordinate system.

In an embodiment, S901 may include: acquiring second posture data when the shooting device collects each frame of image in the second image set; determining the second image according to the second posture data The fifth rotation relationship between the device coordinate system and the reference coordinate system when each frame of image in the set is collected; the first rotation relationship and the fourth rotation relationship are optimized, so that the optimized The difference between the first rotation relationship and the optimized fourth rotation relationship and the corresponding fifth rotation relationship satisfy the preset second minimization condition. The second posture data includes the posture angle information of each frame of the image in the second image set at the time of shooting, and may specifically include the pitch angle, roll angle, and Yaw angle of each frame of image during shooting.

The second algorithm mainly uses the pan/tilt angle information of the images in the second image set at the time of shooting as a reference, and optimizes the fourth rotation relationship obtained based on the vision algorithm. When performing visual algorithm related processing such as Bundle Adjustment (BA) for the images in the image collection, the pan/tilt angle information at the time of image shooting can be added as a reference for constraint, and the corresponding rotation relationship can be further optimized. The basis of optimization is that although the pan/tilt angle information at the time of image shooting is not very accurate, it can still be used as a reference, that is, the rotation relationship optimized by the BA vision algorithm and the rotation relationship calculated based on the pan/tilt angle information cannot deviate too much . Play the role of fixing the optimized rotation relationship near the rotation matrix provided by the pan/tilt angle information.

In an embodiment, the specific optimization method refers to the following formula 5.

Among them, ||u _ij -v _ij || refers to the reprojection error calculated based on the pixels on the images in the second image set under the BA algorithm, and δ _i diff(R _i -R _{ref_i} ) refers to Rotation relationship deviation, the rotation relationship deviation mainly refers to: the rotation relationship R _{i to} be optimized is constrained based on the pan/tilt angle information of the i-th frame image in the second image set at the time of shooting, R _{ref_i} is based on the above formula 1 2. The formula 3 is calculated, and R _{ref_i is} completely calculated based on the corresponding second posture data of the i-th frame image. δ _i is the elimination dimension coefficient. In order to adapt to the reprojection error, the general value is the error variance of the image feature point extraction. In one embodiment, δ _i may be equal to 4. X _j represents the j-th three-dimensional point, P _i represents the pixel point corresponding to X _j on the i-th frame of the second image set, when the three-dimensional point X _j has a projection P _i in the image of the second image set σ _ij =1, otherwise σ _ij =0.

When i=1, it is the first frame image, that is, the above-mentioned first image, and R ₁ is the first rotation relationship mentioned above, which can be directly the first rotation relationship calculated by the above formula 1, formula 2, and formula 3. a first rotational relationship or equation 4 obtained by optimizing the relationship between the first image as a rotation, R ₁ of the first image can not optimized. For the subsequent i-1 frame images, the R _{i of} each other image is obtained by optimizing formula 5 in combination with R ₁ . For example, for the third image when i=2, it is the sum of the reprojection errors calculated based on the BA algorithm of each image in the two frames and the sum of the deviations from the rotation relationship of each image in the two frames. If R _{1 is} known, optimize R ₂ .

Specifically, referring to the aforementioned formula 1, formula 2, and formula 3 based on the second posture data, the first image from the device coordinate system to the reference coordinate system corresponding to each frame of the second image set can be obtained. Five rotation relations, the fifth rotation relation is R _{ref_i in} formula 5. For each frame of image in the second image set, a corresponding R _{ref_i is} calculated, and the optimization constraint is performed through the posture data when these images are taken.

Further, in an embodiment, the difference between the optimized first rotation relationship, the optimized fourth rotation relationship and the corresponding fifth rotation relationship satisfies the determination manner of the preset second minimization condition, It includes: determining that the sum between the first data and the second data satisfies a preset third minimization condition; wherein, the first data is calculated based on the pixel points on each frame of the second image set The sum of the reprojection errors of ∑ _i,j σ _ij ·||u _ij -v _ij || in formula 5, the second data is the corresponding to each frame of image in the second image set The sum of the deviations of the rotation relationship, that is, the summation of ∑ _i δ _i diff (R _i -R _{ref_i} ) in formula 5, the rotation relationship deviation refers to the first rotation relationship or the fourth rotation relationship and the corresponding The difference between the fifth rotation relationship. In one embodiment, the method for calculating the deviation of the rotation relationship may be by performing dot multiplication on the rotation relationship to be optimized and the corresponding rotation relationship of the pan/tilt head.

Then converted to the modulus value of the angle vector. As mentioned above, R ₁ can be directly calculated by formula 1 to formula 3 above, or obtained by optimization by formula 4. It is also possible to optimize the calculation of R ₁ when i=1 by formula 5 after the optimization is obtained by formula 4. After the first rotation relationship R _{1 is} obtained, the fourth rotation relationship R ₂ of the third image when i=2 can be optimally calculated. When R ₁ and R ₂ are known, the fourth rotation relationship when i=3 can be further optimized. The image rotation relationship, and so on. In this way, even if the first rotation relationship R ₁ obtained by optimization is not particularly accurate, the following images have added constraints on the fifth rotation relationship R _{ref_i} . Under the optimization of formula 5, the subsequent images can also be more accurate In S902, the position information of the pixel on the third image in the second image set in the reference coordinate system can be determined more accurately, and the first rotation relationship is not sufficiently accurate. Big impact.

Through the above-mentioned optimization method, the rotation relationship between the device coordinate system where the shooting device is located and the reference coordinate system is made more accurate, so that in the state of linear motion, the conversion of pixels to three-dimensional coordinates is more accurate. In addition, the accuracy of the rotation relationship corresponding to the subsequent images captured by the photographing device is also better guaranteed, thereby ensuring that tasks such as environmental monitoring and mapping can be better completed.

Please refer to FIG. 10 again, which is a schematic structural diagram of a data processing device for a photographing device according to an embodiment of the present invention. The device is set in an image processing device, and the image processing device can obtain posture data of the photographing device. On a mobile platform, and the motion state of the mobile platform is a linear motion state. The image processing device may be a smart device capable of processing related content in each of the foregoing embodiments. The device includes the following structure.

The acquiring module 1001 is used to acquire the first posture data when the shooting device collects the first image; the determining module 1002 is used to determine the equipment of the shooting device when the first image is collected according to the first posture data A first rotation relationship between a coordinate system and a reference coordinate system; the processing module 1003 is configured to determine the position information of a pixel on the first image in the reference coordinate system according to the first rotation relationship.

In one embodiment, the reference coordinate system refers to a geographic coordinate system or a geodetic coordinate system.

In one embodiment, the photographing device is provided on a pan-tilt of the mobile platform.

In one embodiment, the determining module 1002 is specifically configured to obtain the second rotation relationship between the pan/tilt coordinate system of the pan/tilt and the reference coordinate system according to the first posture data, and obtain the The third rotation relationship between the device coordinate system of the photographing device and the pan-tilt coordinate system; according to the second rotation relationship and the third rotation relationship, it is determined that the device coordinate system is The first rotational relationship between reference coordinate systems.

In an embodiment, the third rotation relationship is configured according to the assembly relationship between the photographing device and the pan/tilt.

In one embodiment, the processing module 1003 is specifically configured to perform optimization processing on the first rotation relationship according to a preset first algorithm; and determine the first image on the first image according to the optimized first rotation relationship The position information of the pixel in the reference coordinate system.

In an embodiment, the first attitude data includes a pitch angle, a roll angle, and a yaw angle; the optimization processing of the first rotation relationship includes an optimization processing on the yaw angle.

In one embodiment, the processing module 1003 is specifically configured to obtain the center coordinates of the device coordinate system and the center coordinates of the device coordinate system determined by using sensor data when each frame of the first image set collected by the shooting device is collected. The first coordinate of the center coordinate in the reference coordinate system; based on the center coordinate and the first coordinate, the first rotation relationship is optimized, so that each determined based on the optimized first rotation relationship The difference between the second coordinate of the center coordinate in the reference coordinate system and the first coordinate meets a preset first minimization condition; wherein, the first image set includes the first image And at least one frame of second image.

In one embodiment, the second image is an image adjacent to the first image in acquisition time.

In one embodiment, the processing module 1003 is specifically configured to perform translation between the first rotation relationship, the device coordinate system and the reference coordinate system based on the center coordinates and the first coordinates The relationship, the scaling relationship between the device coordinate system and the reference coordinate system is optimized.

In one embodiment, the processing module 1003 is specifically configured to determine that the pixels on the first image are in the reference according to the optimized first rotation relationship, the optimized translation relationship, and the optimized zoom relationship. Position information in the coordinate system.

In one embodiment, the sensor data includes data collected by a global positioning sensor and data collected by a height sensor.

In one embodiment, the processing module 1003 is configured to determine, according to the first rotation relationship, the pixel point on each frame of the second image set collected by the shooting device in the reference coordinate system Location information; wherein the second image set includes the first image and at least one frame of the third image.

In an embodiment, the first image is the first frame of image in the second image set.

In one embodiment, the processing module 1003 is specifically configured to use a visual algorithm to determine the relative transformation relationship between the third image and the first image; based on the first rotation relationship and the relative transformation relationship , Determining a fourth rotation relationship between the device coordinate system and the reference coordinate system when the third image is acquired; according to the first rotation relationship, determining that a pixel on the first image is in the reference The position information in the coordinate system is determined, and the position information of the pixel on the third image in the reference coordinate system is determined according to the fourth rotation relationship.

In one embodiment, the processing module 1003 is specifically configured to perform optimization processing on the first rotation relationship and the fourth rotation relationship according to a preset second algorithm; according to the optimized first rotation relationship, Determine the position information of the pixel on the first image in the reference coordinate system, and determine the position of the pixel on the third image in the reference coordinate system according to the optimized fourth rotation relationship information.

In one embodiment, the processing module 1003 is specifically configured to obtain second posture data when the shooting device collects each frame of the second image set; according to the second posture data, determine the The fifth rotation relationship between the device coordinate system and the reference coordinate system when each frame of the image in the second image set is collected; the first rotation relationship and the fourth rotation relationship are optimized to The difference between the optimized first rotation relationship and the optimized fourth rotation relationship and the corresponding fifth rotation relationship satisfy the preset second minimization condition.

In one embodiment, the processing module 1003 is specifically configured to determine that the sum between the first data and the second data satisfies a preset third minimization condition; wherein, the first data is based on the second The sum of the reprojection errors calculated from the pixels on each frame of the image in the image set, the second data is the sum of the rotation relation deviations corresponding to each frame of the image in the second image set, the rotation relation deviation Refers to the difference between the first rotation relationship or the fourth rotation relationship and the corresponding fifth rotation relationship.

In the embodiment of the present invention, for the specific implementation of each module in the device, reference may be made to the description of the related content in the foregoing embodiment, which is not repeated here. In the embodiment of the present invention, when the mobile platform is in a linear motion state, the rotation relationship between the device coordinate system where the photographing device is located and the reference coordinate system is determined, and the posture data of the photographing device when the image is collected is referred to. , The rotation relationship between the equipment coordinate system and the reference coordinate system can be determined more accurately, and the accuracy of the conversion from pixel points to three-dimensional coordinates in the state of linear motion is better ensured, so as to better realize environmental monitoring, Tasks such as drawing.

Please refer to FIG. 11 again, which is a schematic structural diagram of an image processing device according to an embodiment of the present invention. The image processing device is a smart device. The image processing device can obtain the posture data of the shooting device, and the shooting device is set on a mobile platform. Above, and the motion state of the mobile platform is a linear motion state. For example, in the scene in Figure 1, the image processing device is a device on the ground, which can receive images captured by a shooting device through an aircraft, and can also receive shooting The posture data of the device when the image is taken, such as data such as the angle information of the pan/tilt on which the camera is mounted. In other embodiments, the photographing device may also be mounted on a mobile platform, and the data required for image processing can be obtained by connecting with the photographing device, pan-tilt and other devices of the mobile platform. Of course, the image processing device itself can also be used as a component of the mobile platform for connecting with shooting devices, pan-tilt and other devices.

The image processing device includes: a communication interface unit 1101, a processing unit 1102, and other structures such as a power supply module, a housing, etc.; the image processing device may include: a user interface unit 1103 and a storage unit 1104 as needed. The user interface unit 1103 may be, for example, a touch screen, which can obtain instructions from the user and can present the user with corresponding original data (such as received image data) and processed data (such as an environmental monitoring map made after image processing). ) And other data.

The storage unit 1104 may include volatile memory (volatile memory), such as random-access memory (RAM); the storage unit 1104 may also include non-volatile memory (non-volatile memory), such as fast Flash memory (flash memory), solid-state drive (SSD), etc.; the storage unit 1104 may also include a combination of the foregoing types of memories.

The processing unit 1102 may be composed of a central processing unit (CPU). The processing unit 1102 may also include a hardware chip. The above-mentioned hardware chip may be, for example, an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be, for example, a field-programmable gate array (FPGA), a general array logic (generic array logic, GAL), etc.

In one embodiment, the storage unit 1104 may be used to store some data, such as the image data of the environment mentioned above, processed drawing data, etc., and the storage unit 1104 may also be used to store program instructions. The processing unit 1102 can call the program instructions to implement the corresponding functions and steps in the foregoing embodiments.

In one embodiment, the communication interface unit 1101 is used to communicate with an external device to obtain data of the external device; the processing unit 1102 is used to obtain the first image collected by the photographing device through the communication interface unit 1101 According to the first posture data, determine the first rotation relationship between the device coordinate system and the reference coordinate system of the shooting device when the first image is collected according to the first posture data; according to the first rotation relationship , Determining the position information of the pixel on the first image in the reference coordinate system.

In one embodiment, the processing unit 1102 is configured to obtain a second rotation relationship between the pan/tilt coordinate system of the pan/tilt and the reference coordinate system according to the first posture data, and obtain the photographing The third rotation relationship between the device coordinate system of the device and the pan-tilt coordinate system; according to the second rotation relationship and the third rotation relationship, it is determined that the device coordinate system and the reference when collecting the first image The first rotational relationship between coordinate systems.

In an embodiment, the processing unit 1102 is configured to perform optimization processing on the first rotation relationship according to a preset first algorithm; determine the image on the first image according to the optimized first rotation relationship The position information of the pixel in the reference coordinate system.

In one embodiment, the processing unit 1102 is configured to obtain the center coordinates of the device coordinate system and the center determined by using sensor data when each frame of the image in the first image set collected by the shooting device is collected. The coordinates are the first coordinates in the reference coordinate system; based on the center coordinates and the first coordinates, the first rotation relationship is optimized, so that each position determined based on the optimized first rotation relationship The difference between the second coordinate of the center coordinate in the reference coordinate system and the first coordinate meets a preset first minimization condition; wherein, the first image set includes the first image and At least one second image.

In one embodiment, the processing unit 1102 is configured to determine the first rotation relationship, the translation relationship between the device coordinate system and the reference coordinate system based on the center coordinate and the first coordinate , The scaling relationship between the device coordinate system and the reference coordinate system is optimized.

In one embodiment, the processing unit 1102 is configured to determine that a pixel on the first image is at the reference coordinate according to the optimized first rotation relationship, the optimized translation relationship, and the optimized zoom relationship. Location information in the department.

In one embodiment, the processing unit 1102 is configured to determine, according to the first rotation relationship, the pixel point on each frame of the second image set collected by the shooting device in the reference coordinate system. Location information; wherein the second image set includes the first image and at least one frame of the third image.

In an embodiment, the processing unit 1102 is configured to use a visual algorithm to determine the relative transformation relationship between the third image and the first image; based on the first rotation relationship and the relative transformation relationship, Determine the fourth rotation relationship between the device coordinate system and the reference coordinate system when the third image is collected; determine that the pixel on the first image is at the reference coordinate according to the first rotation relationship And determine the position information of the pixel on the third image in the reference coordinate system according to the fourth rotation relationship.

In one embodiment, the processing unit 1102 is configured to perform optimization processing on the first rotation relationship and the fourth rotation relationship according to a preset second algorithm; determine according to the optimized first rotation relationship The position information of the pixel on the first image in the reference coordinate system, and the position information of the pixel on the third image in the reference coordinate system is determined according to the optimized fourth rotation relationship .

In one embodiment, the processing unit 1102 is configured to obtain second posture data when the shooting device collects each frame of the second image set; according to the second posture data, determine the first posture data The fifth rotation relationship between the device coordinate system and the reference coordinate system when each frame of the image in the two image sets is collected; the first rotation relationship and the fourth rotation relationship are optimized to make The difference between the optimized first rotation relationship and the optimized fourth rotation relationship and the corresponding fifth rotation relationship satisfy the preset second minimization condition.

In one embodiment, the processing unit 1102 is configured to determine that the sum between the first data and the second data satisfies a preset third minimization condition; wherein, the first data is based on the second image The sum of the re-projection errors calculated from the pixels on each frame of the image in the set, the second data is the sum of the rotation relation deviations corresponding to each frame of the image in the second image collection, and the rotation relation deviation is Refers to the difference between the first rotation relationship or the fourth rotation relationship and the corresponding fifth rotation relationship.

In the embodiment of the present invention, for the specific implementation of the processing unit, reference may be made to the description of the related content in the foregoing embodiment, which is not repeated here. In the embodiment of the present invention, when the mobile platform is in a linear motion state, the rotation relationship between the device coordinate system where the photographing device is located and the reference coordinate system is determined, and the posture data of the photographing device when the image is collected is referred to. , The rotation relationship between the equipment coordinate system and the reference coordinate system can be determined more accurately, and the accuracy of the conversion from pixel points to three-dimensional coordinates in the state of linear motion is better ensured, so as to better realize environmental monitoring, Tasks such as drawing.

A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing relevant hardware through a computer program. The program can be stored in a computer readable storage medium. During execution, it may include the procedures of the above-mentioned method embodiments. Wherein, the storage medium may be a magnetic disk, an optical disc, a read-only memory (Read-Only Memory, ROM), or a random access memory (Random Access Memory, RAM), etc.

The above-disclosed are only some embodiments of the present invention, which of course cannot be used to limit the scope of rights of the present invention. Therefore, equivalent changes made according to the claims of the present invention still fall within the scope of the present invention.

Claims

A data processing method for a photographing device, characterized in that the method is applied to an image processing device that can obtain posture data of the photographing device, the photographing device is set on a mobile platform, and the mobile platform The motion state of is a linear motion state, and the method includes:

Acquiring first posture data when the photographing device collects the first image;

Determine, according to the first posture data, a first rotation relationship between the device coordinate system of the photographing device and a reference coordinate system when the first image is collected;

According to the first rotation relationship, the position information of the pixel on the first image in the reference coordinate system is determined.
The method according to claim 1, wherein the reference coordinate system refers to a geographic coordinate system or a geodetic coordinate system.
The method of claim 1, wherein the photographing device is set on a pan-tilt of the mobile platform.
The method of claim 3, wherein the first rotation relationship between the device coordinate system and the reference coordinate system of the photographing device when the first image is collected is determined according to the first posture data ,include:

Obtain the second rotation relationship between the pan/tilt coordinate system of the pan/tilt and the reference coordinate system according to the first posture data, and obtain the distance between the device coordinate system of the photographing device and the pan/tilt coordinate system The third rotation relationship;

According to the second rotation relationship and the third rotation relationship, a first rotation relationship between the device coordinate system and a reference coordinate system when the first image is acquired is determined.
The method of claim 4, wherein the third rotation relationship is configured according to the assembly relationship between the photographing device and the pan/tilt.
The method according to claim 2, wherein the determining the position information of the pixel on the first image in the reference coordinate system according to the first rotation relationship comprises:

Optimizing the first rotation relationship according to the preset first algorithm;

According to the optimized first rotation relationship, the position information of the pixel on the first image in the reference coordinate system is determined.
The method according to claim 6, wherein the first attitude data includes a pitch angle, a roll angle, and a yaw angle;

The optimization processing of the first rotation relationship includes optimizing the yaw angle.
8. The method according to claim 7, wherein the optimizing the first rotation relationship according to the preset first algorithm comprises:

Acquiring the center coordinates of the device coordinate system and the first coordinates in the reference coordinate system of the center coordinates determined by using sensor data for each frame of image acquisition in the first image set collected by the shooting device;

Based on the center coordinates and the first coordinates, the first rotation relationship is optimized, so that each of the center coordinates determined based on the optimized first rotation relationship is the second in the reference coordinate system. The difference between the coordinate and the corresponding first coordinate and meets the preset first minimization condition;

Wherein, the first image set includes the first image and at least one frame of second image.
8. The method of claim 8, wherein the second image is an image adjacent to the first image in acquisition time.
The method according to claim 8, wherein the optimizing the first rotation relationship based on the center coordinates and the first coordinates comprises:

Based on the center coordinates and the first coordinates, the first rotation relationship, the translation relationship between the device coordinate system and the reference coordinate system, and the relationship between the device coordinate system and the reference coordinate system The zoom relationship is optimized.
The method according to claim 10, wherein the determining the position information of the pixel on the first image in the reference coordinate system according to the optimized first rotation relationship comprises:

According to the optimized first rotation relationship, the optimized translation relationship, and the optimized zoom relationship, the position information of the pixel on the first image in the reference coordinate system is determined.
The method according to claim 8, wherein the sensor data includes data collected by a global positioning sensor and data collected by a height sensor.
The method according to any one of claims 1 to 12, wherein the determining the position information of the pixel on the first image in the reference coordinate system according to the first rotation relationship, include:

Determine, according to the first rotation relationship, position information in the reference coordinate system of pixels on each frame of images in the second image set collected by the photographing device;

Wherein, the second image set includes the first image and at least one frame of third image.
The method of claim 13, wherein the first image is the first frame of the image in the second image set.
The method according to claim 13, wherein, according to the first rotation relationship, it is determined that the pixel on each frame of the image in the second image set collected by the photographing device is in the reference coordinate system Location information, including:

Using a visual algorithm to determine the relative transformation relationship between the third image and the first image;

Based on the first rotation relationship and the relative transformation relationship, determining a fourth rotation relationship between the device coordinate system and the reference coordinate system when acquiring the third image;

According to the first rotation relationship, determine the position information of the pixel on the first image in the reference coordinate system, and determine that the pixel on the third image is in the third image according to the fourth rotation relationship. The position information in the reference coordinate system.
The method according to claim 15, wherein the position information of the pixel on the first image in the reference coordinate system is determined according to the first rotation relationship, and according to the fourth The rotation relationship to determine the position information of the pixel on the third image in the reference coordinate system includes:

Performing optimization processing on the first rotation relationship and the fourth rotation relationship according to a preset second algorithm;

Determine the position information of the pixel on the first image in the reference coordinate system according to the optimized first rotation relationship, and determine the pixel on the third image according to the optimized fourth rotation relationship Position information in the reference coordinate system.
The method according to claim 16, wherein the optimizing the first rotation relationship and the fourth rotation relationship according to a preset second algorithm comprises:

Acquiring second posture data when the shooting device collects each frame of images in the second image set;

Determine, according to the second posture data, a fifth rotation relationship between the device coordinate system and the reference coordinate system when each frame of image in the second image set is collected;

The first rotation relationship and the fourth rotation relationship are optimized, so that the difference between the optimized first rotation relationship, the optimized fourth rotation relationship and the corresponding fifth rotation relationship meets the preset The second minimum condition.
The method according to claim 15, wherein the difference between the optimized first rotation relationship and the optimized fourth rotation relationship and the corresponding fifth rotation relationship satisfy a preset second minimization condition The methods of determination include:

Determining that the sum between the first data and the second data satisfies a preset third minimization condition;

Wherein, the first data is the sum of reprojection errors calculated based on the pixel points on each frame of image in the second image set, and the second data is each frame of image in the second image set The sum of the deviations of the corresponding rotational relations, the deviation of the rotational relations refers to the difference between the first rotational relationship or the fourth rotational relationship and the corresponding fifth rotational relationship.
A data processing device for photographing equipment, characterized in that the device is applied to an image processing equipment, and the image processing equipment can obtain posture data of the photographing equipment, the photographing equipment is set on a mobile platform, and The motion state of is a linear motion state, and the device includes:

An acquiring module, configured to acquire first posture data when the photographing device acquires the first image;

A determining module, configured to determine, according to the first posture data, a first rotation relationship between the device coordinate system of the shooting device and a reference coordinate system when the first image is collected;

The processing module is configured to determine the position information of the pixel on the first image in the reference coordinate system according to the first rotation relationship.
An image processing device, characterized in that the image processing device can acquire posture data of a shooting device, the shooting device is set on a mobile platform, and the motion state of the mobile platform is a linear motion state, the image processing device Including: communication interface unit, processing unit;

The communication interface unit is used to communicate with an external device and obtain data of the external device;

The processing unit is configured to obtain, through the communication interface unit, the first posture data of the shooting device when the first image is collected; according to the first posture data, determine the status of the shooting device when the first image is collected A first rotation relationship between the device coordinate system and a reference coordinate system; according to the first rotation relationship, the position information of a pixel on the first image in the reference coordinate system is determined.
The image processing device according to claim 20, wherein the reference coordinate system refers to a geographic coordinate system or a geodetic coordinate system.
22. The image processing device of claim 20, wherein the photographing device is provided on a pan-tilt of the mobile platform.
The image processing device according to claim 22, wherein the processing unit is configured to

Obtain the second rotation relationship between the pan/tilt coordinate system of the pan/tilt and the reference coordinate system according to the first posture data, and obtain the distance between the device coordinate system of the photographing device and the pan/tilt coordinate system The third rotation relationship;

According to the second rotation relationship and the third rotation relationship, a first rotation relationship between the device coordinate system and a reference coordinate system when the first image is acquired is determined.
23. The image processing device according to claim 23, wherein the third rotation relationship is configured according to the assembly relationship between the photographing device and the pan/tilt.
The image processing device according to claim 21, wherein the processing unit is configured to

Optimizing the first rotation relationship according to the preset first algorithm;

According to the optimized first rotation relationship, the position information of the pixel on the first image in the reference coordinate system is determined.
The image processing device according to claim 25, wherein the first attitude data includes a pitch angle, a roll angle, and a yaw angle;

The optimization processing of the first rotation relationship includes optimizing the yaw angle.
The image processing device according to claim 26, wherein the processing unit is configured to

Acquiring the center coordinates of the device coordinate system and the first coordinates in the reference coordinate system of the center coordinates determined by using sensor data for each frame of image acquisition in the first image set collected by the shooting device;

Based on the center coordinates and the first coordinates, the first rotation relationship is optimized, so that each of the center coordinates determined based on the optimized first rotation relationship is the second in the reference coordinate system. The difference between the coordinate and the corresponding first coordinate and meets the preset first minimization condition;

Wherein, the first image set includes the first image and at least one frame of second image.
The image processing device according to claim 27, wherein the second image is an image adjacent to the first image in acquisition time.
The image processing device according to claim 27, wherein the processing unit is configured to

Based on the center coordinates and the first coordinates, the first rotation relationship, the translation relationship between the device coordinate system and the reference coordinate system, and the relationship between the device coordinate system and the reference coordinate system The zoom relationship is optimized.
The image processing device according to claim 29, wherein the processing unit is configured to

According to the optimized first rotation relationship, the optimized translation relationship, and the optimized zoom relationship, the position information of the pixel on the first image in the reference coordinate system is determined.
28. The image processing device of claim 27, wherein the sensor data includes data collected by a global positioning sensor and data collected by a height sensor.
The image processing device according to any one of claims 20 to 31, wherein the processing unit is configured to

Determine, according to the first rotation relationship, position information in the reference coordinate system of pixels on each frame of images in the second image set collected by the photographing device;

Wherein, the second image set includes the first image and at least one frame of third image.
The image processing device according to claim 32, wherein the first image is a first frame image in the second image set.
The image processing device according to claim 32, wherein the processing unit is configured to

Using a visual algorithm to determine the relative transformation relationship between the third image and the first image;

Based on the first rotation relationship and the relative transformation relationship, determining a fourth rotation relationship between the device coordinate system and the reference coordinate system when acquiring the third image;

According to the first rotation relationship, determine the position information of the pixel on the first image in the reference coordinate system, and determine that the pixel on the third image is in the third image according to the fourth rotation relationship. The position information in the reference coordinate system.
The image processing device according to claim 34, wherein the processing unit is configured to

Optimizing the first rotation relationship and the fourth rotation relationship according to a preset second algorithm;

Determine the position information of the pixel on the first image in the reference coordinate system according to the optimized first rotation relationship, and determine the pixel on the third image according to the optimized fourth rotation relationship Position information in the reference coordinate system.
The image processing device according to claim 35, wherein the processing unit is configured to

Acquiring second posture data when the shooting device collects each frame of images in the second image set;

Determine, according to the second posture data, a fifth rotation relationship between the device coordinate system and the reference coordinate system when each frame of image in the second image set is collected;

The first rotation relationship and the fourth rotation relationship are optimized, so that the difference between the optimized first rotation relationship, the optimized fourth rotation relationship and the corresponding fifth rotation relationship meets the preset The second minimum condition.
The image processing device according to claim 34, wherein the processing unit is configured to

Determining that the sum between the first data and the second data satisfies a preset third minimization condition;

Wherein, the first data is the sum of reprojection errors calculated based on the pixel points on each frame of image in the second image set, and the second data is each frame of image in the second image set The sum of the deviations of the corresponding rotational relations, the deviation of the rotational relations refers to the difference between the first rotational relationship or the fourth rotational relationship and the corresponding fifth rotational relationship.