WO2020181508A1

WO2020181508A1 - Digital surface model construction method, and processing device and system

Info

Publication number: WO2020181508A1
Application number: PCT/CN2019/077896
Authority: WO
Inventors: 黄文杰; 张明磊
Original assignee: 深圳市大疆创新科技有限公司
Priority date: 2019-03-12
Filing date: 2019-03-12
Publication date: 2020-09-17
Also published as: CN111247564A

Abstract

A digital surface model (DSM) construction method, and a processing device and system. Said method comprises: generating an initial DSM according to an acquired image set (S101); determining a hole region in the initial DSM (S102); if the type of an environmental region corresponding to the hole region is a preset type, determining elevation values of various grid cells in adjacent regions of the hole region in the initial DSM (S103); and according to the elevation values of various grid cells in the adjacent regions, updating elevation values of grid cells in the hole region, so as to obtain a DSM (S104). The embodiments of the present invention can acquire a relatively complete DSM.

Description

Method for constructing digital surface model, processing equipment and system

Technical field

The present invention relates to the technical field of image processing, in particular to a method for constructing a digital surface model, processing equipment and a system.

Background technique

True Orthophoto has many advantages such as large amount of information, intuitive image, and easy interpretation, so it is often applied to geographic information systems (GIS) for urban planning, environmental monitoring, emergency response, etc. In the production of real radio images, the key technologies involved are (1) digital surface model (DSM) generation; (2) digital correction; (3) mosaic fusion of orthophotos, etc. Among them, DSM refers to a ground elevation model that includes the height of buildings, bridges, and trees on the ground. DSM is essential for making real radio images.

At present, the methods of generating DSM include: generating dense point clouds based on dense matching of multi-view images, and then directly constructing regular grids or irregular triangular grids (Triangulated Irregular Network, TIN) to obtain DSM. Of course, there are some other technologies, such as directly using radar scanning to obtain dense point clouds, and then constructing a regular grid or TIN model to obtain DSM.

In the process of matching and determining dense point clouds in multiple images, the matching accuracy depends on the richness and distinguishability of the texture of the pixel location. Therefore, the texture is scarce or there is no texture, the discrimination is poor, there is specular reflection, and there is a certain degree of Areas with characteristics such as fluidity, such as large water surface areas, will cause holes in the DSM generated based on multi-view images, which will affect the subsequent generation of real radio images.

Summary of the invention

The embodiment of the present invention provides a method for constructing a digital surface model, processing equipment, and system, which can be processed to obtain a relatively complete digital surface model.

On the one hand, an embodiment of the present invention provides a method for constructing a digital surface model, including:

Generate an initial digital surface model based on the collected image collection;

Determining a hollow area in the initial digital surface model, wherein the hollow area includes grid cells with abnormal elevation values;

If the type of the environmental area corresponding to the cavity area is a preset type, determining the elevation value of each grid cell in the adjacent area of the cavity area on the initial digital surface model;

According to the elevation value of each grid unit in the adjacent area, the elevation value of the grid unit in the hollow area is updated to obtain a digital surface model.

On the other hand, an embodiment of the present invention also provides an image processing device, the image processing device includes: a communication interface unit and a processing unit, wherein:

The communication interface unit is used to receive environmental images;

The processing unit is configured to generate an initial digital surface model according to the collected image set; determine the cavity area in the initial digital surface model, wherein the cavity area includes grid units with abnormal elevation values; If the type of the environmental area corresponding to the cavity area is a preset type, the elevation value of each grid cell in the adjacent area of the cavity area on the initial digital surface model is determined; according to the elevation value of each grid cell in the adjacent area Value, update the elevation value of the grid unit in the hollow area to obtain a digital surface model.

In another aspect, an embodiment of the present invention also provides an image processing system, the image processing system includes: a mobile platform and an image processing device, the mobile platform is provided with an image acquisition device;

The mobile platform is configured to collect multiple environmental images during the movement of the mobile platform through the image acquisition device, and send the multiple collected environmental images to the image processing device;

The image processing device is configured to generate an initial digital surface model according to the collected image set; determine the cavity area in the initial digital surface model, wherein the cavity area includes grid cells with abnormal elevation values; if If the type of the environmental area corresponding to the cavity area is a preset type, the elevation value of each grid unit in the adjacent area of the cavity area on the initial digital surface model is determined; according to the value of each grid unit in the adjacent area The elevation value is to update the elevation value of the grid unit in the hollow area to obtain a digital surface model.

In the embodiment of the present invention, after the initial digital surface model is generated according to the environmental image, if the initial digital surface model has a hole area, and the type of the environmental area corresponding to the hole area is a preset type, then the adjacent Adjacent areas to update the elevation value of the void area to obtain the final DSM. On the one hand, a relatively complete DSM can be obtained. On the other hand, relatively flat areas such as water surface areas are updated based on the elevation values of adjacent areas as a reference, which also makes the final DSM more accurate.

Description of the drawings

In order to explain the embodiments of the present invention or the technical solutions in the prior art more clearly, the following will briefly introduce the drawings needed in the embodiments. Obviously, the drawings in the following description are only some of the present invention. Embodiments, for those of ordinary skill in the art, without creative work, other drawings can be obtained from these drawings.

Figure 1a is a schematic diagram of an application scenario of an embodiment of the present invention;

FIG. 1b is a schematic flowchart of a method for constructing a digital surface model according to an embodiment of the present invention;

FIG. 2 is a schematic flowchart of determining the type of environmental area corresponding to a cavity area according to an embodiment of the present invention;

FIG. 3 is a schematic diagram of one specific implementation of the construction of the digital surface model of the embodiment of the present invention;

[Corrected according to Rule 91 18.07.2019]
Fig. 4 is an exemplary explanatory diagram of a digital surface model according to an embodiment of the present invention;
Fig. 5 is a schematic structural diagram of an image processing device according to an embodiment of the present invention.

detailed description

In the embodiment of the present invention, in order to obtain a relatively complete digital surface model, it is necessary to fill the void areas in the generated initial digital surface model, that is, to update the elevation values of grid cells with abnormal elevation values in the void areas. In the process of filling the cavity area, different filling methods are selected for the cavity area by referring to the type of environment area where the cavity area in the initial digital surface model is located. In the embodiment of the present invention, some environmental types are preset, such as the type of water surface area or other types of mirrored environmental areas. If it is determined that the environmental area corresponding to the cavity area is of these preset types, it can be based on the initial digital surface. The elevation value of each grid cell in the adjacent area of the cavity area on the model is used to update the elevation value of the grid cell in the cavity area. As shown in Figure 1a, during the flight of a drone equipped with camera equipment, based on the captured environment images, it can generate an initial DSM about mountains and lakes including the illustrated environment. At this time, the initial DSM corresponds to the lake. The area is prone to produce a void area. After processing the relevant steps of the embodiment of the present invention, the elevation value of the lake in the initial DSM can be updated based on the elevation value of the lake edge area in the initial DSM, and a relatively complete scene can be constructed. DSM.

Grid cells with abnormal elevation values in the cavity area mainly refer to grid cells that do not have elevation values. Because in the initial DSM calculation process, there are situations where the elevation values are not calculated from the corresponding images. Therefore, in After calculating and constructing the initial DSM, a large number of grid cells without elevation values constitute the void area in the initial DSM. For example, in the scenario shown in Figure 1a, when a mountain lake is monitored to construct a DSM for this area, the area where the lake is located due to insufficient texture and other features will form a void area on the initial DSM.

The adjacent area of the cavity area mainly refers to the area adjacent to the cavity area. There are boundary grid units in the cavity area. Each boundary grid unit has one or more normal grid units closest to the boundary grid. The corresponding elevation value is recorded in the grid cell. In the embodiment of the present invention, these normal grid cells are called adjacent grid cells, and all these adjacent grid cells constitute an adjacent region of the cavity region. When updating the elevation value of the grid cell in the void area, it can be based on the minimum value of the elevation value of all adjacent grid cells in the adjacent area, or from a set of intermediate values determined from the elevation values of all adjacent grid cells Any one of the values of, or the average of the elevation values of all adjacent grid cells, is used as the elevation update value of the grid cell in the cavity area, and then based on the elevation update value, the elevation value of the grid cell in the cavity area is updated to Complete the elevation value filling of the cavity area, for example, directly use the updated elevation value as the elevation value of the grid cell in the cavity area.

Please refer to Figure 1b, which is a schematic flow chart of a method for constructing a digital surface model according to an embodiment of the present invention. The method according to an embodiment of the present invention may be implemented by an image processing device, which is captured by an image capture device. Based on the environmental image, the subsequent processing of the digital surface model is carried out, and the real radio image can even be further constructed based on the digital surface model.

In the embodiment of the present invention, the image processing device can be installed separately from the mobile platform, or the image processing device can be installed in the mobile platform as a part of the mobile platform, or the image acquisition device can be mounted on the mobile platform as a peripheral device. In an embodiment, as shown in Fig. 1a, the movable platform is a drone, such as a rotary-wing drone or a fixed-wing drone, and the image processing device communicates with the drone or the image acquisition device mounted on the drone Ground image processing equipment. The drone can send the environmental images collected by the image acquisition device to the image processing device through wireless transmission. As a result, the image processing device can process based on the received environmental image, first generate the initial DSM, and further analyze the environmental image based on the cavity area on the initial DSM, so as to realize the elevation value of part or all of the cavity area in the initial DSM Update to get a more accurate and complete DSM for subsequent processing such as making real shot images.

The image acquisition device can shoot and obtain environmental images of a certain environmental area (referred to as the target environmental area in the embodiment of the present invention) during the movement process to form an image collection of the target environmental area. The image processing device will base on the acquired image in S101 Assemble to generate an initial digital surface model. In one embodiment, the internal and external parameters of the camera corresponding to the image can be estimated based on the Structure from Motion (SfM) technology, and the multi-view image dense matching is performed on the image collection to obtain the target according to the estimated internal and external camera parameters. The dense point cloud of the environmental area is then divided into grid cells to obtain the initial DSM of the target environmental area. Optionally, after the obtained initial DSM, noise filtering may be performed to perform preliminary optimization processing on the initial DSM.

In the process of calculating the dense point cloud of the target environment area based on the image set, the dense matching of multi-view images depends on the camera internal parameters and camera external parameters when the corresponding environmental image is taken. The camera internal parameters and camera external parameters can be estimated by SfM. The basic process of SfM is: first extract image features using feature extraction algorithms; then perform feature point matching based on the Euclidean distance between two image feature points; for each image matching pair, calculate the epipolar geometry according to the matched feature points, and estimate F Matrix (equivalent to the relative posture between the images), and the matched feature points at the same time constitute the points with the same name (corresponding to the same three-dimensional point); then, all the images form a global graph according to the relative posture between the two. The points with the same name are connected together to form a global energy function. The energy function is optimized by the beam adjustment (Bundle Adjustment, BA), so that the sum of the reprojection errors of all points with the same name projected onto the corresponding image is minimized, and finally Optimize the internal camera parameters and external camera parameters of all images.

After obtaining the camera internal parameters and camera external parameters of the corresponding environment images, the classic method of dense matching of multi-view images can be divided into depth map extraction and depth map fusion steps. The depth map extraction process includes the following processing for each image independently: Use the current image as a reference image, and select several neighbor images according to the number of sparse feature points matching in SfM as a similarity measure; then find matching pixels for each pixel of the reference image in the neighbor image, and the matching accuracy depends on where the pixel is located The texture richness and distinguishability of the position; after finding the matching pixels, the corresponding three-dimensional points can be crossed, and the depth information corresponding to each pixel can be obtained. Depth map fusion is to project the depth maps of all images into three-dimensional space, and fuse the three-dimensional points corresponding to the overlapping areas of adjacent images, and finally obtain the entire dense point cloud of the target environment area.

After the dense point cloud of the scene is obtained, the scene where the target environment area is located is divided into regular grid cells according to the ground sampling distance (Ground sample distance, GSD), and the initial DSM is obtained by interpolation according to the dense three-dimensional points contained in the grid cells. Among them, the key to the above-mentioned SfM algorithm is the accuracy of feature point extraction and matching. To ensure that the two points need to be photographed in the target environment to have richer textures, and to be rigid and stationary during shooting. However, the environment image of the target environment area such as the water surface area has no texture (or too few textures), specular reflection, and certain fluidity, which basically violates the above requirements. Therefore, if most of the image area is covered by water or similar areas , Then the image may not be able to extract the feature points or the feature point matching fails, and the corresponding camera parameters cannot be recovered, and the 3D points and their elevation values at the relevant positions cannot be obtained. Therefore, there are some problems when constructing the initial DSM. The empty area.

After the initial DSM is constructed, the hole area in the initial digital surface model can be determined in S102, wherein the hole area includes grid cells with abnormal elevation values; no elevation value is recorded in the abnormal grid cells . In one embodiment, a hole area can be determined from the initial DSM through a connected area search and detection rule. After one or more hole areas are determined through the connected area detection, the area with a larger area can be used as the required area. The void area to be processed, that is, the void area of the determined void area on the initial digital surface model is greater than the preset void area threshold. After one or more void regions are determined, the boundary of the void can be marked for subsequent follow-up It can be determined whether the cavity area is a preset type based on the cavity boundary.

After determining the cavity area, the image processing device determines in S103 that the type of the environment area corresponding to the cavity area is a preset type, and then determines that the area adjacent to the cavity area on the initial digital surface model The elevation value of each grid cell. And in S104, according to the elevation value of each grid unit in the adjacent area, the elevation value of the grid unit in the hollow area is updated to obtain a digital surface model.

In an embodiment, the S104 may specifically include: determining the obtained elevation update value according to the elevation value of each grid cell in the adjacent area according to the first confirmation rule; and using the elevation update value as the elevation value in the hollow area The elevation value of the grid cell. In an embodiment, the first confirmation rule may refer to the above-mentioned: the minimum value of the elevation values of all adjacent grid cells, or the intermediate value determined from the elevation values of all adjacent grid cells Any one value in the set, or the average value of the elevation values of all adjacent grid cells, will be used as the updated elevation value of the grid cells in the hollow area.

In one embodiment, after obtaining one or more cavity areas, for each cavity area, the image processing device may detect whether the environmental area corresponding to each determined cavity area is a preset type. The preset type mainly refers to the above-mentioned area type without texture (or too little texture), specular reflection, and certain fluidity. The preset type mainly includes the type of water surface area. That is to say, in one embodiment, if the environmental area corresponding to the cavity area is analyzed to be a water surface area, it can be considered that the type of the environmental area corresponding to the cavity area is a preset type.

There are many ways to detect whether the environmental area corresponding to the cavity area is of the preset type. In a simple embodiment, the aircraft can be judged based on the GPS position coordinates of the mobile platform such as the aircraft when the image collection is captured and combined with the area map. Whether there is a larger area of water area below, if so, it is considered that the type of the environmental area corresponding to the cavity area is a preset type.

In one embodiment, in order to ensure the accuracy of detection, it can be determined by image analysis and recognition whether the environmental area corresponding to the cavity area is a preset type. In one embodiment, after determining the cavity area in the initial digital surface model, as shown in FIG. 2, the image processing device separately detecting whether the environmental area corresponding to each determined cavity area is a preset type may include: S201 determines a mapping area in the target environment image of the image set, where the mapping area is the projection of the cavity area on the target environment image; S202 performs image analysis on the mapping area in the target environment image, and According to the analysis result, it is determined whether the type of the environmental area corresponding to the cavity area is a preset type. That is to say, the cavity area is projected onto one or more environmental images in the image collection, and then based on mature image analysis technology, it is determined whether the environmental area corresponding to the cavity area is a preset area, and whether it is a preset type. The detection accuracy and efficiency are improved. If the preset type includes the type of the water surface area, when the analysis result of the image analysis is that the mapping area is a water surface area, the type of the environmental area corresponding to the cavity area is the preset type.

When detecting whether the environment area corresponding to the cavity area in the target environment area is a preset area, in the embodiment of the present invention, the environment images in the image collection of the target environment area are classified, and different types of environment images are used as the target environment During image analysis, different analysis logics are used to confirm whether the environmental area corresponding to the cavity area is a preset area. When projecting the cavity area to the target environment image, different projection processing is also performed according to different types of environment images.

In an embodiment, the S202 includes: determining the image type of the target environment image; if the image type of the target environment image is the first type, analyzing the area of the mapping area in the target environment image; when When the analysis result is that the area of the mapping area is greater than the preset area threshold, it is determined that the type of the environmental area corresponding to the cavity area is the preset type.

In one embodiment, the determination of the image type of the target environment image may be based mainly on whether the camera internal parameters and the camera external parameters can be calculated as a basis. In the process of generating the initial digital surface model based on the collected image set, When the internal camera parameters and external camera parameters of the target environment image in the image collection cannot be determined, the image type of the target environment image is the first type. And when in the process of generating the initial digital surface model based on the collected image collection, the camera internal parameters and camera external parameters of the target environment image in the image collection can be determined, then the image type of the target environment image is The second type. In one embodiment, based on the above-mentioned SFM, the camera internal parameters and camera external parameters of a certain environment image in the image set, namely the target environment image, cannot be obtained, then the target environment image is the first type. If the camera of the target environment image can be obtained Internal parameters and external camera parameters, the target environment image is of the second type.

Further, if the image type of the target environment image is the first type, the determining the mapping area in the target environment image of the image set includes: setting the target environment image according to the shooting parameters when shooting the target environment image. The camera internal parameters and camera external parameters of the target environment image; and the mapping area of the cavity area on the target environment image is determined according to the camera internal parameters and the camera external parameters set for the target environment image.

That is to say, for the image that has not recovered the internal and external parameters of the camera in SfM, directly use the parameter information that comes with the image to set the internal and external parameters of the camera, and then project the void boundary on these target environment images based on the corresponding internal and external camera parameters. According to the area of a mapping area circled on the target environment images after the boundary of the cavity is projected onto the target environment image, it is judged whether the mapping area belongs to a preset type such as a water surface area.

In one embodiment, the setting the camera internal parameters and camera external parameters of the target environment image according to the shooting parameters when the target environment image is taken includes: according to the camera focal length information and the image when the target environment image is taken. The main point information sets the camera internal parameters of the target environment image; the camera's translation vector is set according to the camera positioning information when the target environment image is taken, and the camera's translation vector is set according to the pan/tilt angle information when the target environment image is taken. Rotation matrix; set the camera external parameters of the target environment image according to the translation vector and the rotation matrix.

For the above-mentioned projection of the cavity area on the target environment image, the determination of the mapping area on the target environment image is actually based on the camera internal parameters and camera external parameters of the image acquisition device, and the points on the cavity boundary of the cavity area are in the coordinate system To determine the conversion between. The points on the initial DSM can be points in the geodetic coordinate system. According to the corresponding relationship between the geodetic coordinate system and the world coordinate system, the points in the geodetic coordinate system can be projected to the world coordinate system, and the mapping area of the cavity area is mainly determined by the world The three-dimensional points in the coordinate system are transformed into the image coordinate system after coordinate system conversion. In some scenarios, it is not necessary to convert the points on the initial DSM to the world coordinate system. When determining the mapping area of the cavity area, directly convert the points on the cavity area of the DSM under the geodetic coordinate system from the geodetic coordinate system. Converted to the image coordinate system, the mapping area can be determined on the corresponding target environment image.

The coordinates of any image point (u, v) on the image coordinate system (that is, a certain pixel on the target environment image) in the image space coordinate system are (u, v, f). The world coordinate system uses the x-axis in the north direction, the y-axis in the east direction, and the z-axis in the opposite direction of gravity. Using RTK (longitude, latitude, altitude) positioning information, the coordinates (x, y, z) and PTZ angle (pitch α, roll β, translation γ) can be converted into a rotation matrix R and a translation transformation matrix T from the world coordinate system to the image space coordinate system by the following formula, and from the image space coordinate system to the world coordinate system The rotation matrix of r=R ^-1 , and the translation matrix t=(x,y,z) ^T. The rotation matrix and translation matrix are the external camera parameters. The point in the image space coordinate system is converted to the world coordinate system X=r.(u,v,f)+t. Conversely, the conversion formula for converting a three-dimensional point in the world coordinate system to the image space coordinate system where the camera is located is: (u, v, f) = R.X+T, and then the point on the image coordinate system can be converted and projected (u, v), determine the pixel position of the grid unit at the boundary of the hole on the target environment image.

T=-R(α, β, γ)*(x, y, z) ^T

In an embodiment, the S202 may further include: determining the image type of the target environment image; if the image type is the second type, calling a preset type recognition model to analyze and recognize the target environment image ; Determine whether the type of the environmental area corresponding to the cavity area is a preset type according to the analysis result output by the type recognition model.

Further, in one embodiment, if the image type of the target environment image is the second type, the determining the mapping area in the target environment image of the image collection includes: The internal camera parameters and external camera parameters of the target environment image determined in the process of generating the initial digital surface model are set as the camera internal parameters and camera external parameters of the target environment image; according to the camera internal parameters set for the target environment image And external camera parameters to determine the mapping area of the cavity area on the target environment image. In this case, the camera internal parameters and the camera external parameters may be calculated based on the aforementioned SfM algorithm, for example. For the second type of target environment image, the image processing equipment projects the boundary of the hole on the target environment image that can restore the camera internal and external camera parameters normally based on the SfM algorithm, etc., and combines the texture information and texture information of the second type of target environment image. The structural information of the cavity area is determined by the type recognition model generated by the machine learning method to determine whether the area circled by the cavity boundary on the image belongs to the water area.

In one embodiment, the type recognition model analyzes and obtains the type of the environment area corresponding to the cavity area based on the texture feature information of the target environment image and the structure information of the cavity area. Wherein, in one embodiment, the texture feature information of the target environment image includes: the pixel color feature of the mapping area in the target environment image; the structure information of the cavity area includes: the elevation of the adjacent area of the cavity area Value characteristics, and/or area characteristics of the void area. The type recognition model may be a two-classification model based on a positive sample training database based on the structure information of a preset type of environmental image such as a water surface area and the structural information of the hollow area of the DSM, and a preset type of environmental image such as no water surface area. The negative sample training database of the structural information of the hollow area of its DSM is used to optimize an initial type recognition model to obtain the final type recognition model.

In other embodiments, there may be other void areas that do not belong to the preset type in the initial DSM. In this case, the elevation value of each grid cell in the adjacent area is calculated according to the second confirmation rule to obtain multiple Elevation update value; the obtained multiple elevation update values are used as the elevation values of the grid cells in the other void areas that do not belong to the preset type. The second confirmation rule may refer to some existing rules for filling the hollow areas in the DSM, or the default filling rules. The second confirmation rule may also be that the elevation values of these other hollow areas are not updated. The elevation value in the cavity area is still abnormal.

In the embodiment of the present invention, after the initial digital surface model is generated based on the environmental image, if the initial digital surface model has large void areas, it can be projected on the environmental image based on these void areas. The projection area of the cavity area is used to determine whether these cavity areas are caused by environmental areas such as the water surface. If so, the adjacent adjacent areas are used to update the elevation value of the cavity area to obtain the final DSM. On the one hand, a relatively complete DSM can be obtained. On the other hand, relatively flat areas such as water surface areas are updated based on the elevation values of adjacent areas as a reference, which makes the final DSM more accurate.

Please refer to FIG. 3 again, which is a schematic diagram of one of the specific implementations of the construction of the digital surface model of the embodiment of the present invention. At the same time, the initial DSM of the embodiment of the present invention will be described in conjunction with FIG. 4, and it is understandable that FIG. The initial DSM is only a brief schematic description of each special area and grid unit on the initial DSM of the embodiment of the present invention, and does not represent an actual DSM. In the embodiment of the present invention, the construction of the digital surface model can also be implemented by the aforementioned image processing equipment. In the embodiment of the present invention, the construction process includes the following steps.

S301: Collect a set of images by the drone to obtain an image set of the target environment area.

Optionally, the environment image in the image collection includes RTK (longitude, latitude, altitude) and pan/tilt angle (rotation angle, pitch angle, yaw angle) information when the environment image was taken, and the camera calibration before leaving the factory Camera internal parameter information, which can be recorded in a preset field corresponding to the environment image, for example, it can be an extended XMP field.

S302: Estimate the internal and external camera parameters corresponding to each environmental image in the image set by SfM, that is, the aforementioned camera internal parameters and camera external parameters.

S303: According to the estimated camera internal parameters and camera external parameters, perform dense matching of multiple images to obtain a dense point cloud of the scene, and divide grid units to obtain an initial DSM of the target environment area. Steps S301 to S303 correspond to the steps of generating an initial digital surface model according to the collected image set mentioned in the above embodiment. Based on the camera internal parameters and the camera external parameters to obtain a dense point cloud and obtain the initial DSM, some existing implementation methods can be used, or in other words, the construction of the DSM can be implemented in an existing method.

S304: Mark grid cells that do not contain dense points in the initial DSM as hollow areas.

S305: Perform denoising processing on the initial DSM. Aiming at some isolated noise points contained in the hole region in the initial DSM, a median filter is used to filter the noise in the initial DSM. The above steps S304 and S305 correspond to the step of determining the void area in the initial digital surface model in the previous embodiment.

S306: Determine the void area in the initial DSM. Starting from a grid point in the cavity area, a connected area search is performed to detect all the cavity areas in the DSM, and the area of each cavity area is recorded, and the cavity areas with an area greater than a certain threshold T are selected and recorded as H _big , And mark the boundary grid cell M _{edge of the} hole. As shown in Figure 4, each grid represents a grid cell of the initial DSM, where the part filled in white is the normal area 401 formed by grid cells with recorded elevation values, and the grid filled in gray means no elevation values are recorded. The grid unit is the grid unit corresponding to the void area. These grid cells constitute a hollow area 402 and

hollow areas

403, 404 and so on. Wherein, the area of the cavity region 402 is greater than a certain threshold T, and therefore, the cavity region 402 is the aforementioned H _big . The

void areas

403 and 404 may not be processed or the elevation value filling processing may be performed according to some existing methods. In the hollow area 402, the partial grid cell filled with "cross" is the boundary grid cell 4021, that is, M _edge .

S307: Determine that the images of the camera internal parameters and camera external parameters are not recovered in the SfM (denoted as I _lose ), that is: the first type of target environment image, and perform projection processing on the cavity area to determine the environment corresponding to the cavity area Whether the area is a preset type.

Among them, directly use the image's own focal length f and the principal point coordinates to set the camera internal parameter matrix. Optionally, the focal length and principal point coordinates of the image can be automatically stored in the XMP field of the captured image when the image is captured, and can be directly extracted and obtained during use. The camera internal reference is:

Among them, cx=ImageWidth/2, cy=ImageHeight/2.

Use RTK information to set the camera's translation vector t, use the pan/tilt angle to set the camera's rotation matrix R, and then get the projection matrix P=[R∣t] according to the camera's internal and external parameters. The external parameters of the camera can be determined according to the projection matrix. According to the camera internal parameters and camera external parameters, the boundary grid unit M _{edge of the} cavity area can be projected onto the image I _lose to obtain the two-dimensional point of the image: m _edge =PM _edge , calculate the number of pixels in the area circled by m _edge , if it occupies the image If the ratio of the total number of pixels (the total number of pixels can be considered as the area of the mapping area) is greater than a certain threshold, it is determined as a water surface area, and the type of the cavity area is determined to be a preset type.

S308: Determine the target environment image of the camera internal parameters and the camera external parameters recovered in SfM, that is, the second type of target environment image, classify the images that can recover the camera internal parameters and the camera external parameters based on the classifier, and determine the corresponding cavity area Whether the environment area of the is of the preset type.

The preset type includes the type of the water surface area. In one embodiment, a Support Vector Machines (SVM) classifier used to determine whether the cavity area belongs to the water surface area can be pre-trained. The method is as follows: first collect positive samples DSM, the environmental area corresponding to the void area on the DSM of the positive sample is the water surface area; and the DSM of the negative sample is collected, and the environmental area corresponding to the void area on the DSM of the negative sample is not the water surface area. The positive and negative samples corresponding to DSM statistics were elevation value of the boundary grid cells M _e of the positive and negative samples in the cavity area, calculated border height variance, referred to as E _variance, additional statistics were positive and negative samples of The area of the void area is denoted as S _hole . The target is projected onto the surrounding image M _e SfM successfully recover the camera and the camera external parameters of the internal control (referred to as I _success), the two-dimensional projection image area statistics circled points on the image of the target environment (i.e., the mapping region void region ), calculate the color _median (R _median , G _median , B _median ) and variance (R _variance , G _variance , B _variance ); the boundary elevation variance E _variance , the area S of the cavity area will be obtained _Hole , the color median and variance are combined into an 8-dimensional feature vector, namely:

V _hole = (E _variance , S _hole , R _median , G _median , B _median , R _variance , G _variance , B _variance ).

Then train the SVM classifier according to V _hole . After passing a large number of positive samples and negative samples, the type recognition model can be optimized and trained.

In the type recognition model obtained by training, the feature vector V _{hole is} calculated according to the target environment image that can recover the camera internal parameters and the camera external parameters, and the corresponding DSM hollow area, which is input to the aforementioned SVM classifier, the type recognition model In the process of determining whether the cavity area belongs to the water surface area, if so, it is determined that the environmental area corresponding to the cavity area is a preset type.

S309: After determining that the type of the environmental area corresponding to the cavity area is a preset type, determine the elevation value of each grid cell in the adjacent area of the cavity area on the initial digital surface model. As shown in FIG. 4, the grid cells filled in white and marked by a plurality of diagonal lines constitute an adjacent area 405. Each grid cell in the adjacent area 405 has an elevation value, and the elevation value of each grid cell in the hollow area 402 can be updated with reference to the elevation value of these grid cells. It should still be noted that FIG. 4 is only an example, and the area of the adjacent area 405 may be larger, and more grid cells may be covered.

Whether the type of the environmental area corresponding to the above-mentioned S306 to S309 and the detection cavity area mentioned in the previous embodiment is the preset type, and if the type of the environmental area corresponding to the cavity area is the preset type, it is determined The steps corresponding to the elevation value of each grid cell in the adjacent area of the cavity area on the initial digital surface model.

S310: Sort the elevation values of the grid cells in the adjacent area of the M _edge in the cavity area of the initial DSM, and take the smaller one as the elevation value of the entire water area to fill the elevation value of the cavity area. For example, among 100 elevation values, after sorting from smallest to largest, the 10th smallest elevation value is used to fill the elevation value of the void area. Or, the elevation value that appears most frequently among the elevation values of the grid cells in the adjacent area is used to fill the elevation value of the void area. In step S310, it is mainly aimed at updating and filling the grid cells in the hole region 402.

S311: Use the void boundaries of other void regions as seed points to fill the remaining small voids in the DSM through the method of region growth. The S310 and S311 correspond to the step of updating the elevation value of the grid unit in the hollow area according to the elevation value of each grid unit in the adjacent area in the previous embodiment to obtain a digital surface model. In S311, it is mainly a method of filling small-area void regions such as

void regions

403 and 404. The void boundary of the void region may refer to, for example, a grid unit with an elevation value on the left side of the void region 403, and the method of region growth For example: the elevation value of the first grid cell from the left in the hole area 403 is updated to the elevation value of the hole boundary, and the elevation value of the second grid cell is on the updated elevation value of the first grid cell The value obtained after adding a threshold.

S312: Perform orthoimage correction on each image according to the DSM, and then mosaic and fuse the corrected orthoimage to obtain the overall true image. The method of generating real radio images based on a relatively complete DSM will not be repeated here.

Please refer to FIG. 5 again, which is a schematic structural diagram of an image processing device according to an embodiment of the present invention. The image processing device of the embodiment of the present invention mainly includes a communication interface unit 501 and a processing unit 502. In some embodiments, the image processing device may include multiple functional units in hardware as required, for example, may include a user interface unit 503, a battery power supply unit, a charging unit, a storage unit 504, and so on.

Wherein, the user interface unit 503 may be used for a touch display screen that receives user input and displays corresponding data, images, etc., and may also include a physical button or even a mouse input unit. The communication interface unit 501 may include a wireless communication interface and a wired communication interface. The wireless communication interface may be a WiFi interface, a radio frequency communication interface or even a mobile communication interface (for example, a 4G communication interface, a 5G communication interface, etc.), and the communication interface unit 501 is used to receive data such as environmental images.

The storage unit 504 may include a volatile memory (volatile memory), such as a random-access memory (random-access memory, RAM); the storage unit 504 may also include a non-volatile memory (non-volatile memory), such as fast Flash memory (flash memory), solid-state drive (SSD), etc.; the storage unit 504 may also include a combination of the foregoing types of memories.

The processing unit 502 may be a central processing unit (CPU). The processing unit 502 may further include a hardware chip. The aforementioned hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), etc. The above-mentioned PLD may be a field-programmable gate array (FPGA), a generic array logic (GAL), etc.

Optionally, the storage unit 504 is also used to store program instructions. The processing unit 502 may call the program instructions as needed to implement the relevant methods in the corresponding embodiments in FIG. 1b, FIG. 2 and FIG. 3 respectively. Specifically, the processing unit 502 is configured to generate an initial digital surface model according to the collected image set; determine a hole area in the initial digital surface model, wherein the hole area includes a grid with abnormal elevation values Unit; if the type of the environmental area corresponding to the cavity area is a preset type, determine the elevation value of each grid cell in the adjacent area of the cavity area on the initial digital surface model; according to each grid in the adjacent area The elevation value of the grid unit is updated to obtain the digital surface model by updating the elevation value of the grid unit in the hollow area.

In one embodiment, the processing unit 502 is further configured to determine a mapping area in the target environment image of the image set, where the mapping area is a projection of the cavity area on the target environment image; Perform image analysis on the mapping area in the target environment image, and determine whether the type of the environment area corresponding to the cavity area is a preset type according to the analysis result.

In one embodiment, the processing unit 502 is configured to, if the analysis result of the image analysis is that the mapping area is a water surface area, the type of the environment area corresponding to the cavity area is a preset type.

In one embodiment, the processing unit 502 is configured to: determine the image type of the target environment image; if the image type of the target environment image is the first type, analyze the mapping area in the target environment image When the analysis result is that the area of the mapping area is greater than the preset area threshold, it is determined that the type of the environmental area corresponding to the cavity area is the preset type.

In one embodiment, when in the process of generating the initial digital surface model based on the collected image collection, the internal camera parameters and external camera parameters of the target environment image in the image collection cannot be determined, then the target The image type of the environment image is the first type.

In one embodiment, if the image type of the target environment image is the first type, the processing unit 502 is configured to set the camera internal parameters of the target environment image according to the shooting parameters when shooting the target environment image And camera external parameters; according to the camera internal parameters and camera external parameters set for the target environment image, determine the mapping area of the cavity area on the target environment image.

In one embodiment, the processing unit 502 is configured to set the camera internal parameters of the target environment image according to the camera focal length information and image principal point information when shooting the target environment image; The camera positioning information is used to set the camera's translation vector, and the camera's rotation matrix is set according to the pan/tilt angle information when the target environment image is taken; the camera outside of the target environment image is set according to the translation vector and the rotation matrix Participate.

In one embodiment, the processing unit 502 is configured to determine the image type of the target environment image; if the image type is the second type, call a preset type recognition model to analyze the target environment image Recognition; determining whether the type of the environmental area corresponding to the cavity area is a preset type according to the analysis result output by the type recognition model.

In one embodiment, when in the process of generating the initial digital surface model according to the collected image collection, the internal camera parameters and external camera parameters of the target environment image in the image collection can be determined, then the target environment The image type of the image is the second type.

In one embodiment, if the image type of the target environment image is the second type, the processing unit 502 is configured to combine the image determined in the process of generating the initial digital surface model according to the collected image set. The camera internal parameters and camera external parameters of the target environment image are set as the camera internal parameters and camera external parameters of the target environment image; according to the camera internal parameters and camera external parameters set for the target environment image, it is determined that the cavity area is in the The mapped area on the target environment image.

In one embodiment, the type recognition model analyzes and obtains the type of the environment area corresponding to the cavity area based on the texture feature information of the target environment image and the structure information of the cavity area.

In an embodiment, the texture feature information of the target environment image includes: the pixel color feature of the mapping area in the target environment image; the structure information of the cavity area includes: the elevation value feature of the adjacent area of the cavity area , And/or the area characteristics of the cavity area.

In one embodiment, the cavity area of the determined cavity area on the initial digital surface model is greater than a preset cavity area threshold.

In an embodiment, the void area is determined from the initial digital surface model according to a connected area search detection rule.

In one embodiment, the processing unit 502 is configured to determine the obtained elevation update value according to the elevation value of each grid unit in the adjacent area according to the first confirmation rule; use the elevation update value as the hole area The elevation value of the grid cell.

In one embodiment, the processing unit 502 is further configured to, if the type of the environmental area corresponding to the cavity area is not a preset type, perform a calculation of the elevation value of each grid cell in the adjacent area according to the second confirmation rule Perform calculations to obtain multiple elevation update values; and use the obtained multiple elevation update values as the elevation values of the grid cells in the hollow area.

In the embodiment of the present invention, for the specific implementation of the processing unit, reference may be made to the description of the relevant content in the foregoing embodiment, which is not repeated here. In addition, after the processing unit generates the initial digital surface model based on the environmental image, if the initial digital surface model has large void areas, it can in turn project on the environmental image based on these void areas. The projection area of the cavity area is used to determine whether these cavity areas are caused by environmental areas such as the water surface. If so, the adjacent adjacent areas are used to update the elevation value of the cavity area to obtain the final DSM. On the one hand, a relatively complete DSM can be obtained. On the other hand, relatively flat areas such as water surface areas are updated based on the elevation values of adjacent areas as a reference, which makes the final DSM more accurate.

In addition, an embodiment of the present invention also provides an image processing system, the image processing system includes: a mobile platform and an image processing device, the mobile platform is provided with an image acquisition device; the mobile platform is used to pass the The image capture device captures multiple environmental images during the movement of the mobile platform, and sends the multiple captured environmental images to the image processing device; the image processing device is configured to collect images based on the Collectively generate an initial digital surface model; determine the cavity area in the initial digital surface model, wherein the cavity area includes grid cells with abnormal elevation values; if the type of the environmental area corresponding to the cavity area is pre- Set the type, determine the elevation value of each grid cell in the adjacent area of the cavity area on the initial digital surface model; update the grid cell in the cavity area according to the elevation value of each grid cell in the adjacent area The elevation value of, get the digital surface model. Please refer to FIG. 1 for a schematic diagram of the mobile platform, and the image processing device refers to the image processing device shown in the embodiment corresponding to FIG. 5.

A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing relevant hardware through a computer program. The program can be stored in a computer readable storage medium. During execution, it may include the procedures of the above-mentioned method embodiments. Wherein, the storage medium may be a magnetic disk, an optical disc, a read-only memory (Read-Only Memory, ROM), or a random access memory (Random Access Memory, RAM), etc.

The above-disclosed are only some embodiments of the present invention, which of course cannot be used to limit the scope of rights of the present invention. Therefore, equivalent changes made according to the claims of the present invention still fall within the scope of the present invention.

Claims

A method for constructing a digital surface model, which is characterized in that it comprises:

Generate an initial digital surface model based on the collected image collection;

Determining a hollow area in the initial digital surface model, wherein the hollow area includes grid cells with abnormal elevation values;

If the type of the environmental area corresponding to the cavity area is a preset type, determining the elevation value of each grid cell in the adjacent area of the cavity area on the initial digital surface model;

According to the elevation value of each grid unit in the adjacent area, the elevation value of the grid unit in the hollow area is updated to obtain a digital surface model.
The method according to claim 1, wherein after determining the void area in the initial digital surface model, the method further comprises:

Determining a mapping area in the target environment image of the image set, where the mapping area is a projection of the cavity area on the target environment image;

Perform image analysis on the mapping area in the target environment image, and determine whether the type of the environment area corresponding to the cavity area is a preset type according to the analysis result.
The method of claim 2, further comprising:

If the analysis result of the image analysis is that the mapping area is a water surface area, the type of the environmental area corresponding to the cavity area is a preset type.
The method of claim 2 or 3, wherein the image analysis is performed on the mapped area in the target environment image, and according to the analysis result, it is determined whether the type of the environment area corresponding to the cavity area is a predetermined Set types, including:

Determining the image type of the target environment image;

If the image type of the target environment image is the first type, analyzing the area of the mapping area in the target environment image;

When the analysis result is that the area of the mapping area is greater than the preset area threshold, it is determined that the type of the environmental area corresponding to the hollow area is the preset type.
The method according to claim 4, characterized in that, in the process of generating the initial digital surface model according to the collected image collection, it is impossible to determine the camera internal parameters and the camera external parameters of the target environment image in the image collection. Time reference, the image type of the target environment image is the first type.
The method according to claim 5, wherein if the image type of the target environment image is the first type, the determining a mapping area in the target environment image of the image set comprises:

Setting the camera internal parameters and camera external parameters of the target environment image according to the shooting parameters when shooting the target environment image;

Determine the mapping area of the cavity area on the target environment image according to the camera internal parameters and the camera external parameters set for the target environment image.
7. The method according to claim 6, wherein the setting the camera internal parameters and camera external parameters of the target environment image according to the shooting parameters when the target environment image is taken, comprises:

Setting the camera internal parameters of the target environment image according to the camera focal length information and image principal point information when the target environment image is taken;

Setting the translation vector of the camera according to the camera positioning information when the target environment image is taken, and setting the rotation matrix of the camera according to the pan/tilt angle information when the target environment image is taken;

The external camera parameters of the target environment image are set according to the translation vector and the rotation matrix.
The method of claim 2 or 3, wherein the image analysis is performed on the mapped area in the target environment image, and according to the analysis result, it is determined whether the type of the environment area corresponding to the cavity area is a predetermined Set types, including:

Determining the image type of the target environment image;

If the image type is the second type, call a preset type recognition model to analyze and recognize the target environment image;

According to the analysis result output by the type recognition model, it is determined whether the type of the environmental area corresponding to the cavity area is a preset type.
The method according to claim 8, characterized in that, in the process of generating the initial digital surface model according to the collected image collection, the internal camera parameters and external camera parameters of the target environment image in the image collection can be determined When, the image type of the target environment image is the second type.
The method of claim 9, wherein if the image type of the target environment image is the second type, the determining the mapping area in the target environment image of the image set comprises:

Setting the camera internal parameters and camera external parameters of the target environment image determined in the process of generating the initial digital surface model according to the collected image set as the camera internal parameters and camera external parameters of the target environment image;

Determine the mapping area of the cavity area on the target environment image according to the camera internal parameters and the camera external parameters set for the target environment image.
The method according to claim 8, wherein the type recognition model is based on the texture feature information of the target environment image and the structure information of the cavity region to analyze and obtain the environment region corresponding to the cavity region Types of.
The method of claim 11, wherein:

The texture feature information of the target environment image includes: the pixel color feature of the mapping area in the target environment image;

The structural information of the cavity region includes: the feature of the elevation value of the adjacent region of the cavity region, and/or the area feature of the cavity region.
The method according to claim 1, wherein the cavity area of the determined cavity area on the initial digital surface model is greater than a preset cavity area threshold.
The method according to claim 1, wherein the hollow area is determined from the initial digital surface model according to a connected area search and detection rule.
The method according to claim 1, wherein the updating the elevation value of the grid unit in the hollow area according to the elevation value of each grid unit in the adjacent area comprises:

Determine the obtained elevation update value according to the elevation value of each grid cell in the adjacent area according to the first confirmation rule;

The elevation update value is used as the elevation value of the grid unit in the hollow area.
The method of claim 1, wherein:

If the type of the environmental area corresponding to the hollow area is not the preset type, the elevation value of each grid cell in the adjacent area is calculated according to the second confirmation rule to obtain multiple elevation update values;

The obtained multiple updated elevation values are used as the elevation values of the grid cells in the hollow area.
An image processing device, characterized in that the image processing device comprises: a communication interface unit and a processing unit, wherein:

The communication interface unit is used to receive environmental images;

The processing unit is configured to generate an initial digital surface model according to the collected image set; determine the cavity area in the initial digital surface model, wherein the cavity area includes grid units with abnormal elevation values; If the type of the environmental area corresponding to the cavity area is a preset type, the elevation value of each grid cell in the adjacent area of the cavity area on the initial digital surface model is determined; according to the elevation value of each grid cell in the adjacent area Value, update the elevation value of the grid unit in the hollow area to obtain a digital surface model.
The image processing device according to claim 17, wherein the processing unit is further used for

Determining a mapping area in the target environment image of the image set, where the mapping area is a projection of the cavity area on the target environment image;

Perform image analysis on the mapping area in the target environment image, and determine whether the type of the environment area corresponding to the cavity area is a preset type according to the analysis result.
The image processing device according to claim 18, wherein the processing unit is configured to

If the analysis result of the image analysis is that the mapping area is a water surface area, the type of the environmental area corresponding to the cavity area is a preset type.
The image processing device according to claim 18 or 19, wherein the processing unit is configured to:

Determining the image type of the target environment image;

If the image type of the target environment image is the first type, analyzing the area of the mapping area in the target environment image;

When the analysis result is that the area of the mapping area is greater than the preset area threshold, it is determined that the type of the environmental area corresponding to the hollow area is the preset type.
The image processing device according to claim 20, wherein, in the process of generating the initial digital surface model based on the collected image collection, it is impossible to determine the camera internal parameters of the target environment image in the image collection and When the camera is externally referenced, the image type of the target environment image is the first type.
The image processing device according to claim 21, wherein if the image type of the target environment image is the first type, the processing unit is configured to

Setting the camera internal parameters and camera external parameters of the target environment image according to the shooting parameters when shooting the target environment image;

Determine the mapping area of the cavity area on the target environment image according to the camera internal parameters and the camera external parameters set for the target environment image.
The image processing device according to claim 22, wherein the processing unit is configured to

Setting the camera internal parameters of the target environment image according to the camera focal length information and image principal point information when the target environment image is taken;

Setting the translation vector of the camera according to the camera positioning information when the target environment image is taken, and setting the rotation matrix of the camera according to the pan/tilt angle information when the target environment image is taken;

The external camera parameters of the target environment image are set according to the translation vector and the rotation matrix.
The image processing device according to claim 18 or 19, wherein the processing unit is configured to

Determining the image type of the target environment image;

If the image type is the second type, call a preset type recognition model to analyze and recognize the target environment image;

According to the analysis result output by the type recognition model, it is determined whether the type of the environmental area corresponding to the cavity area is a preset type.
The image processing device according to claim 24, characterized in that, in the process of generating the initial digital surface model based on the collected image collection, the camera internal parameters and camera parameters of the target environment image in the image collection can be determined. For external reference, the image type of the target environment image is the second type.
The image processing device of claim 25, wherein if the image type of the target environment image is the second type, the processing unit is configured to

Setting the camera internal parameters and camera external parameters of the target environment image determined in the process of generating the initial digital surface model according to the collected image set as the camera internal parameters and camera external parameters of the target environment image;

Determine the mapping area of the cavity area on the target environment image according to the camera internal parameters and the camera external parameters set for the target environment image.
The image processing device according to claim 24, wherein the type recognition model is based on the texture feature information of the target environment image and the structural information of the cavity region to obtain the environment corresponding to the cavity region by analysis. The type of area.
The image processing device according to claim 27, wherein:

The texture feature information of the target environment image includes: the pixel color feature of the mapping area in the target environment image;

The structural information of the cavity region includes: the feature of the elevation value of the adjacent region of the cavity region, and/or the area feature of the cavity region.
18. The image processing device according to claim 17, wherein the cavity area of the determined cavity area on the initial digital surface model is greater than a preset cavity area threshold.
The image processing device according to claim 17 or 29, wherein the hollow area is determined from the initial digital surface model according to a connected area search detection rule.
The image processing device according to claim 17, wherein the processing unit is configured to

Determine the obtained elevation update value according to the elevation value of each grid cell in the adjacent area according to the first confirmation rule;

The elevation update value is used as the elevation value of the grid unit in the hollow area.
The image processing device according to claim 17, wherein the processing unit is further used for

If the type of the environmental area corresponding to the hollow area is not the preset type, the elevation value of each grid cell in the adjacent area is calculated according to the second confirmation rule to obtain multiple elevation update values;

The obtained multiple updated elevation values are used as the elevation values of the grid cells in the hollow area.
An image processing system, characterized in that the image processing system comprises: a mobile platform and an image processing device, and an image acquisition device is provided on the mobile platform;

The mobile platform is configured to collect multiple environmental images during the movement of the mobile platform through the image acquisition device, and send the multiple collected environmental images to the image processing device;

The image processing device is configured to generate an initial digital surface model according to the collected image set; determine the cavity area in the initial digital surface model, wherein the cavity area includes grid cells with abnormal elevation values; if If the type of the environmental area corresponding to the cavity area is a preset type, the elevation value of each grid unit in the adjacent area of the cavity area on the initial digital surface model is determined; according to the value of each grid unit in the adjacent area The elevation value is to update the elevation value of the grid unit in the hollow area to obtain a digital surface model.