WO2021046861A1

WO2021046861A1 - Orthographic image generation method and system, and storage medium

Info

Publication number: WO2021046861A1
Application number: PCT/CN2019/105922
Authority: WO
Inventors: 张明磊; 宋孟肖; 梁家斌; 田艺; 李思晋
Original assignee: 深圳市大疆创新科技有限公司
Priority date: 2019-09-12
Filing date: 2019-09-16
Publication date: 2021-03-18

Abstract

An orthographic image generation method (100) and system (700), and a storage medium. The method (100) comprises: acquiring a plurality of images, obtained by means of a plurality of cameras photographing a target measurement area, of the target measurement area, wherein the plurality of cameras are arranged on a mobile platform (S110); acquiring imaging pose information of a main image from among the plurality of images, and generating, according to the main image and the imaging pose information, elevation data for fitting the elevation of the target measurement area (S120); and according to the elevation data and the imaging pose information, correcting other images, apart from the main image, from among the plurality of images, so as to obtain orthographic images corresponding to the other images (S130). According to the orthographic image generation method (100) and system (700), and the storage medium, acquired imaging pose information of a main image and elevation data of a measurement area are used to generate orthographic images corresponding to other images, such that the amount of calculation needed to acquire a plurality of orthographic images in a collection scenario of a plurality of cameras can be reduced, thereby improving the generation efficiency of the orthographic images.

Description

Ortho image generation method, system and storage medium

Manual

Technical field

The present invention generally relates to the technical field of orthophotos, and more specifically to a method, system and storage medium for generating orthophotos.

Background technique

At present, the strategy used to generate orthophotos using multispectral images is to obtain the imaging pose information of each band image, and generate the orthophotos of each band image based on the imaging pose information of each band image. It is worth noting that the non-directional movement of the mobile platform is accompanied by the non-directional movement of the mobile platform during the image acquisition process. For each waveband image, a large number of calculation steps are required to reconstruct the imaging pose information when the image is acquired. Therefore, the above method requires a large amount of calculation due to the need to obtain the imaging pose information of each waveband image, and the generation efficiency of the orthoimage is low.

Summary of the invention

In order to solve the above-mentioned problems, the present invention has been proposed. The present invention provides an orthophoto generation scheme, which uses the imaging pose information of the main image and the elevation data of the survey area in the acquired multispectral image to generate orthoimages corresponding to other images. In this way, the number of multiple cameras can be reduced. The calculation amount of acquiring multiple orthophotos in the acquisition scene improves the generation efficiency of orthophotos. The following briefly describes the orthophoto generation scheme proposed by the present invention, and more details will be described in the specific implementation in conjunction with the accompanying drawings.

According to one aspect of the present invention, there is provided a method for generating orthophotos, the method comprising: acquiring a plurality of images of the target measurement area obtained by multiple cameras shooting the target measurement area, wherein the multiple cameras Set on a mobile platform; acquire imaging pose information of the main image in the plurality of images, and generate elevation data for fitting the elevation of the target survey area according to the main image and the imaging pose information; and According to the elevation data and the imaging pose information, other images in the plurality of images other than the main image are corrected to obtain orthoimages corresponding to the other images.

In an embodiment of the present invention, the acquiring multiple images of the target measurement area obtained by multiple cameras shooting the target measurement area includes: acquiring multiple cameras to shoot the target measurement area within the same exposure period The obtained multiple images of the target measurement area, wherein the duration of the exposure period is less than a preset duration threshold.

In an embodiment of the present invention, the acquiring multiple images of the target measurement area obtained by shooting by multiple cameras on the target measurement area further includes: acquiring multiple cameras on the target measurement area during multiple exposure periods. The multiple images of the target measurement area obtained by shooting, wherein the main images taken in each exposure period are taken by the same camera of the multiple cameras; and the imaging position of the main image is obtained The posture information includes: performing reconstruction processing on the main image captured during the respective exposure periods through the SFM algorithm to obtain the imaging posture information of the main image.

In an embodiment of the present invention, the types of the images include at least two of RGB images, near-infrared images, red-band images, and green-band images.

In an embodiment of the present invention, the main image is an RGB image.

In an embodiment of the present invention, the method further includes: before correcting the other images, preprocessing the other images by correction parameters; and the correction includes: according to the elevation data and the imaging The pose information is used to correct the preprocessed other images to obtain an orthoimage corresponding to the other images.

In an embodiment of the present invention, the preprocessing of the other images by correcting parameters includes: aligning the other images with the main image; and/or comparing the main image and the other images Perform radiation correction processing.

In an embodiment of the present invention, the aligning the other image with the main image includes: determining an image point with the same name between the main image and the other image, and determining the image point with the same name based on the image point with the same name. The other image is mapped to a mapping transformation matrix of the main image, and the other image is mapped to the main image based on the mapping transformation matrix.

In an embodiment of the present invention, the aligning the other image with the main image includes: determining the other image based on respective imaging pose information of the main image and the other image recorded by a sensor A mapping relationship with the main image, and mapping the other images to the main image based on the mapping relationship.

In an embodiment of the present invention, the method further includes: before aligning the other image with the main image, determining respective gradient information of the main image and the other image, and based on the main image The gradient information corresponding to each of the other images aligns the other images with the main image.

In an embodiment of the present invention, the method further includes: before aligning the other image with the main image, performing distortion correction processing on the main image and the other image respectively, and correcting the distortion based on the distortion. The main image and the other images of align the other images with the main image.

In an embodiment of the present invention, the distortion correction processing is based on the internal parameters and distortion parameters of the multiple cameras.

In an embodiment of the present invention, the elevation data is generated based on a point cloud, and the point cloud is generated based on the imaging pose information.

In an embodiment of the present invention, the point cloud is a dense point cloud generated based on the imaging pose information.

In an embodiment of the present invention, the point cloud is based on a sparse point cloud obtained by performing an SFM algorithm on the main image.

In an embodiment of the present invention, the method further includes: mosaic fusion of the respective orthoimages of the images captured by the same camera to generate orthoimages of various bands.

In an embodiment of the present invention, the method further includes: generating a vegetation index map based on the orthophotos of the respective bands.

In an embodiment of the present invention, the method further includes: performing pseudo-color visual rendering on the vegetation index map.

In an embodiment of the present invention, the method further includes: after aligning the other image with the main image, generating a vegetation index map based on the main image and the aligned other image; and The elevation data and the imaging pose information correct the vegetation index map to obtain an orthophoto corresponding to the vegetation index map.

According to another aspect of the present invention, an orthophoto generation system is provided. The system includes a storage device and a processor. The storage device stores a computer program run by the processor. When the processor is running, the orthophoto generating method described in any one of the above is executed.

According to another aspect of the present invention, a storage medium is provided, and a computer program is stored on the storage medium, and the computer program executes the orthoimage generating method described in any one of the above items when the computer program is running.

The orthophoto generating method, system and storage medium according to the embodiments of the present invention only acquire the imaging pose information of the main image in the multispectral image, and generate elevation data fitting the elevation of the target survey area based on the imaging pose information of the main image Finally, based on the imaging pose information of the main image and the generated elevation data, other images other than the main image are corrected to obtain the orthophoto of other images, that is, the imaging pose information of the main image and the elevation of the survey area will be obtained The data is used to generate orthoimages corresponding to other images. In this way, the amount of calculation for acquiring multiple orthoimages in a multi-camera acquisition scenario can be reduced, and the generation efficiency of orthoimages can be improved.

Description of the drawings

Fig. 1 shows a schematic flowchart of a method for generating an orthophoto according to an embodiment of the present invention;

Fig. 2 shows an example diagram of multiple images according to an embodiment of the present invention;

Fig. 3 shows a schematic flowchart of a method for generating an orthophoto according to another embodiment of the present invention;

FIG. 4A shows a rolling-blind superimposed display effect of a green band image and a red edge band image before alignment according to an embodiment of the present invention;

FIG. 4B shows a rolling-blind superimposed display effect of a green band image and a red edge band image after alignment and radiation correction according to an embodiment of the present invention;

FIG. 5 shows a schematic flowchart of a method for generating an orthophoto according to another embodiment of the present invention;

6A to 6C respectively show schematic diagrams of orthoimages of different wavebands obtained by the method for generating orthoimages according to an embodiment of the present invention;

FIG. 6D shows a schematic diagram of a vegetation index map rendered by pseudo-color visualization obtained by the method for generating an orthophoto according to an embodiment of the present invention; and

Fig. 7 shows a schematic block diagram of an orthophoto generation system according to an embodiment of the present invention.

detailed description

In order to make the objectives, technical solutions, and advantages of the present invention more obvious, the exemplary embodiments according to the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments of the present invention, and it should be understood that the present invention is not limited by the exemplary embodiments described herein. Based on the embodiments of the present invention described in the present invention, all other embodiments obtained by those skilled in the art without creative work should fall within the protection scope of the present invention.

In the following description, a lot of specific details are given in order to provide a more thorough understanding of the present invention. However, it is obvious to those skilled in the art that the present invention can be implemented without one or more of these details. In other examples, in order to avoid confusion with the present invention, some technical features known in the art are not described.

It should be understood that the present invention can be implemented in different forms and should not be construed as being limited to the embodiments presented here. On the contrary, the provision of these embodiments will make the disclosure thorough and complete, and will fully convey the scope of the present invention to those skilled in the art.

The purpose of the terms used here is only to describe specific embodiments and not as a limitation of the present invention. When used herein, the singular forms "a", "an" and "the/the" are also intended to include plural forms, unless the context clearly indicates otherwise. It should also be understood that the terms "composition" and/or "including", when used in this specification, determine the existence of the described features, integers, steps, operations, elements and/or components, but do not exclude one or more other The existence or addition of features, integers, steps, operations, elements, components, and/or groups. As used herein, the term "and/or" includes any and all combinations of related listed items.

In order to thoroughly understand the present invention, detailed steps and detailed structures will be proposed in the following description to explain the technical solution proposed by the present invention. The preferred embodiments of the present invention are described in detail as follows. However, in addition to these detailed descriptions, the present invention may also have other embodiments.

As mentioned earlier, the current orthophoto generation method needs to obtain the imaging pose information of each band image, and generate the orthophoto of each band image based on the imaging pose information of each band image. This method is very computationally intensive. , The generation efficiency of orthophotos is low. Based on this, the present invention provides an orthophoto generation solution, and the orthophoto generation solution according to the embodiments of the present invention will be described below with reference to the accompanying drawings.

FIG. 1 shows a schematic flowchart of a method 100 for generating an orthophoto according to an embodiment of the present invention. As shown in FIG. 1, the orthophoto generation method 100 may include the following steps:

In step S110, multiple images of the target measurement area obtained by multiple cameras shooting the target measurement area are acquired, wherein the multiple cameras are arranged on a mobile platform.

In the embodiment of the present invention, the multiple images acquired in step S110 are obtained by shooting the target survey area by multiple cameras, and the multiple cameras are set on a mobile platform (such as a drone). During the movement of the platform, the multiple cameras capture multiple images for the target measurement area. Within the same exposure period (where the duration of the exposure period is less than the preset duration threshold), there may be more overlapping parts between the multiple images captured by the multiple cameras. In different exposure periods, the multiple images captured by the multiple cameras may have less overlap or no overlap.

It should be understood that the multiple images captured by multiple cameras described herein can be understood as: multiple cameras in the same camera capture multiple images each. For example, the same camera includes an RGB camera and a near-infrared camera, and the images taken for the target measurement area may include RGB images and near-infrared images. In addition, the multiple images captured by multiple cameras described in this article can also be understood as: multiple images captured by multiple cameras each. For example, for RGB cameras and near-infrared cameras, the images taken for the target measurement area are RGB images and near-infrared images respectively. In addition, the multiple images captured by multiple cameras described in this article can also be understood as: the same camera in the same camera captures an image, and images of multiple channels or images of multiple formats are obtained from the image. For example, an RGB image captured by an RGB camera can be used to obtain a red band image, a green band image, etc. from the RGB image. In short, based on the multiple cameras arranged on the mobile platform, multiple images of the target measurement area obtained by the multiple cameras shooting the target measurement area can be obtained. The multiple images may be multiple images of the same exposure period, and at this time, the multiple images are images of different wavebands; the multiple images may also be multiple images of multiple exposure periods, and the multiple images may be multiple images of multiple exposure periods. Images in the same exposure period are images of different wavebands, and images in different exposure periods may be images of different wavebands, or images of the same waveband.

The multiple images described above can be understood with reference to FIG. 2. FIG. 2 shows an example diagram of multiple images according to an embodiment of the present invention. As shown in FIG. 2, images with four rows and six columns are exemplarily shown. Among them, the images of each row are images of different wavebands in the same exposure period, and the images of each column are images of the same waveband in different exposure periods. That is to say, in FIG. 2, images of four exposure periods and six bands are exemplarily shown. It should be understood that the example shown in FIG. 2 is only exemplary, and actual applications may include other numbers of exposure periods and other numbers of bands.

Next, referring to FIG. 1, the subsequent steps of the orthophoto generation method 100 according to the embodiment of the present invention will be described.

In step S120, the imaging pose information of the main image in the plurality of images is acquired, and the elevation data for fitting the elevation of the target survey area is generated according to the main image and the imaging pose information.

In the embodiment of the present invention, from the multiple images obtained in step S110, only the imaging pose information of the main image can be obtained. Wherein, the main image may be an image of any waveband among the plurality of images, and all images in the plurality of images other than the main image may be referred to as auxiliary images or other images. For example, continuing to refer to the example of FIG. 2, if only multiple images (such as a row of images in FIG. 2) within the same exposure period are acquired in step S110, the main image may be any one of the multiple images. (For example, any one of the images in a row of images in FIG. 2); if multiple images in multiple exposure periods are acquired in step S110 (for example, images in four rows and six columns in FIG. 2), the main image can be all images. Any column of the multiple images (for example, any column of images in four rows and six columns in FIG. 2). Exemplarily, the main image may be an RGB image. Compared with images of other bands, RGB images have richer texture information. Therefore, it is easy to obtain accurate and reliable results by obtaining the imaging pose information of RGB images for three-dimensional reconstruction.

In one example, it is possible to determine what band of the image is based on the tag (identifier) of each image, or determine whether the image is the main image or other images. In other examples, the main image can also be obtained from multiple images in any other suitable manner, which is not limited in this application.

In the embodiment of the present invention, the main image may be reconstructed through the structure from motion (SFM) algorithm to obtain the imaging pose information of the main image. The imaging pose information of the main image may include positioning information, posture information, in-camera parameters and distortion parameters of the camera that shoots the main image when shooting the main image. Wherein, the positioning information and posture information may be the positioning information and posture information of the mobile platform on which the camera is set. Exemplarily, the posture information may include pan/tilt angle information. Exemplarily, the pan/tilt angle information may include the attitude information of the pan/tilt when the camera shoots the main image, such as the roll, pitch, or yaw angle of the pan/tilt.

After acquiring the imaging pose information of the main image, elevation data for fitting the elevation of the target survey area can be generated according to the main image and its imaging pose information. Exemplarily, a dense point cloud can be generated based on the main image and its imaging pose information, and then a digital elevation model or digital surface model of the target survey area can be generated from the dense point cloud. Exemplarily, the elevation data used to fit the elevation of the target survey area may also be generated based on the sparse point cloud generated in the foregoing SFM process. In addition, in addition to the aforementioned digital elevation model or digital surface model, the elevation data used to fit the elevation of the target survey area may also be other geometric curved surfaces or planes.

Next, referring to FIG. 1, the subsequent steps of the orthophoto generation method 100 according to an embodiment of the present invention will be described.

In step S130, according to the elevation data and the imaging pose information, other images in the plurality of images except the main image are corrected to obtain orthoimages corresponding to the other images.

In the embodiment of the present invention, based on the elevation data obtained in step S120, an orthophoto of the main image can be generated. For example, a digital differential correction algorithm is used to change the angle of view of the main image to an orthographic angle to generate the orthoimage of the main image. . For images other than the main image, in the embodiment of the present invention, the elevation data of the target measurement area obtained based on the imaging pose information of the main image is used to correct the angle of view of other images to generate orthoimages of other images, reducing One by one, the amount of calculation to obtain the imaging pose information of other bands of images, so compared to the way of obtaining the imaging pose information of each band of images to generate orthophotos, no matter how many bands of images are included, the main image can be used The image reconstruction algorithm of this band is used to obtain the imaging pose information, and then the orthoimages of other bands are generated, which can reduce the calculation amount of acquiring multiple orthoimages in the multi-camera acquisition scene, and improve the generation efficiency of orthoimages. , And can avoid the inconsistency of the reconstruction results when each band image is reconstructed, and it also eliminates the need for strict and uniform exposure time for the camera.

Hereinafter, a method for generating an orthophoto according to another embodiment of the present invention will be described with reference to FIG. 3. FIG. 3 shows a schematic flowchart of a method 300 for generating an orthophoto according to another embodiment of the present invention. As shown in FIG. 3, the orthophoto generating method 300 may include the following steps:

In step S310, multiple images of the target measurement area obtained by multiple cameras shooting the target measurement area are acquired, wherein the multiple cameras are arranged on a mobile platform.

In step S320, the imaging pose information of the main image in the plurality of images is acquired, and the elevation data for fitting the elevation of the target survey area is generated according to the main image and the imaging pose information.

In step S330, the other images are preprocessed by correcting parameters.

In step S340, according to the elevation data and the imaging pose information, the preprocessed other images are corrected to obtain orthoimages corresponding to the other images.

Step S310, step S320, and step S340 in the orthophoto generation method 300 according to the embodiment of the present application described with reference to FIG. 3 and step S110 in the orthophoto generation method 100 according to the embodiment of the present application described with reference to FIG. 1 Step S120 is similar to step S130, and for the sake of brevity, it will not be repeated here. Different from the orthophoto generating method 100 according to the embodiment of the present application described with reference to FIG. 1, the orthophoto generating method 300 according to the embodiment of the present application described with reference to FIG. 3 further includes step S330, that is, correcting other images. Previously, the other images were preprocessed by correcting parameters, and then step S340 is based on the elevation data and the imaging pose information to correct the preprocessed other images to obtain orthoimages corresponding to the other images . In this embodiment, by preprocessing other images before generating orthoimages of other images, orthoimages with higher reliability can be obtained.

Exemplarily, the preprocessing of the other images by the correction parameters may include aligning the other images with the main image, and/or performing radiometric correction processing on the main image and the other images. In this example, radiation correction processing can be performed on the main image and other images to eliminate or correct image distortion caused by radiation errors, so that subsequent processing is more accurate. In addition, you can also align other images with the main image (such as texture alignment), so that other images can be mapped to the main image to obtain the same internal and external camera parameters as the main image, so that other images can be generated based on the imaging pose information of the main image. Ortho image is more accurate and reliable. The effects of image alignment before and after alignment and radiation correction can be understood with reference to Figs. 4A and 4B. Fig. 4A shows the effect of rolling over the green band image and the red edge (RedEdeg) band image before alignment. FIG. 4B shows the superimposed display effect of the green band image and the red edge band image after alignment and radiation correction.

In an embodiment of the present invention, the aligning the other image with the main image may include: determining an image point with the same name between the main image and the other image, and determining based on the image point with the same name The other image is mapped to a mapping transformation matrix of the main image, and the other image is mapped to the main image based on the mapping transformation matrix. In one example, four (or more than four) image points with the same name can be determined between the main image and other images, and the homography transformation matrix that maps other images to the main image can be determined based on these image points with the same name, and based on this The homography transformation matrix maps other images onto the main image. In another example, image points with the same name can be determined between the main image and other images, and based on these image points, the similar transformation matrix or affine transformation matrix for mapping other images to the main image can be determined, and based on the similar transformation matrix Or the affine transformation matrix maps other images onto the main image. In other examples, image points with the same name can also be determined between the main image and other images, and based on these image points, any other transformation matrix that maps other images to the main image can be determined, and based on the transformation matrix, other images can be mapped to On the main image. In a further example, the transformation relationship of mapping other images to the main image can also be calculated by an optimization method of image correlation coefficients other than the image points of the same name. In a further example, the above alignment methods can be implemented in various combinations, such as partitioning the image, the first area is aligned by one transformation method, and the second area is aligned by another transformation method; or the alignment is performed in steps. , The first step adopts the first transformation method, the second step adopts the second transformation method, and finally the alignment is realized through multiple steps.

In another embodiment of the present invention, the aligning the other image with the main image may include: determining the respective imaging pose information of the main image and the other image recorded by a sensor A mapping relationship between other images and the main image, and mapping the other images to the main image based on the mapping relationship. In this embodiment, the position and posture of the main image and other images at respective imaging moments can be determined based on the position and posture sensors provided on the mobile platform, and the mapping relationship between the other images and the main image can be further calculated.

It is worth noting that because the main image and other images may be acquired by different camera exposures, and there may be slight differences in exposure time and pose, the brightness of different images will be different, which will lead to the difference between the main image and other images. The inter-mapping transformation is inaccurate, that is, the alignment accuracy is insufficient.

In a further embodiment of the present invention, the method 300 may further include the following steps (not shown in FIG. 3): before aligning the other image with the main image, determining the main image and the other image Respective gradient information, and align the other image with the main image based on the respective gradient information of the main image and the other image. The gradient information of the image can better reflect the edge characteristics of the image. Therefore, in this embodiment, other images are aligned with the main image based on the respective gradient information of the main image and other images, which can accurately distinguish between different objects, and further Improve the accuracy of alignment.

In a further embodiment of the present invention, the method 300 may further include the following steps (not shown in FIG. 3): before aligning the other image with the main image, separately aligning the main image and the other image The image undergoes distortion correction processing, and the other images are aligned with the main image based on the main image and the other images that have been distorted. Distortion correction processing can correct the distortion introduced by the lens accuracy and technology in the imaging process of the camera. Therefore, in this embodiment, the alignment of other images and the main image is implemented based on the other images and the main image after the distortion correction processing, which can further improve The accuracy of subsequent processing. Exemplarily, the distortion correction processing may be based on the internal parameters and distortion parameters of the multiple cameras. Wherein, the internal parameters may include parameter information such as focal length, frame, sensitivity, etc., which may include information collected via a sensor, or may be pre-calibrated. In addition, the aforementioned distortion correction processing can also be used as the aforementioned preprocessing for other images.

Based on the above description, the orthophoto generation method 300 according to another embodiment of the present invention uses the acquired imaging pose information of the main image and the elevation data of the survey area to generate orthoimages corresponding to other images. In this way, it can be reduced The amount of calculation for acquiring multiple orthophotos in a small multi-camera collection scene improves the generation efficiency of orthophotos. In addition, the orthophoto generating method 300 according to another embodiment of the present invention preprocesses the other images before generating the orthoimages of other images, so that an orthoimage with higher reliability can be obtained.

Hereinafter, a method for generating an orthoimage according to another embodiment of the present invention will be described with reference to FIG. 5. FIG. 5 shows a schematic flowchart of a method 500 for generating an orthophoto according to another embodiment of the present invention. As shown in FIG. 5, the orthophoto generation method 500 may include the following steps:

In step S510, multiple images of the target measurement area obtained by multiple cameras shooting the target measurement area are acquired, wherein the multiple cameras are set on a mobile platform.

In step S520, the imaging pose information of the main image in the plurality of images is acquired, and the elevation data for fitting the elevation of the target survey area is generated according to the main image and the imaging pose information.

In step S530, according to the elevation data and the imaging pose information, other images in the plurality of images except the main image are corrected to obtain orthoimages corresponding to the other images.

In step S540, the orthoimages of the respective images captured by the same camera are mosaic-fused to generate orthoimages of various bands.

Step S510, step S520, and step S530 in the orthophoto generation method 500 according to the embodiment of the present application described with reference to FIG. 5 and step S110 in the orthophoto generation method 100 according to the embodiment of the present application described with reference to FIG. 1 Step S120 is similar to step S130, and for the sake of brevity, it will not be repeated here. Different from the orthophoto generation method 100 according to the embodiment of the present application described with reference to FIG. 1, the orthophoto generation method 500 according to the embodiment of the present application described with reference to FIG. 5 further includes step S540, that is, when other images are obtained After the orthoimages of the images, the orthoimages of the images taken by the same camera are mosaic-fused to generate orthoimages of various bands. As described above in conjunction with Figure 1 and Figure 2, multiple images can be multiple images obtained during multiple exposure periods, each exposure period can obtain multiple band images, and each band image corresponds to a camera. And the image of one band in each exposure period is the main image, and the images of the remaining bands are other images. Therefore, after the orthoimages of other images in each exposure period are obtained, the orthoimages of the same waveband images with different exposure periods can be mosaic-fused (that is, the orthoimages of each row of images in Figure 2 are mosaicked). Fusion) to generate orthoimages of each band to obtain a larger range of orthoimages of the target survey area, for example, as shown in Figure 6A to Figure 6C, Figure 6A, Figure 6B and Figure 6C respectively show three Schematic diagram of the orthophoto of the band.

Furthermore, based on the orthophotos of each band, a vegetation index map of the target measurement area can be generated. The vegetation index map can quantitatively provide a basis for fertilization and drug use for precision agriculture, effectively improve the utilization of fertilizers and pesticides, and can also be used for early detection of crops. Pests and diseases reduce losses. Further, in order to better visualize the numerical value of the distinguishing index, in one example, the generated vegetation index map can be rendered in pseudo-color, as shown in Figure 6D (it should be understood that the vegetation index shown in Figure 6D The map should be in color, but considering the requirements of the patent application documents, it is adjusted to the form of grayscale map).

In still another embodiment of the present invention, after aligning other images with the main image, a vegetation index map may be generated based on the main image and other aligned images, and then based on the elevation data and the imaging pose information Correcting the vegetation index map to obtain an orthophoto corresponding to the vegetation index map. In this embodiment, the vegetation index map is first generated based on the main image and other images, and then the orthographic image of the vegetation index map is generated, which can also realize the generation of the vegetation index map from the orthographic perspective.

The above exemplarily describes the orthophoto generation method according to the embodiment of the present invention. Based on the above description, the orthophoto generation method according to the embodiment of the present invention only obtains the imaging pose information of the main image in the multispectral image, and generates elevation data fitting the elevation of the target survey area based on the imaging pose information of the main image Finally, based on the imaging pose information of the main image and the generated elevation data, other images other than the main image are corrected to obtain the orthophoto of other images, that is, the imaging pose information of the main image and the elevation of the survey area will be obtained The data is used to generate orthoimages corresponding to other images. In this way, the amount of calculation for acquiring multiple orthoimages in a multi-camera acquisition scenario can be reduced, and the generation efficiency of orthoimages can be improved. In addition, the orthophoto generation method according to the embodiment of the present invention can preprocess other images before generating the orthoimages of other images, and can obtain orthoimages with higher reliability. Further, the orthophoto generation method according to the embodiment of the present invention can be used to generate a vegetation index map of the target survey area, thereby quantitatively providing a basis for fertilization and drug use for precision agriculture, effectively improving the utilization rate of fertilizers and pesticides, and can also be used in the early stage. Detect crop diseases and insect pests to reduce losses.

The following describes an orthophoto generation system according to another aspect of the present invention with reference to FIG. 7. FIG. 7 shows a schematic block diagram of an orthophoto generation system 700 according to an embodiment of the present invention. The orthophoto generation system 700 includes a storage device 710 and a processor 720.

Wherein, the storage device 710 stores a program for realizing the corresponding steps in the orthophoto generation method according to the embodiment of the present invention. The processor 720 is configured to run a program stored in the storage device 710 to execute the corresponding steps of the orthoimage generating method according to the embodiment of the present invention.

In one embodiment, when the program is executed by the processor 720, the orthophoto generation system 700 is caused to perform the following steps: acquiring multiple images of the target measurement area obtained by shooting the target measurement area by multiple cameras, wherein , The plurality of cameras are set on a mobile platform; the imaging pose information of the main image in the plurality of images is acquired, and according to the main image and the imaging pose information, it is used to fit the target measurement area Elevation data; and based on the elevation data and the imaging pose information, correcting other images in the plurality of images other than the main image to obtain an orthoimage corresponding to the other images.

In an embodiment of the present invention, when the program is executed by the processor 720, the orthophoto generation system 700 executes the acquisition of multiple cameras to photograph the target measurement area. The image includes: acquiring multiple images of the target measurement area obtained by multiple cameras shooting the target measurement area within the same exposure period, wherein the duration of the exposure period is less than a preset duration threshold.

In an embodiment of the present invention, when the program is executed by the processor 720, the orthophoto generation system 700 executes the acquisition of multiple cameras to photograph the target measurement area. The image further includes: acquiring multiple images of the target measurement area obtained by multiple cameras shooting the target measurement area in multiple exposure periods, wherein the main images captured in each exposure period are captured by the multiple cameras And the acquisition of the imaging pose information of the main image includes: using an SFM algorithm to reconstruct the main image taken during each exposure period to obtain the main image The imaging pose information of the image.

In an embodiment of the present invention, the main image is an RGB image.

In an embodiment of the present invention, when the program is executed by the processor 720, the orthophoto generation system 700 is also caused to perform the following steps: before the other images are corrected, the other images are preprocessed by correction parameters And the correction includes: correcting the preprocessed other images according to the elevation data and the imaging pose information to obtain an orthoimage corresponding to the other images.

In an embodiment of the present invention, when the program is executed by the processor 720, the orthophoto generation system 700 executes the preprocessing of the other images through the correction parameters, including: combining the other images with Aligning the main image; and/or performing radiometric correction processing on the main image and the other images.

In an embodiment of the present invention, when the program is executed by the processor 720, the alignment of the other images with the main image performed by the orthophoto generation system 700 includes: The image points of the same name are determined among the other images, the mapping transformation matrix for mapping the other images to the main image is determined based on the image points of the same name, and the other images are mapped to the main image based on the mapping transformation matrix. On the main image.

In an embodiment of the present invention, when the program is executed by the processor 720, the alignment of the other images with the main image performed by the orthophoto generation system 700 includes: based on all the images recorded by the sensor. The respective imaging pose information of the main image and the other images determines the mapping relationship between the other image and the main image, and maps the other image to the main image based on the mapping relationship.

In an embodiment of the present invention, when the program is executed by the processor 720, the orthophoto generating system 700 is also caused to perform the following steps: before aligning the other images with the main image, determining the main image And the respective gradient information of the other image, and align the other image with the main image based on the respective gradient information of the main image and the other image.

In an embodiment of the present invention, when the program is run by the processor 720, the orthophoto image generation system 700 is also caused to perform the following steps: before aligning the other images with the main image, perform the following steps on the main image. The image and the other images are subjected to distortion correction processing, and the other images are aligned with the main image based on the main image and the other images that have been distorted.

In an embodiment of the present invention, when the program is run by the processor 720, the orthophoto generating system 700 is also caused to perform the following steps: mosaic and fusion of the respective orthophotos of the images taken by the same camera to generate each Orthophoto of the band.

In an embodiment of the present invention, when the program is run by the processor 720, the orthophoto generating system 700 is also caused to perform the following steps: generating a vegetation index map based on the orthophotos of each band.

In an embodiment of the present invention, when the program is run by the processor 720, the orthophoto generation system 700 is also caused to perform the following steps: visually rendering the vegetation index map in pseudo-color.

In an embodiment of the present invention, when the program is executed by the processor 720, the orthophoto generation system 700 is also caused to perform the following steps: after aligning the other images with the main image, based on the main image Generating a vegetation index map with the aligned other images; and correcting the vegetation index map according to the elevation data and the imaging pose information to obtain an orthoimage corresponding to the vegetation index map.

In addition, according to an embodiment of the present invention, there is also provided a storage medium on which program instructions are stored, and when the program instructions are executed by a computer or a processor, they are used to execute the orthophoto of the embodiment of the present invention. Generate the corresponding steps of the method. The storage medium may include, for example, a memory card of a smart phone, a storage component of a tablet computer, a hard disk of a personal computer, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a portable compact disk read-only memory (CD-ROM), USB memory, or any combination of the above storage media. The computer-readable storage medium may be any combination of one or more computer-readable storage media.

In one embodiment, the computer program instructions can execute the orthophoto generation method according to the embodiment of the present invention when the computer program instructions are executed by a computer.

In one embodiment, when the computer program instructions are executed by a computer or a processor, the computer or the processor executes the following steps: acquiring multiple images of the target measurement area obtained by multiple cameras shooting the target measurement area, Wherein, the plurality of cameras are arranged on a mobile platform; the imaging pose information of the main image in the plurality of images is acquired, and the main image and the imaging pose information are used to generate the measurement for fitting the target Elevation data of the area elevation; and based on the elevation data and the imaging pose information, correcting other images in the plurality of images except the main image to obtain an orthoimage corresponding to the other images.

In an embodiment of the present invention, when the computer program instructions are executed by the computer or the processor, the acquisition of multiple cameras to shoot the target measurement area is performed by the computer or the processor. The image includes: acquiring multiple images of the target measurement area obtained by multiple cameras shooting the target measurement area in the same exposure period, wherein the duration of the exposure period is less than a preset duration threshold.

In an embodiment of the present invention, when the computer program instructions are executed by the computer or the processor, the acquisition of multiple cameras to shoot the target measurement area is performed by the computer or the processor. The image further includes: acquiring multiple images of the target measurement area obtained by multiple cameras shooting the target measurement area in multiple exposure periods, wherein the main images captured in each exposure period are composed of the multiple The same camera in the camera; and said acquiring the imaging pose information of the main image includes: performing reconstruction processing on the main image captured in each exposure time period through the SFM algorithm to obtain the The imaging pose information of the main image.

In an embodiment of the present invention, the main image is an RGB image.

In an embodiment of the present invention, when the computer program instructions are executed by the computer or the processor, the computer or the processor executes the following steps: before the other images are corrected, the other images are pre-processed by the correction parameters. Processing; and the correction includes: correcting the other pre-processed images according to the elevation data and the imaging pose information to obtain orthoimages corresponding to the other images.

In an embodiment of the present invention, when the computer program instructions are executed by the computer or the processor, the computer or processor executes the preprocessing of the other images through the correction parameters, including: converting the other images Aligning with the main image; and/or performing radiometric correction processing on the main image and the other images.

In an embodiment of the present invention, when the computer program instructions cause the computer or the processor to execute the aligning of the other image with the main image when being executed by the computer or the processor, it includes: Determine the image point with the same name between the image point and the other image, determine a mapping transformation matrix for mapping the other image to the main image based on the image point with the same name, and map the other image to the main image based on the mapping transformation matrix On the main image.

In an embodiment of the present invention, when the computer program instructions cause the computer or processor to execute the aligning of the other image with the main image when being executed by the computer or processor, it includes: The respective imaging pose information of the main image and the other images determine the mapping relationship between the other image and the main image, and map the other image to the main image based on the mapping relationship.

In an embodiment of the present invention, when the computer program instructions are executed by the computer or the processor, the computer or the processor also executes the following steps: before aligning the other images with the main image, determining the main image The respective gradient information of the image and the other image, and align the other image with the main image based on the respective gradient information of the main image and the other image.

In an embodiment of the present invention, when the computer program instructions are executed by the computer or the processor, the computer or the processor also executes the following steps: before aligning the other images with the main image, respectively Distortion correction processing is performed on the main image and the other images, and the other images are aligned with the main image based on the main image and the other images that have been distorted.

In an embodiment of the present invention, when the computer program instructions are executed by the computer or the processor, the computer or the processor also executes the following steps: mosaic and fusion of the respective orthophotos of the images taken by the same camera to generate Orthophoto of each band.

In an embodiment of the present invention, when the computer program instructions are executed by the computer or the processor, the computer or the processor further executes the following steps: generating a vegetation index map based on the orthophotos of the respective bands.

In an embodiment of the present invention, when the computer program instructions are executed by the computer or the processor, the computer or the processor also executes the following steps: visually rendering the vegetation index map in pseudo-color.

In an embodiment of the present invention, when the computer program instructions are executed by the computer or the processor, the computer or the processor also executes the following steps: after aligning the other image with the main image, based on the main image A vegetation index map is generated from the image and the aligned other images; and the vegetation index map is corrected according to the elevation data and the imaging pose information to obtain an orthoimage corresponding to the vegetation index map.

The above exemplarily describes the orthophoto generating method, system and storage medium according to the embodiments of the present invention. Based on the above description, the orthophoto generation method, system and storage medium according to the embodiments of the present invention only acquire the imaging pose information of the main image in the multispectral image, and generate fitting target measurements based on the imaging pose information of the main image. The elevation data of the area elevation, finally based on the imaging pose information of the main image and the generated elevation data to correct other images other than the main image to obtain the orthoimage of other images, that is, the imaging pose information of the main image to be acquired The elevation data of the survey area is used to generate orthoimages corresponding to other images. In this way, the calculation amount of acquiring multiple orthoimages in a multi-camera acquisition scenario can be reduced, and the generation efficiency of orthoimages can be improved.

Although the exemplary embodiments have been described herein with reference to the accompanying drawings, it should be understood that the above-described exemplary embodiments are merely exemplary, and are not intended to limit the scope of the present invention thereto. Those of ordinary skill in the art can make various changes and modifications therein without departing from the scope and spirit of the present invention. All these changes and modifications are intended to be included within the scope of the present invention as claimed in the appended claims.

A person of ordinary skill in the art may be aware that the units and algorithm steps of the examples described in the embodiments disclosed in this document can be implemented by electronic hardware or a combination of computer software and electronic hardware. Whether these functions are executed by hardware or software depends on the specific application and design constraint conditions of the technical solution. Professionals and technicians can use different methods for each specific application to implement the described functions, but such implementation should not be considered as going beyond the scope of the present invention.

In the several embodiments provided in this application, it should be understood that the disclosed device and method may be implemented in other ways. For example, the device embodiments described above are only illustrative. For example, the division of the units is only a logical function division, and there may be other divisions in actual implementation, for example, multiple units or components can be combined or It can be integrated into another device, or some features can be ignored or not implemented.

In the instructions provided here, a lot of specific details are explained. However, it can be understood that the embodiments of the present invention can be practiced without these specific details. In some instances, well-known methods, structures, and technologies are not shown in detail, so as not to obscure the understanding of this specification.

Similarly, it should be understood that in order to simplify the present invention and help understand one or more of the various aspects of the invention, in the description of the exemplary embodiments of the present invention, the various features of the present invention are sometimes grouped together into a single embodiment. , Or in its description. However, the method of the present invention should not be construed as reflecting the intention that the claimed invention requires more features than those explicitly stated in each claim. To be more precise, as reflected in the corresponding claims, the point of the invention is that the corresponding technical problems can be solved with features that are less than all the features of a single disclosed embodiment. Therefore, the claims following the specific embodiment are thus explicitly incorporated into the specific embodiment, wherein each claim itself serves as a separate embodiment of the present invention.

Those skilled in the art can understand that in addition to mutual exclusion between the features, any combination of all features disclosed in this specification (including the accompanying claims, abstract, and drawings) and any method or device disclosed in this manner can be used. Processes or units are combined. Unless expressly stated otherwise, each feature disclosed in this specification (including the accompanying claims, abstract and drawings) may be replaced by an alternative feature providing the same, equivalent or similar purpose.

In addition, those skilled in the art can understand that although some embodiments described herein include certain features included in other embodiments but not other features, the combination of features of different embodiments means that they are within the scope of the present invention. Within and form different embodiments. For example, in the claims, any one of the claimed embodiments can be used in any combination.

The various component embodiments of the present invention may be implemented by hardware, or by software modules running on one or more processors, or by a combination of them. Those skilled in the art should understand that a microprocessor or a digital signal processor (DSP) may be used in practice to implement some or all of the functions of some modules according to the embodiments of the present invention. The present invention can also be implemented as a device program (for example, a computer program and a computer program product) for executing part or all of the methods described herein. Such a program for realizing the present invention may be stored on a computer-readable medium, or may have the form of one or more signals. Such a signal can be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

It should be noted that the above-mentioned embodiments illustrate rather than limit the present invention, and those skilled in the art can design alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses should not be constructed as a limitation to the claims. The invention can be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In the unit claims that list several devices, several of these devices may be embodied in the same hardware item. The use of the words first, second, and third, etc. do not indicate any order. These words can be interpreted as names.

The above are only specific implementations or descriptions of specific implementations of the present invention. The protection scope of the present invention is not limited thereto. Any person skilled in the art can easily fall within the technical scope disclosed by the present invention. Any change or replacement should be included in the protection scope of the present invention. The protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

An orthophoto generation method, characterized in that, the method includes:

Acquiring multiple images of the target measurement area obtained by multiple cameras shooting the target measurement area, wherein the multiple cameras are set on a mobile platform;

Acquiring imaging pose information of a main image of the plurality of images, and generating elevation data for fitting the elevation of the target survey area according to the main image and the imaging pose information; and

According to the elevation data and the imaging pose information, other images in the plurality of images other than the main image are corrected to obtain orthoimages corresponding to the other images.
The method according to claim 1, wherein said acquiring a plurality of images of the target survey area obtained by multiple cameras shooting the target survey area comprises:

Acquire multiple images of the target measurement area obtained by multiple cameras shooting the target measurement area within the same exposure period, wherein the duration of the exposure period is less than a preset duration threshold.
The method according to claim 2, wherein said acquiring a plurality of images of the target survey area obtained by multiple cameras shooting the target survey area, further comprises:

Acquire multiple images of the target measurement area obtained by multiple cameras shooting the target measurement area in multiple exposure periods, wherein the main image captured in each exposure period is taken by the same camera among the multiple cameras Filmed; and

The acquiring the imaging pose information of the main image includes: performing reconstruction processing on the main image captured during the respective exposure periods through an SFM algorithm to obtain the imaging pose information of the main image.
The method according to any one of claims 1 to 3, wherein the type of the image includes at least two of RGB image, near-infrared image, red waveband image, and green waveband image.
The method of claim 4, wherein the main image is an RGB image.
The method according to claim 1, wherein the method further comprises:

Before correcting the other images, preprocessing the other images by correcting parameters; and

The correction includes: correcting the preprocessed other images according to the elevation data and the imaging pose information to obtain orthoimages corresponding to the other images.
The method according to claim 6, wherein said preprocessing said other images by correcting parameters comprises:

Align the other images with the main image; and/or

Perform radiation correction processing on the main image and the other images.
8. The method of claim 7, wherein the aligning the other image with the main image comprises:

Determine a pixel with the same name between the main image and the other image, determine a mapping transformation matrix for mapping the other image to the main image based on the pixel with the same name, and convert the Other images are mapped onto the main image.
8. The method of claim 7, wherein the aligning the other image with the main image comprises:

The mapping relationship between the other image and the main image is determined based on the respective imaging pose information of the main image and the other images recorded by the sensor, and the other image is mapped to all the images based on the mapping relationship. On the main image.
The method according to any one of claims 7-9, wherein the method further comprises:

Before aligning the other image with the main image, determine the respective gradient information of the main image and the other image, and align the other image based on the respective gradient information of the main image and the other image. The image is aligned with the main image.
The method according to any one of claims 7-9, wherein the method further comprises:

Before aligning the other images with the main image, perform distortion correction processing on the main image and the other images respectively, and convert the other images based on the distortion-corrected main image and the other images Align with the main image.
The method according to claim 11, wherein the distortion correction processing is based on internal parameters and distortion parameters of the plurality of cameras.
The method according to claim 1, wherein the elevation data is generated based on a point cloud, and the point cloud is generated based on the imaging pose information.
The method according to claim 13, wherein the point cloud is a dense point cloud generated based on the imaging pose information.
The method according to claim 13, wherein the point cloud is based on a sparse point cloud obtained by performing an SFM algorithm on the main image.
The method according to claim 4, wherein the method further comprises:

Mosaic fusion is performed on the respective orthoimages of the images captured by the same camera to generate orthoimages of various bands.
The method according to claim 16, wherein the method further comprises:

A vegetation index map is generated based on the orthophoto of each band.
The method according to claim 17, wherein the method further comprises:

The vegetation index map is visually rendered in pseudo-color.
The method according to claim 7, wherein the method further comprises:

After aligning the other images with the main image, generating a vegetation index map based on the main image and the aligned other images; and

The vegetation index map is corrected according to the elevation data and the imaging pose information to obtain an orthophoto corresponding to the vegetation index map.
An orthophoto generation system based on multispectral imagery, characterized in that the system includes a storage device and a processor, the storage device stores a computer program run by the processor, and the computer program is The processor executes the multi-spectral image-based orthoimage generation method according to any one of claims 1-19 when the processor is running.
A storage medium, characterized in that a computer program is stored on the storage medium, and the computer program executes the multispectral image-based orthoimage generation according to any one of claims 1-19 when the computer program is running. method.