CN117478840A - River risk inspection method, device and equipment based on 5G unmanned aerial vehicle - Google Patents

Abstract

The application provides a river risk inspection method, device and equipment based on a 5G unmanned aerial vehicle. The method comprises the following steps: sending a patrol instruction to the unmanned aerial vehicle based on the 5G network, wherein the patrol instruction is used for indicating the unmanned aerial vehicle to carry out patrol aerial photography on a preset area of a river channel; receiving river channel image data transmitted by the unmanned aerial vehicle based on a 5G network; processing the river image data to generate a plurality of orthographic images; performing feature extraction processing on the plurality of orthographic images to obtain at least one key feature data; the key characteristic data represents abnormal positions and abnormal problems on a preset area of the patrolled river channel; processing each piece of key feature data to obtain and output risk prompt information corresponding to each piece of key feature data; and the risk prompt information corresponding to each key feature data represents the position information and the abnormal problem of the abnormal position indicated by the key feature data. The river risk inspection efficiency can be improved.

Description

River risk inspection method, device and equipment based on 5G unmanned aerial vehicle

Technical Field

The application relates to the field of river channel detection, in particular to a river channel risk inspection method, device and equipment based on a 5G unmanned aerial vehicle.

Background

Traditional river course inspection generally uses manual work to drive ship or car, and then artificial inspection is carried out to the river course with the unusual position and the unusual problem on the manual determination river course. However, the above-described method is slow in inspection efficiency, inefficient in determining the abnormal position and problem, and has high labor costs. Therefore, a method for efficiently and timely completing the river risk inspection is needed.

Disclosure of Invention

The application provides a river risk inspection method, device and equipment based on a 5G unmanned aerial vehicle, which are used for solving the technical problem of low river risk inspection efficiency.

In a first aspect, the present application provides a river risk inspection method based on a 5G unmanned aerial vehicle, including:

sending a patrol instruction to the unmanned aerial vehicle based on the 5G network, wherein the patrol instruction is used for indicating the unmanned aerial vehicle to carry out patrol aerial photography on a preset area of a river channel, and the patrol instruction indicates a patrol flight route of the river channel;

receiving river channel image data transmitted by the unmanned aerial vehicle based on a 5G network;

processing the river image data to generate a plurality of orthographic images; the method comprises the steps that an orthographic image represents an image of a part of a preset area for inspecting a river channel, and a plurality of orthographic images represent complete images of the preset area for inspecting the river channel; performing feature extraction processing on the plurality of orthographic images to obtain at least one key feature data; the key characteristic data represent abnormal positions and abnormal problems on a preset area of the patrolled river channel;

Processing each piece of key feature data to obtain and output risk prompt information corresponding to each piece of key feature data; the risk prompt information corresponding to each piece of key feature data represents the position information and the abnormal problem of the abnormal position indicated by the key feature data.

Optionally, the river image data includes an image and data information, where the data information characterizes position information and azimuth information corresponding to the image; processing the river image data to generate a plurality of orthographic images, including:

performing image processing on the image in the river image data to obtain a processed image;

adding a preset image control point into the processed image;

according to the preset image control points, sequentially performing analysis air triangulation and modeling on the processed image added with the preset image control points to obtain a digital earth surface model diagram;

splicing the digital earth surface model map based on a preset splicing line to obtain an image corresponding to the preset area; performing image output processing on the image corresponding to the preset area to obtain a digital orthographic image;

And framing the obtained digital orthographic images to obtain a plurality of orthographic images.

Optionally, performing feature extraction processing on the plurality of orthographic images to obtain at least one key feature data, including:

performing feature extraction processing on the plurality of orthographic images based on a preset detection model to obtain at least one node coding data; the node coding data are used for representing the character strings and abnormal positions of abnormal problems on a preset area of the patrolled river channel;

determining key feature data corresponding to each node coding data based on a preset data table; the preset data table represents the corresponding relation between the character strings of the abnormal problems and the abnormal problems.

Optionally, the method further comprises:

acquiring marked orthographic images of different river channels from a preset data warehouse; the preset data warehouse comprises marked orthographic images of a plurality of river channels; the marked orthographic image is obtained by marking the orthographic image of the river channel based on marking information and attribute information of a region corresponding to the orthographic image;

and training the initial model based on the obtained marked orthophotos of different river channels to obtain the preset detection model.

Optionally, the method further comprises:

labeling each of the orthographic images in response to a labeling instruction to obtain each labeled orthographic image; the marking instruction indicates marking information and attribute information of the area corresponding to the orthographic image;

and storing each marked orthophoto image into a preset data warehouse.

Optionally, before processing the river image data to generate a plurality of orthographic images, the method further includes:

and sequentially performing size normalization processing, filtering processing and noise reduction processing on the river image data.

Optionally, the method further comprises:

inputting each piece of key characteristic data into a first preset identification model to obtain a risk level corresponding to each piece of key characteristic data;

and determining the highest risk level as the risk level of the preset area of the patrolled river channel.

Optionally, the method further comprises:

inputting the orthographic images with the key characteristic data into a second preset recognition model aiming at each orthographic image with the key characteristic data to obtain text description information corresponding to the orthographic images; the character description information represents river channel information of an orthographic image with key characteristic data;

Generating processing measure information corresponding to each orthographic image with key characteristic data according to risk prompt information corresponding to the orthographic image; the processing measure information characterizes an abnormal processing mode of the risk prompt information;

and generating a river channel early warning report according to the text description information, the processing measure information and the risk level.

Optionally, the method further comprises:

acquiring voice information; the voice information is prompt voice for the abnormal position; generating a playing instruction based on the voice information, wherein the playing instruction carries the voice information and is used for indicating the unmanned aerial vehicle to fly to an abnormal position indicated by the voice information so as to play the voice;

and sending the playing instruction to the unmanned aerial vehicle based on a 5G network, so that the unmanned aerial vehicle flies to an abnormal position indicated by the voice information based on the playing instruction to play the voice information.

Optionally, before sending the inspection instruction to the unmanned aerial vehicle based on the 5G network, the method further includes: responding to a trigger instruction, and generating the inspection instruction; the trigger instruction indicates flight parameters of the unmanned aerial vehicle;

After sending the inspection instruction to the unmanned aerial vehicle based on the 5G network, the method further comprises the following steps: receiving flight parameters sent by the unmanned aerial vehicle based on a 5G network, and displaying the flight parameters; wherein, the flight parameter characterizes the flight state of the unmanned aerial vehicle.

In a second aspect, the application provides a river risk inspection device based on 5G unmanned aerial vehicle, include:

the system comprises a sending unit, a receiving unit and a control unit, wherein the sending unit is used for sending a patrol instruction to the unmanned aerial vehicle based on a 5G network, the patrol instruction is used for indicating the unmanned aerial vehicle to carry out patrol aerial photography on a preset area of a river channel, and the patrol instruction indicates a patrol flight route of the river channel;

the receiving unit is used for receiving river channel image data transmitted by the unmanned aerial vehicle based on a 5G network;

the generation unit is used for processing the river image data and generating a plurality of orthographic images; the method comprises the steps that an orthographic image represents an image of a part of a preset area for inspecting a river channel, and a plurality of orthographic images represent complete images of the preset area for inspecting the river channel; performing feature extraction processing on the plurality of orthographic images to obtain at least one key feature data; the key characteristic data represent abnormal positions and abnormal problems on a preset area of the patrolled river channel;

The processing unit is used for processing each piece of key characteristic data to obtain and output risk prompt information corresponding to each piece of key characteristic data; the risk prompt information corresponding to each piece of key feature data represents the position information and the abnormal problem of the abnormal position indicated by the key feature data.

In a third aspect, the present application provides an electronic device, comprising: a processor, and a memory communicatively coupled to the processor;

the memory stores computer-executable instructions;

the processor executes computer-executable instructions stored by the memory to implement the method of any one of the first aspects.

In a fourth aspect, the present application provides a computer-readable storage medium having stored therein computer-executable instructions for performing the method of any of the first aspects when executed by a processor.

In a fifth aspect, the present application provides a computer program product comprising a computer program which, when executed by a processor, implements the method according to any of the first aspects.

In a sixth aspect, the present application provides a chip having a computer program stored thereon, which, when executed by the chip, implements the method according to any of the first aspects.

According to the river risk inspection method, device and equipment based on the 5G unmanned aerial vehicle, the river image data collected by the unmanned aerial vehicle inspection is received based on the 5G network, the river image data are processed to obtain an orthographic image, after feature extraction processing is carried out on the orthographic image, key feature data comprising abnormal problems and abnormal positions of the river are obtained, and risk prompt information is output according to the key feature data. According to the method, the unmanned aerial vehicle can be used for carrying out inspection to collect the river image data, compared with the manual inspection method in the prior art, the river risk inspection efficiency is improved, meanwhile, the abnormal problems and abnormal positions of the river are obtained through the feature extraction processing of the orthographic images, and compared with the manual determination of the abnormal problems and abnormal positions in the prior art, the river risk inspection efficiency is greatly improved.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the application and together with the description, serve to explain the principles of the application.

Fig. 1 is a schematic structural diagram of a river channel inspection system provided in the present application;

fig. 2 is a schematic structural diagram of a server of a river channel inspection system provided in the present application;

Fig. 3 is a schematic structural diagram of a service end of another river channel inspection system provided in the present application;

fig. 4 is a schematic flow chart of a river risk inspection method based on a 5G unmanned aerial vehicle provided in the present application;

fig. 5 is a schematic flow chart of a river risk inspection method based on a 5G unmanned aerial vehicle provided by the present application;

fig. 6 is a schematic structural diagram of a river risk inspection device based on a 5G unmanned aerial vehicle provided in the present application;

fig. 7 is a schematic structural diagram of a river risk inspection device based on a 5G unmanned aerial vehicle provided in the present application;

fig. 8 is a schematic structural diagram of an electronic device provided in the present application.

Specific embodiments thereof have been shown by way of example in the drawings and will herein be described in more detail. These drawings and the written description are not intended to limit the scope of the inventive concepts in any way, but to illustrate the concepts of the present application to those skilled in the art by reference to specific embodiments.

Detailed Description

Reference will now be made in detail to exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, the same numbers in different drawings refer to the same or similar elements, unless otherwise indicated. The implementations described in the following exemplary examples are not representative of all implementations consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with some aspects of the present application as detailed in the accompanying claims.

First, terms involved in the present application will be explained:

river course: including lakes, artificial waterways, flood-moving areas, flood-accumulating areas, flood-stagnation areas, etc.

Digital orthophotograph (Digital Orthophoto Map, DOM): the method is characterized in that digital differential correction and mosaic are carried out on aviation (or aerospace) photo, and the digital differential correction and mosaic are generated by cutting according to a certain picture range.

Internal orientation element: refers to parameters describing the relative position between the photographing center and the photo, including three parameters, the sag distance (main distance) from the photographing center to the photo, and the coordinates of the principal point of the image in the frame coordinate system.

External orientation element: the position and attitude of the moment of photography, including the coordinates of the centre of photography in the ground coordinate system, and three angular elements describing the spatial attitude of the photographic beam.

Resolving aerial triangulation: refers to the determination of the external orientation elements of all images in the region by photogrammetry analysis.

Photo control point (image control point): is a control point which is directly arranged and measured in the field for the encryption or mapping of the control point of the photogrammetry.

Digital surface model (Digital Surface Model, DSM): refers to a ground elevation model comprising the heights of ground buildings, bridges, trees and the like.

At present, in the prior art, a river channel is subjected to risk inspection by manually going to a driving ship or a vehicle so as to manually determine whether an abnormal problem and an abnormal position exist on the river channel. However, the inspection efficiency is low, the abnormal position and the abnormal problem need to be manually determined, and the inspection method has high labor cost.

In view of this, the application provides a river risk inspection method based on a fifth generation mobile communication technology (5th Generation Mobile Communication Technology,5G) unmanned aerial vehicle, which uses the unmanned aerial vehicle to perform aerial inspection on a river, and performs processing analysis on image data based on 5G transmission to determine the abnormal problems and abnormal positions of the river. According to the method, the unmanned aerial vehicle aerial photography inspection mode can be used for replacing manual inspection, the mode of processing and analyzing the image data to determine the abnormal problems and abnormal positions of the river channel is utilized for replacing manual determination, and the risk inspection efficiency of the river channel can be improved.

Fig. 1 is a schematic structural diagram of a river channel inspection system provided in the present application. As shown in FIG. 1, the river channel inspection system comprises at least one unmanned aerial vehicle, a server side and a client side. In fig. 1, only one unmanned aerial vehicle is illustrated.

The client is used for providing a front-end display interface and receiving an instruction triggered by a user. The client may be a global Wide area network (Web) client or an Application (app) client, which is not limited in this Application.

The server is used for managing the image data of river channel inspection and the flight data of the unmanned aerial vehicle. The server may be, for example, a server or a server cluster, and may be deployed in any environment, for example, in the cloud, or may be deployed in multiple environments in a distributed manner, which is not limited herein. The execution subject of the present application may be the server.

In one example, the server may be divided into a plurality of modules according to software functions. Fig. 2 is a schematic structural diagram of a server of the river channel inspection system provided in the present application. As shown in fig. 2, the service end may include a flight management module, a data transmission module, an image data processing module, and a river channel information management module.

The flight management module is used for managing flight parameters of the unmanned aerial vehicle, such as flight height, speed, gesture and the like, and providing a front-end display interface to display real-time flight data of the unmanned aerial vehicle so as to monitor the flight state of the unmanned aerial vehicle.

The data transmission module is used for providing a data communication link based on 5G.

The image data processing module is used for processing and analyzing the image data acquired by the unmanned aerial vehicle and determining the abnormal problems and abnormal positions of the river channel.

The river channel information management module is used for providing a front-end display interface so that a user can label, divide, edit and the like the river channel based on the image data of the river channel and manage the information of the river channel.

It should be noted that the deployment device of each module is not limited in this application, for example, the image data processing module and the river channel information management module may be located on the same electronic device, or may be located on different electronic devices, and may specifically be set according to the processing capability of the server.

In another example, the server may be divided into multiple platforms according to hardware functions. Fig. 3 is a schematic structural diagram of a service end of another river channel inspection system provided in the present application. As shown in fig. 3, the server may include an application platform, an analysis platform, an image processing platform, and a data transmission platform.

The application platform is used for managing river channel information, such as labeling, attribute editing and the like, based on the image data of the river channel, and managing a database.

The analysis platform is used for training a machine learning model for processing the image data and managing training data and analyzing the image data.

The image processing platform is used for performing image preprocessing, such as data cleaning, filtering, optimizing and the like, on the collected image data so as to facilitate subsequent image data analysis.

The data transmission platform is used for providing a data transmission link based on 5G.

It should be noted that, the above-mentioned platforms are respectively located on different electronic devices at the server side, for example, the above-mentioned application platform is located on the server 1 at the server side, and the above-mentioned analysis platform may be located on the server 2 at the server side.

The foregoing is merely illustrative of the structure of the server, and the present application does not limit the structure of the server.

The unmanned aerial vehicle is used for acquiring the image data of the river channel through aerial photography. The unmanned aerial vehicle comprises a camera, various sensors, a flight control system and the like, and the flight control system can control the unmanned aerial vehicle to fly and shoot based on flight parameters. The camera can be used for collecting image data of the river channel. The various sensors may be used to collect data from the river and/or flight data from the drone. It should be noted that the present application is not limited to the type and number of unmanned aerial vehicles, and the types of devices configured on the unmanned aerial vehicle.

Optionally, the camera configured on the unmanned aerial vehicle may be a high-resolution camera, for example. Before shooting by the camera, in order to ensure accuracy of shooting data, imaging geometry calibration can be performed on the camera, for example, calibration parameters can be calculated by a photogrammetry adjustment method, and whether each calibration parameter meets a preset threshold value is determined. If the camera is maintained, subjected to severe vibration or impact and the like, the camera needs to be checked again.

The following describes the technical solutions of the present application and how the technical solutions of the present application solve the above technical problems in detail with specific embodiments. The following embodiments may be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments. Embodiments of the present application will be described below with reference to the accompanying drawings.

Fig. 4 is a schematic flow chart of a river risk inspection method based on a 5G unmanned aerial vehicle. As shown in fig. 4, the method includes:

s101, sending a patrol instruction to the unmanned aerial vehicle based on the 5G network.

The inspection instruction is used for indicating the unmanned aerial vehicle to inspect the preset area of the river channel for aerial photography, and the inspection instruction indicates the inspection flight route of the river channel.

In one example, an identification of an unmanned aerial vehicle and a flight parameter of the unmanned aerial vehicle, which are input by a user and sent by a client, are received, the flight parameter includes, for example, a flight altitude, a speed, a flight route, and the like, the inspection instruction is generated based on the flight parameter, the unmanned aerial vehicle corresponding to the identification of the unmanned aerial vehicle is determined according to the identification of the unmanned aerial vehicle, and the inspection instruction is sent to the unmanned aerial vehicle based on a 5G network.

In another example, the inspection instruction of the unmanned aerial vehicle is preset, and may be a timing instruction or an instruction triggered by a user and sent by a receiving client, and the inspection instruction is sent to the unmanned aerial vehicle based on the 5G network.

S102, receiving river channel image data transmitted by the unmanned aerial vehicle based on the 5G network.

The river image data is image data of the river preset area collected by the unmanned aerial vehicle by using a camera and/or a sensor configured by the unmanned aerial vehicle, and may include, for example, image and azimuth data of the river preset area.

In one example, the server may receive river image data collected by the unmanned aerial vehicle in real time based on the 5G network, that is, the unmanned aerial vehicle may collect the river image data and then transmit the river image data to the server in real time; the server may receive a preset number of river image data collected by the unmanned aerial vehicle based on the 5G network, that is, the unmanned aerial vehicle may collect a preset number of image data and then transmit the image data to the server; or, the server receives the river image data of the partial area collected by the unmanned aerial vehicle based on the 5G network, that is, the unmanned aerial vehicle may collect the river image data of the partial area and then transmit the river image data to the server.

S103, processing the river channel image data to generate a plurality of orthographic images.

The orthographic image represents an image of a partial area of a preset area for inspecting the river channel, and the plurality of orthographic images represent complete images of the preset area for inspecting the river channel.

It should be understood that the river image data includes a plurality of river images. In one example, the plurality of river images may be subjected to image preprocessing, for example, size normalization, light and color adjustment processing, distortion correction processing, and the like, on the images in the river image data, to obtain a plurality of preprocessed river images, and then the preprocessed river images are converted into a plurality of orthoimages, to obtain a plurality of orthoimages.

In another example, after the preprocessed river channel image is obtained by adopting the image preprocessing mode, a plurality of frames of river channel images are spliced to obtain a complete image of a preset area of the river channel, the complete image is converted into a complete orthographic image, and the complete orthographic image is subjected to framing according to a preset size to obtain a plurality of orthographic images.

S104, performing feature extraction processing on the plurality of orthographic images to obtain at least one key feature data.

And the key characteristic data represent abnormal positions and abnormal problems on a preset area of the patrolled river channel. The abnormal problem may be, for example, a behavior such as a river being encroached on, an illegal fishing, or the like. The abnormal position may be, for example, latitude and longitude position information, and the abnormal position may be a position corresponding to the abnormal problem, may be a shooting position corresponding to the orthographic image, or may be any point or a preset point in a river channel region corresponding to the orthographic image, which is not limited herein, and may be specifically set according to actual requirements.

In one example, an orthographic image may be input into a predetermined detection model to perform feature extraction processing on the orthographic image, and a detection result of the orthographic image may be output. The detection result may include an abnormality problem and an abnormality position of the orthophoto. If the orthographic image has no abnormal problem, that is, is normal, the abnormal problem in the detection result may be normal, and the abnormal position may be a shooting position of the orthographic image or an arbitrary position in the orthographic image. The key feature data is an abnormal detection result among the detection results of the respective normal images. The detection model may be, for example, a convolutional neural network model, a cyclic neural network model, etc., which is not limited herein, and may be specifically set according to actual requirements.

S105, processing each piece of key feature data to obtain and output risk prompt information corresponding to each piece of key feature data.

The risk prompt information corresponding to each key feature data characterizes the position information and the abnormal problem of the abnormal position indicated by the key feature data.

In one example, according to a preset generation rule, risk prompt information corresponding to each key feature data is generated based on each key feature data. For example, the preset generation rule may be "riverway [ abnormal position ] has [ abnormal problem ], please deal with-! ". The form and content of the generation rule are not limited in this application.

The risk prompt information corresponding to the key feature data may be output to the client in a text prompt manner, or may be output to the client in a voice playing manner, which is not limited in the output manner.

In this embodiment, river image data acquired by unmanned aerial vehicle inspection is received based on a 5G network, the river image data is processed to obtain an orthographic image, and after feature extraction processing is performed on the orthographic image, key feature data including an abnormal problem and an abnormal position of the river is obtained, and risk prompt information is output according to the key feature data. According to the method, the unmanned aerial vehicle can be used for carrying out inspection to collect the river image data, compared with the manual inspection method in the prior art, the river risk inspection efficiency is improved, meanwhile, the abnormal problems and abnormal positions of the river are obtained through the feature extraction processing of the orthographic images, and compared with the manual determination of the abnormal problems and abnormal positions in the prior art, the river risk inspection efficiency is greatly improved.

Fig. 5 is a schematic flow chart of a river risk inspection method based on a 5G unmanned aerial vehicle. As shown in fig. 5, the method includes:

s201, responding to the trigger instruction and generating a patrol instruction.

The triggering instruction indicates flight parameters of the unmanned aerial vehicle.

The method includes the steps that an exemplary trigger instruction sent by a client is received, the trigger instruction comprises flight parameters of the unmanned aerial vehicle, and a patrol instruction is generated in response to the trigger instruction.

S202, sending a patrol instruction to the unmanned aerial vehicle based on the 5G network.

The inspection instruction is used for indicating the unmanned aerial vehicle to inspect the preset area of the river channel for aerial photography, and the inspection instruction indicates the inspection flight route of the river channel. The explanation of this step may refer to the aforementioned step S101.

Optionally, after sending the inspection instruction, receiving a flight parameter sent by the unmanned aerial vehicle based on a 5G network, and displaying the flight parameter on a client; wherein, this flight parameter characterizes unmanned aerial vehicle's flight state. The method can facilitate the user to monitor the flight state of the unmanned aerial vehicle.

S203, receiving river image data transmitted by the unmanned aerial vehicle based on the 5G network.

The river image data comprises image images and data information, and the data information represents position information and azimuth information corresponding to the image images. The image refers to an image captured by a camera, the position information corresponding to the image refers to latitude and longitude information, altitude information, and the like when the image is captured, and the azimuth information corresponding to the image refers to an inner azimuth element, an outer azimuth element, and the like when the image is captured.

The explanation of this step may refer to the aforementioned step S102.

S204, sequentially performing size normalization processing, filtering processing and noise reduction processing on the river image data.

The above size normalization refers to performing normalization processing on the size of the image, and unifying the size to a preset size, so as to facilitate subsequent image processing.

The filtering process is to screen the image after the size normalization, and delete the image with poor shooting effect, such as difficult recognition due to high shielding degree.

The noise reduction processing is to reduce noise of the filtered image, and may be performed by wavelet transform or the like.

S205, performing image processing on the image in the river image data to obtain a processed image.

The image processing refers to preliminary processing of the image by using preset processing software. Illustratively, based on preset processing software, importing an image and data information, and performing preliminary processing on the image to obtain a processed image.

S206, adding preset image control points into the processed image.

It should be noted that after the addition of the image control points, the adjustment may be performed according to the calculation error in each image, and the present application is not limited herein.

S207, according to the preset image control points, sequentially performing analysis air triangulation and modeling on the processed image after the preset image control points are added to obtain a digital earth surface model diagram.

Illustratively, the method of analyzing the aerial triangulation process, such as the model-based approach, the independent model approach and the beam approach, can be adopted, and the application is not limited herein, and specific implementation can refer to the prior art.

The modeling processing refers to performing model processing on the image after analysis aerial triangulation processing to obtain a DSM image. The modeling process may be implemented by reference to the prior art process of generating DSM images, and is not limited herein.

S208, splicing the digital earth surface model map based on a preset splicing line to obtain an image corresponding to the preset area; and performing image output processing on the image corresponding to the preset area to obtain a digital orthophoto.

The image corresponding to the preset area is an integral image corresponding to the preset area, and is obtained by stitching a plurality of DSM images based on a preset stitching line. The image output processing means that the image is uniformly colored, so that the color of the output image is more attractive.

S209, framing the obtained digital orthographic images to obtain a plurality of orthographic images.

Illustratively, the data orthographic images are subjected to framing processing according to a preset size to obtain a plurality of orthographic images.

S210, performing feature extraction processing on a plurality of orthographic images based on a preset detection model to obtain at least one node coding data.

The node coding data are used for representing the character strings and the abnormal positions of the abnormal problems on the preset area of the patrolled river channel. For example, the node-encoded data may be 0000X5060, where 0000X is a character string of an anomaly problem, and 5060 is longitude and latitude information of an anomaly location.

Optionally, each orthographic image may be labeled in response to a labeling instruction sent by the client, so as to obtain each labeled orthographic image. The marking instruction indicates marking information and attribute information of the area corresponding to the orthographic image; and storing each marked orthophoto image into a preset data warehouse.

For example, the labeling information refers to labeling the orthographic image, for example, labeling elements such as houses and drain ports in the image and abnormal problems such as illegal fishing; the attribute information is an attribute of a river such as a river section in the image. For example, the client receives the labeling instruction of the user on the orthographic images, and the server responds to the labeling instruction of the user to label each orthographic image, so as to obtain each labeled orthographic image.

The storage form in the data warehouse is not limited in this application, and for example, an HBase database may be used to manage a large amount of orthophotos.

Alternatively, the marked orthographic images of different river channels can be obtained from a preset data warehouse; the preset data warehouse comprises marked orthographic images of a plurality of river channels; the marked orthographic image is obtained by marking the orthographic image of the river channel based on marking information and attribute information of the region corresponding to the orthographic image. Based on the obtained orthographic images of different river channels after marking, training the initial model to obtain a preset detection model.

For example, as described above, the data warehouse stores the marked orthographic images in advance, and the initial model may be trained according to the marked orthographic images to obtain a preset detection model.

S211, determining key characteristic data corresponding to each node coded data based on a preset data table.

The preset data table represents the corresponding relation between the character strings of the abnormal problems and the abnormal problems.

The above-mentioned preset data table may be shown in table 1, for example. Referring to table 1, key feature data corresponding to each node coded data can be determined according to the preset data table.

TABLE 1

Character string	Abnormality problem
		0000X	Illegal fishing

S212, processing each piece of key feature data to obtain and output risk prompt information corresponding to each piece of key feature data.

The explanation of this step may refer to the aforementioned step S105.

After step S212, steps included in the following several cases may also be performed.

First case:

s213, inputting each piece of key feature data into a first preset recognition model to obtain a risk level corresponding to each piece of key feature data.

The first preset recognition model may be, for example, a convolutional neural network model or a cyclic neural network model, which is not limited herein. The first preset recognition model may be pre-trained based on the sample's key feature data and the sample's risk level.

Through this step, the risk level corresponding to each key feature data can be determined.

And S214, determining the highest risk level as the risk level of the preset area of the patrolled river channel.

For example, as described above, each key feature data corresponds to one risk level, and the highest risk level in each key feature data may be used as the risk level of the preset area of the channel being patrolled.

S215, inputting the orthographic images with the key characteristic data into a second preset recognition model aiming at each orthographic image with the key characteristic data to obtain text description information corresponding to the orthographic images.

The character description information characterizes river channel information of the orthographic image with key characteristic data. Illustratively, the text description information refers to generating a popular text description based on the abnormal problems and abnormal positions of the orthophotos.

The second preset recognition model may be, for example, a contrast language-Image Pre-trained subtitle (Contrastive Language-Image Pre-Training Prefix for Image Captioning, clipCap) model, or may be another deep learning model, which is not limited herein and may be set according to actual requirements.

Taking the second preset recognition model as a ClipCap model for example, the ClipCap model may include an encoder and a decoder, wherein the encoder uses a contrast language-Image Pre-Training (CLIP model), and the decoder uses a generated Pre-Training transform (GPT) model, for example, a GPT-2 model.

S216, generating processing measure information corresponding to each orthographic image with key characteristic data according to risk prompt information corresponding to the orthographic image.

The processing measure information characterizes an abnormal processing mode of the risk prompt information.

For example, the correspondence between the abnormal problem and the processing measure information is preset, and the processing measure information corresponding to the orthographic image may be determined according to the abnormal problem included in the risk prompt information corresponding to the orthographic image.

S217, generating a river channel early warning report according to the text description information, the processing measure information and the risk grades.

For example, a PDF file is generated according to each text description information, each processing measure information, and each risk level, and is used as an early warning report of the river channel. The mode can be convenient for the user to clearly know the abnormal problems and risks existing in the river channel.

Second case:

s218, acquiring voice information; the voice information is prompt voice for the abnormal position; and generating a playing instruction based on the voice information, wherein the playing instruction carries the voice information, and the playing instruction is used for indicating the unmanned aerial vehicle to fly to an abnormal position indicated by the voice information so as to play the voice.

Illustratively, the voice information triggered by the user is received by the client, for example, the voice information sent by the user and collected by a collecting device such as a microphone; the voice information may be preset, and the voice information corresponding to the abnormal problem may be determined according to the abnormal problem in the key feature data.

S219, sending a playing instruction to the unmanned aerial vehicle based on the 5G network, so that the unmanned aerial vehicle flies to an abnormal position indicated by the voice information based on the playing instruction to play the voice information.

The client may receive a flight command triggered by a user, generate a flight parameter, and send the flight parameter and the play command to the unmanned aerial vehicle, so that the unmanned aerial vehicle flies to an abnormal position to perform voice play. The method is helpful for prompting the user to stop the abnormal behavior in real time and timely treating the river risk.

In the embodiment, the unmanned aerial vehicle is controlled to patrol a preset area of a river channel based on a patrol instruction to obtain river channel image data, and after the river channel image data are processed, an orthographic image is obtained; and identifying the orthographic image through a detection model, determining key characteristic data comprising abnormal problems and abnormal positions in a preset area of the river channel, and determining risk prompt information corresponding to the key characteristic data based on the key characteristic data. Meanwhile, the risk level in the preset area of the river channel is determined based on the key characteristic data, an early warning report of the preset area of the river channel is generated, and the unmanned aerial vehicle can be controlled to play voice prompt information at the abnormal position based on the key characteristic data. According to the method, abnormal problems are identified through the unmanned aerial vehicle inspection and detection model, the efficiency of river risk inspection is improved, meanwhile, the key characteristic data are utilized to generate an early warning report and carry out voice playing prompt, and the convenience of river risk inspection result display and the timeliness of river risk processing are improved.

Fig. 6 is a schematic structural diagram of a river risk inspection device based on a 5G unmanned aerial vehicle. As shown in fig. 6, the apparatus 30 includes:

the sending unit 31 is configured to send an inspection instruction to the unmanned aerial vehicle based on the 5G network, where the inspection instruction is used to instruct the unmanned aerial vehicle to perform inspection aerial photography on a preset area of the river channel, and the inspection instruction indicates an inspection flight path of the river channel;

a receiving unit 32, configured to receive river image data transmitted by the unmanned aerial vehicle based on a 5G network;

a generating unit 33, configured to process the river image data to generate a plurality of orthographic images; the method comprises the steps that an orthographic image represents an image of a part of a preset area for inspecting a river channel, and a plurality of orthographic images represent complete images of the preset area for inspecting the river channel; performing feature extraction processing on the plurality of orthographic images to obtain at least one key feature data; the key characteristic data represent abnormal positions and abnormal problems on a preset area of the patrolled river channel;

the processing unit 34 is configured to process each piece of the key feature data, and obtain and output risk prompt information corresponding to each piece of the key feature data; the risk prompt information corresponding to each piece of key feature data represents the position information and the abnormal problem of the abnormal position indicated by the key feature data.

The river risk inspection device based on the 5G unmanned aerial vehicle can execute the river risk inspection method based on the 5G unmanned aerial vehicle in the embodiment of the method, and the implementation principle and the technical effect are similar and are not repeated here.

Fig. 7 is a schematic structural diagram of a river risk inspection device based on a 5G unmanned aerial vehicle. As shown in fig. 7, the apparatus 40 includes:

a sending unit 41, configured to send an inspection instruction to the unmanned aerial vehicle based on the 5G network, where the inspection instruction is used to instruct the unmanned aerial vehicle to perform inspection aerial photography on a preset area of the river channel, and the inspection instruction indicates an inspection flight path of the river channel;

a receiving unit 42, configured to receive river image data transmitted by the unmanned aerial vehicle based on a 5G network;

a first generating unit 43, configured to process the river image data to generate a plurality of orthographic images; the method comprises the steps that an orthographic image represents an image of a part of a preset area for inspecting a river channel, and a plurality of orthographic images represent complete images of the preset area for inspecting the river channel; performing feature extraction processing on the plurality of orthographic images to obtain at least one key feature data; the key characteristic data represent abnormal positions and abnormal problems on a preset area of the patrolled river channel;

The processing unit 44 is configured to process each piece of the key feature data, and obtain and output risk prompt information corresponding to each piece of the key feature data; the risk prompt information corresponding to each piece of key feature data represents the position information and the abnormal problem of the abnormal position indicated by the key feature data.

In one example, the river image data includes an image and data information, wherein the data information characterizes position information and azimuth information corresponding to the image; the first generation unit 43 includes:

the first processing module 431 is configured to perform image processing on the image in the river image data to obtain a processed image;

a joining module 432, configured to join a preset image control point in the processed image;

the second processing module 433 is configured to sequentially perform an analysis aerial triangulation process and a modeling process on the processed image after the preset image control point is added according to the preset image control point, so as to obtain a digital earth surface model map;

a third processing module 434, configured to perform stitching processing on the digital earth model map based on a preset stitching line, so as to obtain an image corresponding to the preset area; performing image output processing on the image corresponding to the preset area to obtain a digital orthographic image;

A fourth processing module 435, configured to perform framing processing on the obtained digital orthographic images to obtain the plurality of orthographic images.

In one example, the first generating unit 43 may further include:

a fifth processing module 436, configured to perform feature extraction processing on the plurality of orthographic images based on a preset detection model, so as to obtain at least one node encoded data; the node coding data are used for representing the character strings and abnormal positions of abnormal problems on a preset area of the patrolled river channel;

a determining module 437, configured to determine key feature data corresponding to each node coded data based on a preset data table; the preset data table represents the corresponding relation between the character strings of the abnormal problems and the abnormal problems.

In one example, the generating unit 43 may be further configured to obtain the annotated orthographic images of different river channels from a preset data warehouse; the preset data warehouse comprises marked orthographic images of a plurality of river channels; the marked orthographic image is obtained by marking the orthographic image of the river channel based on marking information and attribute information of a region corresponding to the orthographic image; and training the initial model based on the obtained marked orthophotos of different river channels to obtain the preset detection model.

In one example, the first generating unit 43 may be further configured to label each of the orthographic images in response to a labeling instruction, to obtain each labeled orthographic image; the marking instruction indicates marking information and attribute information of the area corresponding to the orthographic image; and storing each marked orthophoto image into a preset data warehouse.

In one example, the first generating unit 43 may be further configured to sequentially perform a size normalization process, a filtering process, and a noise reduction process on the river image data.

In one example, the above apparatus may further include a determining unit 45, configured to input each piece of key feature data into a first preset recognition model, to obtain a risk level corresponding to each piece of key feature data; and determining the highest risk level as the risk level of the preset area of the patrolled river channel.

In one example, the determining unit 45 may be further configured to input, for each orthographic image having key feature data, the orthographic image having the key feature data into a second preset recognition model to obtain text description information corresponding to the orthographic image; the character description information represents river channel information of an orthographic image with key characteristic data;

Generating processing measure information corresponding to each orthographic image with key characteristic data according to risk prompt information corresponding to the orthographic image; the processing measure information characterizes an abnormal processing mode of the risk prompt information;

and generating a river channel early warning report according to the text description information, the processing measure information and the risk level.

In one example, the apparatus may further include a playing unit 46 for acquiring voice information; the voice information is prompt voice for the abnormal position; generating a playing instruction based on the voice information, wherein the playing instruction carries the voice information and is used for indicating the unmanned aerial vehicle to fly to an abnormal position indicated by the voice information so as to play the voice; and sending the playing instruction to the unmanned aerial vehicle based on a 5G network, so that the unmanned aerial vehicle flies to an abnormal position indicated by the voice information based on the playing instruction to play the voice information.

In one example, the apparatus may further include a second generating unit 47, where before the sending unit 41 sends the inspection instruction to the unmanned aerial vehicle based on the 5G network, the second generating unit 47 is configured to generate the inspection instruction in response to the trigger instruction; wherein, the trigger instruction indicates the flight parameter of the unmanned aerial vehicle. After the sending unit 41 sends the inspection instruction to the unmanned aerial vehicle based on the 5G network, the second generating unit 47 is configured to receive the flight parameter sent by the unmanned aerial vehicle based on the 5G network, and display the flight parameter; wherein, the flight parameter characterizes the flight state of the unmanned aerial vehicle.

The river risk inspection device based on the 5G unmanned aerial vehicle can execute the river risk inspection method based on the 5G unmanned aerial vehicle in the embodiment of the method, and the implementation principle and the technical effect are similar and are not repeated here.

Fig. 8 is a schematic structural diagram of an electronic device provided in the present application. As shown in fig. 8, the electronic device may include at least one processor 501, a memory 502, and may be, for example, a computer, a tablet computer, or the like, having processing capabilities.

A memory 502 for storing a program. In particular, the program may include program code including computer-operating instructions. The memory 502 may comprise high-speed RAM memory or may further comprise non-volatile memory (non-volatile memory), such as at least one disk memory.

The processor 501 is configured to execute the computer-executable instructions stored in the memory 502, so as to implement the river risk inspection method based on the 5G unmanned aerial vehicle described in the foregoing method embodiment. The processor 501 may be a central processing unit (Central Processing Unit, abbreviated as CPU), or an application specific integrated circuit (Application Specific Integrated Circuit, abbreviated as ASIC), or one or more integrated circuits configured to implement embodiments of the present application.

The electronic device 500 may also include a communication interface 503 such that communication interactions with external devices may be performed through the communication interface 503. The external device may be, for example, a computer, tablet, cell phone, etc.

In a specific implementation, if the communication interface 503, the memory 502, and the processor 501 are implemented independently, the communication interface 503, the memory 502, and the processor 501 may be connected to each other and perform communication with each other through buses. The bus may be an industry standard architecture (Industry Standard Architecture, abbreviated ISA) bus, an external device interconnect (Peripheral Component, abbreviated PCI) bus, or an extended industry standard architecture (Extended Industry Standard Architecture, abbreviated EISA) bus, among others. Buses may be divided into address buses, data buses, control buses, etc., but do not represent only one bus or one type of bus.

Alternatively, in a specific implementation, if the communication interface 503, the memory 502, and the processor 501 are integrated on a chip, the communication interface 503, the memory 502, and the processor 501 may complete communication through internal interfaces.

The present application also provides a computer-readable storage medium, which may include: various media capable of storing program codes, such as a usb disk, a removable hard disk, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), a magnetic disk, or an optical disk, and specifically, the computer-readable storage medium stores computer-executable instructions for the 5G unmanned aerial vehicle-based river risk inspection method in the above embodiment.

The present application also provides a computer program product comprising execution instructions stored in a readable storage medium. The at least one processor of the electronic device 500 may read the execution instructions from the readable storage medium, and execution of the execution instructions by the at least one processor causes the electronic device 500 to implement the 5G unmanned aerial vehicle-based river risk inspection method provided in the various embodiments described above.

The application also provides a chip, wherein the chip is stored with a computer program, and when the computer program is executed by the chip, the river risk inspection method based on the 5G unmanned aerial vehicle provided by various embodiments is realized.

It should be noted that, for simplicity of description, the foregoing method embodiments are all expressed as a series of action combinations, but it should be understood by those skilled in the art that the present application is not limited by the order of actions described, as some steps may be performed in other order or simultaneously in accordance with the present application. Further, those skilled in the art will also appreciate that the embodiments described in the specification are all alternative embodiments, and that the acts and modules referred to are not necessarily required in the present application.

It should be further noted that, although the steps in the flowchart are sequentially shown as indicated by arrows, the steps are not necessarily sequentially performed in the order indicated by the arrows. The steps are not strictly limited to the order of execution unless explicitly recited herein, and the steps may be executed in other orders. Moreover, at least a portion of the steps in the flowcharts may include a plurality of sub-steps or stages that are not necessarily performed at the same time, but may be performed at different times, the order in which the sub-steps or stages are performed is not necessarily sequential, and may be performed in turn or alternately with at least a portion of the sub-steps or stages of other steps or other steps.

It should be understood that the above-described device embodiments are merely illustrative, and that the device of the present application may be implemented in other ways. For example, the division of the units/modules in the above embodiments is merely a logic function division, and there may be another division manner in actual implementation. For example, multiple units, modules, or components may be combined, or may be integrated into another system, or some features may be omitted or not performed.

In addition, each functional unit/module in each embodiment of the present application may be integrated into one unit/module, or each unit/module may exist alone physically, or two or more units/modules may be integrated together, unless otherwise specified. The integrated units/modules described above may be implemented either in hardware or in software program modules.

In the foregoing embodiments, the descriptions of the embodiments are emphasized, and for parts of one embodiment that are not described in detail, reference may be made to related descriptions of other embodiments. The technical features of the above embodiments may be combined in any way, and for brevity, all of the possible combinations of the technical features of the above embodiments are not described, but should be considered as the scope of the description

Other embodiments of the present application will be apparent to those skilled in the art from consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the application following, in general, the principles of the application and including such departures from the present disclosure as come within known or customary practice within the art to which the application pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the application being indicated by the following claims.

It is to be understood that the present application is not limited to the precise arrangements and instrumentalities shown in the drawings, which have been described above, and that various modifications and changes may be effected without departing from the scope thereof. The scope of the application is limited only by the appended claims.

Claims (13)

1. A river risk inspection method based on a 5G unmanned aerial vehicle is characterized by comprising the following steps:

sending a patrol instruction to the unmanned aerial vehicle based on the 5G network, wherein the patrol instruction is used for indicating the unmanned aerial vehicle to carry out patrol aerial photography on a preset area of a river channel, and the patrol instruction indicates a patrol flight route of the river channel;

receiving river channel image data transmitted by the unmanned aerial vehicle based on a 5G network;

processing the river image data to generate a plurality of orthographic images; the method comprises the steps that an orthographic image represents an image of a part of a preset area for inspecting a river channel, and a plurality of orthographic images represent complete images of the preset area for inspecting the river channel; performing feature extraction processing on the plurality of orthographic images to obtain at least one key feature data; the key characteristic data represent abnormal positions and abnormal problems on a preset area of the patrolled river channel;

Processing each piece of key feature data to obtain and output risk prompt information corresponding to each piece of key feature data; the risk prompt information corresponding to each piece of key feature data represents the position information and the abnormal problem of the abnormal position indicated by the key feature data.

2. The method of claim 1, wherein the river image data includes an image and data information, wherein the data information characterizes position information and azimuth information corresponding to the image; processing the river image data to generate a plurality of orthographic images, including:

performing image processing on the image in the river image data to obtain a processed image;

adding a preset image control point into the processed image;

according to the preset image control points, sequentially performing analysis air triangulation and modeling on the processed image added with the preset image control points to obtain a digital earth surface model diagram;

splicing the digital earth surface model map based on a preset splicing line to obtain an image corresponding to the preset area; performing image output processing on the image corresponding to the preset area to obtain a digital orthographic image;

And framing the obtained digital orthographic images to obtain a plurality of orthographic images.

3. The method of claim 1, wherein performing feature extraction processing on the plurality of orthographic images to obtain at least one key feature data comprises:

performing feature extraction processing on the plurality of orthographic images based on a preset detection model to obtain at least one node coding data; the node coding data are used for representing the character strings and abnormal positions of abnormal problems on a preset area of the patrolled river channel;

determining key feature data corresponding to each node coding data based on a preset data table; the preset data table represents the corresponding relation between the character strings of the abnormal problems and the abnormal problems.

4. A method according to claim 3, characterized in that the method further comprises:

acquiring marked orthographic images of different river channels from a preset data warehouse; the preset data warehouse comprises marked orthographic images of a plurality of river channels; the marked orthographic image is obtained by marking the orthographic image of the river channel based on marking information and attribute information of a region corresponding to the orthographic image;

And training the initial model based on the obtained marked orthophotos of different river channels to obtain the preset detection model.

5. The method according to claim 4, wherein the method further comprises:

labeling each of the orthographic images in response to a labeling instruction to obtain each labeled orthographic image; the marking instruction indicates marking information and attribute information of the area corresponding to the orthographic image;

and storing each marked orthophoto image into a preset data warehouse.

6. The method of claim 1, further comprising, prior to processing the river image data to generate a plurality of orthographic images:

and sequentially performing size normalization processing, filtering processing and noise reduction processing on the river image data.

7. The method according to claim 1, wherein the method further comprises:

inputting each piece of key characteristic data into a first preset identification model to obtain a risk level corresponding to each piece of key characteristic data;

and determining the highest risk level as the risk level of the preset area of the patrolled river channel.

8. The method of claim 7, wherein the method further comprises:

inputting the orthographic images with the key characteristic data into a second preset recognition model aiming at each orthographic image with the key characteristic data to obtain text description information corresponding to the orthographic images; the character description information represents river channel information of an orthographic image with key characteristic data;

generating processing measure information corresponding to each orthographic image with key characteristic data according to risk prompt information corresponding to the orthographic image; the processing measure information characterizes an abnormal processing mode of the risk prompt information;

and generating a river channel early warning report according to the text description information, the processing measure information and the risk level.

9. The method according to any one of claims 1-8, further comprising:

acquiring voice information; the voice information is prompt voice for the abnormal position; generating a playing instruction based on the voice information, wherein the playing instruction carries the voice information and is used for indicating the unmanned aerial vehicle to fly to an abnormal position indicated by the voice information so as to play the voice;

And sending the playing instruction to the unmanned aerial vehicle based on a 5G network, so that the unmanned aerial vehicle flies to an abnormal position indicated by the voice information based on the playing instruction to play the voice information.

10. The method of any one of claims 1-8, further comprising, prior to sending the inspection instructions to the drone based on the 5G network: responding to a trigger instruction, and generating the inspection instruction; the trigger instruction indicates flight parameters of the unmanned aerial vehicle;

after sending the inspection instruction to the unmanned aerial vehicle based on the 5G network, the method further comprises the following steps: receiving flight parameters sent by the unmanned aerial vehicle based on a 5G network, and displaying the flight parameters; wherein, the flight parameter characterizes the flight state of the unmanned aerial vehicle.

11. River risk inspection device based on 5G unmanned aerial vehicle, its characterized in that, the device includes:

the system comprises a sending unit, a receiving unit and a control unit, wherein the sending unit is used for sending a patrol instruction to the unmanned aerial vehicle based on a 5G network, the patrol instruction is used for indicating the unmanned aerial vehicle to carry out patrol aerial photography on a preset area of a river channel, and the patrol instruction indicates a patrol flight route of the river channel;

the receiving unit is used for receiving river channel image data transmitted by the unmanned aerial vehicle based on a 5G network;

The generation unit is used for processing the river image data and generating a plurality of orthographic images; the method comprises the steps that an orthographic image represents an image of a part of a preset area for inspecting a river channel, and a plurality of orthographic images represent complete images of the preset area for inspecting the river channel; performing feature extraction processing on the plurality of orthographic images to obtain at least one key feature data; the key characteristic data represent abnormal positions and abnormal problems on a preset area of the patrolled river channel;

the processing unit is used for processing each piece of key characteristic data to obtain and output risk prompt information corresponding to each piece of key characteristic data; the risk prompt information corresponding to each piece of key feature data represents the position information and the abnormal problem of the abnormal position indicated by the key feature data.

12. An electronic device, comprising: a processor, and a memory communicatively coupled to the processor;

the memory stores computer-executable instructions;

the processor executes computer-executable instructions stored in the memory to implement the method of any one of claims 1 to 10.

13. A computer readable storage medium having stored therein computer executable instructions which when executed by a processor are adapted to carry out the method of any one of claims 1 to 10.