CN116703860A

CN116703860A - Construction hidden danger detection method, training device, construction hidden danger detection equipment and storage medium

Info

Publication number: CN116703860A
Application number: CN202310667351.7A
Authority: CN
Inventors: 孟庆鲁; 杜福之; 刘畅
Original assignee: China United Network Communications Group Co Ltd
Current assignee: China United Network Communications Group Co Ltd
Priority date: 2023-06-06
Filing date: 2023-06-06
Publication date: 2023-09-05

Abstract

The application provides a construction hidden danger detection method, a training device, equipment and a storage medium, relates to the technical field of image processing, and is used for improving the efficiency of construction hidden danger detection. The method comprises the following steps: acquiring an image to be detected of a supervision area, wherein the supervision area is an area with a communication line; inputting the image to be detected into a construction hidden danger detection model, and outputting a construction hidden danger detection result, wherein the construction hidden danger detection result is used for indicating whether construction hidden danger exists in a supervision area corresponding to the image to be detected; and under the condition that the construction hidden danger detection result indicates that the construction hidden danger exists in the supervision area corresponding to the image to be detected, sending out alarm information, wherein the alarm information is used for indicating the supervision area corresponding to the image to be detected to maintain.

Description

Construction hidden danger detection method, training device, construction hidden danger detection equipment and storage medium

Technical Field

The present application relates to the field of image processing technologies, and in particular, to a method, a training method, a device, equipment, and a storage medium for detecting construction hidden danger.

Background

With the advancement of the national strategy of network, it is very important for operators to ensure that the communication line is not damaged by urban construction, however, the communication line is usually buried underground, and urban construction may bring about hidden danger of damage to the safety of the communication line, so that frequent inspection of the communication line is required. At present, the mode of guaranteeing the safety of the communication line mainly comprises the steps of arranging personnel every day to patrol the line, and because the communication line is longer, more personnel are needed, the line patrol time is longer, the efficiency of detecting construction hidden danger is lower, the construction hidden danger cannot be timely treated, the communication line is easily damaged by urban construction, and a user cannot normally use a communication network, so that the use experience of the user is influenced.

Disclosure of Invention

The application provides a construction hidden danger detection method, a training device, equipment and a storage medium, which are used for improving the efficiency of construction hidden danger detection.

In order to achieve the above purpose, the present application adopts the following technical scheme.

In a first aspect, a method for detecting a construction hidden trouble is provided, the method comprising:

acquiring an image to be detected of a supervision area, wherein the supervision area is an area with a communication line;

inputting the image to be detected into a construction hidden danger detection model, and outputting a construction hidden danger detection result, wherein the construction hidden danger detection result is used for indicating whether construction hidden danger exists in a supervision area corresponding to the image to be detected;

and under the condition that the construction hidden danger detection result indicates that the construction hidden danger exists in the supervision area corresponding to the image to be detected, sending out alarm information, wherein the alarm information is used for indicating the supervision area corresponding to the image to be detected to maintain.

The technical scheme provided by the application has at least the following beneficial effects: through with waiting to detect the image input to construction hidden danger detection model in to this detects the construction hidden danger, detects whether there is the construction hidden danger by the mode of manual inspection for present, has realized the automated inspection of construction hidden danger, has promoted the efficiency that the construction hidden danger detected. And under the condition that the construction hidden danger detection result indicates that the construction hidden danger exists in the supervision area corresponding to the image to be detected, alarm information is sent out, related personnel are timely reminded of maintaining the supervision area corresponding to the image to be detected, and the probability of damaging the communication line of the supervision area corresponding to the image to be detected is reduced.

Optionally, the construction hidden danger detection model comprises a feature extraction network and a feature fusion network, wherein the feature fusion network comprises a plurality of fusion paths, and different fusion paths are used for carrying out feature fusion on feature images with different resolutions; the feature extraction network is used for extracting feature images in the images to be detected and outputting the feature images of the images to be detected to target fusion paths in a plurality of fusion paths included in the feature fusion network; the target fusion path is used for carrying out feature fusion on the feature images of the images to be detected and outputting a construction hidden danger detection result; the target fusion path is a fusion path matched with the resolution of the characteristic image of the image to be detected in the multiple fusion paths.

Optionally, the construction hidden danger detection model is a construction hidden danger detection model based on a convolutional neural network algorithm.

In a second aspect, a training method for a construction hidden danger detection model is provided, and the method includes:

acquiring a training sample set, wherein the training sample set comprises one or more images marked with construction hidden danger;

performing augmentation treatment on the training sample set to obtain a treated training sample set; wherein the augmentation process comprises one or more of: adding noise, rotating samples, scaling samples, clipping samples, or flipping samples;

And training an initial construction hidden danger detection model according to the processed training sample set until a construction hidden danger detection model after training is completed is obtained.

Optionally, the construction hidden danger detection model comprises a feature extraction network and a feature fusion network; the feature fusion network comprises a plurality of fusion paths, and different fusion paths are used for carrying out feature fusion on feature images with different resolutions.

Optionally, the feature extraction network includes a CBS module with a convolution kernel size of 3×3, a res_unit1 module, a CBS module with a convolution kernel size of 3×3, a CBS module with a convolution kernel size of 1×1, a CBS module with a convolution kernel size of 3×3, a res_unit2 module and a CBS module with a convolution kernel size of 3×3, which are sequentially connected.

Optionally, the plurality of fusion paths includes a first fusion path, a second fusion path, and a third fusion path; the first fusion path is used for carrying out feature fusion on the feature images with the first resolution, the second fusion path is used for carrying out feature fusion on the feature images with the second resolution, the third fusion path is used for carrying out feature fusion on the feature images with the third resolution, and the first resolution is larger than the second resolution, and the second resolution is larger than the third resolution.

Optionally, the first fusion path includes a CBS module with a convolution kernel size of 3×3, a res_unit2 module, a CBS module with a convolution kernel size of 1×1, and a CBS module with a convolution kernel size of 3×3, which are sequentially connected; the second fusion path comprises a CBS module with a convolution kernel size of 3 multiplied by 3, a Res_Unit2 module, an up-sampling layer, a characteristic splicing layer, a Res_Unit2 module, a CBS module with a convolution kernel size of 1 multiplied by 1 and a CBS module with a convolution kernel size of 3 multiplied by 3 which are connected in sequence; the third fusion path comprises a CBS module with a convolution kernel size of 3 multiplied by 3, a Res_Unit2 module, an up-sampling layer, a feature splicing layer, a Res_Unit2 module, a CBS module with a convolution kernel size of 1 multiplied by 1 and a CBS module with a convolution kernel size of 3 multiplied by 3 which are connected in sequence.

Optionally, one CBS module includes a convolution layer, a batch normalization (Batch normalization, BN) layer, and a SiLU activation function connected in sequence; one Res_Unit1 module comprises a CBS module with a convolution kernel size of 3 multiplied by 3, a CBS module with a convolution kernel size of 1 multiplied by 1, an ABS module and a characteristic addition layer which are connected in sequence; an ABS module comprises an asymmetric convolution layer with a convolution kernel size of 3 multiplied by 1, an asymmetric convolution layer with a convolution kernel size of 1 multiplied by 3, a BN layer and a SiLU activation function which are connected in sequence; the Res_Unit2 module comprises a plurality of feature extraction paths and feature splicing layers which are connected in parallel, wherein one feature extraction path comprises a CBS module with a convolution kernel size of 3 multiplied by 3, a feature splicing layer, a CBS module with a convolution kernel size of 3 multiplied by 3, a feature addition layer and a CBS module with a convolution kernel size of 1 multiplied by 1, which are sequentially connected, and the CBS module with the convolution kernel size of 3 multiplied by 3 and the ABS module with the convolution kernel size of 1 multiplied by 1 are connected in parallel between the CBS module with the convolution kernel size of 3 multiplied by 3 and the feature splicing layer.

The embodiment of the application provides a training method of a construction hidden danger detection model, which improves the precision of the construction hidden danger detection model for the construction hidden danger detection by improving the model structure of the construction hidden danger detection model based on a convolutional neural network, further improves the precision of the construction hidden danger detection result, and is beneficial to reducing the probability of the communication line of a supervision area corresponding to an image to be detected being damaged.

In a third aspect, a construction hidden trouble detection device is provided, including:

the acquisition unit is used for acquiring an image to be detected of a supervision area, wherein the supervision area is an area with a communication line; the processing unit is used for inputting the image to be detected into the construction hidden danger detection model, outputting a construction hidden danger detection result, and indicating whether the construction hidden danger exists in the supervision area corresponding to the image to be detected; the sending unit is used for sending out alarm information when the construction hidden danger detection result indicates that the supervision area corresponding to the image to be detected has construction hidden danger, and the alarm information is used for indicating the supervision area corresponding to the image to be detected to maintain.

In a fourth aspect, a training device for a construction hidden danger detection model is provided, including:

the acquisition unit is used for acquiring a training sample set, wherein the training sample set comprises one or more images marked with construction hidden danger; a processing unit for: performing augmentation treatment on the training sample set to obtain a treated training sample set; wherein the augmentation process comprises one or more of: adding noise, rotating samples, scaling samples, clipping samples, or flipping samples; and training the initial construction hidden danger detection model according to the processed training sample set until the construction hidden danger detection model after training is completed is obtained.

Optionally, one CBS module includes a convolution layer, a batch standardized BN layer, and a SiLU activation function connected in sequence; one Res_Unit1 module comprises a CBS module with a convolution kernel size of 3 multiplied by 3, a CBS module with a convolution kernel size of 1 multiplied by 1, an ABS module and a characteristic addition layer which are connected in sequence; an ABS module comprises an asymmetric convolution layer with a convolution kernel size of 3 multiplied by 1, an asymmetric convolution layer with a convolution kernel size of 1 multiplied by 3, a BN layer and a SiLU activation function which are connected in sequence; the Res_Unit2 module comprises a plurality of feature extraction paths and feature splicing layers which are connected in parallel, wherein one feature extraction path comprises a CBS module with a convolution kernel size of 3 multiplied by 3, a feature splicing layer, a CBS module with a convolution kernel size of 3 multiplied by 3, a feature addition layer and a CBS module with a convolution kernel size of 1 multiplied by 1, which are sequentially connected, and the CBS module with the convolution kernel size of 3 multiplied by 3 and the ABS module with the convolution kernel size of 1 multiplied by 1 are connected in parallel between the CBS module with the convolution kernel size of 3 multiplied by 3 and the feature splicing layer.

In a fifth aspect, there is provided a network device comprising: a processor and a memory; the memory stores instructions executable by the processor; the processor is configured to, when executing the instructions, cause the network device to implement the method as provided in the first or second aspect described above.

In a sixth aspect, there is provided a computer readable storage medium storing computer instructions that, when run on a computer, cause the computer to perform the method provided in the first or second aspect.

In a seventh aspect, there is provided a computer program product comprising computer instructions which, when run on a computer, cause the computer to perform the method of the first or second aspect.

Technical effects caused by any possible implementation manners of the third aspect to the seventh aspect may be referred to technical effects caused by corresponding implementation manners of the first aspect or the second aspect, and are not described herein.

Drawings

The accompanying drawings are included to provide a further understanding of the application and are incorporated in and constitute a part of this specification, illustrate and do not limit the application.

FIG. 1 is a schematic diagram of a construction hidden danger detection system according to an embodiment of the present application;

fig. 2 is a schematic flow chart of a method for detecting construction hidden danger according to an embodiment of the present application;

fig. 3 is a schematic structural diagram of a CBS module according to an embodiment of the present application;

FIG. 4 is a schematic diagram of an ABS module according to an embodiment of the present application;

FIG. 5 is a schematic diagram of a Res_Unit1 module according to an embodiment of the present application;

FIG. 6 is a schematic diagram of a Res_Unit2 module according to an embodiment of the present application;

fig. 7 is a schematic diagram of a processing flow of a feature extraction network to a feature image according to an embodiment of the present application;

fig. 8 is a schematic diagram of a processing flow of a feature fusion network to a feature image according to an embodiment of the present application;

FIG. 9 is a schematic diagram of an image to be detected according to an embodiment of the present application;

FIG. 10 is a schematic flow chart of a training method of a construction hidden danger detection model according to an embodiment of the present application;

FIG. 11 is a schematic diagram of a construction hidden trouble detecting device according to an embodiment of the present application;

FIG. 12 is a schematic diagram of a training device for a construction hidden danger detection model according to an embodiment of the present application;

fig. 13 is a schematic hardware structure of a network device according to an embodiment of the present application.

Detailed Description

The following description of the embodiments of the present application will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present application, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the application without making any inventive effort, are intended to be within the scope of the application.

It should be noted that all directional indicators (such as up, down, left, right, front, and rear … …) in the embodiments of the present application are merely used to explain the relative positional relationship, movement, etc. between the components in a particular posture (as shown in the drawings), and if the particular posture is changed, the directional indicator is changed accordingly.

The terms "first," "second," and the like, are used for descriptive purposes only and are not to be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defining "a first" or "a second" may explicitly or implicitly include one or more such feature. In the description of the present application, unless otherwise indicated, the meaning of "a plurality" is two or more.

In the description of the present application, it should be noted that, unless explicitly stated and limited otherwise, the terms "connected," "connected," and "connected" are to be construed broadly, and may be, for example, fixedly connected, detachably connected, or integrally connected. The specific meaning of the above terms in the present application will be understood in specific cases by those of ordinary skill in the art. In addition, when describing a pipeline, the terms "connected" and "connected" as used herein have the meaning of conducting. The specific meaning is to be understood in conjunction with the context.

In embodiments of the application, words such as "exemplary" or "such as" are used to mean serving as an example, instance, or illustration. Any embodiment or design described herein as "exemplary" or "e.g." in an embodiment should not be taken as preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "such as" is intended to present related concepts in a concrete fashion.

At present, operators mainly arrange personnel to patrol the line every day in a mode of guaranteeing the safety of the communication line, and because the communication line is longer, the required personnel are more, the line patrol time is longer, and the labor-intensive daily line patrol scheduling mode is time-consuming and labor-consuming, and can not meet the requirements of intellectualization and modernization. How to promote the efficiency to construction hidden danger detection is the technical problem who needs to be solved urgently.

Based on the above, the embodiment of the application provides a construction hidden danger detection method, which is used for detecting the construction hidden danger by acquiring the image to be detected and inputting the image to be detected into a construction hidden danger detection model, so that the automatic detection of the construction hidden danger is realized and the efficiency of the detection of the construction hidden danger is improved compared with the existing detection of the construction hidden danger by a manual inspection mode. And under the condition that the construction hidden trouble detection result indicates that the construction hidden trouble exists in the region corresponding to the image to be detected, alarm information is sent out, related personnel are timely reminded of maintaining the region corresponding to the image to be detected, and the probability of damaging communication lines of the region corresponding to the image to be detected is reduced.

Embodiments of the present application will be described below with reference to the accompanying drawings.

As shown in fig. 1, the embodiment of the application provides a composition schematic diagram of a construction hidden danger detection system. This construction hidden danger detecting system includes: a construction hidden trouble detecting device 10 and a photographing device 20. The construction hidden trouble detecting device 10 and the photographing device 20 may be connected by a wired or wireless method.

The camera 20 may be disposed near the surveillance area. For example, taking a supervision area as an area where a communication line is located as an example, the imaging device 20 may be installed at a position where an image of the area where the communication line is located can be imaged, such as on top of a building where the communication line is located. The embodiment of the present application does not limit the specific installation manner and specific installation position of the photographing device 20.

The photographing device 20 may be used to photograph an image to be detected of the surveillance area.

In some embodiments, camera 20 may employ a color camera to capture color images.

The color camera may be an RGB camera, for example. The RGB camera adopts an RGB color mode, and obtains various colors through the changes of three color channels of red (red, R), green (G), blue (B) and the superposition of the three color channels. Typically, an RGB camera gives three basic color components from three different cables, and three separate charge coupled device (charge coupled device, CCD) sensors are used to acquire the three color signals.

In some embodiments, the camera may employ a depth camera to capture depth images.

By way of example, the depth camera may be a time of flight (TOF) camera. The TOF camera adopts TOF technology, and the imaging principle of the TOF camera is as follows: the method comprises the steps of emitting modulated pulse infrared light according to a laser light source, reflecting the modulated pulse infrared light after encountering an object, receiving the light source reflected by the object by a light source detector, converting the distance between a TOF camera and a shot object by calculating the time difference or the phase difference between the emission and the reflection of the light source, and further obtaining the depth value of each point in a scene according to the distance between the TOF camera and the shot object.

The construction hidden danger detection device 10 is configured to obtain an image to be detected captured by the capturing device 20, and determine whether a construction hidden danger exists in an area corresponding to the image to be detected based on the image to be detected captured by the capturing device 20.

In some embodiments, the construction hidden trouble detecting device 10 may be an independent server, a server cluster or a distributed system formed by a plurality of servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content distribution networks, and big data service networks.

In some embodiments, the construction hazard detection apparatus 10 may be a cell phone, tablet, desktop, laptop, handheld computer, notebook, ultra-mobile personal computer (UMPC), netbook, cell phone, personal digital assistant (personal digital assistant, PDA), augmented reality (augmented reality, AR) \virtual reality (VR) device, or the like.

In some embodiments, the construction hazard detection device 10 may communicate with other terminal devices in a wired or wireless manner, for example, communicate with the terminal device of the patrol personnel to send an alarm message to the terminal device of the patrol personnel.

It should be understood that fig. 1 is an exemplary schematic diagram, and the number of devices included in the construction hazard detection system shown in fig. 1 is not limited, for example, the number of image capturing apparatuses is not limited. The construction hazard detection system shown in fig. 1 may include other devices in addition to the device shown in fig. 1, and is not limited thereto.

The execution main body of the construction hidden danger detection method provided by the embodiment of the application is a construction hidden danger detection device. Alternatively, the construction hidden trouble detecting device may be the construction hidden trouble detecting device 10 described above; alternatively, the construction hazard detection device may be a processor in the construction hazard detection device 10; still alternatively, the construction hidden trouble detection device may be an Application (APP) for executing the construction hidden trouble detection method installed in the construction hidden trouble detection device 10; alternatively, the construction hidden trouble detecting device may be a functional module having an image processing function in the construction hidden trouble detecting device 10. The present examples are not limited in this regard.

Next, as shown in fig. 2, a flow chart of a method for detecting hidden construction hazards according to an embodiment of the present application is provided, where the method includes the following steps:

s101, acquiring an image to be detected of a supervision area.

The image to be detected is an image obtained by shooting the monitoring area by the shooting device. The supervision area is an area needing supervision on whether construction hidden danger exists or not. Such as the area in which the communication lines are located, etc., or the area with the communication lines.

In some embodiments, the supervision area may be determined by a construction hazard detection device. For example, a plurality of photographing devices are connected to the construction hidden trouble detecting device, and the construction hidden trouble detecting device may consider an area where each of the plurality of photographing devices is located as a supervision area.

In some embodiments, the surveillance area may be determined by the user in a direct or indirect manner. For example, an area where the communication line is located is provided with M photographing devices, and the user may select to close the N photographing devices, and the construction hidden trouble detecting device may select an area where each of the M to N photographing devices is located as the supervision area. Wherein M and N are positive integers.

In some embodiments, the image to be detected is used to record an image of the surveillance area at the current time.

In some embodiments, the construction hidden danger detection device executes the construction hidden danger detection method provided by the embodiment of the application after the construction hidden danger detection function is started. Correspondingly, if the construction hidden danger detection device closes the construction hidden danger detection function, the construction hidden danger detection device does not execute or stops executing the construction hidden danger detection method provided by the embodiment of the application.

In an alternative implementation, the construction hidden danger detection device defaults to turn on the construction hidden danger detection function.

In another alternative implementation, the construction hidden danger detection device periodically starts the construction hidden danger detection function. For example, the construction hazard detection device is 6 in the morning: 00-night 12:00 automatically opens construction hidden danger detection function, at night 12: 00-morning 6: and 00, the detection function of the hidden danger of construction is automatically closed.

In another alternative implementation manner, the construction hidden danger detection device determines to turn on/off the construction hidden danger detection function according to the instruction of the terminal device.

In some embodiments, the construction hidden danger detection device acquires the image to be detected through the photographing device after the construction hidden danger detection device starts the construction hidden danger detection function.

In some embodiments, the construction hidden danger detection device obtains the image to be detected of the supervision area through the shooting device, and may be specifically implemented as: the construction hidden danger detection device sends a shooting instruction to the shooting device, wherein the shooting instruction is used for instructing the shooting device to shoot an image of a supervision area; and then, the construction hidden danger detection device receives an image to be detected from the monitoring area of the shooting device.

Optionally, the image to be identified may be captured by the capturing device before the capturing instruction is received, or may be captured by the capturing device after the capturing instruction is received.

S102, inputting the image to be detected into a construction hidden danger detection model, and outputting a construction hidden danger detection result.

In some embodiments, the memory of the construction hidden danger detection device stores a construction hidden danger detection model after training is completed in advance, and after the construction hidden danger detection device obtains the image to be detected, the image to be detected can be input into the construction hidden danger detection model to obtain a construction hidden danger detection result of the supervision area.

In some embodiments, the construction hidden danger detection model may be a convolutional neural network model (convolutional neural networks, CNN), for example, implemented using a model structure of VGG-16.

Optionally, the construction hidden danger detection model comprises a feature extraction network and a feature fusion network. The construction hidden danger detection model comprises a feature extraction network and a feature fusion network, wherein the feature fusion network comprises a plurality of fusion paths, and different fusion paths are used for carrying out feature fusion on feature images with different resolutions; the feature extraction network is used for extracting feature images in the images to be detected and outputting the feature images of the images to be detected to target fusion paths in a plurality of fusion paths included in the feature fusion network; the target fusion path is used for carrying out feature fusion on the feature images of the images to be detected and outputting a construction hidden danger detection result; the target fusion path is a fusion path matched with the resolution of the characteristic image of the image to be detected in the multiple fusion paths.

In some embodiments, the feature extraction network is a backbone network of construction hidden danger detection models based on convolutional neural networks, including a CBS module, an ABS module, a res_unit1 module, and a res_unit2 module.

The following describes each module constituting the feature extraction network one by one.

Fig. 3 is a schematic structural diagram of a CBS module according to an embodiment of the present application. Referring to fig. 3, one CBS module includes a convolution layer, a batch normalization (Batch normalization, BN) layer, and a sulu activation function connected in sequence. The processing procedure of the CBS module for the image comprises the following steps: the feature extraction is performed by using a convolution layer, the batch standardization is performed by using a BN layer, the nonlinear transformation is performed by using a SiLU activation function, and the SiLU activation function can enable a model to better fit a linear function, so that a model learning process is faster and more stable. The image processing procedure of the CBS module may be as shown in the following formula (1):

Wherein G is _CBSO Representing the output characteristic diagram of the CBS module, G _CBSI Input representing CBS modules

The characteristic diagram is entered into in the feature diagram,representing the convolutional layer, B () representing the BN layer and S () representing the SiLU activation function.

Next, as shown in fig. 4, a schematic structural diagram of an ABS module according to an embodiment of the present application is shown. Referring to fig. 4, one ABS module includes an asymmetric convolution layer with a convolution kernel size of 3×1, an asymmetric convolution layer with a convolution kernel size of 1×3, a BN layer, and a SiLU activation function connected in sequence. The processing process of the ABS module for the image comprises the following steps: and extracting the characteristics by using an asymmetric convolution layer. It can be appreciated that the asymmetric convolution layer reduces the number of parameters and the amount of calculation while ensuring the feature extraction capability, reduces the information redundancy, and lightens the model. Next, batch normalization is performed using the BN layer and nonlinear transformation is performed using the SiLU activation function. The image processing process of the ABS module may be as shown in the following formula (2):

wherein G is _ABSO Output characteristic diagram of ABS module, G _ABSI Is an input characteristic diagram of the ABS module,for an asymmetric convolution layer with a convolution kernel size of 3 x 1>Is an asymmetric convolution layer with a convolution kernel size of 1 x 3.

Next, as shown in fig. 5, a schematic structural diagram of a res_unit1 module according to an embodiment of the present application is shown. Referring to fig. 5, one res_unit1 module includes a CBS module having a convolution kernel size of 3×3, a CBS module having a convolution kernel size of 1×1, an ABS module, and a feature addition layer, which are sequentially connected. The processing procedure of the Res_Unit1 module for the image comprises the following steps: for an input feature map, firstly, a CBS module with a convolution kernel size of 3 multiplied by 3 is used for feature extraction, then a CBS module with a convolution kernel size of 1 multiplied by 1 is used for feature extraction, then an ABS module is used for feature extraction, the process is circulated for N times, and the extracted feature map and the input feature map are subjected to feature addition through feature addition so as to realize short connection of a neural network, optimize gradient transfer and accelerate a model training process, and an output feature map of a Res_Unit1 module is obtained. Wherein N is an integer.

Next, as shown in fig. 6, a schematic structural diagram of a res_unit2 module provided in this embodiment of the present application, where a res_unit2 module includes a plurality of feature extraction paths connected in parallel and a feature stitching layer, and an exemplary case is that the res_unit2 module includes 3 feature extraction paths, referring to fig. 6, one feature extraction path includes a CBS module with a convolution kernel size of 3×3, a feature stitching layer, a CBS module with a convolution kernel size of 3×3, a feature addition layer, and a CBS module with a convolution kernel size of 1×1, which are sequentially connected, and a CBS module with a convolution kernel size of 1×1 and an ABS module are connected in parallel between the CBS module with a convolution kernel size of 3×3 and the feature stitching layer. The processing procedure of the Res_Unit2 module for the image comprises the following steps:

first, for the first feature extraction path, for the input feature map, feature map G is first extracted using CBS module with convolution kernel size 3×3 _r1 The process may be as shown in the following formula (3).

G _r1 ＝C ₃ (G _rui ) Formula (3)

Wherein G is _rui To input the feature map, C ₃ () Is a CBS module with a convolution kernel size of 3 x 3.

Then the CBS module and the ABS module with the convolution kernel size of 1 multiplied by 1 are utilized to respectively extract the feature map G _r2 And G _r3 The feature map G will then _r2 And feature map G _r3 Concat channel splicing is carried out through the feature splicing layer to obtain a feature graph G _r4 The features extracted by the convolution kernels of different receptive fields are fused, so that the extracted features are more abundant, and then the CBS module with the convolution kernel size of 3 multiplied by 3 is utilized to extract the features from the feature graph G _r4 Extracting features to obtain a feature map G _r5 Then the characteristic diagram G _r1 And feature map G _r5 Feature addition is carried out through a feature addition layer to obtain a feature graph G _r6 By constructing the neural network short connection, optimizing gradient transfer, and then utilizing a CBS module with the convolution kernel size of 3 multiplied by 3 to obtain a characteristic diagram G _r6 Extracting features to obtain a feature map G _r7 The procedure is shown in the following formula (4).

G _r7 ＝C ₁ (C ₃ (Concat(C ₁ (G _r1 )，A(G _r1 )))+G _r1 ) Formula (4)

Wherein C is ₁ () Is convolution kernel size 1 x 1And a CBS module, wherein A () is an ABS module, and Concat () is a channel splicing operation.

For the second feature extraction path, feature G is extracted using a CBS module with a convolution kernel size of 3×3 for the input feature map _r8 The procedure is shown in the following formula (5).

G _r8 ＝C ₃ (G _rui ) Formula (5)

Then the CBS module with the convolution kernel size of 1 multiplied by 1 is used for the characteristic graph G _r8 Extracting features to obtain a feature graph G _r9 The ABS module is utilized to make a characteristic diagram G _r8 Extracting features to obtain a feature graph G _r10 . Feature G is then followed _r9 Graph and feature graph G _r10 Concat channel splicing is carried out through the feature splicing layer to obtain a feature graph G _r11 The feature map G is then mapped using a CBS module with a convolution kernel size of 3×3 _r11 Extracting features to obtain a feature map G _r12 Then the characteristic diagram G _r8 And feature map G _r12 Feature addition is carried out through a feature addition layer to obtain a feature graph G _r13 The feature map G is then mapped using a CBS module with a convolution kernel size of 3×3 _r13 Extracting features to obtain a feature map G _r14 The process is shown in formula (6).

G _r14 ＝C ₁ (C ₃ (Concat(C ₁ (G _r8 )，A(G _r8 )))+G _r8 ) Formula (6)

For the third feature extraction path, feature map G is first extracted from the input feature map using a CBS module with a convolution kernel size of 3×3 _r15 The procedure is shown in the following formula (7).

G _r15 ＝C ₃ (G _rui ) Formula (7)

Then the CBS module with the convolution kernel size of 1 multiplied by 1 is used for the characteristic graph G _r8 Extracting features to obtain a feature graph G _r16 The ABS module is utilized to make a characteristic diagram G _r15 Extracting features to obtain a feature graph G _r17 . The feature map G will then be _r16 And feature map G _r17 Concat channel splicing is carried out through the feature splicing layer to obtain a feature graph G _r18 Then use convolution kernel size 3X3 CBS Module from feature map G _r18 Extracting feature map G _r19 Then the characteristic diagram G _r15 And feature map G _r19 Feature addition is carried out through a feature addition layer to obtain a feature graph G _r20 The feature map G is then mapped using a CBS module with a convolution kernel size of 3×3 _r20 Extracting to obtain a characteristic diagram G _r21 The procedure is shown in the following formula (8).

G _r21 ＝C ₁ (C ₃ (Concat(C ₁ (G _r15 )，A(G _r15 )))+G _r15 ) Formula (8)

Finally, the extracted feature map Gr7, the feature map Gr14 and the feature map Gr21 are spliced through a Concat channel by a feature splicing layer, the feature maps extracted by three feature extraction paths are fused, the feature extraction capacity is enhanced, the extracted features are enriched, the final detection result is optimized, and the output feature G of the Res_Unit2 module is obtained _r22 The procedure is shown in the following formula (9).

G _r21 ＝Concat(G _r7 ，G _r14 ，G _r21 ) Formula (9)

It should be noted that, in the above embodiment, the first feature extraction path performs feature extraction first, then the second feature extraction path performs feature extraction, and finally the third feature extraction path performs feature extraction, which is merely exemplary, and the multiple feature extraction paths may also perform feature extraction on the input feature map at the same time.

The above is an exemplary description of the CBS module, the ABS module, the res_unit1 module, and the res_unit2 module, and the following describes a processing procedure of the feature extraction network of the construction hidden danger detection model provided by the embodiment of the present application in combination with the above description of each module that composes the feature extraction network.

In some embodiments, the feature extraction network includes a CBS module with a convolution kernel size of 3×3, a res_unit1 module, a CBS module with a convolution kernel size of 3×3, a CBS module with a convolution kernel size of 1×1, a CBS module with a convolution kernel size of 3×3, a res_unit2 module, and a CBS module with a convolution kernel size of 3×3, connected in sequence.

Exemplary, as shown in fig. 7, a schematic diagram of a processing flow of an image by a feature extraction network according to an embodiment of the present application is shown. The processing procedure of the feature extraction network for the feature image will be described below with reference to fig. 7.

For an input image to be detected, a CBS module with a convolution kernel size of 3 multiplied by 3 is used for extracting a feature graph G ₁ The process is shown in the following formula (10).

G ₁ ＝C ₃ (F) Formula (10)

Wherein F represents an input image to be detected, C ₃ Is a CBS module with a convolution kernel size of 3 x 3. Then extract feature map G using Res_Unit1 Module ₂ N is set to 1 to ensure that the original features of the image to be detected remain to the maximum extent, as shown in the following equation (11).

G ₂ ＝R ₁₁ (G ₁ ) Formula (11)

Wherein R is ₁₁ () Res_Unit1 block with N being 1. Feature map G is then extracted using a CBS module with a convolution kernel size of 3×3 ₃ Feature map G using Res_Unit1 module with N3 ₃ Extracting features to obtain a feature map G ₄ Thus, nonlinear transformation of the features is ensured, and a model fitting process is optimized, wherein the process is shown in a formula (12).

G ₄ ＝R ₁₃ (C ₃ (G ₂ ) Formula (12)

Wherein R is ₁₃ () Res_Unit1 block with N3. The feature map G is then mapped using a CBS module with a convolution kernel size of 3×3 ₄ Extracting to obtain a characteristic diagram G ₅ Feature map G using CBS modules with convolution kernel size of 1×1 ₅ Extracting features to obtain a feature map G ₆ Then, the Res_Unit1 module with N being 5 is used for the feature graph G ₆ And extracting to obtain a characteristic diagram G7. Thus, through multi-layer short connection, gradient descent is guaranteedThe effectiveness, the procedure is shown in the following equation (13).

G ₇ ＝R ₁₅ (C ₁ (C ₃ (G ₄ ) Formula (13))

Wherein R is ₁₅ () Res_Unit1 block with N5. Then, feature extraction is carried out on the feature graph G7 by utilizing a CBS module with the convolution kernel size of 3 multiplied by 3 to obtain a feature graph G ₈ Feature map G using CBS modules with convolution kernel size of 1×1 ₈ Extracting features to obtain a feature map G ₉ Feature map G using CBS modules with convolution kernel size of 3×3 ₉ Extracting features to obtain a feature map G ₁₀ Feature map G by Res_Unit2 module ₁₀ Extracting features to obtain a feature map G ₁₁ . Thus, the Res_Unit2 module is utilized to multiplex and extract the features, the feature extraction process is enriched, the feature is guaranteed to be transmitted to the end of the network, and the process is shown in a formula (14).

G ₁₁ ＝R ₂ (C ₃ (C ₁ (C ₃ (G ₄ ) ) formula (14)

Wherein R is ₂ () Representing a Res _ Unit2 module. Finally, feature map G is mapped using CBS modules with convolution kernel size 3×3 ₁₁ Extracting features to obtain a feature map G ₁₂ ，G ₁₂ I.e. the feature image extracted from the image to be detected by the feature extraction network.

In some embodiments, a feature image G is extracted from the image to be detected at a feature extraction network ₁₂ After that, the feature image G can be ₁₂ Outputting the feature images G and the multiple fusion paths to the feature fusion network ₁₂ Is a target fusion path with matched resolution. The target fusion path is used for carrying out feature fusion on the feature images of the images to be detected and outputting detection results of construction hidden danger. The construction hidden danger detection result is used for indicating whether construction hidden danger exists in the supervision area corresponding to the image to be detected.

It will be appreciated that one feature image is of resolution. After the feature extraction network performs feature extraction on the image to be detected to obtain the feature image of the image to be detected, the feature image of the image to be detected is input into a target fusion path matched with the resolution of the feature image of the image to be detected in a plurality of fusion paths included in the feature fusion network, so that the feature fusion path performs feature fusion on the feature image of the image to be detected, the feature fusion matching degree can be improved, and the accuracy of construction hidden danger detection results is improved.

In some embodiments, the plurality of fusion paths included in the feature fusion network includes a first fusion path, a second fusion path, and a third fusion path; the first fusion path is used for carrying out feature fusion on the feature images with the first resolution, the second fusion path is used for carrying out feature fusion on the feature images with the second resolution, the third fusion path is used for carrying out feature fusion on the feature images with the third resolution, and the first resolution is larger than the second resolution, and the second resolution is larger than the third resolution.

Illustratively, the first resolution is 96×96, the second resolution is 64×64, and the third resolution is 32×32. It should be noted that, the values of the resolutions are merely exemplary, and the embodiments of the present application are not limited to the values of the resolutions.

That is, assuming that the resolution of the feature image obtained after feature extraction is performed on the image to be detected by the feature extraction network is 96×96, the feature extraction network determines that the target fusion path is the first fusion path, and then the feature extraction network inputs the feature image with the resolution of 96×96 into the first fusion path, and feature fusion of the feature image of the image to be detected is completed by the first fusion path, so as to output a final detection result of hidden construction hazards.

In some embodiments, the first fusion path includes a CBS module with a convolution kernel size of 3×3, a res_unit2 module, a CBS module with a convolution kernel size of 1×1, and a CBS module with a convolution kernel size of 3×3 connected in sequence; the second fusion path comprises a CBS module with a convolution kernel size of 3 multiplied by 3, a Res_Unit2 module, an up-sampling layer, a characteristic splicing layer, a Res_Unit2 module, a CBS module with a convolution kernel size of 1 multiplied by 1 and a CBS module with a convolution kernel size of 3 multiplied by 3 which are connected in sequence; the third fusion path comprises a CBS module with a convolution kernel size of 3 multiplied by 3, a Res_Unit2 module, an up-sampling layer, a feature splicing layer, a Res_Unit2 module, a CBS module with a convolution kernel size of 1 multiplied by 1 and a CBS module with a convolution kernel size of 3 multiplied by 3 which are connected in sequence.

The processing procedure of the feature fusion network for the feature image will be described below with reference to a specific example. Fig. 8 is a schematic diagram of a processing flow of a feature fusion network to a feature image according to an embodiment of the present application.

Optionally, for the feature image G of the image to be detected output by the feature extraction network ₁₂ If the characteristic image G ₁₂ If the resolution of the target fusion path is the first resolution, determining the target fusion path as the first fusion path, and using the first fusion path to detect the characteristic image G of the image to be detected ₁₂ Fusion is performed.

Illustratively, in the case where the target fusion path is the first fusion path, the feature image G is first derived using a CBS module with a convolution kernel size of 3×3 ₁₂ Extracting features G from ₁₃ Then from feature G using Res_Unit2 module ₁₃ Extracting features G from ₁₄ Feature G is derived from a feature using a CBS module with a convolution kernel size of 1×1 ₁₄ Extracting feature G ₁₅ Feature G is derived from a feature using a CBS module with a convolution kernel size of 3×3 ₁₅ Extracting feature G ₁₆ The characteristic value G ₁₆ As a predicted value, a predicted value is further output. I.e. feature G ₁₆ And (5) as a construction hidden danger detection result, and outputting the construction hidden danger detection result.

In some embodiments, a plurality of features G are obtained at the end ₁₆ Can be based on non-maximum suppression (maximum suppression, NMS) algorithm from multiple features G ₁₆ Extracting target feature G ₁₆ And then target feature G ₁₆ And outputting the result as a final detection result of the hidden danger of construction.

Wherein, NMS is an algorithm for removing non-maximum value, which is commonly used for edge detection, object recognition, face detection, target detection (DPM, YOLO, SSD, fast R-CNN) and the like in computer vision. The idea is to search for local maxima and suppress non-maxima.

Optionally, continue to take part inReferring to FIG. 8, if the feature image G ₁₂ If the resolution of the image to be detected is the second resolution, determining the target fusion path as a second fusion path, and using the second fusion path to detect the characteristic image G of the image to be detected ₁₂ Fusion is performed.

Illustratively, in the case where the target fusion path is the second fusion path, the CBS module with the convolution kernel size of 3×3 is first utilized to extract G from the feature image ₁₂ Extracting feature G ₁₃ Then from feature G using Res_Unit2 module ₁₃ Extracting features G from ₁₄ Then feature G ₁₄ Input to an upsampling layer for upsampling, e.g. feature G ₁₄ Performing double up-sampling, and then performing double up-sampling on the characteristic G ₁₄ Input to a feature stitching layer for feature stitching, e.g. with feature G extracted by a feature extraction network ₈ Concat feature stitching is carried out to obtain a feature G ₁₇ . The procedure is shown in the following equation (15).

G ₁₇ ＝Concat(E(G ₁₄ )，G ₈ ) Formula (15)

Where E () is double upsampling. It will be appreciated that the double up-sampled feature G ₁₄ Features G extracted from feature extraction network ₈ Concat feature stitching is performed to fuse features of the shallow network and the deep network, wherein the shallow network contains more detailed information, the deep network contains more semantic information, and the features of the shallow network and the deep network are fused (namely, features G of the feature extraction network are fused ₈ Feature G of feature fusion network ₁₄ Fusion) can ensure the combination of details and semantics, and optimize the final prediction result (namely, optimize the final detection result of hidden construction hazards).

With continued reference to FIG. 8 above, a feature G is obtained ₁₇ Thereafter, feature G is then removed from the memory using Res_Unit2 module ₁₇ Extracting features G from ₁₈ Feature G is derived from a feature using a CBS module with a convolution kernel size of 1×1 ₁₈ Extracting features G from ₁₉ Feature G is derived from a feature using a CBS module with a convolution kernel size of 3×3 ₁₉ Extracting features G from ₂₀ Feature G ₂₀ As a predicted value, a predicted value is further output. Also is provided withI.e. feature G ₂₀ And (5) as a construction hidden danger detection result, and outputting the construction hidden danger detection result.

Similarly, if the second fusion path finally outputs a plurality of features G ₂₀ Then from a plurality of features G according to NMS ₂₀ Is selected to be the target feature G ₂₀ Then the target feature G ₂₀ And outputting the detection result as a final construction hidden danger detection result.

Alternatively, with continued reference to fig. 8 above, if the feature image G ₁₂ If the resolution of the target fusion path is the third resolution, determining the target fusion path as a third fusion path, and utilizing the third fusion path to detect the characteristic image G of the image to be detected ₁₂ Fusion is performed.

Exemplary, in the case where the target fusion path is the third fusion path, first, in the case where the target fusion path is the second fusion path, first, G is derived from the feature image using a CBS module having a convolution kernel size of 3×3 ₁₂ Extracting feature G ₁₃ Then from feature G using Res_Unit2 module ₁₃ Extracting features G from ₁₄ Then feature G ₁₄ Input to an upsampling layer for upsampling, e.g. feature G ₁₄ Performing double up-sampling, and then performing double up-sampling on the characteristic G ₁₄ Input to a feature stitching layer for feature stitching, e.g. with feature G extracted by a feature extraction network ₈ Concat feature stitching is carried out to obtain a feature G ₁₇ Then from feature G using Res_Unit2 module ₁₇ Extracting features G from ₁₈ Then feature G ₁₈ Input to an upsampling layer for upsampling, e.g. feature G ₁₈ Performing double up-sampling, and then performing double up-sampling on the characteristic G ₁₈ Input to a feature stitching layer for feature stitching, e.g. with feature G extracted by a feature extraction network ₅ Concat feature stitching is carried out to obtain a feature G ₂₁ The process is shown in the following formula (16).

G ₂₁ ＝Concat(E(G ₁₈ )，G ₅ ) Formula (16)

It will be appreciated that feature G ₅ And feature G ₁₈ Performing feature stitching to complete shallow network and deep networkThe final prediction result (namely, the detection result of the hidden construction trouble) can be optimized.

With continued reference to FIG. 8 above, a feature G is obtained ₂₁ Thereafter, feature G is then removed from the memory using Res_Unit2 module ₂₁ Extracting features G from ₂₂ Feature G is derived from a feature using a CBS module with a convolution kernel size of 1×1 ₂₂ Extracting features G from ₂₃ Feature G is derived from a feature using a CBS module with a convolution kernel size of 3×3 ₂₃ Extracting features G from ₂₄ Feature G ₂₄ As a predicted value, a predicted value is further output. I.e. feature G ₂₄ And (5) as a construction hidden danger detection result, and outputting the construction hidden danger detection result.

Similarly, if the second fusion path finally outputs a plurality of features G ₂₄ Then from a plurality of features G according to NMS ₂₄ Is selected to be the target feature G ₂₄ Then the target feature G ₂₄ And outputting the detection result as a final construction hidden danger detection result.

And S103, sending out alarm information under the condition that the construction hidden danger detection result indicates that the construction hidden danger exists in the supervision area corresponding to the image to be detected.

The alarm information is used for indicating a supervision area corresponding to the image to be detected to be maintained.

By way of example, taking a construction hidden danger as an excavator, when the construction hidden danger detection result indicates that the excavator exists in the supervision area corresponding to the image to be detected, the construction hidden danger detection device can send alarm information to the terminal device of the inspection personnel so as to prompt the inspection personnel to maintain the supervision area corresponding to the image to be detected.

For example, taking the area corresponding to the image to be detected as the area a, the content of the alarm information may be that "the area a has a construction hidden trouble, and please process in time".

In some embodiments, the alarm information may include an image to be detected, where the image to be detected may display a current time and a current position, so that a patrol person can quickly locate a position of a supervision area where a construction hidden danger exists, and timely notify a relevant attended person to carry out nursing and rush repair. The location may be the longitude and latitude of the supervision area corresponding to the image to be detected.

Exemplary, as shown in fig. 9, a schematic diagram of an image to be detected according to an embodiment of the present application is provided. The image to be detected shows time 2022-08-18:10:24:35, and longitude is

116.880332, latitude 36.648828. The patrol personnel can quickly locate the position of the area with construction hidden danger based on the image to be detected.

It should be noted that, taking the construction hidden danger as an excavator as an example only, the construction hidden danger may be other, such as a pile driver, etc., which is not limited by the embodiment of the present application.

Based on the embodiment shown in fig. 2, the to-be-detected image is input into the construction hidden danger detection model, so that the detection of the construction hidden danger is performed, compared with the existing detection of whether the construction hidden danger exists in a manual inspection mode, the automatic detection of the construction hidden danger is realized, the uninterrupted detection of the construction hidden danger of the communication line for 7×24 hours is realized, the discovery time of the original construction hidden danger is shortened from an hour level to a minute level, and the efficiency of the detection of the construction hidden danger is improved. And under the condition that the construction hidden danger detection result indicates that the construction hidden danger exists in the supervision area corresponding to the image to be detected, alarm information is sent out, related personnel are timely reminded of maintaining the supervision area corresponding to the image to be detected, and the probability of damaging the communication line of the supervision area corresponding to the image to be detected is reduced.

In some embodiments, the present application further provides a training method of the construction hidden danger detection model, where the method is performed by a training device of the construction hidden danger detection model, and the training device of the construction hidden danger detection model may be independent of the construction hidden danger detection device or may be integrated with the construction hidden danger detection device. In the case that the training device of the construction hidden danger detection model is independent of the construction hidden danger detection device, the training device of the construction hidden danger detection model may be a device with a processing function, and in particular, the description of the training device of the construction hidden danger detection model may refer to the description of the construction hidden danger detection device in fig. 1, which is not repeated herein.

As shown in fig. 10, the method includes the steps of:

s201, acquiring a training sample set.

Wherein the training sample set includes one or more images labeled with construction hazards.

In some embodiments, the training device of the construction hidden danger detection model may take a live photograph of the construction hidden danger of the communication line in real time through the photographing device, or the training device of the construction hidden danger detection model obtains a live image of the construction hidden danger of the communication line by opening a source from a network, and the image includes different photographing heights, lighting conditions, weather conditions and photographing distances. The data acquired by the open source may find text detection identification data of the open source, such as ICDAR2019-LSVT, which contains 50,000 pictures with complete annotations. The data synthesized by the tool can use a text_render open source item (item on the github), and the text_render open source item has some randomness, such as random fonts, random font sizes, random picture text contents and the like, so that the diversity of the constructed image results is ensured, and the data can also be synthesized by using other modes.

And then coding the face and the license plate in the image, and marking the construction hidden trouble in the image by using a marking tool to obtain a training sample set.

S202, performing augmentation treatment on the training sample set to obtain a treated training sample set.

Wherein the augmentation process comprises one or more of: adding noise, rotating samples, scaling samples, cropping samples, or flipping samples.

It can be appreciated that under the condition of insufficient data volume, the data augmentation can often improve the training effect of the model, and the data augmentation process adopted by the application comprises the following steps: adding noise, rotating samples, scaling samples, clipping samples, flipping samples, etc., may also include warping samples, telescoping samples, dialysis samples, blurring samples, dithering samples, color inverting samples, etc., to increase the amount of training data.

In some embodiments, the rotation angle includes 90 degrees, 180 degrees, and 270 degrees when rotating the sample. It can be appreciated that the robustness of the construction hidden danger detection model to the construction hidden danger detection in all directions can be enhanced by rotating the image.

In some embodiments, the added noise includes: gaussian noise, poisson noise, multiplicative noise, and pretzel noise.

S203, training an initial construction hidden danger detection model according to the processed training sample set until the construction hidden danger detection model after training is completed is obtained.

In some embodiments, after the processed training sample set is obtained, the processed training sample set may be divided into a training set and a test set according to a preset ratio. And further training the initial construction hidden danger detection model according to the training set, and testing the trained construction hidden danger detection model according to the test until the construction hidden danger detection model after the training is completed is obtained.

Optionally, the preset ratio is 3:1.

In some embodiments, the construction hidden danger detection model includes a feature extraction network and a feature fusion network, and for specific descriptions of the feature extraction network and the feature fusion network, reference may be made to corresponding descriptions in the embodiment shown in fig. 2, which are not described herein.

In some embodiments, when the training device of the construction hidden danger detection model is independent of the construction hidden danger detection device, after the training device of the construction hidden danger detection model obtains the construction hidden danger detection model after training, the construction hidden danger detection model after training can be issued to the construction hidden danger detection device, so that the construction hidden danger detection device can detect the construction hidden danger according to the construction hidden danger detection model after training.

Based on the embodiment shown in fig. 10, by improving the model structure of the construction hidden danger detection model based on the convolutional neural network, the precision of the construction hidden danger detection model on the construction hidden danger detection is improved, and further the precision of the construction hidden danger detection result is improved, so that the probability of the communication line of the supervision area corresponding to the image to be detected being damaged is reduced.

The foregoing description of the solution provided by the embodiments of the present application has been mainly presented in terms of a method. To achieve the above functions, it includes corresponding hardware structures and/or software modules that perform the respective functions. Those of skill in the art will readily appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as hardware or combinations of hardware and computer software. Whether a function is implemented as hardware or computer software driven hardware depends upon the particular application and design constraints imposed on the solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.

The present application may divide the functional modules of the network device according to the above method example, for example, each functional module may be divided corresponding to each function, or two or more functions may be integrated into one processing module. The integrated modules may be implemented in hardware or in software functional modules. It should be noted that, the division of the modules in the present application is illustrative, and is merely a logic function division, and other division manners may be implemented in practice.

As shown in fig. 11, an embodiment of the present application provides a construction hidden trouble detection device for executing the construction hidden trouble detection method shown in any one of the above. This construction hidden danger detection device 2000 includes: an acquisition unit 2001, a processing unit 2002, and a transmission unit 2003. In some embodiments, the construction hazard detection device 2000 may further include a storage unit 2004.

In some embodiments, the acquiring unit 2001 is configured to acquire an image to be detected of a supervision area, where the supervision area is an area having a communication line.

The processing unit 2002 is configured to input an image to be detected into a construction hidden danger detection model, and output a construction hidden danger detection result, where the construction hidden danger detection result is used to indicate whether a construction hidden danger exists in a supervision area corresponding to the image to be detected.

And the sending unit 2003 is configured to send out alarm information when the construction hidden danger detection result indicates that the construction hidden danger exists in the supervision area corresponding to the image to be detected, where the alarm information is used to indicate that the supervision area corresponding to the image to be detected is maintained.

In some embodiments, the construction hidden danger detection model comprises a feature extraction network and a feature fusion network, wherein the feature fusion network comprises a plurality of fusion paths, and different fusion paths are used for carrying out feature fusion on feature images with different resolutions; the feature extraction network is used for extracting feature images in the images to be detected and outputting the feature images of the images to be detected to target fusion paths in a plurality of fusion paths included in the feature fusion network; the target fusion path is used for carrying out feature fusion on the feature images of the images to be detected and outputting a construction hidden danger detection result; the target fusion path is a fusion path matched with the resolution of the characteristic image of the image to be detected in the multiple fusion paths. In some embodiments, the storage unit 2004 is used to store a construction hazard detection model.

In some embodiments, the construction hidden danger detection model is a construction hidden danger detection model based on a convolutional neural network algorithm.

As shown in fig. 12, an embodiment of the present application provides a training apparatus for a construction hidden danger detection model, which is configured to execute the training method for the construction hidden danger detection model shown in any one of the above. This construction hidden danger detects trainer 3000 of model includes: an acquisition unit 3001, and a processing unit 3002. In some embodiments, the training device 3000 for the construction hidden danger detection model may further include a transmitting unit 3003.

In some embodiments, the acquiring unit 3001 is configured to acquire a training sample set, where the training sample set includes one or more images marked with construction hazards.

A processing unit 3002 configured to: performing augmentation treatment on the training sample set to obtain a treated training sample set; wherein the augmentation process comprises one or more of: adding noise, rotating samples, scaling samples, clipping samples, or flipping samples; and training the initial construction hidden danger detection model according to the processed training sample set until the construction hidden danger detection model after training is completed is obtained.

In some embodiments, the construction hidden danger detection model includes a feature extraction network and a feature fusion network; the feature fusion network comprises a plurality of fusion paths, and different fusion paths are used for carrying out feature fusion on feature images with different resolutions.

In some embodiments, the plurality of fusion paths includes a first fusion path, a second fusion path, and a third fusion path; the first fusion path is used for carrying out feature fusion on the feature images with the first resolution, the second fusion path is used for carrying out feature fusion on the feature images with the second resolution, the third fusion path is used for carrying out feature fusion on the feature images with the third resolution, and the first resolution is larger than the second resolution, and the second resolution is larger than the third resolution.

In some embodiments, one CBS module comprises a convolution layer, a batch normalization BN layer, and a SiLU activation function connected in sequence; one Res_Unit1 module comprises a CBS module with a convolution kernel size of 3 multiplied by 3, a CBS module with a convolution kernel size of 1 multiplied by 1, an ABS module and a characteristic addition layer which are connected in sequence; an ABS module comprises an asymmetric convolution layer with a convolution kernel size of 3 multiplied by 1, an asymmetric convolution layer with a convolution kernel size of 1 multiplied by 3, a BN layer and a SiLU activation function which are connected in sequence; the Res_Unit2 module comprises a plurality of feature extraction paths and feature splicing layers which are connected in parallel, wherein one feature extraction path comprises a CBS module with a convolution kernel size of 3 multiplied by 3, a feature splicing layer, a CBS module with a convolution kernel size of 3 multiplied by 3, a feature addition layer and a CBS module with a convolution kernel size of 1 multiplied by 1, which are sequentially connected, and the CBS module with the convolution kernel size of 3 multiplied by 3 and the ABS module with the convolution kernel size of 1 multiplied by 1 are connected in parallel between the CBS module with the convolution kernel size of 3 multiplied by 3 and the feature splicing layer.

In some embodiments, the sending unit 3003 is configured to send the trained construction hidden danger detection model to the construction hidden danger detection device.

The units in fig. 11 and 12 may also be referred to as modules, for example, the processing units may be referred to as processing modules.

Each of the units in fig. 11 and 12 may be stored in a computer-readable storage medium if implemented in the form of software functional modules and sold or used as a separate product. Based on such understanding, the technical solution of the embodiments of the present application may be essentially or a part contributing to the prior art or all or part of the technical solution may be embodied in the form of a software product stored in a storage medium, including several instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) or a processor (processor) to perform all or part of the steps of the method described in the embodiments of the present application. The storage medium storing the computer software product includes: a U-disk, a removable hard disk, a read-only memory (ROM), a random access memory (random access memory, RAM), a magnetic disk, or an optical disk, or other various media capable of storing program codes.

In the case of implementing the functions of the integrated modules in the form of hardware, the embodiment of the present application further provides a schematic hardware structure of a network device, as shown in fig. 13, where the network device 4000 includes a processor 4001, and optionally, includes a memory 4002 and a communication interface 4003 connected to the processor 4001. The processor 4001, the memory 4002 and the communication interface 4003 are connected through the bus 4004.

The processor 4001 may be a central processing unit (central processing unit, CPU), a general purpose processor network processor (network processor, NP), a digital signal processor (digital signal processing, DSP), a microprocessor, a microcontroller, a programmable logic device (programmable logic device, PLD), or any combination thereof. The processor 4001 may also be any other apparatus having a processing function, such as a circuit, a device, or a software module. The processor 4001 may also include a plurality of CPUs, and the processor 4001 may be one single-core (single-CPU) processor or a multi-core (multi-CPU) processor. A processor herein may refer to one or more devices, circuits, or processing cores for processing data (e.g., computer program instructions).

Memory 4002 may be a read-only memory (ROM) or other type of static storage device that can store static information and instructions, a random access memory (random access memory, RAM) or other type of dynamic storage device that can store information and instructions, or an electrically erasable programmable read-only memory (electrically erasable programmable read-only memory, EEPROM), a compact disc read-only memory (compact disc read-only memory) or other optical disk storage, optical disk storage (including compact disc, laser disc, optical disc, digital versatile disc, blu-ray disc, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer, as embodiments of the application are not limited in this regard. The memory 4002 may be separate or integrated with the processor 4001. Wherein the memory 4002 may comprise computer program code. The processor 4001 is configured to execute computer program codes stored in the memory 4002, thereby implementing the construction hidden danger detection method and the training method of the construction hidden danger detection model provided by the embodiment of the present application.

The communication interface 4003 may be used to communicate with other devices or communication networks (e.g., ethernet, radio access network (radio access network, RAN), wireless local area network (wireless local area networks, WLAN), etc.). The communication interface 4003 may be a module, a circuit, a transceiver, or any device capable of enabling communication.

Bus 4004 may be a peripheral component interconnect standard (peripheral component interconnect, PCI) bus or an extended industry standard architecture (extended industry standard architecture, EISA) bus, among others. The bus 4004 may be divided into an address bus, a data bus, a control bus, and the like. For ease of illustration, only one thick line is shown in fig. 13, but not only one bus or one type of bus.

The embodiment of the application also provides a computer readable storage medium, which comprises computer execution instructions, when the computer readable storage medium runs on a computer, the computer is caused to execute any one of the construction hidden danger detection method and the training method of the construction hidden danger detection model provided by the embodiment.

The embodiment of the application also provides a computer program product containing computer execution instructions, which when run on a computer, cause the computer to execute any one of the construction hidden danger detection method and the training method of the construction hidden danger detection model provided by the embodiment.

The embodiment of the application also provides a chip, which comprises: the processor is coupled with the memory through the interface, and when the processor executes the computer program or the computer execution instruction in the memory, the processor causes any one of the construction hidden danger detection method and the training method of the construction hidden danger detection model provided in the above embodiment to be executed.

In the above embodiments, it may be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using a software program, it may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer-executable instructions. When the computer-executable instructions are loaded and executed on a computer, the processes or functions described in accordance with embodiments of the present application are produced, in whole or in part. The computer may be a general purpose computer, a special purpose computer, a computer network, or other programmable apparatus. The computer-executable instructions may be stored in or transmitted from one computer-readable storage medium to another, for example, from one website, computer, server, or data center by wired (e.g., coaxial cable, fiber optic, digital subscriber line (digital subscriber line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). Computer readable storage media can be any available media that can be accessed by a computer or data storage devices including one or more servers, data centers, etc. that can be integrated with the media. The usable medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a Solid State Disk (SSD)), or the like.

Although the application is described herein in connection with various embodiments, other variations to the disclosed embodiments can be understood and effected by those skilled in the art in practicing the claimed application, from a study of the drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other elements or steps, and the "a" or "an" does not exclude a plurality. A single processor or other unit may fulfill the functions of several items recited in the claims. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.

Although the application has been described in connection with specific features and embodiments thereof, it will be apparent that various modifications and combinations can be made without departing from the spirit and scope of the application. Accordingly, the specification and drawings are merely exemplary illustrations of the present application as defined in the appended claims and are considered to cover any and all modifications, variations, combinations, or equivalents that fall within the scope of the application. It will be apparent to those skilled in the art that various modifications and variations can be made to the present application without departing from the spirit or scope of the application. Thus, it is intended that the present application also include such modifications and alterations insofar as they come within the scope of the appended claims or the equivalents thereof.

The foregoing is merely illustrative of specific embodiments of the present application, and the scope of the present application is not limited thereto, but any changes or substitutions within the technical scope of the present application should be covered by the scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. The construction hidden danger detection method is characterized by comprising the following steps of:

and sending alarm information when the construction hidden danger detection result indicates that the construction hidden danger exists in the supervision area corresponding to the image to be detected, wherein the alarm information is used for indicating maintenance of the supervision area corresponding to the image to be detected.

2. The method of claim 1, wherein the construction hidden danger detection model comprises a feature extraction network and a feature fusion network, the feature fusion network comprising a plurality of fusion paths, different fusion paths being used for feature fusion of feature images of different resolutions;

The feature extraction network is used for extracting feature images in the images to be detected and outputting the feature images of the images to be detected to target fusion paths in a plurality of fusion paths included in the feature fusion network;

the target fusion path is used for carrying out feature fusion on the feature images of the images to be detected and outputting the construction hidden danger detection result; the target fusion path is a fusion path matched with the resolution of the characteristic image of the image to be detected in the fusion paths.

3. The method according to claim 1 or 2, wherein the construction hidden danger detection model is a construction hidden danger detection model based on a convolutional neural network algorithm.

4. The training method of the construction hidden danger detection model is characterized by comprising the following steps of:

and training an initial construction hidden danger detection model according to the processed training sample set until the construction hidden danger detection model after training is completed is obtained.

5. The method of claim 4, wherein the construction hazard detection model comprises a feature extraction network and a feature fusion network; the feature fusion network comprises a plurality of fusion paths, and different fusion paths are used for carrying out feature fusion on feature images with different resolutions.

6. The method of claim 5, wherein the step of determining the position of the probe is performed,

the feature extraction network comprises a CBS module with a convolution kernel size of 3 multiplied by 3, a Res_Unit1 module, the CBS module with the convolution kernel size of 3 multiplied by 3, the CBS module with the convolution kernel size of 1 multiplied by 1, the CBS module with the convolution kernel size of 3 multiplied by 3, a Res_Unit2 module and the CBS module with the convolution kernel size of 3 multiplied by 3 which are connected in sequence.

7. The method of claim 5, wherein the plurality of fusion paths comprises a first fusion path, a second fusion path, and a third fusion path;

the first fusion path is used for carrying out feature fusion on the feature images with the first resolution, the second fusion path is used for carrying out feature fusion on the feature images with the second resolution, the third fusion path is used for carrying out feature fusion on the feature images with the third resolution, the first resolution is larger than the second resolution, and the second resolution is larger than the third resolution.

8. The method of claim 7, wherein the step of determining the position of the probe is performed,

the first fusion path comprises a CBS module with a convolution kernel size of 3 multiplied by 3, a Res_Unit2 module, the CBS module with the convolution kernel size of 1 multiplied by 1 and the CBS module with the convolution kernel size of 3 multiplied by 3 which are connected in sequence;

the second fusion path comprises a CBS module with a convolution kernel size of 3 multiplied by 3, a Res_Unit2 module, an upsampling layer, a feature splicing layer, a Res_Unit2 module, a CBS module with a convolution kernel size of 1 multiplied by 1 and a CBS module with a convolution kernel size of 3 multiplied by 3 which are connected in sequence;

the third fusion path comprises a CBS module with a convolution kernel size of 3 multiplied by 3, a Res_Unit2 module, an upsampling layer, a feature splicing layer, a Res_Unit2 module, a CBS module with a convolution kernel size of 1 multiplied by 1 and a CBS module with a convolution kernel size of 3 multiplied by 3 which are sequentially connected.

9. The method according to claim 6 or 8, wherein,

one CBS module comprises a convolution layer, a batch standardized BN layer and a SiLU activation function which are sequentially connected;

one Res_Unit1 module comprises a CBS module with a convolution kernel size of 3 multiplied by 3, a CBS module with a convolution kernel size of 1 multiplied by 1, an ABS module and a characteristic addition layer which are connected in sequence;

One ABS module comprises an asymmetric convolution layer with a convolution kernel of 3 multiplied by 1, an asymmetric convolution layer with a convolution kernel of 1 multiplied by 3, a BN layer and a SiLU activation function which are connected in sequence;

one Res_Unit2 module comprises a plurality of feature extraction paths and feature splicing layers which are connected in parallel, wherein one feature extraction path comprises a CBS module with a convolution kernel size of 3 multiplied by 3, a feature splicing layer, a CBS module with a convolution kernel size of 3 multiplied by 3, a feature addition layer and a CBS module with a convolution kernel size of 1 multiplied by 1, which are sequentially connected, and the CBS module with the convolution kernel size of 3 multiplied by 3 and the feature splicing layer are connected in parallel with the CBS module with the convolution kernel size of 1 multiplied by 1 and the ABS module.

10. The utility model provides a construction hidden danger detection device which characterized in that includes:

the acquisition unit is used for acquiring an image to be detected of a supervision area, wherein the supervision area is an area with a communication line;

the processing unit is used for inputting the image to be detected into a construction hidden danger detection model and outputting a construction hidden danger detection result, wherein the construction hidden danger detection result is used for indicating whether construction hidden danger exists in a supervision area corresponding to the image to be detected;

the sending unit is used for sending out alarm information when the construction hidden danger detection result indicates that the supervision area corresponding to the image to be detected has construction hidden danger, and the alarm information is used for indicating maintenance of the supervision area corresponding to the image to be detected.

11. The apparatus of claim 10, wherein the construction hidden danger detection model comprises a feature extraction network and a feature fusion network, the feature fusion network comprising a plurality of fusion paths, different fusion paths being used for feature fusion of feature images of different resolutions;

12. The apparatus of claim 10 or 11, wherein the construction hidden danger detection model is a construction hidden danger detection model based on a convolutional neural network algorithm.

13. The utility model provides a construction hidden danger detects trainer of model which characterized in that includes:

the acquisition unit is used for acquiring a training sample set, wherein the training sample set comprises one or more images marked with construction hidden danger;

A processing unit for: performing augmentation treatment on the training sample set to obtain a treated training sample set; wherein the augmentation process comprises one or more of: adding noise, rotating samples, scaling samples, clipping samples, or flipping samples; and training an initial construction hidden danger detection model according to the processed training sample set until the construction hidden danger detection model after training is completed is obtained.

14. The apparatus of claim 13, wherein the construction hazard detection model comprises a feature extraction network and a feature fusion network; the feature fusion network comprises a plurality of fusion paths, and different fusion paths are used for carrying out feature fusion on feature images with different resolutions.

15. The apparatus of claim 14, wherein the feature extraction network comprises a CBS module with a convolution kernel size of 3 x 3, a res_unit1 module, the CBS module with a convolution kernel size of 3 x 3, the CBS module with a convolution kernel size of 1 x 1, the res_unit1 module, the CBS module with a convolution kernel size of 3 x 3, the CBS module with a convolution kernel size of 1 x 1, a CBS module with a convolution kernel size of 3 x 3, a res_unit2 module, and the CBS module with a convolution kernel size of 3 x 3 connected in sequence.

16. The apparatus of claim 14, wherein the plurality of fusion paths comprises a first fusion path, a second fusion path, and a third fusion path;

17. The apparatus of claim 16, wherein the device comprises a plurality of sensors,

18. The apparatus according to claim 15 or 17, wherein,

19. A network device, comprising: a processor and a memory;

the memory stores instructions executable by the processor;

The processor is configured to, when executing the instructions, cause the network device to implement the method of any one of claims 1-3 or the method of any one of claims 4-9.

20. A computer readable storage medium comprising computer instructions which, when run on a computer, cause the computer to perform the method of any one of claims 1-3 or the method of any one of claims 4-9.