CN112184651A

CN112184651A - Photovoltaic power station part looseness detection system and method based on artificial intelligence

Info

Publication number: CN112184651A
Application number: CN202011004973.4A
Authority: CN
Inventors: 李灵芝; 廖一峰
Original assignee: Zhengzhou Maitou Information Technology Co ltd
Current assignee: Zhengzhou Maitou Information Technology Co ltd
Priority date: 2020-09-21
Filing date: 2020-09-21
Publication date: 2021-01-05

Abstract

The invention relates to the technical field of artificial intelligence, in particular to a photovoltaic power station part looseness detection system and method based on artificial intelligence. The invention comprises an image acquisition module, an instance segmentation module, an attitude feature extraction module, a relative feature extraction module, a feature fusion module and a looseness judgment module; the image acquisition module is used for acquiring part image information; the example segmentation module is used for obtaining example masks of different parts; the attitude feature extraction module is used for acquiring the attitude feature of each part; the relative feature extraction module is used for acquiring relative features of each part and other parts around the part; the characteristic fusion module is used for fusing the example mask, the posture characteristic and the relative characteristic into the loosening judgment module; and the looseness judging module is used for judging the part according to the preset looseness grade by combining the part image information and the fused features. The invention can clearly and efficiently acquire the loosening degree of the part through the multi-feature discrimination network.

Description

Photovoltaic power station part looseness detection system and method based on artificial intelligence

Technical Field

The invention relates to the technical field of artificial intelligence, in particular to a photovoltaic power station part looseness detection system and method based on artificial intelligence.

Background

The photovoltaic power station is composed of electronic elements such as a crystalline silicon plate, an inverter and the like which utilize solar energy and adopt special materials. Due to the influence of weather and geographic environment of the photovoltaic power station, parts of components of the photovoltaic power station are inevitably loosened or fall off, and serious consequences can be caused.

In the existing part detection technology, part picture information is input through a single network for training. And inputting the current part picture information into a network to obtain a network output result. For photovoltaic power plant because part kind is many, and the tie point is many, when judging that the part is not become flexible, probably have the condition that present part is not become flexible but its accessory connection's part is not flexible. Under the condition, the current part looseness is easily judged by using a single network, so that misjudgment and misjudgment are caused, and the judgment is not accurate.

Disclosure of Invention

In order to solve the technical problems, the invention aims to provide a photovoltaic power station part looseness detection system and method based on artificial intelligence, and the adopted technical scheme is as follows:

the invention provides a photovoltaic power station part looseness detection system based on artificial intelligence, which comprises an image acquisition module, an instance segmentation module, an attitude feature extraction module, a relative feature extraction module, a feature fusion module and a looseness judgment module, wherein the image acquisition module is used for acquiring an image of a part of a photovoltaic power station;

the image acquisition module is used for acquiring part image information; the part image information is divided into initial image information and current image information;

the example segmentation module is used for segmenting the initial image information and outputting an example mask of each part;

the attitude feature extraction module is used for acquiring the attitude feature of each part by processing the example mask;

the relative feature extraction module is used for obtaining an example neighborhood mask by processing the example mask, and the example neighborhood mask represents accessory parts around the parts; acquiring relative characteristics of each part and peripheral accessory parts thereof by processing the example neighborhood mask and the example mask;

the feature fusion module is used for fusing the example mask, the posture feature and the relative feature into the loosening judgment module respectively;

the loosening judging module is used for judging the part according to the preset loosening grade by combining the image information and the feature fused by the feature fusion module.

Further, the image acquisition module acquires the image information of the part by using a line scanning camera or an area-array camera.

Further, the pose feature extraction module includes: the image processing device comprises a first image cutting module and an edge extraction module;

the first image cutting module is used for cutting the example mask to obtain part pixel data;

the edge extraction module is used for processing the part pixel data through a pre-trained edge extraction network to obtain the edge feature of each part, and the edge feature represents the posture feature of each part.

Further, the following steps: the relative feature extraction module includes: the system comprises an example neighborhood mask extraction module, a second image cropping module, a relative feature encoder and a relative feature decoder;

the example neighborhood mask extraction module is used for processing the example mask and outputting the example neighborhood mask;

the second image cutting module is used for cutting the example neighborhood mask to obtain neighborhood pixel data of the part;

a relative feature encoder for extracting features of the part pixel data and the part neighborhood pixel data as inputs to the relative feature decoder;

and the relative feature decoder is used for sampling the input of the relative feature decoder and outputting the relative feature.

Further, the following steps: the feature fusion module comprises 3 identical fusion units: a first fusion unit, a second fusion unit and a third fusion unit;

the fusion unit respectively processes the example mask, the attitude feature and the relative feature by using an attention mechanism, and inputs the processed fusion feature information into the loosening module, so that the loosening judgment module concentrates more on the fusion feature information to perform operation judgment when processing the image information;

furthermore, the looseness judging module comprises a current information coding layer, an initial information coding layer, a first full-connection layer, a second full-connection layer and a third full-connection layer;

the weight value is shared when the current information coding layer and the initial information coding layer are processed; the current information coding layer is used for processing the current image information; the initial information coding layer is used for processing the initial image information, inputting a processing result into the fusion unit, and fusing a loosening judgment module with the example mask, the posture feature and the relative feature respectively;

the current information coding layer transmits the processing result to the first full-connection layer, and the initial information coding layer transmits the processing result to the second full-connection layer; the first full connection layer and the second full connection layer share weight when processing information;

and the processing results of the first full connection layer and the second full connection layer are jointly input into the third full connection layer, and the third full connection layer outputs the preset loosening grade.

Further, the fusion unit includes: the device comprises a multiplication unit, a first fusion encoder and a second fusion encoder;

taking the feature to be fused as the input of the first fusion encoder; and the multiplication unit multiplies the output of the initial information discrimination encoder and the output of the first fusion encoder, and inputs the multiplied output into the loosening module through the second fusion encoder.

The invention also provides a photovoltaic power station part looseness detection method based on artificial intelligence, which comprises the following steps:

acquiring part image information; the part image information is divided into initial image information and current image information;

segmenting the initial image information through an example segmentation network to obtain an example mask of each part;

processing the example mask through an edge extraction network to obtain the attitude feature of each part;

obtaining an example neighborhood mask by processing the example mask, wherein the example neighborhood mask represents accessory parts around the parts of the example neighborhood mask; the relative feature extraction network acquires the relative features of each part and the peripheral accessory parts thereof by processing the example neighborhood mask and the example mask;

respectively fusing the example mask, the attitude characteristic and the relative characteristic into a loosening judgment network;

and the loosening judgment network is used for judging the parts according to the preset loosening grade by combining the image information and the fused features.

Further, the method for acquiring the relative features comprises the following steps:

the example mask is cut to obtain part pixel data;

the example neighborhood mask is cut to obtain neighborhood pixel data of the part;

and the relative extraction network obtains the relative characteristics of the part and the peripheral accessory parts according to the part pixel data and the part neighborhood pixel data.

Further, fusing the instance mask, the pose feature, and the relative feature using an attention mechanism; multiplying and then processing the characteristics to be fused and the output of the loosening judgment network, and fusing the characteristics to be fused and the output of the loosening judgment network into the loosening judgment network.

The invention has the following beneficial effects:

1. the looseness degree of the part is obtained through a looseness judging network based on fusion of various features, and the three aspects of whether the part exists, the posture of the part and the relative features of accessory parts connected with the part are considered. And respectively fusing the example mask, the posture characteristic and the relative characteristic into a loosening judgment network by using an attention mechanism, so that the loosening judgment network is more concerned about the input characteristic when judging the picture information, processing and comparing the picture information from the input characteristic, and outputting the picture similarity, namely the part loosening grade. The accuracy of the discrimination is guaranteed.

2. The position and the category of the loose part are determined by shooting the position of the camera of the loose part and the output part example, so that inspection personnel can overhaul the loose part in time.

Drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions and advantages of the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and other drawings can be obtained by those skilled in the art without creative efforts.

FIG. 1 is a structural block diagram of part looseness detection of a photovoltaic power station based on artificial intelligence of the invention;

FIG. 2 is a block diagram of a system implemented with a line scan camera according to the present invention;

FIG. 3 is a block diagram of the fusion unit of the fusion module according to the present invention;

FIG. 4 is a block diagram of a system implemented with an area-array camera according to the present invention;

FIG. 5 is a flow chart of a photovoltaic power station part looseness detection method based on artificial intelligence.

Detailed Description

In order to further illustrate the technical means and effects of the present invention adopted to achieve the predetermined objects, the following detailed description of a system for detecting loosening of parts of a photovoltaic power station according to the present invention with reference to the accompanying drawings and preferred embodiments shows the following detailed descriptions of the specific implementation, structure, features and effects thereof. In the following description, different "one embodiment" or "another embodiment" refers to not necessarily the same embodiment. Furthermore, the particular features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

The following specifically describes a specific scheme of the photovoltaic power station part looseness detection system provided by the invention with reference to the accompanying drawings.

Referring to fig. 1, a block diagram of a system for detecting loose parts in a photovoltaic power plant based on artificial intelligence is shown. The system comprises: the system comprises an image acquisition module 101, an instance segmentation module 102, a posture extraction module 103, a relative feature extraction module 104, a feature fusion module 105 and a looseness judgment module 106.

The loosening judgment module 106 is a main network and is used for judging the loosening level of the real-time image information acquired by the image acquisition module 101; the example segmentation module 102 comprises an example segmentation network, and performs example segmentation on the image input by the image acquisition module 101 to obtain an example mask of each part; the gesture extraction module 103 extracts edge features of the example mask to obtain gesture features of the part; the relative feature extraction module 104 processes the part as an example mask to obtain an example neighborhood mask of the part, i.e. a mask of the part around the part. Obtaining relative characteristics of the part through the example mask and the example neighborhood mask; the feature fusion module 105 fuses the example mask, the relative feature and the posture feature through a fusion network, and guides the main network to respectively judge whether the part exists, whether the posture of the part changes and whether the part around the part changes.

The output of the main network is the level of looseness of the part. In this embodiment, the preset loosening grade is divided into three stages: slight, obvious and severe looseness. Slight loosening means no loosening or a degree of loosening that is not obvious, which is acceptable without repair; obvious looseness indicates that the part obviously deviates from the original position and needs to be repaired manually; severe loosening, so-called sloughing, can have serious consequences and requires timely repair. When obvious loosening or serious loosening occurs, the inspection personnel is informed to repair the parts. The position of the photovoltaic assembly which is loosened or falls off is determined by utilizing the position of the track camera, the type and the position of a falling or loosening part on an image are determined according to an example segmentation result, and the positions are notified to inspection personnel.

In summary, by processing the initial image information, an example mask, pose features, and relative features for each part are obtained. The looseness judging module is integrated with the characteristics, so that the looseness judging module pays more attention to the integrated characteristics when processing information, and the looseness degree is judged from the three aspects of whether parts exist, the self postures of the parts and the relative characteristics of the parts and the accessory parts connected with the parts, so that misjudgment is avoided, and the main network is more accurately judged. And can shoot the part example of position and output according to the camera, clear acquisition becomes flexible the position and the type of part, notifies the personnel of patrolling and examining in time to overhaul.

Based on the above system, two embodiments of the present invention are provided, please refer to fig. 2, in which fig. 2 shows a first embodiment.

The whole system comprises an image acquisition module 101, an instance segmentation module 102, a posture extraction module 103, a relative feature extraction module 104, a feature fusion module 105 and a looseness judgment module 106. Specifically, the method comprises the following steps:

the image acquisition module 101 specifically operates as follows: a track camera is installed on each row of photovoltaic modules, and the cameras adopt line scanning cameras 2101, so that the cameras are opposite to the back supports of the photovoltaic modules. And acquiring image data of the whole row of photovoltaic modules by moving the camera at a constant speed. The image data is input to the system after being cut into images of the same size. When no part is loosened immediately after the photovoltaic power station is installed, image data obtained by scanning with the camera is initial image information 2103. The line scanning camera 2101 acquires a back bracket image of the photovoltaic module at regular intervals, and the acquired image information is current image information 2102.

The instance segmentation module 102 is used to segment individual part instances in the image information using an instance segmentation network. Common example segmentation Networks include an FCN (full Convolutional network), an Encoder-Decoder (Encoder-Decoder) semantic segmentation network, a hole convolution, a lightweight semantic segmentation network and the like, and the example segmentation network is constructed by adopting the Encoder-Decoder semantic segmentation.

The instance partitioning module 102 includes: an example partition encoder 2201 and an example partition decoder 2202. The example segmentation module 102 is a pre-trained segmentation network, and is configured to obtain examples of each part of the initial image information 2103, including a vertical purlin, a vertical metal bearing column, a screw, a triangular bracket, a metal component of a fixed bracket, a hinge, and the like. The example segmentation module 102 acquires an example mask of each part through the initial image information 2103, and outputs mask binarization processing.

The pose feature extraction module 103 is configured to extract an edge feature of the part using the edge extraction network, where the pose of the part is represented by its edge, and thus the edge feature of the part can be regarded as the pose feature of the part.

Common Edge extraction networks such as HED (monolithic-Nested Edge Detection), CASNet (Category-Aware Semantic Edge Detection algorithm), CEDN (full convolutional encoder-decoder network), and the like are widely used and will not be described herein.

The posture feature extraction module 103 includes: a first image cropping module 2301 and an edge extraction module 2302. The first image cropping module 2301 crops out the pixel data of the part by using the example mask output by the example segmentation network, inputs the pixel data of the part into the edge extraction module 2302, and obtains the edge features of the part by using the edge extraction network trained in advance.

The relative feature extraction module 104 also performs processing by using a coding/decoding structure network, specifically:

the relative feature extraction module 104 includes: an example neighborhood mask extraction module 2401, a second image cropping module 2402, a relative feature encoder 2403, a relative feature decoder 2404. The example neighborhood mask extraction module 2401 obtains an example neighborhood mask, that is, a mask of parts around the part, by extracting an example mask output by the example segmentation network 102. The example neighborhood mask is cut by the second image cutting module 2402, pixel data of a part around the part is output, the pixel data of the part cut by the first image cutting module 2301 and the pixel data of the part are input into the relative feature encoder 2403 to be subjected to feature extraction, and then the relative features of the part and the part around are output after sampling by the relative feature decoder 2402.

The feature fusion module 105 includes: a first fusion unit 2501, a second fusion unit 2502, and a third fusion unit 2503. The loosening determination module 106 includes: a first current information discrimination encoder 2601, a first initial information discrimination encoder 2602, a second current information discrimination encoder 2603, a second initial information discrimination encoder 2604, a third current information encoder 2605, a third initial information discrimination encoder 2606, a first fully-connected layer 2607, a second fully-connected layer 2608, and a third fully-connected layer 2609.

The looseness judging module 106 utilizes a plurality of encoders to process the image information and fuses the image information with the fused feature information fused by the feature fusion module 105, so that the looseness judging module focuses more on the fused features when processing the image. And finally, processing results by the full connection layer, calculating the similarity of the two image information, and outputting a judgment result. Specifically, the specific working modes of the feature fusion module 105 and the looseness judging module 106 are as follows:

the first current information discrimination encoder 2601 processes the input current image information 2102, and then transfers the processed current image information to the second current information discrimination encoder 2603. The first initial information discrimination encoder 2602 processes the initial image information 2103, and transfers the processed image information to the first fusion unit 2501 and the second initial information discrimination encoder 2604. First fusing section 2501 inputs the fused features to second initial information discrimination encoder 2604. The first current information discrimination encoder 2601 and the first initial information discrimination encoder 2602 share a weight value when processing information.

The second current information discrimination encoder 2603 inputs the processing result to the third current information encoder 2605, and the second initial information discrimination encoder 2604 inputs the processing result to the second fusion unit 2502 and the third initial information discrimination encoder 2606. The second fusion unit 2502 processes the posture feature output from the posture feature extraction module 103 and the output from the second initial discrimination encoder 2604, and inputs the processed result to the third initial encoder 2606. The second current information discrimination encoder 2603 and the second initial information discrimination encoder 2604 share a weight value when processing information.

The third current information discrimination encoder 2605 processes the result and then inputs the result into the first fully-connected layer 2607, and the third initial information discrimination encoder 2606 receives the input and then inputs the processed result into the third merging unit 2503 and the second fully-connected layer 2608. The third fusion encoder 2503 processes the relative feature output by the relative feature extraction module 104 and the output of the third initial information discrimination encoder, and inputs the processed result to the second full connection layer. And the third current information discrimination encoder and the third initial information discrimination encoder share weight values when processing information.

After the above steps, the loosening determination module 106 fuses the required features through the feature fusion module 105, respectively. So that the fused features are more focused on during the judgment.

The first fully-connected layer 2607 and the second fully-connected layer 2608 share the processed information time weight, and the result is input to the third fully-connected layer 2609. Finally, the third fully-connected layer 2609 outputs the determined loosening level.

In addition, the training data sets of each network shown in fig. 2 are acquired by a line scanning camera, and the data set acquisition method may be as follows:

1) because rigid structures such as metal can be easily simulated realistically by a simulator, the invention simulates a photovoltaic module by a simulator such as a phantom 4 engine or Unity3D, including between a panel and metal.

2) And constructing a real three-dimensional model of the photovoltaic module and vivid mapping data. Different natural lights are simulated by the light of the simulator. Image data is acquired using a virtual line scan camera.

3) In the simulator, the acquisition difficulty of the data set is reduced, and the labeling difficulty is smaller.

4) The data set was divided into 3 classes by the loosening class: slight, obvious and severe looseness. Each part will comprise a plurality of images, which are acquired by the line scanning camera, real data representing image data without part loosening or falling, label data being image data of the loosened parts, a mask of each part in the real data, and an edge mask of the part.

5) The example segmentation network, edge extraction network, and relative feature extraction network of fig. 2 are trained separately and independently.

6) And (5) taking 80% of the data set as a training set, taking the rest 20% as a testing set, and training the network by using a cross entropy loss function and a random gradient descent method.

Referring to fig. 3, fig. 3 is a schematic diagram of a fusion unit in the fusion module according to the present invention.

The fusion unit uses an attention mechanism to make the main network in the loose discrimination module focus on several fused features. Specifically, the multiplying unit 3101, a first fusion encoder 3102, and a second fusion encoder 3103 are included.

The first fusion encoder 3102 is used for receiving the transmitted features to be fused, processing the features to be fused by the first fusion encoder 3102, transmitting the processed features to the multiplication unit 3101, receiving the input transmitted by the looseness judging module, multiplying the input by the features to be fused, and transmitting the multiplied input to the second fusion encoder 3103. The second fusion encoder 3103 processes the transferred data and inputs the processing result into the loosening determination module.

Referring to fig. 4, fig. 4 shows a second embodiment of the present invention.

The second embodiment of the present invention is the same in concept as the first embodiment, except that the line scanning camera is replaced with an area-array camera. The area-array camera is arranged on the rail, a row of images of the photovoltaic modules are collected in a uniform-speed motion mode, and each part can obtain image data of multiple visual angles. The image data containing the same part is divided into a set, one part corresponds to a unique image set, and the image set contains the image data of the part from different view angles. After the area-array camera acquires images, a plurality of image sets can be generated, and because at least two parts exist in one image, the same picture can appear in different sets.

According to the same principle, the resulting image data is divided into a current image set and an initial image set. And the current image set and the initial image set are respectively input into a looseness judging module through a merging module.

And for each initial image set, performing example segmentation on the images in the set to segment different photovoltaic module examples, namely acquiring an example mask set of parts of the images in each initial image set. And deleting example masks which do not belong to the part set, deleting images with small mask areas, finally enabling different sets to have the same number of images, and outputting an example mask set of each part.

As in the first embodiment, the obtained example mask set is input to the posture feature extraction module and the relative feature extraction module, and the posture feature and the relative feature of each part are obtained. The feature outputs are also a set, namely a set of pose features and a set of relative features, representing features from different perspectives.

And respectively merging the example mask set, the posture feature set and the relative feature set together through a merging module and inputting the merged mask set, the posture feature set and the relative feature set into a feature fusion network. And then the features are fused into a looseness judging module through a feature fusion module which is the same as that of the first embodiment. The loosening judgment module outputs the same loosening and falling grades of the parts as the first embodiment.

The two embodiments provided by the invention have complementary disadvantages, the first embodiment obtains image information through a line scanning camera, the judgment capability on tiny looseness is higher, but information of a sheltering place or a camera dead angle place cannot be obtained. The second embodiment acquires image information through an area-array camera, and has low capability of judging tiny looseness, but can acquire looseness information of more parts through multi-view information. The two embodiments complement each other to verify that more accurate results can be obtained.

Referring to fig. 5, fig. 5 shows a method for detecting loosening of parts of a photovoltaic power station based on artificial intelligence according to the present invention.

S1, acquiring part image information; the part image information is divided into initial image information and current image information;

s2, segmenting the initial image information and outputting an example mask of each part;

s3, processing the example mask to obtain the posture characteristic of each part;

s4, obtaining an example neighborhood mask by processing the example mask, wherein the example neighborhood mask represents accessory parts around the parts; acquiring relative characteristics of each part and peripheral accessory parts thereof by processing the example neighborhood mask and the example mask;

s5, fusing the example mask, the posture characteristic and the relative characteristic into a loose judgment network respectively;

and S6, the loosening judgment network is combined with the image information and the fused features to judge the parts according to the preset loosening grade.

It should be noted that: the precedence order of the above embodiments of the present invention is only for description, and does not represent the merits of the embodiments. And specific embodiments thereof have been described above. Other embodiments are within the scope of the following claims. In some cases, the actions or steps recited in the claims may be performed in a different order than in the embodiments and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some embodiments, multitasking and parallel processing may also be possible or may be advantageous.

The embodiments in the present specification are described in a progressive manner, and the same and similar parts among the embodiments are referred to each other, and each embodiment focuses on the differences from the other embodiments.

The above description is only for the purpose of illustrating the preferred embodiments of the present invention and is not to be construed as limiting the invention, and any modifications, equivalents, improvements and the like that fall within the spirit and principle of the present invention are intended to be included therein.

Claims

1. The utility model provides a not hard up detecting system of photovoltaic power plant part based on artificial intelligence which characterized in that: the system comprises an image acquisition module, an instance segmentation module, an attitude feature extraction module, a relative feature extraction module, a feature fusion module and a looseness judgment module;

2. The artificial intelligence based photovoltaic power plant part looseness detection system of claim 1, wherein: the image acquisition module adopts a line scanning camera or an area-array camera to acquire the image information of the part.

3. The artificial intelligence based photovoltaic power plant part looseness detection system of claim 1, wherein: the attitude feature extraction module includes: the image processing device comprises a first image cutting module and an edge extraction module;

4. The artificial intelligence based photovoltaic power plant part looseness detection system of claim 1, wherein: the relative feature extraction module includes: the system comprises an example neighborhood mask extraction module, a second image cropping module, a relative feature encoder and a relative feature decoder;

5. The artificial intelligence based photovoltaic power plant part looseness detection system of claim 1, wherein: the feature fusion module comprises 3 identical fusion units: a first fusion unit, a second fusion unit and a third fusion unit;

the fusion unit respectively processes the example mask, the posture feature and the relative feature by using an attention mechanism, and inputs the processed fusion feature information into the loosening module, so that the loosening judgment module concentrates more on the fusion feature information for operation judgment when processing the image information.

6. The artificial intelligence based photovoltaic power plant part looseness detection system of claim 1, wherein: the looseness judging module comprises a current information coding layer, an initial information coding layer, a first full-connection layer, a second full-connection layer and a third full-connection layer;

7. The artificial intelligence based photovoltaic power plant part looseness detection system of claim 5, wherein: the fusion unit includes: the device comprises a multiplication unit, a first fusion encoder and a second fusion encoder;

8. A photovoltaic power station part looseness detection method based on artificial intelligence is characterized by comprising the following steps: the photovoltaic power station part looseness detection method based on artificial intelligence comprises the following steps:

9. The artificial intelligence based photovoltaic power station part looseness detection method of claim 8, wherein: the method for acquiring the relative characteristics comprises the following steps:

the example mask is cut to obtain part pixel data;

10. The artificial intelligence based photovoltaic power station part looseness detection method of claim 8, wherein: fusing the instance mask, the pose feature, and the relative feature using an attention mechanism; multiplying and then processing the characteristics to be fused and the output of the loosening judgment network, and fusing the characteristics to be fused and the output of the loosening judgment network into the loosening judgment network.