CN118038008A - Hydropower plant personnel positioning method and system based on ptz multi-camera linkage - Google Patents
Hydropower plant personnel positioning method and system based on ptz multi-camera linkage Download PDFInfo
- Publication number
- CN118038008A CN118038008A CN202410445115.5A CN202410445115A CN118038008A CN 118038008 A CN118038008 A CN 118038008A CN 202410445115 A CN202410445115 A CN 202410445115A CN 118038008 A CN118038008 A CN 118038008A
- Authority
- CN
- China
- Prior art keywords
- target
- personnel
- hydropower plant
- images
- representing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 39
- 230000004927 fusion Effects 0.000 claims description 28
- 238000010586 diagram Methods 0.000 claims description 15
- 238000003062 neural network model Methods 0.000 claims description 15
- 238000000605 extraction Methods 0.000 claims description 14
- 230000009466 transformation Effects 0.000 claims description 13
- 238000001914 filtration Methods 0.000 claims description 12
- 238000013507 mapping Methods 0.000 claims description 12
- 239000011159 matrix material Substances 0.000 claims description 11
- 238000007781 pre-processing Methods 0.000 claims description 10
- 239000013598 vector Substances 0.000 claims description 8
- 238000009432 framing Methods 0.000 claims description 6
- 210000002569 neuron Anatomy 0.000 claims description 6
- 238000013528 artificial neural network Methods 0.000 claims description 4
- 238000005259 measurement Methods 0.000 claims description 4
- 230000001629 suppression Effects 0.000 claims description 2
- 230000008569 process Effects 0.000 description 10
- 238000001514 detection method Methods 0.000 description 7
- 238000013527 convolutional neural network Methods 0.000 description 6
- 230000007246 mechanism Effects 0.000 description 6
- 238000012512 characterization method Methods 0.000 description 3
- 238000010276 construction Methods 0.000 description 3
- 238000010606 normalization Methods 0.000 description 3
- 230000008901 benefit Effects 0.000 description 2
- 230000000903 blocking effect Effects 0.000 description 2
- 238000012423 maintenance Methods 0.000 description 2
- 238000013473 artificial intelligence Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
Landscapes
- Image Analysis (AREA)
- Image Processing (AREA)
Abstract
The invention relates to a hydropower plant personnel positioning method and system based on ptz multi-camera linkage, comprising the following steps: fusing the plurality of registration images to obtain a plurality of fused images; extracting feature images of the fused images to obtain a plurality of extracted feature images, detecting a plurality of target personnel images in the extracted feature images, and calculating a plurality of personnel category probabilities of the target personnel images; calculating the intra-plant distance between the target hydropower plant personnel and the target camera, and dividing the background three-dimensional scene of the target hydropower plant personnel from the three-dimensional reconstruction scene; performing personnel coarse positioning on the target hydropower plant personnel in the three-dimensional reconstruction scene to obtain personnel coarse positions; and combining the rough positions of the personnel, splicing the path tracks of the target hydropower plant personnel in the hydropower plant, and dynamically positioning the target hydropower plant personnel based on the path tracks to obtain the positioning result of the hydropower plant personnel in the hydropower plant. The invention can be used for positioning hydropower plant personnel with accuracy and timeliness.
Description
Technical Field
The invention relates to the field of image processing, in particular to a ptz multi-camera linkage-based hydropower plant personnel positioning method and system.
Background
At present, the personnel identification technology mainly uses a convolutional neural network algorithm, wherein the convolutional neural network algorithm comprises an R-CNN algorithm, a YOLO algorithm and the like, the R-CNN algorithm uses SELECTIVE SEARCH to select a candidate region, a candidate region target feature map is obtained through a CNN basic network, and in the second stage, targets are classified through SVM and regression operation to obtain target boundary frame coordinates, and the output result is long in time consumption due to the fact that the model of the R-CNN algorithm is complex; the YOLO algorithm, however, uses the grid to directly predict the target, and although the detection speed is increased by omitting the selection of the candidate region, it loses a certain accuracy, and secondly, the equipment, the building and the personnel in the hydropower plant are complex, and there are a certain number of equipment and buildings with the same appearance, and when the ptz camera is used for shooting, the situation that the same plurality of equipment or the same plurality of buildings are overlapped in the shot image is unavoidable, and in this case, it is difficult to locate the position of the personnel based on the position of the personnel relative to the equipment. Therefore, the accuracy and timeliness of positioning hydropower plant personnel under the overlapping of image scenes are insufficient.
Disclosure of Invention
In order to solve the problems, the invention provides a hydropower plant personnel positioning method and system based on ptz multi-camera linkage, which can ensure that hydropower plant personnel positioning accuracy and timeliness under the condition of overlapping image scenes have the capability.
In a first aspect, the invention provides a hydropower plant personnel positioning method based on ptz multi-camera linkage, comprising the following steps:
Shooting a plurality of hydropower plant images in a hydropower plant by using a plurality of panoramic high-speed dome cameras, performing image preprocessing on the plurality of hydropower plant images to obtain a plurality of preprocessed images, registering the plurality of preprocessed images to obtain a plurality of registered images, and fusing the plurality of registered images to obtain a plurality of fused images;
Extracting feature images of the multiple fusion images to obtain multiple extracted feature images, detecting multiple target personnel images in the multiple extracted feature images, calculating multiple personnel category probabilities of the multiple target personnel images, detecting target hydropower plant personnel in the multiple fusion images according to the multiple personnel category probabilities, selecting target images corresponding to the target hydropower plant personnel from the multiple fusion images, and selecting a target camera for shooting the target images from the multiple panoramic speed dome cameras;
Calculating the intra-plant distance between the target hydropower plant personnel and the target camera, and reconstructing a three-dimensional scene of the target hydropower plant personnel based on the intra-plant distance and the target image to obtain a three-dimensional reconstruction scene, and dividing the background three-dimensional scene of the target hydropower plant personnel from the three-dimensional reconstruction scene;
Acquiring an actual in-plant scene corresponding to the hydropower plant image, and performing personnel coarse positioning on the target hydropower plant personnel in the three-dimensional reconstruction scene according to the actual in-plant scene to obtain a personnel coarse position;
and combining the rough positions of the staff, splicing the path tracks of the target hydropower plant staff in the hydropower plant, and dynamically positioning the target hydropower plant staff based on the path tracks to obtain a hydropower plant staff positioning result in the hydropower plant.
In a second aspect, the present invention provides a ptz multi-camera linkage based hydropower plant personnel positioning system, the system comprising:
The image fusion module is used for shooting a plurality of hydropower plant images in the hydropower plant by utilizing a plurality of panoramic high-speed dome cameras, carrying out image preprocessing on the plurality of hydropower plant images to obtain a plurality of preprocessed images, registering the plurality of preprocessed images to obtain a plurality of registered images, and fusing the plurality of registered images to obtain a plurality of fused images;
The camera selection module is used for extracting feature images of the multiple fusion images to obtain multiple extracted feature images, detecting multiple target personnel images in the multiple extracted feature images, calculating multiple personnel category probabilities of the multiple target personnel images, detecting target hydropower plant personnel in the multiple fusion images according to the multiple personnel category probabilities, selecting target images corresponding to the target hydropower plant personnel from the multiple fusion images, and selecting target cameras for shooting the target images from the multiple panoramic speed dome cameras;
The reconstruction scene module is used for calculating the intra-plant distance between the target hydropower plant personnel and the target camera, reconstructing the three-dimensional scene of the target hydropower plant personnel based on the intra-plant distance and the target image to obtain a three-dimensional reconstruction scene, and dividing the background three-dimensional scene of the target hydropower plant personnel from the three-dimensional reconstruction scene;
The personnel coarse positioning module is used for acquiring an actual in-plant scene corresponding to the hydropower plant image, and performing personnel coarse positioning on the target hydropower plant personnel in the three-dimensional reconstruction scene according to the actual in-plant scene to obtain a personnel coarse position;
and the dynamic positioning module is used for combining the rough positions of the personnel, splicing the path tracks of the target hydropower plant personnel in the hydropower plant, and dynamically positioning the target hydropower plant personnel based on the path tracks to obtain the hydropower plant personnel positioning result in the hydropower plant.
Compared with the prior art, the technical principle and beneficial effect of this scheme lie in:
The embodiment of the invention converts the value range of the gray value of the pixel in the image from 0 to 255 to 0 to 1 on the premise of ensuring that the information of the images is unchanged by carrying out image preprocessing on the plurality of hydropower plant images, further, the embodiment of the invention maps one image to the other image by utilizing space transformation so as to achieve the purpose of information fusion by utilizing space transformation to map the points corresponding to the same position in the two images to one by one, further, the embodiment of the invention is used for splicing incomplete scenes shot by cameras with different angles into complete scenes by fusing the plurality of registration images, the embodiment of the invention is used for extracting target features in the plurality of fusion images by utilizing convolution kernels, the embodiment of the invention detects a plurality of target personnel graphs in the plurality of extracted feature graphs by detecting the plurality of target personnel graphs in the plurality of extracted feature graphs, selecting candidate frames only containing personnel from the plurality of extracted feature graphs, carrying out target detection by referring to a default candidate frame mechanism and a regression mechanism of an RPN network and adopting a multi-scale feature characterization idea, so that an algorithm of target extraction has the characteristics of high detection speed and high precision at the same time, further, the embodiment of the invention is used for identifying the categories of the personnel in the target personnel graphs by calculating the category of the personnel in the target personnel graph, in the embodiment of the invention, when the target hydropower plant personnel is blocked by the hydropower plant internal equipment, the blocking equipment is used as a reference to determine the intra-plant distance between the area where the target hydropower plant personnel is located and the target camera, and the three-dimensional reconstruction scene is used for roughly positioning the target hydropower plant personnel according to the actual intra-plant scene, so that the detected area position where the overlapped equipment or building is located is used as the personnel rough position under the condition that the image scene is overlapped, and the embodiment of the invention is used for splicing the path track of the target hydropower plant personnel in the hydropower plant by combining the personnel rough position, so as to dynamically track the positioning of the target hydropower plant personnel in the hydropower plant, thereby determining the specific position of the target hydropower plant personnel in the vicinity of the overlapped equipment or building based on the path track of the target hydropower plant personnel in the hydropower plant, and distinguishing the specific position of the target hydropower plant personnel in the vicinity of the overlapped equipment or building. Therefore, the hydropower plant personnel positioning method and system based on ptz multi-camera linkage can ensure that the hydropower plant personnel can be positioned quickly and accurately under the condition of overlapping image scenes.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and together with the description, serve to explain the principles of the invention.
In order to more clearly illustrate the embodiments of the invention or the technical solutions of the prior art, the drawings which are used in the description of the embodiments or the prior art will be briefly described, and it will be obvious to a person skilled in the art that other drawings can be obtained from these drawings without inventive effort.
FIG. 1 is a schematic flow chart of a hydropower plant personnel positioning method based on ptz multi-camera linkage according to an embodiment of the invention;
FIG. 2 is a schematic flow chart of one step of the ptz multi-camera linkage-based hydropower plant personnel positioning method provided by the invention in FIG. 1;
FIG. 3 is a schematic flow chart of another step of the ptz multi-camera linkage based hydropower plant personnel positioning method provided in FIG. 1;
Fig. 4 is a schematic block diagram of a ptz multi-camera linkage-based hydropower station personnel positioning system according to an embodiment of the invention.
Detailed Description
It should be understood that the detailed description is presented by way of example only and is not intended to limit the invention.
The embodiment of the invention provides a hydropower plant personnel positioning method based on ptz multi-camera linkage, and an execution subject of the hydropower plant personnel positioning method based on ptz multi-camera linkage comprises, but is not limited to, at least one of a server, a terminal and the like which can be configured to execute the hydropower plant personnel positioning method provided by the embodiment of the invention. In other words, the ptz multi-camera linkage-based hydropower plant personnel positioning method can be executed by software or hardware installed in a terminal device or a server device, and the software can be a blockchain platform. The service end includes but is not limited to: a single server, a server cluster, a cloud server or a cloud server cluster, and the like. The server may be an independent server, or may be a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN), and basic cloud computing services such as big data and artificial intelligence platforms.
Referring to fig. 1, a flow chart of a ptz multi-camera linkage-based hydropower station personnel positioning method according to an embodiment of the invention is shown, which includes:
S1, shooting a plurality of hydropower plant images in a hydropower plant by using a plurality of panoramic high-speed dome cameras, performing image preprocessing on the plurality of hydropower plant images to obtain a plurality of preprocessed images, registering the plurality of preprocessed images to obtain a plurality of registered images, and fusing the plurality of registered images to obtain a plurality of fused images.
In this embodiment, the panoramic speed dome camera is also referred to as a panoramic staring camera system, and is a combination of a panoramic camera and a speed dome camera (i.e. a PTZ camera), and is a high-definition network camera having both the global wide-view angle advantage of the panoramic camera and the local close-up staring advantage of the speed dome camera; the hydropower plant image refers to images of all places in the hydropower plant, which can be shot by the PTZ camera, and comprises a factory building, a hydroelectric generating set, a substation, power transmission equipment, workers and the like.
Further, in this embodiment, the image preprocessing is performed on the plurality of hydropower plant images, so that the value range of the pixel gray values in the images is converted from 0 to 255 to 0 to 1 on the premise of ensuring that the information of the images is unchanged.
And meanwhile, the plurality of hydropower plant images are subjected to image preprocessing, and a plurality of preprocessed images are obtained through image normalization of the plurality of hydropower plant images.
Optionally, the process of performing image normalization on the hydropower plant image is: dividing the pixel value in the image by the maximum pixel value of the image, for example, if the original image is an 8-bit gray image, then the maximum value of the read pixel matrix is 256, the minimum value is 1, the pixel matrix in the image is defined as I, j=i/256, J is the normalized image matrix, that is, all the pixel values are in the [0,1] interval after normalization.
Further, the embodiment is used for mapping one image to another image by utilizing spatial transformation through registering the plurality of preprocessed images, so that points corresponding to the same position in space in the two images are in one-to-one correspondence, and the purpose of information fusion is achieved;
And the plurality of registration images are fused to be used for splicing incomplete scenes shot by cameras with different angles into complete scenes.
Specifically, the fusing the plurality of registration images to obtain a plurality of fused images includes: identifying feature matching points between the plurality of registered images; calculating a mapping matrix between each of the feature matching points by using the following formula:
；
；
Wherein, Representing a mapping matrix,/>Representing the coordinates of one of the feature points,Representing another AND/>Coordinates of feature points constituting feature matching points between corresponding feature points,Representation/>And/>Vector cross product between;
and fusing the plurality of registration images by using the mapping matrix to obtain a plurality of fused images.
S2, extracting feature images of the multiple fusion images to obtain multiple extracted feature images, detecting multiple target personnel images in the multiple extracted feature images, calculating multiple personnel category probabilities of the multiple target personnel images, detecting target hydropower plant personnel in the multiple fusion images according to the multiple personnel category probabilities, selecting target images corresponding to the target hydropower plant personnel from the multiple fusion images, and selecting a target camera for shooting the target images from the multiple panoramic speed dome cameras.
Thus, the present embodiment is used to extract target features in the plurality of fused images by performing feature map extraction on the plurality of fused images, and to compress the feature map dimensions using downsampled layers and keep the network space undeformed.
Specifically, the feature map extracting the multiple fused images to obtain multiple extracted feature maps includes: and filtering the first feature map of the fused images by using the following formula to obtain a plurality of filtering feature maps:
；
Wherein, Pixel values representing pixel points in the plurality of filter feature maps,/>Representing the coordinates in the plurality of fused images as/>Pixel value of pixel point of/>Representing coordinates of pixel points in the plurality of fused images,Representing the co-ordinates/>, in a convolution kernel used in first feature map filtering the plurality of fused imagesCorresponding coordinates/>Representing the coordinates in the convolution kernel as/>Pixel values of the pixel points of (a);
Performing downsampling operation on the plurality of filtering feature images to obtain a plurality of downsampled feature images; and performing second feature map filtering on the plurality of downsampled feature maps to obtain the plurality of extracted feature maps.
Optionally, the process of performing the second feature map filtering on the plurality of downsampled feature maps to obtain the plurality of extracted feature maps is similar to the above-described process of performing the first feature map filtering on the plurality of fused images to obtain a plurality of filtered feature maps, which is not further described herein.
Therefore, in the embodiment, the multiple target person diagrams in the multiple extraction feature diagrams are detected to be used for detecting the target person diagrams in the multiple extraction feature diagrams, the candidate frames only containing the person are selected from the multiple extraction feature diagrams, and the target detection is performed by referring to the default candidate frame mechanism and the regression mechanism of the RPN network and adopting the concept of multi-scale feature characterization, so that the target extraction algorithm has the characteristics of high detection speed and high precision.
Referring to fig. 2, detecting a plurality of target person diagrams in the plurality of extracted feature diagrams includes:
s201, performing multi-scale feature extraction on the plurality of extracted feature images to obtain a multi-scale feature image;
S202, carrying out feature map framing on the multi-scale feature map to obtain a framing feature map;
and S203, performing non-maximum suppression on the frame feature map to obtain the plurality of target personnel maps.
Optionally, the process of extracting the multi-scale features from the plurality of extracted feature images to obtain the multi-scale feature image includes: and checking the plurality of extracted feature images by utilizing convolution of a plurality of different scales to extract features, so as to obtain a plurality of feature image extraction results of different scales.
In this embodiment, performing feature map framing on the multi-scale feature map to obtain a framed feature map, including: calculating candidate frame parameters of the multi-scale feature map by using the following formula:
；
；
Wherein, Candidate box parameters representing the multi-scale feature map,/>Representing coordinates of pixel points selected from the multi-scale feature map by the neural network model,/>Expressed as/>Width of rectangle with coordinates as center,/>Expressed as/>Height of rectangle with coordinates as center,/>Representing the size of the candidate frame corresponding to the candidate frame parameter,/>Parameter representing minimum scale of candidate frame initially set in neural network structure,/>Parameter representing maximum scale of candidate frame initially set in neural network structure,/>Representing the number of feature maps of a plurality of different scales in the multi-scale feature map,/>Serial number representing a feature map of the multiscale feature map,/>Representing an ith feature map of the multiscale feature map,/>Representation/>Size,/>，/>Representing an aspect ratio preset according to different sizes of the multi-scale feature map;
And simultaneously, carrying out parameter transformation on the candidate frame parameters by using the following formula to obtain transformation parameters:
；
Wherein, Representing the transformation parameters,/>Candidate box parameters representing the multi-scale feature map,/>Representing a fixed measurement scale preset in a neural network model, wherein the fixed measurement scale is used for measuring the mapping relation between the candidate frame parameters and the transformation parameters;
And taking the candidate frame corresponding to the transformation parameter as the frame division characteristic diagram.
Therefore, the method is used for identifying the classes of the personnel in the target personnel images by calculating the multiple personnel class probabilities of the multiple target personnel images, wherein the classes of the personnel comprise various classes such as a hydropower plant personnel and a non-hydropower plant personnel, the hydropower plant personnel further comprise various classes such as a patrol man, a safety man, a maintenance man and a cleaner, the non-hydropower plant personnel comprise various classes such as a constructor and a designer, whether the non-hydropower plant personnel comprise various classes such as a patrol man, a safety man, a maintenance man and a cleaner, or various classes such as a constructor and a designer, and each class comprises various types of subdivision classes due to a level system, for example, the patrol man further comprises a low-level patrol man and a high-level patrol man. When the category of the person is other, the person is indicated as the category which does not relate to all other persons, namely, the non-human object in the hydropower plant.
Further, the calculating a plurality of person class probabilities of the plurality of target person graphs includes: calculating the personnel category index of the plurality of target personnel graphs using the following formula:
；
Wherein, Person class index representing the plurality of target person diagrams,/>Weights representing classification layers in neural network model,/>Bias vector representing classification layer in neural network model,/>Representing a vector matrix composed of feature vectors in the plurality of target person diagrams;
according to the personnel category index, calculating a plurality of personnel category probabilities of the plurality of target personnel graphs by using the following formula:
；
Wherein, Multiple person class probabilities representing the multiple target person graphs,/>A personnel category index representing the plurality of target personnel diagrams, namely input data of an output layer in the neural network model, wherein u represents a neuron serial number of the output layer in the neural network model, N represents the total number of neurons of the output layer in the neural network model, v represents the neuron serial number of the output layer in the neural network model, and/>N。
Further, detecting the target hydropower plant personnel in the plurality of fused images according to the plurality of personnel category probabilities, including: selecting a maximum probability from the plurality of personnel category probabilities; identifying the personnel category corresponding to the maximum probability; when the personnel category accords with a preset category, determining that hydropower plant personnel exist in the plurality of fusion images; and when hydropower plant personnel exist in the fused images, taking a plurality of target personnel graphs corresponding to the personnel types as the target hydropower plant personnel.
Wherein, the preset category refers to the rest category among others.
Optionally, the selecting the target image corresponding to the target hydropower plant personnel from the multiple fusion images is to use an image including the target hydropower plant personnel in the multiple fusion images as the target image.
And S3, calculating the intra-plant distance between the target hydropower plant personnel and the target camera, reconstructing the three-dimensional scene of the target hydropower plant personnel based on the intra-plant distance and the target image to obtain a three-dimensional reconstruction scene, and dividing the background three-dimensional scene of the target hydropower plant personnel from the three-dimensional reconstruction scene.
Therefore, the method and the device are used for determining the intra-plant distance between the area where the target hydropower plant personnel is located and the target camera by using the shielding device as a reference object when the target hydropower plant personnel is shielded by the intra-hydropower plant device by calculating the intra-plant distance between the target hydropower plant personnel and the target camera.
In this embodiment, calculating the intra-plant distance between the target hydropower plant personnel and the target camera includes: calculating a focal length between the target hydropower plant personnel and the target camera by using the following formula:
；
；
Wherein, Representing the focal length of the target camera,/>Representing a distance sample between an object sample and the target camera,/>Representing the size of the object sample in the image obtained by photographing the object sample by the target camera,/>Representing the actual size of the object sample;
Acquiring the neighborhood equipment size of the target hydropower plant personnel in the target image; acquiring a horizontal deflection angle and a vertical deflection angle when the target camera shoots the target image, and calculating the device-camera distance between the neighborhood device of the target camera under the horizontal deflection angle and the target camera based on the vertical deflection angle by using the following formula:
；
；
Wherein, Representing device-camera distance,/>Representing the height of the target camera,/>Representing the vertical deflection angle;
And taking the device-camera distance as the in-plant distance when the focal length calculated by using the device-camera distance, the actual size of the neighborhood device and the neighborhood device size is consistent with the focal length of the target camera.
The horizontal deflection angle refers to the angle of the target camera rotating left and right in the horizontal direction, and the vertical deflection angle refers to the angle of the target camera relative to the wall.
Optionally, the process of calculating the focal length by using the device-camera distance, the actual size of the neighboring device and the focal length of the target camera is: substituting the device-camera distance, the actual size of the neighborhood device, and the neighborhood device size into a formula。
Further, the process of reconstructing the three-dimensional scene of the target hydropower plant personnel based on the intra-plant distance and the target image to obtain the three-dimensional reconstructed scene is as follows: and based on the connection of the neighborhood equipment corresponding to the intra-plant distance and the target hydropower plant personnel on the appearance in the target image, carrying out three-dimensional model construction on the target hydropower plant personnel in the target image, wherein in the process, equipment or other objects connected with the neighborhood equipment exist, and the equipment or other objects also need to carry out three-dimensional model construction.
Specifically, referring to fig. 3, segmenting the background three-dimensional scene of the target hydropower plant personnel from the three-dimensional reconstructed scene includes:
S301, inquiring a background image of the target hydropower plant personnel in the target image;
s302, extracting a background three-dimensional scene corresponding to the background image from the three-dimensional reconstruction scene based on the mapping relation between the background image and the three-dimensional reconstruction scene.
The background image refers to a two-dimensional image of the neighborhood device, the device connected with the neighborhood device or other objects.
S4, acquiring an actual in-plant scene corresponding to the hydropower plant image, and performing rough positioning on the target hydropower plant personnel in the three-dimensional reconstruction scene according to the actual in-plant scene to obtain a rough personnel position.
Therefore, the embodiment can perform coarse positioning on the personnel of the target hydropower plant in the three-dimensional reconstruction scene, so that the position of the area where the overlapped equipment or building can be detected is used as the coarse position of the personnel when the image scene is overlapped.
Further, according to the actual in-plant scene, performing coarse positioning on the target hydropower plant personnel in the three-dimensional reconstruction scene to obtain a coarse personnel position, including: and taking the area where the equipment in the actual in-plant scene accords with the appearance characteristics of the neighborhood equipment in the three-dimensional reconstruction scene as the coarse position of the personnel.
And S5, combining the rough positions of the staff, splicing the path tracks of the target hydropower plant staff in the hydropower plant, and dynamically positioning the target hydropower plant staff based on the path tracks to obtain a hydropower plant staff positioning result in the hydropower plant.
Therefore, the embodiment combines the rough positions of the personnel, and splices the path tracks of the target hydropower plant personnel in the hydropower plant so as to dynamically track the positioning of the target hydropower plant personnel in the hydropower plant, namely, the specific positions of the target hydropower plant personnel near overlapped equipment or buildings can be determined based on the path tracks of the target hydropower plant personnel in the hydropower plant, so that the positions of the target hydropower plant personnel near the overlapped equipment or buildings can be distinguished.
Combining the coarse positions of the personnel, splicing the path tracks of the personnel of the target hydropower plant in the hydropower plant, and comprising the following steps: inquiring passable paths between every two coarse positions of the personnel in an actual in-plant scene; extracting a first segment of path from the traversable paths; identifying whether contact exists between the target hydropower plant personnel and objects within the hydropower plant; when there is contact between the target hydropower plant personnel and an object within the hydropower plant, calculating a first average moving speed of the target hydropower plant personnel using the following formula:
；
；
；
；
；
Wherein, 、/>、/>Representing a first average moving speed,/>Representing the length of the first segment path,/>Representing the period of walking the first path,/>Indicating that the kth segment path does not have a contact point in the first segment path, and/>, when the kth segment path does not have a contact point in the first segment path，/>The expression slave/>Remaining period after the period of removing the contact point of contact,/>Indicating that the kth segment path exists at the contact point in the first segment path, and/>, when the kth segment path exists at the contact point in the first segment pathK represents the sequence number of the remaining paths except the first path;
When there is no contact between the target hydropower plant personnel and an object within the hydropower plant, calculating a second average movement speed of the target hydropower plant personnel using the following formula:
；
Wherein, Representing a second average movement speed,/>Representing the length of the first segment path,/>A period representing a first segment of the path of travel;
Calculating the path length between each two rough positions of the personnel by using the first average moving speed, the second average moving speed and the moment when the target hydropower plant personnel reach each rough position of the personnel; and when the path length of the passable path accords with the path length between every two rough positions of the personnel, splicing the passable path into the path track of the personnel of the target hydropower plant in the hydropower plant.
The first path is a path between a first personnel rough position and a second personnel rough position from an entrance, and it is required to be noted that when contact exists between the target hydropower plant personnel and an object in the hydropower plant, the target hydropower plant personnel is indicated to be a non-patrol crowd, such as a maintainer, a constructor and the like, the non-patrol crowd also includes overhaul or construction time besides normal walking, when contact exists between the target hydropower plant personnel and the object in the hydropower plant, the target hydropower plant personnel is indicated to be a patrol crowd, the patrol crowd only includes normal walking time, and the contact point refers to overhauled equipment, a built object and the like.
Optionally, when the path length of the passable path accords with the path length between the coarse positions of every two persons, the process of splicing the passable path into the path track of the target hydropower plant person in the hydropower plant is as follows: because a plurality of passable paths exist between every two rough positions of the personnel, when the path length of the passable paths accords with the path length between every two rough positions of the personnel, a unique one path can be selected from the plurality of passable paths, so that the selected unique paths between every two rough positions of the personnel can be spliced, and the travelling path can be obtained.
The process of dynamically positioning the target hydropower plant personnel based on the path track to obtain the hydropower plant personnel positioning result in the hydropower plant refers to taking the walking record of the target hydropower plant personnel on the path track as the hydropower plant personnel positioning result in the hydropower plant.
Therefore, in the embodiment, firstly, image preprocessing is performed on a plurality of hydropower plant images so as to convert the value range of the pixel gray values in the images from 0-255 to 0-1 on the premise of ensuring that the information of the images is unchanged, and image mapping is performed, so that the aim of information fusion is achieved, incomplete scenes shot by cameras with different angles are spliced into complete scenes, feature map extraction is performed on the plurality of fusion images so as to extract target features in the plurality of fusion images by utilizing convolution kernels, target personnel maps in the plurality of extraction feature maps are detected, and then a plurality of personnel category probabilities of the plurality of target personnel maps are calculated so as to identify the categories of personnel in the target personnel maps.
In the process, a default candidate frame mechanism and a regression mechanism of the RPN network are used for reference, and meanwhile, the thought of multi-scale feature characterization is adopted for target detection, so that an algorithm for target extraction has the characteristics of high detection speed and high precision.
Meanwhile, in the embodiment, when the target hydropower plant personnel is blocked by the hydropower plant internal equipment, the blocking equipment is used as a reference to determine the intra-plant distance between the area where the target hydropower plant personnel is located and the target camera, a three-dimensional reconstruction scene is constructed, the target hydropower plant personnel in the three-dimensional reconstruction scene is roughly positioned, the position of the area where the overlapped equipment or building can be detected is used as the personnel rough position under the condition that the image scene is overlapped, the rough position of the personnel is further combined, the path track of the target hydropower plant personnel in the hydropower plant is spliced, the positioning of the target hydropower plant personnel in the hydropower plant is dynamically tracked, and meanwhile, the specific position of the target hydropower plant personnel in the vicinity of the overlapped equipment or building can be determined according to the path track of the target hydropower plant personnel in the hydropower plant, so that the specific position of the target hydropower plant personnel in the vicinity of the overlapped equipment or building can be distinguished.
Therefore, the embodiment can ensure quick Zhu Zhunque positioning of hydropower plant personnel under the condition of overlapping image scenes.
As shown in fig. 4, another embodiment of the present invention provides a hydropower plant personnel positioning system 400 based on ptz multi-camera linkage, which can implement the above hydropower plant personnel positioning method based on ptz multi-camera linkage, where the hydropower plant personnel positioning system 400 includes an image fusion module 401, a camera selection module 402, a scene reconstruction module 403, a personnel rough positioning module 404, and a dynamic positioning module 405. Specifically, the functions for each module/unit are as follows:
The image fusion module 401 is configured to capture a plurality of hydropower plant images in a hydropower plant by using a plurality of panoramic high-speed dome cameras, perform image preprocessing on the plurality of hydropower plant images to obtain a plurality of preprocessed images, register the plurality of preprocessed images to obtain a plurality of registered images, and fuse the plurality of registered images to obtain a plurality of fused images;
The camera selection module 402 is configured to perform feature map extraction on the multiple fused images to obtain multiple extracted feature maps, detect multiple target person maps in the multiple extracted feature maps, calculate multiple person class probabilities of the multiple target person maps, detect target hydropower plant persons in the multiple fused images according to the multiple person class probabilities, select a target image corresponding to the target hydropower plant persons from the multiple fused images, and select a target camera for shooting the target image from the multiple panoramic speed dome cameras;
The reconstruction scene module 403 is configured to calculate an intra-plant distance between the target hydropower plant personnel and the target camera, reconstruct a three-dimensional scene of the target hydropower plant personnel based on the intra-plant distance and the target image, obtain a three-dimensional reconstruction scene, and segment a background three-dimensional scene of the target hydropower plant personnel from the three-dimensional reconstruction scene;
The personnel coarse positioning module 404 is configured to obtain an actual in-plant scene corresponding to the hydropower plant image, and perform personnel coarse positioning on the target hydropower plant personnel in the three-dimensional reconstructed scene according to the actual in-plant scene, so as to obtain a personnel coarse position;
The dynamic positioning module 405 is configured to combine the coarse positions of the personnel, splice the path trajectories of the target hydropower plant personnel in the hydropower plant, and dynamically position the target hydropower plant personnel based on the path trajectories, so as to obtain a positioning result of the hydropower plant personnel in the hydropower plant.
In detail, the modules in the ptz multi-camera linkage-based hydropower station personnel positioning device 400 in the embodiment of the invention adopt the same technical means as the ptz multi-camera linkage-based hydropower station personnel positioning method described in fig. 1 to 3 and can produce the same technical effects when in use, and are not described herein.
The foregoing is only a specific embodiment of the invention to enable those skilled in the art to understand or practice the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims (10)
1. A hydropower plant personnel positioning method based on ptz multi-camera linkage is characterized by comprising the following steps:
Shooting a plurality of hydropower plant images in a hydropower plant by using a plurality of panoramic high-speed dome cameras, performing image preprocessing on the plurality of hydropower plant images to obtain a plurality of preprocessed images, registering the plurality of preprocessed images to obtain a plurality of registered images, and fusing the plurality of registered images to obtain a plurality of fused images;
Extracting feature images of the multiple fusion images to obtain multiple extracted feature images, detecting multiple target personnel images in the multiple extracted feature images, calculating multiple personnel category probabilities of the multiple target personnel images, detecting target hydropower plant personnel in the multiple fusion images according to the multiple personnel category probabilities, selecting target images corresponding to the target hydropower plant personnel from the multiple fusion images, and selecting a target camera for shooting the target images from the multiple panoramic speed dome cameras;
Calculating the intra-plant distance between the target hydropower plant personnel and the target camera, and reconstructing a three-dimensional scene of the target hydropower plant personnel based on the intra-plant distance and the target image to obtain a three-dimensional reconstruction scene, and dividing the background three-dimensional scene of the target hydropower plant personnel from the three-dimensional reconstruction scene;
Acquiring an actual in-plant scene corresponding to the hydropower plant image, and performing personnel coarse positioning on the target hydropower plant personnel in the three-dimensional reconstruction scene according to the actual in-plant scene to obtain a personnel coarse position;
and combining the rough positions of the staff, splicing the path tracks of the target hydropower plant staff in the hydropower plant, and dynamically positioning the target hydropower plant staff based on the path tracks to obtain a hydropower plant staff positioning result in the hydropower plant.
2. The method of claim 1, wherein fusing the plurality of registered images to obtain a plurality of fused images comprises:
Identifying feature matching points between the plurality of registered images;
calculating a mapping matrix between each of the feature matching points by using the following formula:
；
；
Wherein, Representing a mapping matrix,/>Representing the coordinates of one of the feature matching points,/>Representing another AND/>Coordinates of feature points constituting feature matching points between corresponding feature points,/>Representation ofAnd/>Vector cross product between;
and fusing the plurality of registration images by using the mapping matrix to obtain a plurality of fused images.
3. The method of claim 1, wherein performing feature map extraction on the plurality of fused images to obtain a plurality of extracted feature maps comprises:
and filtering the first feature map of the fused images by using the following formula to obtain a plurality of filtering feature maps:
；
Wherein, Pixel values representing pixel points in the plurality of filter feature maps,/>Representing the coordinates in the plurality of fused images as/>Pixel value of pixel point of/>Representing coordinates of pixel points in the plurality of fused images,Representing the co-ordinates/>, in a convolution kernel used in first feature map filtering the plurality of fused imagesCorresponding coordinates/>Representing the coordinates in the convolution kernel as/>Pixel values of the pixel points of (a);
performing downsampling operation on the plurality of filtering feature images to obtain a plurality of downsampled feature images;
and performing second feature map filtering on the plurality of downsampled feature maps to obtain the plurality of extracted feature maps.
4. The method of claim 1, wherein detecting a plurality of target person graphs in the plurality of extracted feature graphs comprises:
Performing multi-scale feature extraction on the plurality of extracted feature images to obtain a multi-scale feature image;
carrying out feature map framing on the multi-scale feature map to obtain a framing feature map;
And performing non-maximum suppression on the frame feature map to obtain the plurality of target personnel maps.
5. The method of claim 4, wherein framing the multi-scale feature map to obtain a framed feature map comprises:
Calculating candidate frame parameters of the multi-scale feature map by using the following formula:
；
；
Wherein, Candidate box parameters representing the multi-scale feature map,/>Representing coordinates of pixel points selected from the multi-scale feature map by the neural network model,/>Expressed as/>The width of the rectangle with the coordinates as the center,Expressed as/>Height of rectangle with coordinates as center,/>Representing the size of the candidate frame corresponding to the candidate frame parameter,/>Parameter representing minimum scale of candidate frame initially set in neural network structure,/>Parameter representing maximum scale of candidate frame initially set in neural network structure,/>Representing the number of feature maps of a plurality of different scales in the multi-scale feature map,/>Serial number representing a feature map of the multiscale feature map,/>Representing an ith feature map of the multiscale feature map,/>Representation/>Size,/>，/>Representing an aspect ratio preset according to different sizes of the multi-scale feature map;
And carrying out parameter transformation on the candidate frame parameters by using the following formula to obtain transformation parameters:
；
Wherein, Representing the transformation parameters,/>Candidate box parameters representing the multi-scale feature map,/>Representing a fixed measurement scale preset in a neural network model, wherein the fixed measurement scale is used for measuring the mapping relation between the candidate frame parameters and the transformation parameters;
And taking the candidate frame corresponding to the transformation parameter as the frame division characteristic diagram.
6. The method of claim 1, wherein calculating a plurality of person class probabilities for the plurality of target person graphs comprises:
calculating the personnel category index of the plurality of target personnel graphs using the following formula:
；
Wherein, Person class index representing the plurality of target person diagrams,/>Weights representing classification layers in neural network model,/>Bias vector representing classification layer in neural network model,/>Representing a vector matrix composed of feature vectors in the plurality of target person diagrams;
according to the personnel category index, calculating a plurality of personnel category probabilities of the plurality of target personnel graphs by using the following formula:
；
Wherein, Multiple person class probabilities representing the multiple target person graphs,/>A person class index representing the plurality of target person graphs, u representing a neuron number of an output layer in the neural network model, N representing a total number of neurons of the output layer in the neural network model, v representing a neuron number of the output layer in the neural network model,/>N。
7. The method of claim 1, wherein calculating the in-plant distance between the target hydropower plant personnel and the target camera comprises:
calculating a focal length between the target hydropower plant personnel and the target camera by using the following formula:
；
；
Wherein, Representing the focal length of the target camera,/>Representing a distance sample between an object sample and the target camera,/>Representing the size of the object sample in the image obtained by photographing the object sample by the target camera,/>Representing the actual size of the object sample;
acquiring the neighborhood equipment size of the target hydropower plant personnel in the target image;
Acquiring a horizontal deflection angle and a vertical deflection angle when the target camera shoots the target image, and calculating the device-camera distance between the neighborhood device of the target camera under the horizontal deflection angle and the target camera based on the vertical deflection angle by using the following formula:
；
；
Wherein, Representing device-camera distance,/>Representing the height of the target camera,/>Representing the vertical deflection angle;
And taking the device-camera distance as the in-plant distance when the focal length calculated by using the device-camera distance, the actual size of the neighborhood device and the neighborhood device size is consistent with the focal length of the target camera.
8. The method of claim 1, wherein segmenting the background three-dimensional scene of the target hydropower plant personnel from the three-dimensional reconstructed scene comprises:
inquiring a background image of the target hydropower plant personnel in the target image;
and extracting a background three-dimensional scene corresponding to the background image from the three-dimensional reconstruction scene based on the mapping relation between the background image and the three-dimensional reconstruction scene.
9. The method of claim 1, wherein said combining said coarse positions of said personnel to stitch up path trajectories of said target hydropower plant personnel within said hydropower plant comprises:
inquiring passable paths between every two coarse positions of the personnel in an actual in-plant scene;
extracting a first segment of path from the traversable paths;
identifying whether contact exists between the target hydropower plant personnel and objects within the hydropower plant;
When there is contact between the target hydropower plant personnel and an object within the hydropower plant, calculating a first average moving speed of the target hydropower plant personnel using the following formula:
；
；
；
；
；
Wherein, 、/>、/>Representing a first average moving speed,/>Representing the length of the first segment path,/>Representing the period of walking the first path,/>Indicating that the kth segment path does not have a contact point in the first segment path, and/>, when the kth segment path does not have a contact point in the first segment path，/>The expression slave/>Remaining period after the period of removing the contact point of contact,/>Indicating that the kth segment path exists at the contact point in the first segment path, and/>, when the kth segment path exists at the contact point in the first segment pathK represents the sequence number of the remaining paths except the first path;
When there is no contact between the target hydropower plant personnel and an object within the hydropower plant, calculating a second average movement speed of the target hydropower plant personnel using the following formula:
；
Wherein, Representing a second average movement speed,/>Representing the length of the first segment path,/>A period representing a first segment of the path of travel;
Calculating the path length between each two rough positions of the personnel by using the first average moving speed, the second average moving speed and the moment when the target hydropower plant personnel reach each rough position of the personnel;
And when the path length of the passable path accords with the path length between every two rough positions of the personnel, splicing the passable path into the path track of the personnel of the target hydropower plant in the hydropower plant.
10. Hydropower plant personnel positioning system for implementing the hydropower plant personnel positioning method according to any one of claims 1-9, characterized in that the system comprises:
The image fusion module is used for shooting a plurality of hydropower plant images in the hydropower plant by utilizing a plurality of panoramic high-speed dome cameras, carrying out image preprocessing on the plurality of hydropower plant images to obtain a plurality of preprocessed images, registering the plurality of preprocessed images to obtain a plurality of registered images, and fusing the plurality of registered images to obtain a plurality of fused images;
The camera selection module is used for extracting feature images of the multiple fusion images to obtain multiple extracted feature images, detecting multiple target personnel images in the multiple extracted feature images, calculating multiple personnel category probabilities of the multiple target personnel images, detecting target hydropower plant personnel in the multiple fusion images according to the multiple personnel category probabilities, selecting target images corresponding to the target hydropower plant personnel from the multiple fusion images, and selecting target cameras for shooting the target images from the multiple panoramic speed dome cameras;
The reconstruction scene module is used for calculating the intra-plant distance between the target hydropower plant personnel and the target camera, reconstructing the three-dimensional scene of the target hydropower plant personnel based on the intra-plant distance and the target image to obtain a three-dimensional reconstruction scene, and dividing the background three-dimensional scene of the target hydropower plant personnel from the three-dimensional reconstruction scene;
The personnel coarse positioning module is used for acquiring an actual in-plant scene corresponding to the hydropower plant image, and performing personnel coarse positioning on the target hydropower plant personnel in the three-dimensional reconstruction scene according to the actual in-plant scene to obtain a personnel coarse position;
and the dynamic positioning module is used for combining the rough positions of the personnel, splicing the path tracks of the target hydropower plant personnel in the hydropower plant, and dynamically positioning the target hydropower plant personnel based on the path tracks to obtain the hydropower plant personnel positioning result in the hydropower plant.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410445115.5A CN118038008B (en) | 2024-04-15 | Hydropower plant personnel positioning method and system based on ptz multi-camera linkage |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410445115.5A CN118038008B (en) | 2024-04-15 | Hydropower plant personnel positioning method and system based on ptz multi-camera linkage |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN118038008A true CN118038008A (en) | 2024-05-14 |
| CN118038008B CN118038008B (en) | 2024-07-12 |
Family
ID=
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2021196294A1 (en) * | 2020-04-03 | 2021-10-07 | 中国科学院深圳先进技术研究院 | Cross-video person location tracking method and system, and device |
| CN115546710A (en) * | 2022-08-09 | 2022-12-30 | 国网湖北省电力有限公司黄龙滩水力发电厂 | Method, device and equipment for locating personnel in hydraulic power plant and readable storage medium |
| CN116704377A (en) * | 2022-12-30 | 2023-09-05 | 中国电力科学研究院有限公司 | Non-inductive personnel positioning method and system based on digital twin model |
| CN117635721A (en) * | 2022-08-16 | 2024-03-01 | 华为云计算技术有限公司 | Target positioning method, related system and storage medium |
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2021196294A1 (en) * | 2020-04-03 | 2021-10-07 | 中国科学院深圳先进技术研究院 | Cross-video person location tracking method and system, and device |
| CN115546710A (en) * | 2022-08-09 | 2022-12-30 | 国网湖北省电力有限公司黄龙滩水力发电厂 | Method, device and equipment for locating personnel in hydraulic power plant and readable storage medium |
| CN117635721A (en) * | 2022-08-16 | 2024-03-01 | 华为云计算技术有限公司 | Target positioning method, related system and storage medium |
| CN116704377A (en) * | 2022-12-30 | 2023-09-05 | 中国电力科学研究院有限公司 | Non-inductive personnel positioning method and system based on digital twin model |
Non-Patent Citations (1)
| Title |
|---|
| 徐安丽 等: "一种基于实景三维模型的定位方法", 《科学技术与工程》, no. 02, 18 January 2016 (2016-01-18), pages 224 - 228 * |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Attard et al. | Vision-based change detection for inspection of tunnel liners | |
| CN108230237B (en) | Multispectral image reconstruction method for electrical equipment online detection | |
| CN113111727B (en) | Feature alignment-based method for detecting rotating target in remote sensing scene | |
| CN111932596B (en) | Method, device and equipment for detecting camera occlusion area and storage medium | |
| CN112541403B (en) | Indoor personnel falling detection method by utilizing infrared camera | |
| US20220210375A1 (en) | Method and system for optical monitoring of unmanned aerial vehicles based on three-dimensional light field technology | |
| CN111383204A (en) | Video image fusion method, fusion device, panoramic monitoring system and storage medium | |
| CN113052066A (en) | Multi-mode fusion method based on multi-view and image segmentation in three-dimensional target detection | |
| CN112613668A (en) | Scenic spot dangerous area management and control method based on artificial intelligence | |
| CN111967396A (en) | Processing method, device and equipment for obstacle detection and storage medium | |
| CN112084892B (en) | Road abnormal event detection management device and method thereof | |
| CN113569981A (en) | Power inspection bird nest detection method based on single-stage target detection network | |
| CN114842340A (en) | Robot binocular stereoscopic vision obstacle sensing method and system | |
| Wang et al. | Real-time damaged building region detection based on improved YOLOv5s and embedded system from UAV images | |
| CN110636248B (en) | Target tracking method and device | |
| CN113158963A (en) | High-altitude parabolic detection method and device | |
| Dutta et al. | Real testbed for autonomous anomaly detection in power grid using low-cost unmanned aerial vehicles and aerial imaging | |
| CN118038008B (en) | Hydropower plant personnel positioning method and system based on ptz multi-camera linkage | |
| CN117115412A (en) | Small target detection method based on weighted score label distribution | |
| CN114648736B (en) | Robust engineering vehicle identification method and system based on target detection | |
| CN118038008A (en) | Hydropower plant personnel positioning method and system based on ptz multi-camera linkage | |
| CN116051876A (en) | Camera array target recognition method and system of three-dimensional digital model | |
| CN114708544A (en) | Intelligent violation monitoring helmet based on edge calculation and monitoring method thereof | |
| CN114912536A (en) | Target identification method based on radar and double photoelectricity | |
| CN114639084A (en) | Road side end vehicle sensing method based on SSD (solid State disk) improved algorithm |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant |