CN116012892A - Signal device and signal transmission method for tower crane signal engineering - Google Patents
Signal device and signal transmission method for tower crane signal engineering Download PDFInfo
- Publication number
- CN116012892A CN116012892A CN202310300309.1A CN202310300309A CN116012892A CN 116012892 A CN116012892 A CN 116012892A CN 202310300309 A CN202310300309 A CN 202310300309A CN 116012892 A CN116012892 A CN 116012892A
- Authority
- CN
- China
- Prior art keywords
- signal
- tower crane
- worker
- gesture
- joint
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 47
- 230000008054 signal transmission Effects 0.000 title claims abstract description 22
- 230000011664 signaling Effects 0.000 claims abstract description 28
- 238000001514 detection method Methods 0.000 claims abstract description 10
- 230000008569 process Effects 0.000 claims abstract description 8
- 230000011218 segmentation Effects 0.000 claims abstract description 8
- 238000012549 training Methods 0.000 claims description 17
- 238000002372 labelling Methods 0.000 claims description 10
- 210000001145 finger joint Anatomy 0.000 claims description 9
- 230000000007 visual effect Effects 0.000 claims description 8
- 210000002310 elbow joint Anatomy 0.000 claims description 7
- 210000000323 shoulder joint Anatomy 0.000 claims description 7
- 230000001360 synchronised effect Effects 0.000 claims description 7
- 238000005259 measurement Methods 0.000 claims description 6
- 230000005540 biological transmission Effects 0.000 claims description 5
- 238000005457 optimization Methods 0.000 claims description 4
- 210000003857 wrist joint Anatomy 0.000 claims description 4
- 230000002776 aggregation Effects 0.000 claims description 3
- 238000004220 aggregation Methods 0.000 claims description 3
- 238000004891 communication Methods 0.000 claims description 3
- 238000004519 manufacturing process Methods 0.000 claims description 3
- 210000000707 wrist Anatomy 0.000 claims description 3
- 238000010845 search algorithm Methods 0.000 abstract description 3
- 230000006870 function Effects 0.000 description 6
- 238000010586 diagram Methods 0.000 description 3
- 210000000988 bone and bone Anatomy 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 239000000463 material Substances 0.000 description 2
- 230000006978 adaptation Effects 0.000 description 1
- 238000007792 addition Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 230000004927 fusion Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000005192 partition Methods 0.000 description 1
- 230000004044 response Effects 0.000 description 1
Images
Landscapes
- Image Analysis (AREA)
Abstract
The invention discloses a signal device and a signal transmission method for a tower crane signaling worker, and belongs to the field of signal devices. Firstly, acquiring gesture images of a tower crane signaling worker by using a high-definition camera, marking joint points by image processing, and calculating information of the joint points by using a three-dimensional coordinate system; secondly, recognizing and classifying the coordinates of the points by using a K nearest neighbor search algorithm, and providing a space segmentation method for dividing the joint points so as to ensure the high efficiency of the recognition and classification process; finally, to achieve real-time signal transmission, a deployment method is described for arranging the detection device on a field programmable gate array. The method and the device can acquire the signal gesture in real time, immediately plan the driving intention in real time, and effectively improve the working efficiency and the safety; the invention can recognize the tower crane signal worker and the tower crane signal worker gesture as true value information, and effectively improve the safety of the tower crane work while improving the working efficiency of the tower crane.
Description
Technical Field
The invention belongs to the field of signal devices, and particularly relates to a signal device and a signal transmission method for a tower crane signaling worker.
Background
When the traditional tower crane is used for lifting and hoisting materials, the height of a building is increased along with the promotion of construction progress, and blind areas often exist when the tower crane is used for lifting and hoisting materials; the blind area operation needs the tower crane driver and the signal worker to be perfectly matched to achieve lifting and hoisting without errors, but sometimes the interphone can not transmit the command in real time, clearly and completely due to long distance between the tower crane driver and the signal worker or a series of interference, so that the tower crane driver can not accurately and timely cope with various emergency situations, and great potential safety hazards are brought to the lifting and hoisting operation. In this scenario, it is important how to accurately and completely identify and transmit the tower crane signal work order.
Disclosure of Invention
Aiming at the technical problems in the prior art, the invention provides a signal device and a signal transmission method for a tower crane signaling worker, which are reasonable in design, overcome the defects in the prior art and have good effects.
In order to achieve the above purpose, the present invention adopts the following technical scheme: a signal device for a tower crane signal worker comprises three cameras, a gesture recognition signal sending unit and a tower crane vehicle end signal receiving unit; the camera is configured to acquire tower crane signal work targets, positions and gesture characteristic data; the gesture recognition signal sending unit is configured to recognize gesture signals of a signaler in real time and transmit the signals to the tower crane vehicle end signal receiving unit; the tower crane vehicle end signal receiving unit is configured to receive gesture signals so as to facilitate corresponding operation of a tower crane driver.
Preferably, the tower crane vehicle end signal receiving unit comprises a wireless signal receiving module, a wireless signal identifying module, a display screen and a voice broadcasting module; the wireless signal receiving module is configured to be used for receiving the wireless signal transmitted by the gesture recognition signal transmitting unit by the tower crane; the wireless signal recognition module is configured to recognize the wireless signal transmitted by the gesture recognition signal transmitting unit by the tower crane; the display screen is configured to display gestures of a tower crane signal worker in real time; the voice broadcasting module is configured to be used for playing gestures of the tower crane signal worker in real time.
Preferably, three cameras are respectively disposed at the left side, the upper side and the right side of the gesture recognition signal transmission unit; the cameras shoot and record synchronous video information of the tower crane signal worker from three different directions, the detected video content comprises all information in an active area of the tower crane signal worker, the cameras are calibrated independently by using a world coordinate system, and the cameras are calibrated in a joint mode by using a epipolar geometry constraint method, so that the signal worker gestures can be positioned and identified accurately.
Preferably, after the gesture recognition signal sending unit collects and marks the target, the position and gesture characteristic data of the tower crane signaling worker, a data set is formed; the recognition and detection of each camera is trained by adopting a new frame optimization KNN algorithm of space segmentation, namely a K nearest neighbor search algorithm, a mean square error loss function MSE based on distance measurement is adopted in the training process so as to improve recognition precision, and recognized tower crane signal workers and signal worker gestures are transmitted to a signal receiving device through wireless transmission.
Preferably, the tower crane vehicle end signal receiving unit receives the gesture signal and then projects a tower crane signal worker into a display screen of the tower crane according to a scaling coefficient, the display screen of the tower crane and the voice broadcasting module respectively display and broadcast command prompt voice of the gesture of the signal worker in real time, a tower crane driver can operate according to the instruction, and meanwhile the tower crane driver can zoom in and zoom out the rotation and gesture information details of the tower crane signal worker in a touch mode.
In addition, the invention also relates to a signal transmission method for a tower crane signal worker, which adopts the signal device for the tower crane signal worker and comprises the following steps: step 1: the ground camera, the gesture recognition signal transmitting unit and the tower crane vehicle end signal receiving unit are arranged and calibrated; step 2: acquiring, calibrating and manufacturing a data set of tower crane signal work targets, positions and gesture characteristic data; step 3: training a tower crane signal target, position determination and gesture recognition algorithm; step 4: the tower crane vehicle end signal receiving unit recognizes the signal tool gesture and transmits the signal tool gesture to the tower crane vehicle end device through a wireless communication technology.
Preferably, in step 1, the signal transmission device is placed at a position at a safe distance from the tower crane, and three cameras are respectively mounted above, left and right sides of the signal transmission device, and an area 2m×2m right in front of the signal transmission device is set as an active area of the tower crane signal worker.
Preferably, in step 2, using a ground camera to shoot synchronous videos of the tower crane at different visual angles, and taking the videos as a training data set; the trained dataset labeling benchmarks are as follows: a) Defining a tower crane signal engineering coordinate system; the initial standing position of the tower crane signaling worker is set as an origin, the visual direction is the positive direction of the y axis, the extending direction of the right arm vertical to the x axis is the positive direction of the x axis, and the upward direction vertical to the planes of the x axis and the y axis is the positive direction of the z axis; b) Labeling the signal class of the tower crane; the classification at the time of labeling is: the tower crane signal worker safety helmet; before identifying the tower crane signal worker gesture, accurately identifying the signal worker target, taking whether the tower crane signal worker is characterized by wearing a tower crane signal worker safety helmet or not, marking the tower crane signal worker target wearing the tower crane signal worker safety helmet as a signal worker type, and simultaneously carrying out mirror image marking on the tower crane signal worker direction, wherein the command direction is determined by the signal worker direction and the gesture togetherThe method comprises the steps of carrying out a first treatment on the surface of the c) Labeling a tower crane signal worker arm joint point and a finger joint point; the command gesture of the tower crane signal worker is completed by the left and right arms, namely, only 10 finger joints, 1 wrist joint, 1 elbow joint and 1 shoulder joint of each arm of the tower crane signal worker are marked, each joint detected by each camera contains three dimensional coordinate information, and the finger joint (f) 1 …f 10 ) Coordinate (f) x1 , f y1 , f z1 )…(f x20 , f y20 , f z20 ) Wrist node w coordinate (w x1 , w y1 , w z1 ) (w x2 , w y2 , w z2 ) Elbow joint point e coordinate (e x1 , e y1 , e z1 ) (e x2 , e y2 , e z2 ) And shoulder joint point s coordinate (s x1 , s y1 , s z1 ) (s x2 , s y2 , s z2 ) All the joint Points form a point aggregation list Points, and all the joint Points are connected into a line to form a skeleton annotation list; firstly, solving the distance between each joint point of the signal worker and the sensor, and then obtaining the real position of each joint point of the signal worker in the three-dimensional space by using a similarity relationship through a world coordinate system established by the signal worker; d) Calibrating a gesture of a tower crane signaling worker; according to different gestures of an actual tower crane signal worker, classification and calibration are carried out for different gestures, and the tower crane signal worker is used for training of different gesture recognition.
Preferably, in step 3, a new frame of space division is adopted to optimize the KNN algorithm to train the 3D joint point detection of each camera, wherein the KNN algorithm is to search K points closest to the target point; firstly, defining a preset searching range (Rin), wherein the range ensures that all nodes can be searched; dividing the point set to obtain a group of three-dimensional space with the side length equal to Rin; if a three-dimensional space contains too many points, a threshold T is set max Dividing the three-dimensional space into a plurality of subspaces; secondly, constructing a novel joint point data structure; by traversing all points to find a bounding box, the minimum (lowbound) and maximum (highbound) ranges on each axis are calculated as AABB (coordinate axis alignment bounding box) of the point cloud； 
； />
The method comprises the steps of carrying out a first treatment on the surface of the Wherein x is max ，y max ，z max Is the maximum value of the coordinates of the joint points; x is x min ，y min ，z min Is the minimum value of the coordinates of the joint points; and dividing the AABB into a set of spaces VS; />
； />
The method comprises the steps of carrying out a first treatment on the surface of the Wherein VS is x ，VS y ，VS z The number of spaces per axis; r is R re For the space side length, R re Is not less than Rin; assigning an index vp to each space and defining a hash function hash (p); />
； 
The method comprises the steps of carrying out a first treatment on the surface of the Wherein, voxel (p) is any point in space; vp x ，vp y ，vp z Index coordinates; through the above procedure, a spatial index including a specific point can be calculated; when the number of points in a space exceeds a threshold T max When it is divided into a group of subspaces with K points at most; the same joint point from different cameras measures and calculates 3D joint point information of the tower crane signal worker arm in space through a three-dimensional coordinate system; the training process adopts a mean square error loss function MSE based on distance measurement, and the formula is as follows: />
The method comprises the steps of carrying out a first treatment on the surface of the Wherein (1)>
For the real coordinate values of all joint points in space, +.>
And detecting coordinate values for the space of all the joint points, wherein n is the acquisition times, and stopping acquisition when the MSE is less than or equal to 0.01.
The invention has the beneficial technical effects that: 1) According to the invention, under the condition that a tower crane worker with a visual field blind area cannot timely acquire the signal worker gesture information of the tower crane, the signal worker gesture can be acquired in real time, and the driving intention planning can be immediately performed in real time, so that the working efficiency and the safety can be effectively improved; 2) The method can recognize the tower crane signal worker and the tower crane signal worker gesture as true value information, effectively improve the safety of the tower crane work while improving the working efficiency of the tower crane, and can complete all relevant detection, recognition and transmission tasks under the condition of low cost by using only a camera and wireless transmission equipment.
Drawings
FIG. 1 is a schematic diagram of the arrangement of high definition cameras of a ground device and the positional relationship with a tower crane; FIG. 2 is a flow chart of the method of the present invention; FIG. 3 is a schematic diagram of a ground high definition camera and gesture recognition signal transmission unit; FIG. 4 is a flow chart of a new frame-optimized KNN algorithm for space division; FIG. 5 is a schematic diagram of a tower crane end gesture signal receiving device; FIG. 6 is a schematic illustration of a signaler arm joint and bone dataset annotation; wherein 1-a first camera; 2-a second camera; 3-a third camera; 4-a gesture recognition signal transmission unit; 5-a tower crane end signal receiving unit; 6-a display screen; 7-voice broadcasting device.
Detailed Description
The invention is described in further detail below with reference to the attached drawings and detailed description: example 1: as shown in fig. 1, a signaling device for a tower crane signaling worker comprises three cameras, a gesture recognition signal sending unit 4 and a tower crane vehicle end signal receiving unit 5; the camera is configured to acquire tower crane signal work targets, positions and gesture characteristic data; the three cameras are a first camera 1, a second camera 2 and a third camera 3 respectively; the first camera 1, the second camera 2, and the third camera 3 are disposed above, left, and right of the gesture recognition signal transmission unit 4, respectively.
The gesture recognition signal sending unit 4 is configured to recognize gesture signals of a signaler in real time and send the signals to the tower crane vehicle end signal receiving unit 5; the tower crane vehicle end signal receiving unit 5 is configured to receive gesture signals so as to facilitate corresponding operation of a tower crane driver.
The tower crane vehicle end signal receiving unit 5, as shown in fig. 5, comprises a wireless signal receiving module, a wireless signal identifying module, a display screen 6 and a voice broadcasting device 7; the wireless signal receiving module is configured to be used for receiving the wireless signal transmitted by the gesture recognition signal transmitting unit by the tower crane; the wireless signal recognition module is configured to recognize the wireless signal transmitted by the gesture recognition signal transmitting unit by the tower crane; a display screen 6 configured to display a gesture of a tower crane signaling worker in real time; the voice broadcasting device 7 is configured to play the gesture of the tower crane signaling worker in real time.
The cameras shoot and record synchronous video information of the tower crane signal worker from three different directions, the detected video content comprises all information in an active area of the tower crane signal worker, the cameras are calibrated independently by using a world coordinate system, and the cameras are calibrated in a joint mode by using a epipolar geometry constraint method, so that the signal worker gestures can be positioned and identified accurately.
After the gesture recognition signal sending unit 4 collects the target, position and gesture characteristic data of the calibration tower crane signal worker, a data set is formed; the recognition and detection of each camera are trained by adopting a new frame optimization KNN algorithm of space segmentation, a mean square error loss function MSE based on distance measurement is adopted in the training process to improve recognition accuracy, and recognized tower crane signal work and signal work gestures are transmitted to a signal receiving device through wireless transmission.
The tower crane vehicle end signal receiving unit 5 projects a tower crane signal worker into the display screen 6 of the tower crane according to a certain scaling factor after receiving the gesture signal, the display screen 6 of the tower crane and the voice broadcasting device 7 display and broadcast command prompt voice of the signal worker gesture in real time, a tower crane driver can operate according to the instruction, and meanwhile the tower crane driver can zoom in and zoom out the rotation and gesture information details of the tower crane signal worker in a touch mode.
Example 2: on the basis of the embodiment 1, the invention also relates to a signal transmission method for a tower crane signaling worker, the flow of which is shown in fig. 2, and the method comprises the following steps: step 1: the method comprises the steps of arranging and calibrating a plurality of cameras, gesture recognition signal sending units and tower crane vehicle end signal receiving units on the ground; step 2: acquiring, calibrating and manufacturing a data set of tower crane signal work targets, positions and gesture characteristic data; step 3: training a tower crane signal target, position determination and gesture recognition algorithm; step 4: the tower crane vehicle end signal receiving unit recognizes the signal tool gesture and transmits the signal tool gesture to the tower crane vehicle end device through a wireless communication technology.
The specific content of the step 1 is as follows: and a ground signal transmitting device is arranged at a position with a certain safety distance from the tower crane, and three high-definition cameras are fixedly connected above, left side and right side of the signal transmitting device respectively. The area 2m multiplied by 2m right in front of the screen of the transmitting device is set as an active area of a tower crane signal worker, the ground high-definition cameras shoot and record synchronous high-definition video information of the tower crane signal worker from three different directions, and all video contents detected by the high-definition cameras should contain all information in the active area of the tower crane signal worker; the high-definition cameras are calibrated independently by using a world coordinate system, and all the high-definition cameras are calibrated in a combined mode by using an epipolar geometry constraint method, so that signal gestures can be positioned and identified accurately; the tower crane vehicle end signal receiving unit is formed by combining a wireless signal receiving module, a wireless signal identifying module, a display screen and a voice broadcasting module; the wireless signal receiving module and the wireless signal identifying module are used for receiving and identifying wireless signals transmitted by the ground signal transmitting device by the tower crane; the display screen and the voice broadcasting module are used for playing gestures of a tower crane signal worker in real time.
The specific content of the step 2 is as follows: shooting synchronous high-definition videos of tower crane signal workers at different visual angles by using a ground high-definition camera, wherein the high-definition videos are used as a training data set; the trained dataset labeling benchmarks are as follows: a) Defining a tower crane signal engineering coordinate system; the initial standing position of the signaler is set as an origin, the visual direction is the positive direction of the y axis, the extending direction of the right arm vertical to the x axis is the positive direction of the x axis, and the upward direction vertical to the planes of the x axis and the y axis is the positive direction of the z axis.
b) Marking signal classes; the classification at the time of labeling is: safety helmet for signal worker; before recognizing the signal worker gesture, accurately recognizing the signal worker target, marking the signal worker target wearing the signal worker safety helmet as a signal worker type according to the characteristic of whether the signal worker is wearing the signal worker safety helmet, and simultaneously marking the signal worker direction in a mirror image mode, wherein the signal worker direction and the gesture jointly determine a command direction; c) Marking a signal worker arm joint point and a finger joint point; the command gesture of the signal worker is jointly completed by the left arm and the right arm, namely, only 10 finger joint points, 1 wrist joint point, 1 elbow joint point and 1 shoulder joint point of each arm of the signal worker are marked, each joint point detected by each ground high-definition camera contains three dimensional coordinate information, and the finger joint point (f 1 …f 10 ) Coordinate (f) x1 , f y1 , f z1 )…(f x20 , f y20 , f z20 ) Wrist node w coordinate (w x1 , w y1 , w z1 ) (w x2 , w y2 , w z2 ) Elbow joint point e coordinate (e x1 , e y1 , e z1 ) (e x2 , e y2 , e z2 ) And shoulder joint point s coordinate (s x1 , s y1 , s z1 ) (s x2 , s y2 , s z2 ) All the joint Points form a point aggregation list Points, and all the joint Points are connected into a line to form a skeleton annotation list; firstly, solving the distance between each joint point of the signal worker and the sensor, and then obtaining the real position of each joint point of the signal worker in the three-dimensional space by using a similarity relationship through a world coordinate system established by the signal worker; the method comprises the following steps: points= [ (f) x1 , f y1 , f z1 )…(f xk , f yk , f zk ), (w x1 , w y1 , w z1 ),(w x2 , w y2 , w z2 ),(e x1 , e y1 , e z1 ), (e x2 , e y2 , e z2 ),(s x1 , s y1 , s z1 ), (s x2 , s y2 , s z2 )]；Lines＝[(f 1 , f 2 ), (f 3 , f 4 ),( f 5 , f 6 ),( f 7 , f 8 ),( f 9 , f 10 ), (f 11 , f 12 ), (f 13 , f 14 ), (f 15 , f 16 ), (f 17 , f 18 ), (f 19 , f 20 ), (f 2 , w 1 ), (f 4 , w 1 ), (f 6 , w 1 ), (f 8 , w 1 ), (f 10 , w 1 ), (f 12 , w 2 ), (f 14 , w 2 ), (f 16 , w 2 ), (f 18 , w 2 ), (f 20 , w 2 ), (w 1 , e 1 ), (w 2 , e 2 ), (e 1 , s 1 ), (e 2 , s 2 ), (s 1 , s 2 ) Wherein w is wrist joint point coordinates; e is elbow joint point coordinates; s is the coordinate of the shoulder joint point; FIG. 6 is a schematic illustration of a signaller arm node and bone dataset annotation.
d) Calibrating a gesture of a signaler; and classifying and calibrating the different gestures according to the different gestures of the actual signal worker, and training for identifying the different gestures.
The specific content of the step 3 is as follows: a) And (3) training the 3D joint point detection of each high-definition camera by adopting a new frame optimization KNN algorithm of space segmentation. The KNN algorithm (the flow of which is shown in fig. 4) is a very simple search algorithm, namely, searching K closest points near the target point. Its application is very extensive, but it is difficult to achieve very high real-time requirements.
For this purpose, a predefined search range Rin is first defined, within which it is ensured that all nodes can be searched. The point set is divided to obtain a group of three-dimensional space with the side length equal to Rin. If a three-dimensional space contains too many points, a threshold T is set max The three-dimensional space is divided into a plurality of subspaces. Secondly, constructing a novel joint point data structure; by traversing all pointsTo a bounding box, computing a minimum range lowerbound and a maximum range highbound on each axis as AABB of the point cloud;
； 
the method comprises the steps of carrying out a first treatment on the surface of the Wherein x is max ，y max ，z max Is the maximum value of the coordinates of the joint points; x is x min ，y min ，z min Is the minimum value of the coordinates of the joint points; and dividing the AABB into a set of spaces VS;
； />
the method comprises the steps of carrying out a first treatment on the surface of the Wherein VS is x ，VS y ，VS z The number of spaces per axis; r is R re For the space side length, R re Is not less than Rin; assigning an index vp to each space and defining a hash function hash (p); />
； 
The method comprises the steps of carrying out a first treatment on the surface of the Wherein, voxel (p) is any point in space; vp x ，vp y ，vp z Index coordinates.
Through the above procedure, a spatial index including a specific point can be calculated; when the number of points in a space exceeds a threshold T max When it is further partitioned into a set of subspaces with at most K points inside, the reason is as follows: the second segmentation can eliminate a large number of redundant searches, and only K nearest neighbors are found; the second segmentation allows efficient searching even in uneven point concentrations, preventing space-intensive.
In addition, in order to ensure the real-time performance of the device, a detection algorithm is applied to a Field Programmable Gate Array (FPGA). During the hardware search process, external memory access points limit performance. Meanwhile, the limited amount of block read-write memory (RAM) at the Programmable Logic (PL) end is insufficient for a large-scale reference set. In response to the above problems, a custom data set buffer is designed. The high data rate stream interface is used when transferring the point set from the external memory to the PL side. The block RAM of the circular partition is used while accessing points in one space or subspace. The query point is updated continuously, ensuring that points around the current query point are always in the data buffer.
b) The same joint point from different high-definition cameras measures and calculates 3D joint point information of a signal worker arm in space through a three-dimensional coordinate system; in order to improve the detection precision of the 3D joint point position, a mean square error loss function MSE based on distance measurement is adopted in the training process, and the formula is as follows:
the method comprises the steps of carrying out a first treatment on the surface of the Wherein (1)>
For the real coordinate values of all joint points in space, +.>
And detecting coordinate values for the space of all the joint points, wherein n is the acquisition times, and stopping acquisition when the MSE is less than or equal to 0.01.
The specific content of the step 4 is as follows: and after the high-definition camera finishes calibration and algorithm training, information fusion is carried out on target information detected by the ground equipment at multiple visual angles as a whole. When the high-definition camera on the ground detects and recognizes the tower crane signal worker and the signal worker gesture, the tower crane signal worker is projected into a tower crane vehicle-mounted screen according to a certain scaling coefficient, a display of the tower crane vehicle-mounted device displays and broadcasts command prompt voice of the signal worker gesture in real time, and a tower crane driver can operate according to the instruction; meanwhile, a tower crane driver can rotate a tower crane signaler in a touch mode and zoom in and zoom out the details of gesture information.
It should be understood that the above description is not intended to limit the invention to the particular embodiments disclosed, but to limit the invention to the particular embodiments disclosed, and that the invention is not limited to the particular embodiments disclosed, but is intended to cover modifications, adaptations, additions and alternatives falling within the spirit and scope of the invention.
Claims (9)
1. A signalling device for tower crane signal worker, its characterized in that: the device comprises three cameras, a gesture recognition signal sending unit and a tower crane vehicle end signal receiving unit;
the camera is configured to acquire tower crane signal work targets, positions and gesture characteristic data;
the gesture recognition signal sending unit is configured to recognize gesture signals of a signaler in real time and transmit the signals to the tower crane vehicle end signal receiving unit;
the tower crane vehicle end signal receiving unit is configured to receive gesture signals so as to facilitate corresponding operation of a tower crane driver.
2. A signaling device for a tower crane signaling appliance as claimed in claim 1, wherein: the tower crane vehicle end signal receiving unit comprises a wireless signal receiving module, a wireless signal identifying module, a display screen and a voice broadcasting module;
the wireless signal receiving module is configured to be used for receiving the wireless signal transmitted by the gesture recognition signal transmitting unit by the tower crane;
the wireless signal recognition module is configured to recognize the wireless signal transmitted by the gesture recognition signal transmitting unit by the tower crane;
the display screen is configured to display gestures of a tower crane signal worker in real time;
the voice broadcasting module is configured to be used for playing gestures of the tower crane signal worker in real time.
3. A signaling device for a tower crane signaling appliance as claimed in claim 1, wherein: the three cameras are respectively arranged at the left side, the upper side and the right side of the gesture recognition signal sending unit; the cameras shoot and record synchronous video information of the tower crane signal worker from three different directions, the detected video content comprises all information in an active area of the tower crane signal worker, the cameras are calibrated independently by using a world coordinate system, and the cameras are calibrated in a joint mode by using a epipolar geometry constraint method, so that the signal worker gestures can be positioned and identified accurately.
4. A signaling device for a tower crane signaling appliance as claimed in claim 1, wherein: after the gesture recognition signal sending unit collects target, position and gesture characteristic data of the calibration tower crane signal worker, a data set is formed; the recognition and detection of each camera are trained by adopting a new frame optimization KNN algorithm of space segmentation, a mean square error loss function MSE based on distance measurement is adopted in the training process to improve recognition accuracy, and recognized tower crane signal work and signal work gestures are transmitted to a signal receiving device through wireless transmission.
5. A signaling device for a tower crane signaling appliance as claimed in claim 2, wherein: the tower crane vehicle end signal receiving unit receives the gesture signals and then projects tower crane signal workers in a display screen of the tower crane according to scale scaling factors, the display screen of the tower crane and the voice broadcasting module respectively display and broadcast command prompt voices of the gesture of the signal workers in real time, and meanwhile, a tower crane driver can zoom in and zoom out details of rotation and gesture information of the tower crane signal workers in a touch mode.
6. A signal transmission method for a tower crane signal worker is characterized by comprising the following steps of: a signaling device for a tower crane signaling appliance according to claim 1, comprising the steps of:
step 1: the ground camera, the gesture recognition signal transmitting unit and the tower crane vehicle end signal receiving unit are arranged and calibrated;
step 2: acquiring, calibrating and manufacturing a data set of tower crane signal work targets, positions and gesture characteristic data;
step 3: training a tower crane signal target, position determination and gesture recognition algorithm;
step 4: the tower crane vehicle end signal receiving unit recognizes the signal tool gesture and transmits the signal tool gesture to the tower crane vehicle end device through wireless communication.
7. The signal transmission method for a tower crane signaling tool according to claim 6, wherein: in step 1, a signal transmission device is placed at a position at a safe distance from the tower crane, three cameras are respectively installed above, left and right sides of the signal transmission device, and an area 2m×2m in front of the signal transmission device is set as an active area of the tower crane signal worker.
8. The signal transmission method for a tower crane signaling tool according to claim 6, wherein: in the step 2, shooting synchronous videos of a tower crane signal under different visual angles by using a ground camera, and taking the videos as a training data set; the trained dataset labeling benchmarks are as follows:
(a) Defining a tower crane signal engineering coordinate system;
the initial standing position of the tower crane signaling worker is set as an origin, the visual direction is the positive direction of the y axis, the extending direction of the right arm vertical to the x axis is the positive direction of the x axis, and the upward direction vertical to the planes of the x axis and the y axis is the positive direction of the z axis;
(b) Labeling the signal class of the tower crane;
the classification at the time of labeling is: the tower crane signal worker safety helmet;
marking a tower crane signal worker target of the tower crane signal worker safety helmet as a signal worker class according to the characteristic of whether the tower crane signal worker safety helmet is worn or not, and simultaneously marking the mirror image of the tower crane signal worker direction, wherein the command direction is determined by the signal worker direction and the gesture;
(c) Labeling a tower crane signal worker arm joint point and a finger joint point;
the command gesture of the tower crane signal worker is completed by the left and right arms, namely, only 10 finger joints, 1 wrist joint, 1 elbow joint and 1 shoulder joint of each arm of the tower crane signal worker are marked, each joint detected by each camera contains three dimensional coordinate information, and the finger joint (f) 1 …f 10 ) Coordinate (f) x1 , f y1 , f z1 )…(f x20 , f y20 , f z20 ) Wrist jointNode w coordinate (w) x1 , w y1 , w z1 ) (w x2 , w y2 , w z2 ) Elbow joint point e coordinate (e x1 , e y1 , e z1 ) (e x2 , e y2 , e z2 ) And shoulder joint point s coordinate (s x1 , s y1 , s z1 ) (s x2 , s y2 , s z2 ) All the joint points form a point aggregation list, and all the joint points are connected into a line to form a skeleton annotation list; firstly, solving the distance between each joint point of the signal worker and the sensor, and then obtaining the real position of each joint point of the signal worker in the three-dimensional space by using a similarity relationship through a world coordinate system established by the signal worker;
(d) Calibrating a gesture of a tower crane signaling worker;
according to different gestures of an actual tower crane signal worker, classification and calibration are carried out for different gestures, and the tower crane signal worker is used for training of different gesture recognition.
9. The signal transmission method for a tower crane signaling tool according to claim 8, wherein: in the step 3, a new frame of space segmentation is adopted to optimize KNN algorithm to train 3D joint point detection of each camera, wherein the KNN algorithm is to search K points closest to the target point;
firstly, defining a preset searching range Rin, wherein the range ensures that all nodes can be searched; dividing the point set to obtain a group of three-dimensional space with the side length equal to Rin; if a three-dimensional space contains too many points, a threshold T is set max Dividing the three-dimensional space into a plurality of subspaces;
secondly, constructing a novel joint point data structure; finding a bounding box by traversing all points, and calculating a minimum range lowerbound and a maximum range highbound on each axis as AABB of the point cloud;
and dividing the AABB into a set of spaces VS;
Assigning an index vp to each space and defining a hash function hash (p);
through the above procedure, a spatial index including a specific point can be calculated; when the number of points in a space exceeds a threshold T max When it is divided into a group of subspaces with K points at most;
the same joint point from different cameras measures and calculates 3D joint point information of the tower crane signal worker arm in space through a three-dimensional coordinate system; the training process adopts a mean square error loss function MSE based on distance measurement, and the formula is as follows:
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310300309.1A CN116012892A (en) | 2023-03-27 | 2023-03-27 | Signal device and signal transmission method for tower crane signal engineering |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310300309.1A CN116012892A (en) | 2023-03-27 | 2023-03-27 | Signal device and signal transmission method for tower crane signal engineering |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116012892A true CN116012892A (en) | 2023-04-25 |
Family
ID=86019639
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310300309.1A Pending CN116012892A (en) | 2023-03-27 | 2023-03-27 | Signal device and signal transmission method for tower crane signal engineering |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116012892A (en) |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN207312961U (en) * | 2017-10-25 | 2018-05-04 | 宜昌华鼎建筑工程有限公司 | A kind of crane machine monitoring system with gesture identification wireless camera |
| CN114283497A (en) * | 2021-12-27 | 2022-04-05 | 吉林大学 | Traffic police gesture recognition method under congested intersection scene |
| US20220148281A1 (en) * | 2020-07-22 | 2022-05-12 | Shanghaitech University | Efficient k-nearest neighbor search algorithm for three-dimensional (3d) lidar point cloud in unmanned driving |
-
2023
- 2023-03-27 CN CN202310300309.1A patent/CN116012892A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN207312961U (en) * | 2017-10-25 | 2018-05-04 | 宜昌华鼎建筑工程有限公司 | A kind of crane machine monitoring system with gesture identification wireless camera |
| US20220148281A1 (en) * | 2020-07-22 | 2022-05-12 | Shanghaitech University | Efficient k-nearest neighbor search algorithm for three-dimensional (3d) lidar point cloud in unmanned driving |
| CN114283497A (en) * | 2021-12-27 | 2022-04-05 | 吉林大学 | Traffic police gesture recognition method under congested intersection scene |
Non-Patent Citations (1)
| Title |
|---|
| HAO SUN等: "Efficient FPGA Implementation of K-Nearest-Neighbor Search Algorithm for 3D LIDAR Localization and Mapping in Smart Vehicles", 《IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS—II: EXPRESS BRIEFS》, vol. 67, no. 9, pages 1644 - 1648 * |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110780305B (en) | Track cone detection and target point tracking method based on multi-line laser radar | |
| CA2950791C (en) | Binocular visual navigation system and method based on power robot | |
| CN108012325B (en) | Navigation positioning method based on UWB and binocular vision | |
| CN112085003B (en) | Automatic recognition method and device for abnormal behaviors in public places and camera equipment | |
| JP3895238B2 (en) | Obstacle detection apparatus and method | |
| CN103353758B (en) | A kind of Indoor Robot navigation method | |
| CN103400392B (en) | Binocular vision navigation system and method based on Intelligent Mobile Robot | |
| US20180003498A1 (en) | Visual positioning system and method based on high reflective infrared identification | |
| CN109737981B (en) | Unmanned vehicle target searching device and method based on multiple sensors | |
| CN111735445B (en) | Monocular vision and IMU (inertial measurement Unit) integrated coal mine tunnel inspection robot system and navigation method | |
| US20230236280A1 (en) | Method and system for positioning indoor autonomous mobile robot | |
| CN109828267A (en) | The Intelligent Mobile Robot detection of obstacles and distance measuring method of Case-based Reasoning segmentation and depth camera | |
| CN110189375B (en) | Image target identification method based on monocular vision measurement | |
| CN106370160A (en) | Robot indoor positioning system and method | |
| CN104236540A (en) | Indoor passive navigation and positioning system and indoor passive navigation and positioning method | |
| CN113914407B (en) | Excavator excavation tunnel accurate control system based on BIM and machine vision | |
| JP2015212629A (en) | Detection device and manipulator operation control including detection device | |
| Momeni-k et al. | Height estimation from a single camera view | |
| CN112665575A (en) | SLAM loop detection method based on mobile robot | |
| CN105373130A (en) | Special device accident on-site information detection system based on stereo modeling | |
| CN112068567A (en) | Positioning method and positioning system based on ultra-wideband and visual image | |
| CN114299039B (en) | Robot and collision detection device and method thereof | |
| Cheng et al. | 3D radar and camera co-calibration: A flexible and accurate method for target-based extrinsic calibration | |
| CN107818587A (en) | A kind of machine vision high-precision locating method based on ROS | |
| CN115144879B (en) | Multi-machine multi-target dynamic positioning system and method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20230425 |