CN112489125A - Automatic detection method and device for storage yard pedestrians - Google Patents

Automatic detection method and device for storage yard pedestrians Download PDF

Info

Publication number
CN112489125A
CN112489125A CN202011420496.XA CN202011420496A CN112489125A CN 112489125 A CN112489125 A CN 112489125A CN 202011420496 A CN202011420496 A CN 202011420496A CN 112489125 A CN112489125 A CN 112489125A
Authority
CN
China
Prior art keywords
pedestrian
yard
camera
storage yard
pedestrians
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202011420496.XA
Other languages
Chinese (zh)
Inventor
王永锋
冯志
陈环
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Shanghai Yumo Information Technology Co ltd
Original Assignee
Shanghai Yumo Information Technology Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Shanghai Yumo Information Technology Co ltd filed Critical Shanghai Yumo Information Technology Co ltd
Priority to CN202011420496.XA priority Critical patent/CN112489125A/en
Publication of CN112489125A publication Critical patent/CN112489125A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B66—HOISTING; LIFTING; HAULING
    • B66C—CRANES; LOAD-ENGAGING ELEMENTS OR DEVICES FOR CRANES, CAPSTANS, WINCHES, OR TACKLES
    • B66C13/00—Other constructional features or details
    • B66C13/16—Applications of indicating, registering, or weighing devices
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B66—HOISTING; LIFTING; HAULING
    • B66C—CRANES; LOAD-ENGAGING ELEMENTS OR DEVICES FOR CRANES, CAPSTANS, WINCHES, OR TACKLES
    • B66C15/00—Safety gear
    • B66C15/06—Arrangements or use of warning devices
    • B66C15/065—Arrangements or use of warning devices electrical
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/20—Analysis of motion
    • G06T7/246—Analysis of motion using feature-based methods, e.g. the tracking of corners or segments
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/80—Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10028—Range image; Depth image; 3D point clouds
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30196—Human being; Person
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30232—Surveillance
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30241—Trajectory

Abstract

The invention discloses a method and a device for automatically detecting pedestrians in a storage yard, wherein the method comprises the following steps: setting a pedestrian danger area of the storage yard according to the scene and the positions of the cart and the trolley in the storage yard; acquiring images and point cloud information of pedestrian danger areas in a storage yard through a camera and a laser radar; under a container loading and unloading scene of a storage yard, acquiring the height of a camera from the ground in real time through a laser radar and an encoder, acquiring camera scale information according to the height and an image in the camera, and acquiring pedestrian coordinates based on a deep learning image pedestrian detection method; in a yard cart advancing scene, after an interested area is selected through a camera, acquiring pedestrian coordinates by combining point cloud information acquired by a laser radar; and if the coordinates of the pedestrian fall into or are about to enter the dangerous area of the pedestrian in the yard, decelerating or stopping the crane. The method and the device for automatically detecting the pedestrian in the storage yard provided by the invention respectively judge whether the coordinate of the pedestrian falls into the dangerous area of the pedestrian in the storage yard under different scenes, thereby ensuring the safety of the pedestrian.

Description

Automatic detection method and device for storage yard pedestrians
Technical Field
The invention relates to the field of crane loading and unloading, in particular to a method and a device for automatically detecting pedestrians in a storage yard.
Background
In recent years, the intelligent operation level of port storage yards is increasingly enhanced, more and more high-tech technologies are applied to docks, automation and intellectualization gradually become the trend of port development in the future, and intelligent safety monitoring of storage yards ensures the important basis of port automation operation. At present, port yard safety monitoring is mainly to monitor yard environment through camera images, however, the camera images are only used as an auxiliary tool for judging the yard environment by a driver, the driver is still required to manually judge whether pedestrians exist in a yard area, and the safety monitoring of the pedestrians in the yard can not be carried out with human eyes to participate in monitoring judgment. The port storage yards are numerous, a large number of cameras are required to be equipped for monitoring for 24 hours, and therefore the great image data generated by the monitoring yards need to be judged manually by drivers, and huge labor cost is brought to port automation operation. In addition, the phenomenon of missing detection or false detection of the safety condition of the storage yard environment often occurs, so that the safety and the high efficiency of the actual operation of the storage yard are reduced, and unsafe hidden troubles are brought to the operation of the storage yard.
Therefore, it is necessary to provide an automatic detection method for pedestrians in a storage yard, which can ensure the safety of the pedestrians in the storage yard.
Disclosure of Invention
The invention aims to solve the technical problem of providing a storage yard pedestrian automatic detection method and a storage yard pedestrian automatic detection device, which respectively judge whether the coordinates of storage yard pedestrians fall into a storage yard pedestrian dangerous area under a storage yard container loading and unloading scene and a storage yard cart advancing scene, thereby ensuring the safety of the storage yard pedestrians.
The technical scheme adopted by the invention for solving the technical problems is to provide an automatic detection method for the pedestrians in the storage yard, which comprises the following steps:
setting a pedestrian danger area of the storage yard according to the scene and the positions of the cart and the trolley in the storage yard;
acquiring images and point cloud information of the pedestrian danger areas of the storage yard through a camera and a laser radar;
under a loading and unloading scene of a container in a storage yard, the lifting direction of the container is a dangerous area of the pedestrian in the storage yard, the height of the camera from the ground is obtained in real time through a laser radar and an encoder which are arranged on the trolley, the dimension information of the camera is obtained according to the height and the image in the camera, and the coordinate of the pedestrian is obtained based on a deep learning image pedestrian detection method;
in a yard cart traveling scene, the traveling direction of a cart is a dangerous area of the pedestrian in the yard, after an area of interest is selected by the camera, the coordinates of the pedestrian are acquired by combining point cloud information acquired by the laser radar;
and judging whether the pedestrian enters a dangerous area or not according to the coordinates of the pedestrian and the dangerous area of the yard pedestrian, and if the coordinates of the pedestrian falls into the dangerous area of the yard pedestrian or the pedestrian is predicted to enter the dangerous area according to the movement track of the pedestrian, decelerating or stopping the crane.
Preferably, the method further comprises the following steps:
the camera and the laser radar are arranged on the lifting appliance and the cart, and the camera arranged on the lifting appliance moves up and down along with the lifting appliance to provide more images;
before yard operations, a hazardous area is described under a unified coordinate system based on the camera and the external reference unified coordinate system of the lidar.
Preferably, the pedestrian detection method based on the deep learning image includes the steps of:
the method comprises the steps of manufacturing a stock yard pedestrian data set, collecting and labeling stock yard pedestrian images aiming at port scenes, and generating a stock yard pedestrian data set;
carrying out classification network training, namely training on a public pedestrian detection data set to generate a training pre-model, and continuing training on the yard pedestrian data set to accelerate the speed of network feature extraction;
selecting a dimension clustering frame, namely selecting the dimension clustering frame by using a clustering algorithm so as to improve the detection capability of deep learning;
and multi-scale training, wherein pictures with different scales are set, and one scale is randomly selected for training every set number of times during training.
Preferably, in the yard container loading and unloading scene, a plane rectangular coordinate system is established with a trolley direction as an x axis and a trolley direction as a y axis, the camera automatically detects pixel position coordinates P (x, y) of yard pedestrians in an image of the yard pedestrian danger area, and relative coordinates of the yard pedestrians in the camera coordinate system are calculated and converted into absolute coordinates under system coordinates through the following formula:
X＝h*x/f
Y＝h*y/f
wherein X and Y are the actual distances of the storage yard pedestrians in the X direction and the Y direction of the camera center coordinate system respectively, f is the camera internal reference focal length solved by the camera calibration algorithm, and h is the real-time height of the camera from the ground.
Preferably, in the cart traveling scene, a space rectangular coordinate system is established with the cart direction as an x-axis, the direction perpendicular to the ground as a y-axis and the cart direction as a z-axis, and the coordinate position of the region of interest in the laser radar coordinate system is determined by combining the calibration parameters of the camera and the laser radar, so that the coordinate of the pedestrian is determined.
Preferably, the predicting that the pedestrian is about to enter the dangerous area according to the motion trail of the pedestrian comprises recording the positions of the pedestrian in all frames in the sliding window when the pedestrian is detected to be in the field of view, and predicting the coordinates of the pedestrian at the next moment according to the position change.
Preferably, when a certain number of pedestrians exist in the sliding window and the coordinates of the pedestrians are within a first threshold value from the crane to the predicted next moment, the crane decelerates;
and when the pedestrians with specific frame numbers exist in the sliding window and the coordinates of the pedestrians are within a second threshold value when the crane is far away from the predicted next moment, stopping the crane.
Preferably, when a certain number of pedestrians exist in the sliding window and the coordinates of the pedestrian at the next predicted time from the crane are within a third threshold, a voice broadcast warning and/or a light flashing warning are/is performed.
Preferably, if no pedestrian exists in the pedestrian danger area of the storage yard, the current storage yard operation is recovered, and the voice broadcast warning and/or the light flicker warning are/is released.
The invention also provides an automatic detection device for the pedestrians in the storage yard for solving the technical problems, and the automatic detection method for the pedestrians in the storage yard is used.
Compared with the prior art, the invention has the following beneficial effects: according to the automatic detection method and device for the storage yard pedestrians, the image and the point cloud information of the storage yard pedestrian dangerous area are obtained through the camera and the laser radar which are installed on the crane and the lifting appliance, whether the coordinates of the storage yard pedestrians fall into the storage yard pedestrian dangerous area or not is judged respectively under the storage yard container loading and unloading scene and the storage yard cart driving scene, or whether the pedestrians are about to enter the dangerous area is predicted according to the movement track of the pedestrians, so that the safety of the storage yard pedestrians is guaranteed;
further, under a loading and unloading scene of a container in a storage yard, the lifting direction of the container is a dangerous area of the pedestrian in the storage yard, the height of the camera from the ground is obtained in real time through a laser radar and an encoder which are arranged on the trolley, the scale information of the camera is obtained according to the height and the image in the camera, and the coordinate of the pedestrian in the storage yard is accurately obtained based on a deep learning image pedestrian detection method;
further, under a yard crane advancing scene, the crane advancing direction is a yard pedestrian dangerous area, after an area of interest is selected through the camera, the coordinates of yard pedestrians are accurately acquired by combining point cloud information acquired by the laser radar;
further, it includes when detecting that the pedestrian appears in the field of vision to foresee the pedestrian and be about to get into the danger area according to pedestrian's motion trajectory, records pedestrian's position in all frames in the sliding window, predicts pedestrian's coordinate at next moment according to the position change to accurate prediction pedestrian's coordinate change carries out accurate judgement, thereby operates the hoist and slows down or stop, guarantees pedestrian's safety.
Drawings
FIG. 1 is a flow chart of an automatic detection method for pedestrians in a storage yard according to an embodiment of the present invention;
FIG. 2 is a schematic illustration of pedestrian detection in a yard according to the automatic detection method for pedestrian in a yard of the present invention;
FIG. 3 is a schematic diagram of the detection of the pedestrian in the yard in the loading and unloading scene of the container in the automatic detection method of the pedestrian in the yard according to the embodiment of the invention;
fig. 4 is a schematic diagram illustrating the detection of the pedestrian in the yard in the scene of the travel of the cart in the yard by the automatic detection method for the pedestrian in the yard in the embodiment of the invention;
fig. 5 is a schematic front view illustrating the detection of the yard pedestrians by the automatic detection method of the yard pedestrians in the scene of the travel of the yard cart in the embodiment of the present invention.
Detailed Description
The invention is further described below with reference to the figures and examples.
In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present invention. However, it will be apparent to one of ordinary skill in the art that the present invention may be practiced without these specific details. Accordingly, the particular details set forth are merely exemplary, and the particular details may be varied from the spirit and scope of the present invention and still be considered within the spirit and scope of the present invention.
Referring now to fig. 1, fig. 1 is a flow chart of a method for automatically detecting pedestrians in a yard according to an embodiment of the present invention. An automatic detection method for pedestrians in a storage yard comprises the following steps:
s101: setting a pedestrian danger area of the storage yard according to the scene and the positions of the cart and the trolley in the storage yard;
s102: acquiring images and point cloud information of the pedestrian danger areas of the storage yard through a camera and a laser radar;
s103: under a loading and unloading scene of a container in a storage yard, the lifting direction of the container is a dangerous area of the pedestrian in the storage yard, the height of the camera from the ground is obtained in real time through a laser radar and an encoder which are arranged on the trolley, the dimension information of the camera is obtained according to the height and the image in the camera, and the coordinate of the pedestrian is obtained based on a deep learning image pedestrian detection method;
s104: in a yard cart traveling scene, the traveling direction of a cart is a dangerous area of the pedestrian in the yard, after an area of interest is selected by the camera, the coordinates of the pedestrian are acquired by combining point cloud information acquired by the laser radar;
s105: and judging whether the pedestrian enters a dangerous area or not according to the coordinates of the pedestrian and the dangerous area of the yard pedestrian, and if the coordinates of the pedestrian falls into the dangerous area of the yard pedestrian or the pedestrian is predicted to enter the dangerous area according to the movement track of the pedestrian, decelerating or stopping the crane.
Referring now to fig. 2, fig. 2 is a schematic diagram of a pedestrian detection method in a yard according to an embodiment of the present invention. In specific implementation, the pedestrian danger area of the yard is a pedestrian movement danger area defined according to yard operation experience, and the area outside the danger area is a safety area, so that whether pedestrians exist in the danger area defined by the yard is judged in real time. When a cart and a lifting appliance are started to operate, after a camera and a laser radar are started to collect data in real time, whether pedestrians exist in a storage yard limited dangerous area or not is judged by adopting a deep learning image pedestrian detection method and a laser ranging algorithm, and finally whether operation is continued or not is judged according to a pedestrian detection result, so that the safety and the efficiency of storage yard operation are guaranteed.
The camera and the laser radar are installed on a lifting appliance and a cart, the camera installed on the lifting appliance moves up and down along with the lifting appliance to provide more images, and before storage yard operation, a coordinate system is unified based on the camera and the laser radar, and a dangerous area is described under the unified coordinate system.
In specific implementation, the pedestrian target in the storage yard area is detected by adopting a target detection algorithm based on deep learning, and whether the pedestrian is in a dangerous area or not is judged by combining real-time laser radar data. The pedestrian detection method based on the deep learning image comprises the following steps: the method comprises the steps of manufacturing a stock yard pedestrian data set, collecting and labeling stock yard pedestrian images aiming at port scenes, and generating a stock yard pedestrian data set; carrying out classification network training, namely training on a public pedestrian detection data set to generate a training pre-model, and continuing training on the yard pedestrian data set to accelerate the speed of network feature extraction; selecting a dimension clustering frame, namely selecting the dimension clustering frame by using a clustering algorithm so as to improve the detection capability of deep learning; and multi-scale training, wherein pictures with different scales are set, and one scale is randomly selected for training every set number of times during training. Compared with the traditional image detection algorithm, the algorithm has the advantages of high accuracy, strong robustness and good real-time performance in the aspect of target detection.
The detection process is specifically divided into two cases: pedestrian detection in a yard container loading and unloading scene and pedestrian detection in a yard cart driving scene.
Referring now to fig. 3, fig. 3 is a schematic diagram of a yard pedestrian detection method in a yard container loading and unloading scenario according to an embodiment of the present invention. Under a loading and unloading scene of a container in a storage yard, establishing a rectangular plane coordinate system by taking the direction of a trolley as an x axis and the direction of a cart as a y axis, automatically detecting pixel position coordinates P (x, y) of pedestrian in the storage yard in an image of a pedestrian danger area in the storage yard by a camera, calculating relative coordinates of the pedestrian in the storage yard in a camera coordinate system by the following formula, and converting the relative coordinates into absolute coordinates under system coordinates:
X＝h*x/f
Y＝h*y/f
the method comprises the steps that X and Y are actual distances of a storage yard pedestrian in the X direction and the Y direction of a camera center coordinate system respectively, f is a camera internal reference focal length solved by a camera calibration algorithm, h is a real-time height of the camera from the ground detected in the descending process of a lifting appliance, and the real-time height is obtained according to a laser radar and an encoder on a trolley and is used for judging whether the pedestrian is in a dangerous area or not. For example, in fig. 3, the pixel position coordinates of 3 pedestrians are (x) respectively1，y1，)、(x2，y2) And (x)3，y3) The relative coordinates of the 3 pedestrians in the camera coordinate system can be calculated and converted through the formulaIs absolute coordinate (X) under system coordinate1，Y1，)、(X2，Y2) And (X)3，Y3)。
Referring now to fig. 4 and 5, fig. 4 is a schematic diagram of yard pedestrian detection in a yard cart traveling scene by the yard pedestrian automatic detection method in the embodiment of the present invention, and fig. 5 is a schematic diagram of front view of yard pedestrian detection in a yard cart traveling scene by the yard pedestrian automatic detection method in the embodiment of the present invention. Under the scene of the marching of the large vehicle in the storage yard, a space rectangular coordinate system is established by taking the direction of the small vehicle as an x axis, the direction vertical to the ground as a y axis and the direction of the large vehicle as a z axis. Specifically, X in FIG. 41Y1Z1The pedestrian position, X, in the image can be obtained by an image recognition method for a camera coordinate system2Y2Z2And the point cloud of the image target position can be found according to the calibration result of the camera and the laser radar, and the distance between the target pedestrian and the crane in the directions of the cart and the trolley can be obtained from the point cloud. Namely, the coordinate position of the region of interest in a laser radar coordinate system is determined by combining the calibration parameters of the camera and the laser radar, the relative positions of the storage yard pedestrians and the cart are determined, and whether the pedestrians are in a dangerous area or not is judged.
In the concrete implementation, in the process of detecting the pedestrian in the yard, the step of predicting that the pedestrian is about to enter the dangerous area according to the motion track of the pedestrian comprises the steps of recording the positions of the pedestrian in all frames in the sliding window when the pedestrian is detected to appear in the field of view, and predicting the coordinate of the pedestrian at the next moment according to the position change.
And when the pedestrians with specific frame numbers exist in the sliding window and the coordinates of the pedestrians, which are close to the predicted next moment, of the crane are within a first threshold value, the crane decelerates. The first threshold is 20m, that is, when a certain number of pedestrians exist in the sliding window and the coordinates of the crane from the predicted pedestrian at the next moment are within 20m, the crane decelerates.
And when the pedestrians with specific frame numbers exist in the sliding window and the coordinates of the pedestrians are within a second threshold value when the crane is far away from the predicted next moment, stopping the crane. The second threshold is 10m, that is, when there are pedestrians of a specific number of frames in the sliding window and the coordinates of the crane from the predicted pedestrian at the next time are within 10m, the crane is stopped.
And when the pedestrians with specific frame numbers exist in the sliding window and the coordinates of the pedestrians of the crane at the next predicted moment are within a third threshold value, carrying out voice broadcast warning and/or light flicker warning. The third threshold is 15m, that is, when pedestrians with a specific frame number exist in the sliding window and the coordinates of the crane from the predicted pedestrian at the next moment are within 15m, voice broadcasting warning and/or light flashing warning are/is carried out. And if no pedestrian exists in the pedestrian danger area of the storage yard, recovering the current storage yard operation and relieving the voice broadcast warning and/or the light flicker warning.
The first threshold, the second threshold, and the third threshold may be set according to the needs of a scene, and the above values are only used as examples, and are not limited by the values in specific implementation.
The embodiment of the invention also provides an automatic detection device for the pedestrians in the storage yard, which uses the automatic detection method for the pedestrians in the storage yard.
In summary, according to the method and the device for automatically detecting the pedestrian in the storage yard provided by the embodiment, the image and the point cloud information of the dangerous area of the pedestrian in the storage yard are acquired through the camera and the laser radar which are installed on the crane and the lifting appliance, and whether the coordinates of the pedestrian in the storage yard fall into the dangerous area of the pedestrian in the storage yard or not or whether the pedestrian is about to enter the dangerous area is predicted according to the movement track of the pedestrian is respectively judged in the loading and unloading scene of the container in the storage yard and the driving scene of the cart in the storage yard, so that the safety;
further, under a loading and unloading scene of a container in a storage yard, the lifting direction of the container is a dangerous area of the pedestrian in the storage yard, the height of the camera from the ground is obtained in real time through a laser radar and an encoder which are arranged on the trolley, the scale information of the camera is obtained according to the height and the image in the camera, and the coordinate of the pedestrian in the storage yard is accurately obtained based on a deep learning image pedestrian detection method;
further, under a yard crane advancing scene, the crane advancing direction is a yard pedestrian dangerous area, after an area of interest is selected through the camera, the coordinates of yard pedestrians are accurately acquired by combining point cloud information acquired by the laser radar;
further, it includes when detecting that the pedestrian appears in the field of vision to foresee the pedestrian and be about to get into the danger area according to pedestrian's motion trajectory, records pedestrian's position in all frames in the sliding window, predicts pedestrian's coordinate at next moment according to the position change to accurate prediction pedestrian's coordinate change carries out accurate judgement, thereby operates the hoist and slows down or stop, guarantees pedestrian's safety.
Although the present invention has been described with respect to the preferred embodiments, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention as defined by the appended claims.

Claims (10)

1. An automatic detection method for pedestrians in a storage yard is characterized by comprising the following steps:
setting a pedestrian danger area of the storage yard according to the scene and the positions of the cart and the trolley in the storage yard;
acquiring images and point cloud information of the pedestrian danger areas of the storage yard through a camera and a laser radar;
under a loading and unloading scene of a container in a storage yard, the lifting direction of the container is a dangerous area of the pedestrian in the storage yard, the height of the camera from the ground is obtained in real time through a laser radar and an encoder which are arranged on the trolley, the dimension information of the camera is obtained according to the height and the image in the camera, and the coordinate of the pedestrian is obtained based on a deep learning image pedestrian detection method;
in a yard cart traveling scene, the traveling direction of a cart is a dangerous area of the pedestrian in the yard, after an area of interest is selected by the camera, the coordinates of the pedestrian are acquired by combining point cloud information acquired by the laser radar;
and judging whether the pedestrian enters a dangerous area or not according to the coordinates of the pedestrian and the dangerous area of the yard pedestrian, and if the coordinates of the pedestrian falls into the dangerous area of the yard pedestrian or the pedestrian is predicted to enter the dangerous area according to the movement track of the pedestrian, decelerating or stopping the crane.
2. The method for automatically detecting the pedestrian in the yard according to claim 1, further comprising the steps of:
the camera and the laser radar are arranged on the lifting appliance and the cart, and the camera arranged on the lifting appliance moves up and down along with the lifting appliance to provide more images;
before yard operations, a hazardous area is described under a unified coordinate system based on the camera and the external reference unified coordinate system of the lidar.
3. The method for automatically detecting pedestrians in the storage yard according to claim 1, wherein the method for detecting pedestrians based on the deep learning image comprises the following steps:
the method comprises the steps of manufacturing a stock yard pedestrian data set, collecting and labeling stock yard pedestrian images aiming at port scenes, and generating a stock yard pedestrian data set;
carrying out classification network training, namely training on a public pedestrian detection data set to generate a training pre-model, and continuing training on the yard pedestrian data set to accelerate the speed of network feature extraction;
selecting a dimension clustering frame, namely selecting the dimension clustering frame by using a clustering algorithm so as to improve the detection capability of deep learning;
and multi-scale training, wherein pictures with different scales are set, and one scale is randomly selected for training every set number of times during training.
4. The method for automatically detecting pedestrians in a yard according to claim 1,
under the loading and unloading scene of the container in the storage yard, a plane rectangular coordinate system is established by taking the direction of a trolley as an x axis and the direction of a cart as a y axis, the camera automatically detects the pixel position coordinates P (x, y) of the pedestrian in the storage yard in the image of the pedestrian danger area in the storage yard, and the relative coordinates of the pedestrian in the storage yard in the camera coordinate system are calculated by the following formula and converted into the absolute coordinates under the system coordinates:
X＝h*x/f
Y＝h*y/f
wherein X and Y are the actual distances of the storage yard pedestrians in the X direction and the Y direction of the camera center coordinate system respectively, f is the camera internal reference focal length solved by the camera calibration algorithm, and h is the real-time height of the camera from the ground.
5. The method according to claim 2, wherein the pedestrian detection device is a pedestrian detection device,
and under the cart advancing scene, establishing a space rectangular coordinate system by taking the direction of a trolley as an x axis, the direction vertical to the ground as a y axis and the direction of a cart as a z axis, and determining the coordinate position of the region of interest in a laser radar coordinate system by combining the camera and the calibration parameters of the laser radar so as to determine the coordinates of the pedestrian.
6. The method for automatically detecting pedestrians in a yard according to claim 1,
the step of predicting that the pedestrian is about to enter the dangerous area according to the motion trail of the pedestrian comprises the steps of recording positions of the pedestrian in all frames in the sliding window when the pedestrian is detected to appear in the visual field, and predicting the coordinate of the pedestrian at the next moment according to the position change.
7. The method according to claim 6, wherein the pedestrian detection device is a pedestrian detection device,
when the pedestrians with specific frame numbers exist in the sliding window and the coordinates of the pedestrians, at the next predicted moment, of the crane are within a first threshold value, the crane decelerates;
and when the pedestrians with specific frame numbers exist in the sliding window and the coordinates of the pedestrians are within a second threshold value when the crane is far away from the predicted next moment, stopping the crane.
8. The method for automatically detecting the pedestrian in the storage yard according to claim 6, wherein when a certain number of pedestrians exist in a sliding window and the coordinates of the pedestrian at the next predicted time from the crane are within a third threshold, a voice broadcast warning and/or a light flashing warning are/is performed.
9. The method for automatically detecting the pedestrian in the storage yard according to claim 8, wherein if no pedestrian exists in the pedestrian danger area of the storage yard, the current storage yard operation is resumed and the voice broadcast warning and/or the light flicker warning are/is released.
10. An automatic storage yard pedestrian detection device using the automatic storage yard pedestrian detection method according to any one of claims 1 to 9.
CN202011420496.XA 2020-12-07 2020-12-07 Automatic detection method and device for storage yard pedestrians Pending CN112489125A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202011420496.XA CN112489125A (en) 2020-12-07 2020-12-07 Automatic detection method and device for storage yard pedestrians

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202011420496.XA CN112489125A (en) 2020-12-07 2020-12-07 Automatic detection method and device for storage yard pedestrians

Publications (1)

Publication Number Publication Date
CN112489125A true CN112489125A (en) 2021-03-12

Family

ID=74940541

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202011420496.XA Pending CN112489125A (en) 2020-12-07 2020-12-07 Automatic detection method and device for storage yard pedestrians

Country Status (1)

Country Link
CN (1) CN112489125A (en)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114639220A (en) * 2022-03-16 2022-06-17 国能榆林能源有限责任公司 Coal mining area alarm method, system and storage medium
CN116428996A (en) * 2023-06-06 2023-07-14 北京斯年智驾科技有限公司 Detection method and detection device for lifting appliance height

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114639220A (en) * 2022-03-16 2022-06-17 国能榆林能源有限责任公司 Coal mining area alarm method, system and storage medium
CN116428996A (en) * 2023-06-06 2023-07-14 北京斯年智驾科技有限公司 Detection method and detection device for lifting appliance height
CN116428996B (en) * 2023-06-06 2023-09-01 北京斯年智驾科技有限公司 Detection method and detection device for lifting appliance height

Similar Documents

Publication Publication Date Title
WO2021004077A1 (en) Method and apparatus for detecting blind areas of vehicle
CN106647776B (en) Method and device for judging lane changing trend of vehicle and computer storage medium
CN110430401B (en) Vehicle blind area early warning method, early warning device, MEC platform and storage medium
CN103366569B (en) The method and system of real-time grasp shoot traffic violation vehicle
CN112700470B (en) Target detection and track extraction method based on traffic video stream
CN111369831A (en) Road driving danger early warning method, device and equipment
CN105620489A (en) Driving assistance system and real-time warning and prompting method for vehicle
JP2002083297A (en) Object recognition method and object recognition device
CN104378539B (en) The video structural extraction of semantics video camera and its method of scene adaptive
EP2813973B1 (en) Method and system for processing video image
CN112489125A (en) Automatic detection method and device for storage yard pedestrians
CN113822285A (en) Vehicle illegal parking identification method for complex application scene
CN107458308B (en) Driving assisting method and system
JP2015133078A (en) Object recognition device for control, mobile apparatus control system, and program for object recognition for control
CN111178119A (en) Intersection state detection method and device, electronic equipment and vehicle
KR101239718B1 (en) System and method for detecting object of vehicle surroundings
CN112489383A (en) Early warning system and method for preventing red light running accident based on machine vision
CN111967384A (en) Vehicle information processing method, device, equipment and computer readable storage medium
CN115147587A (en) Obstacle detection method and device and electronic equipment
JP2004086417A (en) Method and device for detecting pedestrian on zebra crossing
CN212570057U (en) Road driving danger early warning device and equipment
CN114084129A (en) Fusion-based vehicle automatic driving control method and system
CN113917453A (en) Multi-sensor fusion method based on radar and video
KR20220067271A (en) Image acquisition apparatus and image acquisition method
JP2009241648A (en) Driving support device

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination