CN116704766A - Highway AI front situation perception system - Google Patents
Highway AI front situation perception system Download PDFInfo
- Publication number
- CN116704766A CN116704766A CN202310711256.2A CN202310711256A CN116704766A CN 116704766 A CN116704766 A CN 116704766A CN 202310711256 A CN202310711256 A CN 202310711256A CN 116704766 A CN116704766 A CN 116704766A
- Authority
- CN
- China
- Prior art keywords
- data
- construction
- edge
- video
- front situation
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Withdrawn
Links
- 230000008447 perception Effects 0.000 title claims description 19
- 238000010276 construction Methods 0.000 claims abstract description 115
- 238000012544 monitoring process Methods 0.000 claims abstract description 46
- 238000004458 analytical method Methods 0.000 claims description 40
- 238000007726 management method Methods 0.000 claims description 40
- 238000012545 processing Methods 0.000 claims description 20
- 230000005540 biological transmission Effects 0.000 claims description 18
- 238000000034 method Methods 0.000 claims description 17
- 238000005516 engineering process Methods 0.000 claims description 15
- 230000036544 posture Effects 0.000 claims description 11
- 230000001133 acceleration Effects 0.000 claims description 9
- 238000004364 calculation method Methods 0.000 claims description 9
- 238000004422 calculation algorithm Methods 0.000 claims description 8
- 230000000007 visual effect Effects 0.000 claims description 8
- 206010000117 Abnormal behaviour Diseases 0.000 claims description 7
- 230000006399 behavior Effects 0.000 claims description 7
- 230000008569 process Effects 0.000 claims description 7
- 206010034719 Personality change Diseases 0.000 claims description 6
- 230000002776 aggregation Effects 0.000 claims description 6
- 238000004220 aggregation Methods 0.000 claims description 6
- 238000006073 displacement reaction Methods 0.000 claims description 6
- 238000013500 data storage Methods 0.000 claims description 4
- 230000002159 abnormal effect Effects 0.000 claims description 3
- 238000007405 data analysis Methods 0.000 claims description 3
- 239000000428 dust Substances 0.000 claims description 3
- 230000003993 interaction Effects 0.000 claims description 3
- 238000003860 storage Methods 0.000 claims description 3
- 238000012916 structural analysis Methods 0.000 claims description 3
- 238000010205 computational analysis Methods 0.000 claims 1
- 230000004048 modification Effects 0.000 abstract description 4
- 238000012986 modification Methods 0.000 abstract description 4
- 230000009286 beneficial effect Effects 0.000 abstract description 2
- 238000007689 inspection Methods 0.000 description 8
- 230000006870 function Effects 0.000 description 6
- 238000004891 communication Methods 0.000 description 5
- 238000009430 construction management Methods 0.000 description 5
- 238000009434 installation Methods 0.000 description 4
- 230000007246 mechanism Effects 0.000 description 4
- 238000012790 confirmation Methods 0.000 description 3
- 238000011161 development Methods 0.000 description 3
- 238000010586 diagram Methods 0.000 description 3
- 238000012423 maintenance Methods 0.000 description 3
- 101100494773 Caenorhabditis elegans ctl-2 gene Proteins 0.000 description 2
- 101100112369 Fasciola hepatica Cat-1 gene Proteins 0.000 description 2
- 101100005271 Neurospora crassa (strain ATCC 24698 / 74-OR23-1A / CBS 708.71 / DSM 1257 / FGSC 987) cat-1 gene Proteins 0.000 description 2
- 230000008859 change Effects 0.000 description 2
- 238000006243 chemical reaction Methods 0.000 description 2
- QVFWZNCVPCJQOP-UHFFFAOYSA-N chloralodol Chemical compound CC(O)(C)CC(C)OC(O)C(Cl)(Cl)Cl QVFWZNCVPCJQOP-UHFFFAOYSA-N 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 230000005012 migration Effects 0.000 description 2
- 238000013508 migration Methods 0.000 description 2
- 230000006855 networking Effects 0.000 description 2
- 230000007613 environmental effect Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000002452 interceptive effect Effects 0.000 description 1
- 230000007774 longterm Effects 0.000 description 1
- 230000014759 maintenance of location Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 230000002035 prolonged effect Effects 0.000 description 1
- 238000007619 statistical method Methods 0.000 description 1
- 238000011179 visual inspection Methods 0.000 description 1
Classifications
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D30/00—Reducing energy consumption in communication networks
- Y02D30/70—Reducing energy consumption in communication networks in wireless communication networks
Landscapes
- Alarm Systems (AREA)
Abstract
The invention discloses an AI front situation awareness system of a highway, which comprises an edge system and a background system, wherein the edge system is formed by a plurality of front situation awareness stations distributed and deployed, the front situation awareness stations comprise edge terminals, 5G routers, fixed type Internet of things sensors, power supplies, monitoring cameras and mobile type Internet of things sensors, and the background system comprises cloud servers and terminal equipment. The beneficial effects are that: the front situation sensing capability of the construction site is effectively provided, remote command is facilitated, and the working efficiency is improved; the workload and the working pressure of management staff are reduced; the fine management, responsibility definition and examination of each construction unit provide facts and data basis; the potential safety hazard of the construction site is reported in time, and the construction unit is informed of the modification in time, so that the accident is prevented; the workload of manual management is effectively reduced, and the management efficiency is effectively improved.
Description
Technical Field
The invention relates to the technical field of perception systems, in particular to an AI front situation perception system of an expressway.
Background
The urbanization is a necessary result of the development of human society and economy, and the improvement of the urbanization level taking the urban scale, population, resident life quality and the like as indexes is always limited by factors such as an industrial structure, a production mode, traffic transportation and the like, so the urbanization level is one of the marks of the economic development degree of a national society. The appearance of highways, which are mainly characterized by high speed, safety, high land utilization efficiency and high traffic capacity, is key to the realization of the planning goal of an on-surface transportation network layout which is optimized and an 'hour or several hours traffic circle' which arrives or arrives at any place in a round trip area within a limited time. Whereby the expressway is an inevitable product of economic development.
In order to improve construction efficiency and construction safety, construction management and construction safety management are required at the expressway construction stage, and in the construction management and construction safety management, a method capable of acquiring a situation in front of a construction site in the prior art mainly adopts a manual field management mode, and a video monitoring system and a plurality of independent environment sensors are built on the site to serve as technical means of auxiliary management. In the prior art, only manual inspection or post inspection can be performed on a construction site, and meanwhile, a large amount of manual observation is needed to finish the front situation sensing capability of the construction site, so that the workload of acquiring the front situation of the construction site is large and the working efficiency is low.
Disclosure of Invention
In order to solve the above problems, and in particular to solve the disadvantages of the prior art, the present invention provides a highway AI front situation awareness system capable of solving the above problems.
In order to achieve the above purpose, the invention adopts the following technical means:
the highway AI front situation awareness system comprises an edge system and a background system, wherein the edge system is composed of a plurality of front situation awareness stations which are distributed, each front situation awareness station comprises an edge terminal, a 5G router, a fixed type Internet of things sensor, a power supply, a monitoring camera and a mobile type Internet of things sensor, and the background system comprises a cloud server and terminal equipment;
the edge terminal is used for AI reasoning, AI calculation analysis and video, picture, AI data, internet of things data and control instructions transmission;
the 5G router is used for wireless data transmission;
the fixed type internet of things sensor is used for acquiring sensing data related to the field environment;
the power supply supplies power to the edge terminal, the 5G router and the monitoring camera;
the monitoring camera is used for acquiring videos and pictures of the field environment;
the mobile internet of things sensor is used for acquiring sensing data of human body gestures and the gestures of construction equipment (including construction vehicles);
the cloud server is used for data aggregation and analysis, data storage and WEB release;
the terminal equipment is used for data display and instruction transmission;
the edge terminal, the 5G router and the monitoring camera are respectively and electrically connected with the power supply, the 5G router, the fixed type Internet of things sensor, the monitoring camera and the mobile type Internet of things sensor are respectively and electrically connected with the edge terminal through data transmission, the edge terminal is connected with the cloud server through the 5G router, and the cloud server is connected with the terminal equipment through data transmission.
According to a further scheme, the edge terminal, the 5G router, the fixed type Internet of things sensor, the power supply and the monitoring camera are integrally arranged on the monitoring upright rod.
According to a further scheme, the edge terminal, the 5G router, the fixed type Internet of things sensor and the power supply are connected with the monitoring vertical rod through the edge equipment box, and the monitoring camera is connected to one side of the top of the monitoring vertical rod.
According to a further scheme, the bottom of the monitoring vertical rod is connected with a fixed base.
According to the invention, the bottom of the fixed base is connected with a positioning rod.
The invention further adopts the scheme that the edge terminal is a small embedded super computer, the edge terminal is a basic unit for calculating, controlling and storing of a front situation awareness system, and the edge terminal is realized by adopting NVIDIA Jetson series edge computers.
The invention further provides a further scheme that the AI reasoning and the calculation analysis of the edge terminal are based on an open source multimedia framework GStreamer to build a modularized intelligent video analysis pipeline, and the method comprises the following specific processing steps:
s1: video signal input, wherein an edge terminal reads real-time super-high-definition video streams of construction sites above 2K from a monitoring camera at a rate of 30 frames per second;
s2: the method comprises the steps that video signal image processing is carried out, an edge terminal carries out a series of image processing such as H.264/265 decoding, encoding, multiplexing, demultiplexing, OSD displaying and the like on a construction site ultra-high definition video stream, and the video signal image processing is pushed to a cloud background to be presented to a user in a live broadcast mode;
s3: the method comprises the steps of target identification, performing structural analysis on an ultra-high definition video stream by an edge terminal after video decoding by running an expressway construction safety management AI algorithm system, and automatically identifying six construction safety management targets of construction operators, faces, human body postures, safety helmets, safety belts and edge protection fences in a video perception three-dimensional physical space of each front situation perception site;
s4: the intelligent video analysis pipeline of the edge terminal adopts a mode of combining three reasoning modes of basic reasoning, secondary reasoning and parallel reasoning to finish the AI reasoning process of each front situation perception site video perception three-dimensional physical space, and the AI reasoning is combined with the ROI area analysis technology and real-time perception data of the sensors of the Internet of things to automatically judge the logic relationship of constructors and construction equipment (including construction vehicles) illegal operation, wherein the logic relationship is as follows:
1) Safety helmet violations: reversely associating the safety helmet target and the humanoid target in each video frame, and judging that the safety helmet is illegal when the safety helmet target is not in the region of each personnel target frame detected by AI in the video frame;
2) Seat belt violations: and drawing an ROI area to be managed on a video displayed by the background system by using a mouse or a touch screen, wherein the ROI area is reversely associated between the safety belt target and the humanoid target. If the safety belt target of the person is greater than 0 and equal to 0, judging that the safety belt is illegal; a person greater than 0 and a person greater than the seat belt target is determined to be a seat belt violation; the person is more than 0 and equal to the safety belt target, the image human body posture analysis technology is adopted to correlate the sensing data of various mobile internet of things sensors such as an acceleration sensor, a height sensor, beidou/GPS reported human body posture data and the like to carry out target identity correlation, the identity recognition tracking of constructors in the ROI area is completed, and if the condition of the hook of the internet of things is detected to be closed, the safety belt is judged to be illegal;
3) Critical protection violations: drawing an ROI area to be managed on a video displayed by a background system by using a mouse or a touch screen, wherein a critical edge protection fence target in the ROI area is equal to or smaller than a set engineering threshold value, and judging that the critical edge protection fence target is critical edge protection violation;
4) Deformation of slopes and bridges: drawing an ROI area to be managed on a video displayed by a background system by using a mouse or a touch screen, automatically acquiring a displacement scale and an inclination angle of a critical edge protection fence in the ROI area under an image coordinate system, and judging that the side slope and the bridge deform if the displacement scale and the inclination angle exceed a set engineering threshold;
5) Construction equipment (including construction vehicles) tilt: automatically acquiring attitude change data such as acceleration and angle of construction equipment (including construction vehicles) through a mobile internet of things sensor such as an acceleration sensor and a gyroscope, and judging that the construction equipment or the construction vehicles incline if the attitude change data exceeds a set engineering threshold;
s5: the intelligent video analysis pipeline of the edge terminal uses an image target tracking technology to judge whether the target is the same target in continuous video frames, and perceives the real quantity of various targets in a construction site;
s6: the intelligent video analysis pipeline of the edge terminal automatically judges the construction security violations and abnormal behaviors after AI reasoning, intercepts the on-site high-definition pictures in real time and automatically uploads the pictures to the cloud server for management staff to use;
s7: the method comprises the steps of data analysis, real-time reporting of all target attributes automatically perceived by an edge terminal through an AI, including coordinate information and posture information of a rectangular frame in an image coordinate system, target number of continuous frames, identity information of face recognition and abnormal operation behavior data judged after AI reasoning to a cloud server through kafka streaming data through a 5G router, and displaying mass data to a user in a visual chart mode according to different periods of time, day, week, month and the like after aggregation, statistics and duplication removal of mass data by using an OLAP online analysis data warehouse technology by the cloud server;
s8: the environment sensor is used for converging sensing data of various fixed type Internet of things sensors such as environment noise, dust emission, PM2.5, temperature, humidity, wind power and the like related to a construction site by the edge terminal, sending the sensing data to the cloud server and displaying the sensing data to a user in real time;
s9: the system comprises a cloud background, an AI front situation sensing system, a distributed edge terminal, a unified api interface, a video and data of each edge terminal, a construction drawing, a video live broadcast, a screenshot image and a data chart, wherein the distributed edge terminals are used as core calculation, control and storage units, the lightweight cloud background invokes videos and data of each edge terminal through the unified api interface, each deployed front situation sensing site is arranged on the construction drawing in the cloud background, the front situation sensing site can perform visual interaction based on the construction drawing with a user, and the user can sense the construction site universe, the whole process, the whole period and the warning and early warning of construction security violations and abnormal behaviors automatically judged by the AI in real time through video live broadcast, screenshot images and data charts.
According to a further scheme, the database of the cloud server adopts an OLAP online analysis data warehouse.
The further scheme of the invention is that the terminal equipment is PC equipment or intelligent mobile equipment.
The invention has the beneficial effects that:
1. the invention can effectively provide the front situation awareness capability of the construction site, is convenient for remote command and improves the working efficiency.
2. The invention can effectively and automatically monitor the standard operation of constructors, and lighten the workload and the working pressure of management staff.
3. The invention can be seen as a reality and can be seen as a speaking, and the invention provides facts and data basis for fine management, responsibility definition and examination of each construction unit.
4. The invention can improve the prediction capability of the possibly caused construction safety accidents, and timely report the potential safety hazards of the construction site and timely inform the construction unit of modification through a normalized inspection mechanism so as to prevent the accident.
5. The invention can promote the conversion of the construction safety management from manual management to AI auxiliary management, effectively reduce the workload of manual management and effectively improve the management efficiency.
Drawings
FIG. 1 is a schematic diagram of the structure of a front situation awareness site of the present invention;
FIG. 2 is a schematic diagram of the structure of the edge equipment cabinet of the present invention;
FIG. 3 is a network architecture diagram of the AI front situation awareness system of the invention;
FIG. 4 is a flow chart of the AI reasoning process of the invention;
FIG. 5 is a flow chart of a system for front situation awareness in accordance with the present invention;
reference numerals: positioning rod 1, unable adjustment base 2, control pole setting 3, edge equipment case 4, edge terminal 5, 5G router 6, fixed thing networking sensor 7, power 8, surveillance camera 9, portable thing networking sensor 10, high in the clouds server 11, terminal equipment 12.
Detailed Description
The following description of the embodiments of the present invention will be made apparent and fully in view of the accompanying drawings, in which some, but not all embodiments of the invention are shown. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
In the description of the present invention, it should be noted that the directions or positional relationships indicated by the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc. are based on the directions or positional relationships shown in the drawings, are merely for convenience of describing the present invention and simplifying the description, and do not indicate or imply that the devices or elements referred to must have a specific orientation, be configured and operated in a specific orientation, and thus should not be construed as limiting the present invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and are not to be construed as indicating or implying relative importance.
In the description of the present invention, it should be noted that, unless explicitly specified and limited otherwise, the terms "mounted," "connected," and "connected" are to be construed broadly, and may be either fixedly connected, detachably connected, or integrally connected, for example; can be mechanically or electrically connected; can be directly connected or indirectly connected through an intermediate medium, and can be communication between two elements. The specific meaning of the above terms in the present invention will be understood in specific cases by those of ordinary skill in the art.
Example 1
As shown in fig. 1 to 3, the invention provides a highway AI front situation awareness system, which comprises an edge system and a background system, wherein the edge system consists of a plurality of front situation awareness stations distributed and deployed, the front situation awareness stations comprise an edge terminal 5, a 5G router 6, a fixed internet of things sensor 7, a power supply 8, a monitoring camera 9 and a mobile internet of things sensor 10, and the background system comprises a cloud server 11 and terminal equipment 12;
the 5G router 6 is used for information wireless transmission, the fixed type Internet of things sensor 7 is used for acquiring sensing data related to a field environment, the power supply 8 supplies power to the edge terminal 5, the 5G router 6 and the monitoring camera 9, the monitoring camera 9 is used for acquiring videos and pictures of the field environment, the mobile type Internet of things sensor 10 is used for acquiring sensing data of human body gestures and gestures of construction equipment (including construction vehicles), the cloud server 11 is used for data aggregation and analysis, data storage and WEB release, and the terminal equipment 12 is used for data display and instruction transmission;
the edge terminal 5, the 5G router 6 and the monitoring camera 9 are respectively and electrically connected with the power supply 8, the 5G router 6, the fixed type Internet of things sensor 7, the monitoring camera 9 and the mobile type Internet of things sensor 10 are respectively and respectively connected with the edge terminal 5 in a data transmission manner, the edge terminal 5 is connected with the cloud server 11 in a data transmission manner through the 5G router 6, and the cloud server 11 is connected with the terminal equipment 12 in a data transmission manner.
The edge terminal 5, the 5G router 6, the fixed internet of things sensor 7, the power supply 8 and the monitoring camera 9 are integrally arranged on the monitoring upright 3. The installation mode of the monitoring vertical rod 3 is deployed at each point of a construction site, and the monitoring vertical rod is migrated along with the change of construction progress of the construction site, so that the structure setting simplifies management work such as installation, power supply, migration, maintenance and the like to the greatest extent, and is simple and convenient to use in engineering application.
The edge terminals 5, the 5G router 6, the fixed type Internet of things sensor 7 and the power supply 8 are connected with the monitoring upright 3 through the edge equipment box 4, and the arrangement can prevent the external environment from damaging the edge terminals 5, the 5G router 6, the fixed type Internet of things sensor 7 and the power supply 8, so that the service lives of the edge terminals 5, the 5G router 6, the fixed type Internet of things sensor 7 and the power supply 8 are effectively prolonged; the monitoring camera 9 is connected to one side of the top of the monitoring upright 3, and the shooting effect of the monitoring camera 9 can be effectively improved through the arrangement, and the working efficiency of the monitoring camera 9 is effectively improved.
The bottom of the monitoring upright 3 is connected with a fixed base 2. This kind of setting can be convenient for monitor pole setting 3 and setpoint and fix a position the installation, effectively improves the installation effectiveness of control pole setting 3.
The bottom of the fixed base 2 is connected with a positioning rod 1. The compactness between control pole setting 3 and the setpoint is effectively improved to this kind of setting, effectively improves the stability after control pole setting 3 location.
The edge terminal 5 is a small embedded super computer, the edge terminal 5 is a basic unit for calculating, controlling and storing of a front situation awareness system, and the edge terminal 5 is realized by adopting NVIDIA Jetson series edge computers;
as shown in fig. 4 to fig. 5, the AI reasoning and the computing analysis of the edge terminal 5 are based on the open source multimedia framework GStreamer to build a modularized intelligent video analysis pipeline, and the specific processing steps are as follows:
s1: video signal input, the edge terminal 5 reads real-time over 2K construction site ultra-high definition video stream from the monitoring camera 9 at the rate of 30 frames per second;
s2: the video signal image processing, the edge terminal 5 carries out a series of image processing such as H.264/265 decoding, encoding, multiplexing, demultiplexing, OSD display and the like on the construction site ultra-high definition video stream, and pushes the video signal image to a cloud background to be presented to a user in a live broadcast mode;
s3: the method comprises the steps that target identification is carried out, an edge terminal 5 operates an expressway construction safety management AI algorithm system after video decoding to carry out structural analysis on an ultra-high definition video stream, and construction safety management targets of six types, namely construction operators, faces, human body postures, safety helmets, safety belts, edge protection fences and related personal safety, in a video perception three-dimensional physical space of each front situation perception site are automatically identified;
s4: the intelligent video analysis pipeline of the edge terminal 5 adopts a mode of combining three reasoning modes of basic reasoning, secondary reasoning and parallel reasoning to finish the AI reasoning process of each front situation perception site video perception three-dimensional physical space, and the AI reasoning is combined with the ROI area analysis technology and real-time perception data of the sensors of the Internet of things to automatically judge the logic relationship of constructors and construction equipment (including construction vehicles) illegal operation, wherein the logic relationship is as follows:
1) Safety helmet violations: reversely associating the safety helmet target and the humanoid target in each video frame, and judging that the safety helmet is illegal when the safety helmet target is not in the region of each personnel target frame detected by AI in the video frame;
2) Seat belt violations: and drawing an ROI area to be managed on a video displayed by the background system by using a mouse or a touch screen, wherein the ROI area is reversely associated between the safety belt target and the humanoid target. If the safety belt target of the person is greater than 0 and equal to 0, judging that the safety belt is illegal; a person greater than 0 and a person greater than the seat belt target is determined to be a seat belt violation; the person is more than 0 and equal to the safety belt target, the image human body posture analysis technology is adopted to correlate the sensing data of various mobile internet of things sensors 10 such as an acceleration sensor, a height sensor, beidou/GPS reported human body posture data and the like to carry out target identity correlation, the identity recognition tracking of constructors in the ROI area is completed, and if the condition of the hook of the internet of things is detected to be closed, the safety belt is judged to be illegal;
3) Critical protection violations: drawing an ROI area to be managed on a video displayed by a background system by using a mouse or a touch screen, wherein a critical edge protection fence target in the ROI area is equal to or smaller than a set engineering threshold value, and judging that the critical edge protection fence target is critical edge protection violation;
4) Deformation of slopes and bridges: drawing an ROI area to be managed on a video displayed by a background system by using a mouse or a touch screen, automatically acquiring a displacement scale and an inclination angle of a critical edge protection fence in the ROI area under an image coordinate system, and judging that the side slope and the bridge deform if the displacement scale and the inclination angle exceed a set engineering threshold;
5) Construction equipment (including construction vehicles) tilt: automatically acquiring attitude change data such as acceleration and angle of construction equipment through a movable internet of things sensor such as an acceleration sensor and a gyroscope, and judging that the construction equipment or a construction vehicle is inclined if the attitude change data exceeds a set engineering threshold;
s5: the intelligent video analysis pipeline of the edge terminal 5 uses an image target tracking technology to judge whether the targets are the same target in continuous video frames, and perceives the real quantity of various targets in a construction site;
s6: the construction security violations and abnormal behaviors are automatically captured, and the intelligent video analysis pipeline of the edge terminal 5 captures on-site high-definition pictures in real time and automatically uploads the pictures to the cloud server for management staff to use after the construction security violations and abnormal behaviors which are automatically judged after AI reasoning;
s7: the data analysis, the edge terminal 5 automatically perceives all target attributes through AI, including coordinate information and gesture information of a rectangular frame in an image coordinate system, the target number of continuous frames, the identity information of face recognition and the abnormal operation behavior data judged after AI reasoning are reported to a cloud server in real time through kafka stream data through a 5G router 6, and the cloud server uses an OLAP online analysis data warehouse technology to aggregate, count and remove the mass data and then displays the mass data to a user in a visual chart mode according to different periods of time, day, week, month and the like;
s8: the environment sensor, the edge terminal gathers the sensing data of various fixed type Internet of things sensors 7 such as the relevant environmental noise, dust, PM2.5, temperature, humidity, wind power and the like of the construction site, sends the sensing data to the cloud server and displays the sensing data to the user in real time;
s9: the cloud background, the AI front situation sensing system takes the distributed edge terminals 5 as core calculation, control and storage units, the lightweight cloud background calls videos and data of all edge terminals 6 through unified api interfaces, each deployed front situation sensing site is arranged on a construction drawing in the cloud background, the front situation sensing sites can perform visual interaction based on the construction drawing with a user, and the user can sense construction site universe, full flow, full period and alarm and early warning of construction security violations and abnormal behaviors automatically judged by AI in real time through video live broadcast, screenshot images and data charts.
The database of the cloud server adopts an OLAP online analysis data warehouse, the OLAP is a real-time data warehouse technology, and the OLAP data warehouse is installed on the cloud server and provides capability for parallel processing of massive real-time data.
The terminal equipment is PC equipment or intelligent mobile equipment. The PC equipment or the intelligent mobile equipment can efficiently display data and send instructions to the cloud server, and all edge terminals are controlled through the instructions of the cloud server, so that the working efficiency of the terminal equipment is effectively improved.
Example 2
As shown in fig. 1 to 5, the present invention provides a highway AI front situation awareness system, which adopts the simplest system architecture: edge system + background system.
The edge system consists of a plurality of front situation awareness sites distributed and deployed, the front situation awareness sites comprise edge terminals 5, 5G routers 6, fixed type Internet of things sensors 7, power supplies 8 and monitoring cameras 9, meanwhile, the edge system can manage mobile type Internet of things sensors carried by constructors in a construction area and mobile type Internet of things sensors 10 installed on construction equipment (including construction vehicles), and the background system comprises cloud servers 11 and terminal equipment 12. The cloud server 11 is light in function and is only used for data aggregation and analysis, data storage and WEB release, and public cloud or private cloud can be adopted. The AI reasoning and calculation analysis of the edge terminal 5 are completed by the front distributed NVIDIA JETSON series edge terminal, and the analysis result is sent to the cloud server 11 in real time through 5G communication.
The edge terminal 5 is a small embedded super computer, the edge terminal is a basic unit for calculating, controlling and storing a front situation awareness system, the edge terminal is realized by adopting NVIDIA Jetson series edge computers, and the edge terminal 5 and the monitoring camera 9 are connected with the 5G router 6 through network cables to form a local area network of a front situation awareness site. Data of the front situation awareness sites are wirelessly transmitted to the cloud server 11 through the 5G router 6, data of the fixed type Internet of things sensor 7 are transmitted to the edge terminal through USB communication, the mobile type Internet of things sensor 10 is converged by the edge terminal 5 of the front situation awareness sites through 4G/CAT1, NB-IoT or LoRa communication and submitted to the AI algorithm running on the edge terminal 5 to be matched with target recognition unified analysis of ultra-high definition videos, and a plurality of local area networks of the front situation awareness sites and a background system form the whole front situation awareness system together.
System functions:
1. AI algorithm application
The AI front situation awareness system of the expressway applies a self-defined AI algorithm system for expressway construction safety management in engineering.
2. Edge computation
Edge computing is a distributed computing architecture that handles the computation of applications, data, and services by hub nodes moving to edge nodes on the network logic. Edge computation breaks up large services that would otherwise be handled entirely by the central node, cuts into smaller and more manageable parts, and disperses to the edge nodes for processing.
The highway AI front situation awareness system introduces a processing mechanism of edge calculation into construction management and construction safety management, deploys the AI front situation awareness system in the whole domain of a construction site, effectively solves the problem that the AI processing is interrupted due to video frame dropping caused by network delay of ultra-high definition video signal wireless remote transmission, and meanwhile, the whole system is supported by the powerful computing power of an edge terminal, has powerful processing capacity of real-time, parallel and stable processing of mass data related to AI such as ultra-high definition video signals, internet of things signals and the like, and single equipment faults do not influence the stable operation of the whole system.
The highway AI front situation awareness system adopts an edge calculation processing mechanism, takes NVIDIA Jetson series edge terminals as a platform, and builds a real-time IVA intelligent video analysis pipeline at the edge terminals based on a GStreamer open source multimedia framework. The IVA pipeline comprises various modular functions such as video input, encoding and decoding, multiplexing and demultiplexing, basic reasoning, parallel reasoning, secondary reasoning, target tracking, OSD display, ROI analysis, kafka message release, video output, automatic screenshot, automatic video recording and the like, and is convenient for function expansion and engineering application.
3. Video transmission
The highway AI front situation sensing system supports RTSP, RTMP, FLV, HLS, GB28181-2016, WEB-RTC, NMOS and other video streaming media transmission protocols and cloud deck control of the monitoring camera 9 at the edge terminal 5, a streaming media platform is not required to be built at the cloud, the system architecture is simplified to the greatest extent, the cloud processing pressure is reduced, the system stability is improved, real-time and stable ultra-high definition video live broadcast service of a construction site is provided for a user, and construction managers are enabled to be 'in full'.
4. Front end publication
The expressway AI front situation awareness system deploys an interactive, visual, simple and easy-to-use WEB service system at the cloud end, and data statistics and analysis and other related service functions are realized by matching with the edge terminal.
5. Illegal screenshot
After the construction security violation set by the expressway AI front situation awareness system according to a set management strategy is automatically analyzed through an IVA pipeline, a violation picture is automatically intercepted according to the position number of the edge terminal 5 and the occurrence time of a violation event, and is synchronously uploaded to the cloud server 11, and the violation retention time can be set according to the requirements of users.
6. Data statistics
The edge system database of the highway AI front situation awareness system adopts a kafka flow database to gather various data sent by the distributed edge terminals in real time. The kafka is a low-delay (millisecond level), high-throughput, durable and distributed message publishing and subscribing queue system with excellent performance, supports the establishment of a kafka data cluster, builds a reliable real-time data pipeline between the system and an application, and builds the application capable of processing and responding to real-time data flow.
The cloud background database of the cloud server 11 of the highway AI front situation awareness system adopts an OLAP online analysis data warehouse to gather kafka stream data of N edge systems to automatically analyze various target data, effectively eliminates false alarms of algorithms by a mathematical statistical method such as weighted average in a certain period of time according to a statistical strategy, automatically counts various offensive behaviors formulated according to a management strategy in real time, makes 'see-through-image speaking' through an automatically generated dynamic visual chart, combines 'see-through' of real-time video live broadcast, and achieves the awareness of multi-level front offensive behaviors such as highway construction work areas, punctuations, universe and the like, and provides auxiliary decisions for construction site management and improves management efficiency.
7. Block inspection
The highway AI front situation sensing system considers that the highway construction site area is large, the video shot by the monitoring camera 9 cannot be 100% covered, the function of safety management inspection is provided, the inspection can be periodically performed according to a set management strategy, a mobile phone automatically generates a visual inspection track according to a walking path among different sites after self-timer identity confirmation, and the inspection record is reserved in the cloud for the inquiry and confirmation of management personnel.
8. Other business
In order to facilitate remote management and refined management of construction safety for users and improve efficiency, the expressway AI front situation awareness system also comprises other businesses such as personnel management, construction management and the like. In order to simplify and facilitate engineering application, identity confirmation can be performed through a real-time face recognition algorithm by inputting a picture into a face library of site constructors.
9. Internet of things equipment
The highway AI front situation sensing system can integrate various IOT sensors and IOT equipment on the basis of AI+IoT technology, and automatically judge the use state of the safety belt hook by combining the pressure sensor, the photoelectric sensor, the infrared sensor or the distance sensor and other mobile IOT sensors with the 4G/CAT1, NB-IoT or LoRa communication technology, and the IOT equipment adopts solar batteries for auxiliary power supply, so that manual maintenance can be reduced to the greatest extent during long-term work.
Engineering effect of the highway AI front situation awareness system in the highway construction safety management work:
(1) The system is simple and easy to use: automatically judging the illegal behaviors of constructors and automatically intercepting the picture records of the illegal behaviors through video-aware live broadcasting and data-aware AI analysis, wherein the whole process does not interfere with the normal operation of the constructors;
(2) The system construction is simple: the equipment of the front situation awareness site is fixed on the upright post and is transmitted by 5G wireless. The construction difficulty is low, and the equipment migration is convenient to follow the construction progress;
(3) The system stability is good: the device can continuously run for a long time within 7 x 24 hours, and a construction unit is matched with power supply. The fault tolerance rate of the distributed deployed edge system is high, and the overall situation awareness capability of the work area, the standard section and the whole area is not affected by single equipment faults. The use is simple and the maintenance is simple.
Application value of the expressway AI front situation awareness system in expressway construction safety management:
(1) The front situation sensing capability of the construction site can be effectively provided, remote command is facilitated, and the working efficiency is improved;
(2) The normal operation of constructors can be effectively and automatically supervised, and the pressure of management staff is reduced;
(3) The system can be used for realizing 'seeing as reality' and 'speaking through the pictures', and providing facts and data basis for fine management, responsibility definition and examination of each construction unit;
(4) The prediction capability of the possibly caused construction safety accidents can be improved, the potential safety hazards of the construction site are timely reported and the construction unit is timely notified to rectify and change through a normalized inspection mechanism, so that the accident is prevented;
(5) The method can promote the conversion of manual management to AI auxiliary management mode in construction management and construction safety management.
The invention is illustrated by way of example and not by way of limitation. Other variations or modifications of the above description will be apparent to those of ordinary skill in the art, and it is not necessary or exhaustive of all embodiments, and obvious variations or modifications are contemplated as falling within the scope of the invention.
Claims (9)
1. The highway AI front situation awareness system is characterized by comprising an edge system and a background system, wherein the edge system is composed of a plurality of front situation awareness stations distributed and deployed, each front situation awareness station comprises an edge terminal, a 5G router, a fixed type Internet of things sensor, a power supply, a monitoring camera and a mobile type Internet of things sensor, and the background system comprises a cloud server and terminal equipment;
the edge terminal is used for AI reasoning, AI calculation analysis and video, picture, AI data, internet of things data and control instructions transmission;
the 5G router is used for wireless data transmission;
the fixed type internet of things sensor is used for acquiring sensing data related to the field environment;
the power supply supplies power to the edge terminal, the 5G router and the monitoring camera;
the monitoring camera is used for acquiring videos and pictures of the field environment;
the mobile internet of things sensor is used for acquiring sensing data of human body gestures and the gestures of construction equipment (including construction vehicles);
the cloud server is used for data aggregation and analysis, data storage and WEB release;
the terminal equipment is used for data display and instruction transmission;
the edge terminal, the 5G router and the monitoring camera are respectively and electrically connected with the power supply, the 5G router, the fixed type Internet of things sensor, the monitoring camera and the mobile type Internet of things sensor are respectively and electrically connected with the edge terminal through data transmission, the edge terminal is connected with the cloud server through the 5G router, and the cloud server is connected with the terminal equipment through data transmission.
2. The highway AI front situation awareness system of claim 1 wherein the edge terminal, 5G router, stationary internet of things sensor, power supply, monitoring camera are integrally mounted on a monitoring pole.
3. The highway AI front situation awareness system according to claim 2, wherein the edge terminal, the 5G router, the fixed internet of things sensor and the power supply are connected with the monitoring upright rod through an edge equipment box, and the monitoring camera is connected to one side of the top of the monitoring upright rod.
4. A highway AI front situation awareness system according to claim 3, wherein the bottom of the monitoring pole is connected with a fixed base.
5. The highway AI front situation awareness system according to claim 4, wherein a positioning rod is connected to the bottom of the fixing base.
6. The highway AI front situation awareness system of claim 1 wherein the edge terminal is a small embedded supercomputer, the edge terminal is a basic unit for computing, controlling and storing the highway AI front situation awareness system, and the edge terminal is implemented by using an NVIDIAJetson series edge computer.
7. The expressway AI front situation awareness system of claim 6, wherein AI reasoning and computational analysis of the edge terminal is based on an open source multimedia framework GStreamer to build a modularized intelligent video analysis pipeline, and the specific processing steps are as follows:
s1: video signal input, wherein an edge terminal reads real-time super-high-definition video streams of construction sites above 2K from a monitoring camera at a rate of 30 frames per second;
s2: the method comprises the steps that video signal image processing is carried out, an edge terminal carries out a series of image processing such as H.264/265 decoding, encoding, multiplexing, demultiplexing, OSD displaying and the like on a construction site ultra-high definition video stream, and the video signal image processing is pushed to a cloud background to be presented to a user in a live broadcast mode;
s3: the method comprises the steps of target identification, performing structural analysis on an ultra-high definition video stream by an edge terminal after video decoding by running an expressway construction safety management AI algorithm system, and automatically identifying six construction safety management targets of construction operators, faces, human body postures, safety helmets, safety belts and edge protection fences in a video perception three-dimensional physical space of each front situation perception site;
s4: the intelligent video analysis pipeline of the edge terminal adopts a mode of combining three reasoning modes of basic reasoning, secondary reasoning and parallel reasoning to finish the AI reasoning process of each front situation perception site video perception three-dimensional physical space, and the AI reasoning is combined with the ROI area analysis technology and real-time perception data of the sensors of the Internet of things to automatically judge the logic relationship of constructors and construction equipment (including construction vehicles) illegal operation, wherein the logic relationship is as follows:
1) Safety helmet violations: reversely associating the safety helmet target and the humanoid target in each video frame, and judging that the safety helmet is illegal when the safety helmet target is not in the region of each personnel target frame detected by AI in the video frame;
2) Seat belt violations: and drawing an ROI area to be managed on a video displayed by the background system by using a mouse or a touch screen, wherein the ROI area is reversely associated between the safety belt target and the humanoid target. If the safety belt target of the person is greater than 0 and equal to 0, judging that the safety belt is illegal; a person greater than 0 and a person greater than the seat belt target is determined to be a seat belt violation; the person is more than 0 and equal to the safety belt target, the image human body posture analysis technology is adopted to correlate the sensing data of various mobile internet of things sensors such as an acceleration sensor, a height sensor, beidou/GPS reported human body posture data and the like to carry out target identity correlation, the identity recognition tracking of constructors in the ROI area is completed, and if the condition of the hook of the internet of things is detected to be closed, the safety belt is judged to be illegal;
3) Critical protection violations: drawing an ROI area to be managed on a video displayed by a background system by using a mouse or a touch screen, wherein a critical edge protection fence target in the ROI area is equal to or smaller than a set engineering threshold value, and judging that the critical edge protection fence target is critical edge protection violation;
4) Deformation of slopes and bridges: drawing an ROI area to be managed on a video displayed by a background system by using a mouse or a touch screen, automatically acquiring a displacement scale and an inclination angle of a critical edge protection fence in the ROI area under an image coordinate system, and judging that the side slope and the bridge deform if the displacement scale and the inclination angle exceed a set engineering threshold;
5) Construction equipment (including construction vehicles) tilt: automatically acquiring attitude change data such as acceleration and angle of construction equipment (including construction vehicles) through a mobile internet of things sensor such as an acceleration sensor and a gyroscope, and judging that the construction equipment or the construction vehicles incline if the attitude change data exceeds a set engineering threshold;
s5: the intelligent video analysis pipeline of the edge terminal uses an image target tracking technology to judge whether the target is the same target in continuous video frames, and perceives the real quantity of various targets in a construction site;
s6: the intelligent video analysis pipeline of the edge terminal automatically judges the construction security violations and abnormal behaviors after AI reasoning, intercepts the on-site high-definition pictures in real time and automatically uploads the pictures to the cloud server for management staff to use;
s7: the method comprises the steps of data analysis, real-time reporting of all target attributes automatically perceived by an edge terminal through an AI, including coordinate information and posture information of a rectangular frame in an image coordinate system, target number of continuous frames, identity information of face recognition and abnormal operation behavior data judged after AI reasoning to a cloud server through kafka streaming data through a 5G router, and displaying mass data to a user in a visual chart mode according to different periods of time, day, week, month and the like after aggregation, statistics and duplication removal of mass data by using an OLAP online analysis data warehouse technology by the cloud server;
s8: the environment sensor is used for converging sensing data of various fixed type Internet of things sensors such as environment noise, dust emission, PM2.5, temperature, humidity, wind power and the like related to a construction site by the edge terminal, sending the sensing data to the cloud server and displaying the sensing data to a user in real time;
s9: the system comprises a cloud background, an AI front situation sensing system, a distributed edge terminal, a unified api interface, a video and data of each edge terminal, a construction drawing, a video live broadcast, a screenshot image and a data chart, wherein the distributed edge terminals are used as core calculation, control and storage units, the lightweight cloud background invokes videos and data of each edge terminal through the unified api interface, each deployed front situation sensing site is arranged on the construction drawing in the cloud background, the front situation sensing site can perform visual interaction based on the construction drawing with a user, and the user can sense the construction site universe, the whole process, the whole period and the warning and early warning of construction security violations and abnormal behaviors automatically judged by the AI in real time through video live broadcast, screenshot images and data charts.
8. The highway AI front situation awareness system according to claim 1, wherein the database of the cloud server is an OLAP online analysis data warehouse.
9. The highway AI front situation awareness system according to claim 1, wherein the terminal device is a PC device or an intelligent mobile device.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310711256.2A CN116704766A (en) | 2023-06-15 | 2023-06-15 | Highway AI front situation perception system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310711256.2A CN116704766A (en) | 2023-06-15 | 2023-06-15 | Highway AI front situation perception system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116704766A true CN116704766A (en) | 2023-09-05 |
Family
ID=87830841
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310711256.2A Withdrawn CN116704766A (en) | 2023-06-15 | 2023-06-15 | Highway AI front situation perception system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116704766A (en) |
-
2023
- 2023-06-15 CN CN202310711256.2A patent/CN116704766A/en not_active Withdrawn
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111402595B (en) | Intelligent management method and system for high-speed service area | |
| CN106251578A (en) | Artificial abortion's early warning analysis method and system based on probe | |
| CN112288984A (en) | Three-dimensional visual unattended substation intelligent linkage system based on video fusion | |
| CN105187771A (en) | Plant-level comprehensive supervision platform | |
| CN101795395A (en) | System and method for monitoring crowd situation | |
| CN107122909A (en) | A kind of river course datamation monitoring and acquisition method based on BIM+GIS | |
| CN107707881A (en) | A kind of building site remote monitoring system based on internet and mobile terminal | |
| CN105182873A (en) | Urban fire-control remote monitoring system | |
| CN107454364B (en) | Distributed real-time image acquisition and processing system in video monitoring field | |
| KR20100135947A (en) | Operating assistance system for a road network with quality of service | |
| FR2946440A1 (en) | DEVICE FOR SIMULATION OF AN ENVIRONMENT OF A SYSTEM OF SUPERVISION OF AN INFRASTRUCTURE | |
| CN109472895A (en) | A kind of security protection integrated application management system and its management implementation method | |
| CN101996494A (en) | Intelligent analysis and monitoring system of traffic images based on video discriminating technology | |
| CN115248880A (en) | Smart city security monitoring system | |
| CN210222962U (en) | Intelligent electronic fence system | |
| CN107272488A (en) | A kind of electric power logistics comprehensive monitoring system | |
| CN112185121A (en) | Traffic accurate law enforcement method and equipment based on 5G | |
| CN115410354B (en) | Safety early warning method and device and safety early warning system for industrial factory | |
| CN103517040A (en) | Interactive foundation construction site safety management system | |
| CN103617691A (en) | Forest fire early warning monitoring center | |
| CN115797125B (en) | Rural digital intelligent service platform | |
| CN108322511A (en) | A kind of visualized presence monitoring system having point-to-point interactive function | |
| CN115190278A (en) | Monitoring system and method based on digital converter station | |
| CN104202576A (en) | Intelligent video analyzing method and intelligent video analyzing system | |
| CN118433231A (en) | Intelligent building site construction safety monitoring cloud edge cooperative early warning system based on edge mobile monitoring station, control system thereof and control method thereof |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| WW01 | Invention patent application withdrawn after publication | ||
| WW01 | Invention patent application withdrawn after publication |
Application publication date: 20230905 |