KR102206662B1 - Vision camera system to manage an entrance and exit management of vehicles and to recognize objects of camera video data in a port container terminal and method thereof - Google Patents

Abstract

Disclosed is a vision camera system for managing vehicle access and recognizing objects in a port container terminal to increase loading/unloading productivity and a method thereof. According to the present invention, the vision camera system comprises: a plurality of cameras installed in each area of a port gate, a yard/block, a block entrance, an automated rail mounted gantry crane (ARMGC), and a quay crane (QC) in a port container terminal; and an FPGA-based embedded vision system (TLEM) receiving a plurality of pieces of camera image data to detect objects and recognize characters from camera images in accordance with learning data by a deep learning module, recognizing a lane, a vehicle license number, a container number (ISO code), damage to a container, and a vehicle entering the block entrance and detecting a vehicle and a worker entering a danger zone entry vehicle, a reverse driving vehicle, whether a yard worker wears protective gear/safety vest, and the location of loading and unloading equipment to broadcast a warning through a corresponding speaker whenever an event occurs, and transmitting a learned object extraction event (text) and data of an image of an extracted object marked with a rectangular box to middleware. Moreover, a non-GPU-based FPGA-based embedded vision system (TLEM) is connected to a terminal operation system (TOS) through the module.

Description

Vision camera system to manage an entrance and exit management of vehicles and to recognize objects of camera video data in a port container terminal and method thereof}

The present invention relates to a port container terminal system, and more particularly, a non-GPU-based deep learning module that is linked with a plurality of cameras installed in each area of a port gate, yard/block, block entrance, ARMGC, and QC in a port container terminal. Developed an FPGA-based embedded vision system (TLEM, IoT device) equipped with (YOLO, SSD) to detect objects in camera images, detect containers, vehicles, people, and unloading equipment, and block/ Block entrance vehicle detection, lane recognition, vehicle number recognition, text recognition of container number, hazardous area worker and vehicle detection, reverse driving vehicle detection, detection of whether workers are wearing safety protective equipment/safety vests, and containers being loaded and unloaded from QC into the ship It relates to a vision camera system that recognizes a number (ISO code), manages vehicle access and recognizes an object in a port container terminal.

In addition to the 4th industrial technology, IoT, AR/VR, Big Data, and Drone technologies are advancing in recent years, and it is urgent to secure core common base technologies for the expansion and industrialization of vision systems in the port logistics industry and port container terminals.

As the global logistics center moves to Asia, competition for global hub ports is intensifying, and the proportion of Asian shipments in the past is expected to be 47.1% in 2000 and 54.0% in 2015 → 59.4% of future Asian shipments. And the current trend is expected to continue after 2020. With less investment and lack of research compared to other logistics fields, the national port logistics technology industry needs to strengthen the competitiveness of port logistics technology with other countries.

The handling capacity of a port container terminal is determined by the individual capabilities of the berth, transfer vehicle, yard, and gate, as well as the comprehensive linkage capability. As the port container terminal improves the handling capacity, the ship's return time can be reduced, and thus more profits can be obtained with limited resources.

The handling capacity of the container terminal can be evaluated as the berth throughput (productivity) in the end, and as a method to improve the berth throughput, i) improve the speed of the quay crane, ii) apply a new concept crane, iii) the number of cranes working simultaneously on one ship Consider the increase.

The berth throughput is determined by the throughput of yards and transport vehicles as well as its own throughput according to the number of crane inputs.

Currently, the productivity (move/hr) of QC (Quay Crane), which is the berth processing rate of domestic and foreign ports, is recording a maximum of 27 move/hr. QC average productivity of domestic container terminals accounts for 19~24 move/hr, while overseas container terminals account for an average of 22~27 move/hr (20% higher than domestic).

Most of the new ports in Busan are semi-automated terminals, similar to the overseas level with 26~30 move/hr, but the Shanghai automated terminal, which is a fully automated terminal, showed high productivity at 40 move/hr and reduced the manpower by 70%.

Starting from the “RFID-based Port Logistics Efficiency Demonstration Project” conducted by the Ministry of Maritime Affairs and Fisheries since 2005 under the purpose of'low carbon green growth' by applying ICT convergence and complex technology in the port logistics field, many IoT technologies have been applied to the government. It was conducted under the leadership, but only some technologies were applied.

In port container terminals, social safety awareness is increasing, but the risk of accidents is increasing, and it is urgent to prepare a plan to ensure safety for workers in hazardous areas.

-Automation and mechanization in ports, exposure to worker risk due to the increase in vehicles loaded with containers, and increased risk situations outside the range of perception

-There is a limit to real-time location and status identification of resources (containers, vehicles, equipment) in the port

-Port security area monitoring and danger area detection required

Therefore, the'Recognition and Location Information' system, which is an essential element of a fully unmanned automated terminal, uses real-time image data to detect objects and characters, and position correction GPS. ① Container location and character recognition, ② Real-time vehicle location and ID recognition, ③ External fence detection, ④ Worker safety helmet, whether or not to wear a safety vest, ⑤ Detection of vehicles and personnel in hazardous areas, ⑥ Vehicle reverse running/safe speed detection to improve work processing efficiency In addition, there is a need to develop technologies that can improve worker safety.

In addition, in the port container terminal, the ship moves slightly by wind and waves in the section where the container is unloaded and loaded from the ship, and thus the gantry crane also moves to perform the work. Therefore, it changes to the point where the container is unloaded and unloaded using the trailer/chassis. And the type of container is also largely classified into two types (40 feet, 20 feet), so it is difficult to stop at the working position by the driver alone.

As a related prior art 1, in Patent Registration No. 10-0794410, the “port trailer automatic stop position monitoring system” is registered in which the driver of the trailer carrying the port container identifies the stop position and stops the trailer by himself without an assistant worker. Has been.

1 is a block diagram of a conventional port trailer automatic stop position monitoring system.

The port trailer automatic stop position monitoring system includes a CCD camera (100), a frame grabber (FG) (102), a personal computer (PC) (104), a programmable logic controller (PLC) (106), and a display. It is composed of an indicator 108 and a plurality of laser sensors 110. The system can be configured simply by attaching the laser sensor 110 and the CCD camera 100 to the existing gantry crane.

The CCD camera 100 and the laser sensor 110 are installed at the upper and lower ends of the crane structure 10, respectively, and the analog signal input to the CCD camera 100 is a digital image signal from the frame grabber (FG) 102. Is converted into, and this digital video signal is input to the PC 104. The digital image signal input to the PC 104 captures the movement of the trailer 20 using an "Optical Flow" image processing technique. That is, the PC 104 detects a width of 20 m in the horizontal axis inside the crane by using an image having 640 × 48 (pixel) pixels in which the vertical column is reduced to 1/10. Therefore, it can have a resolution of 3.1 cm per pixel, and a more precise stop position can be detected using the laser sensor 110 of the chassis.

The CCD camera 100 is installed on the upper end of the crane structure 10, and is preferably installed so as not to interfere with the work of the spread and to be firmly fixed so that the accuracy is not degraded by shaking during work.

Port trailer automatic stop position monitoring system

In loading and unloading cargo using a crane on a ship,

At least one laser sensor installed at the lower end of the crane structure to detect the position from the side of the trailer, and a CCD camera installed at the upper end of the crane structure to detect the position at the upper end of the crane. Sensing means for automatically recognizing the stop position of the trailer according to the position change;

A display means for notifying a driver of a stop position of the trailer through a visual signal;

A frame grabber for converting the analog signal transmitted from the CCD camera into a digital signal;

A personal computer for capturing the motion of the trailer by using the "Optical flow" image processing technique of the digital signal input from the frame grabber;

And a PLC for calculating and processing the bit string calculated by the personal computer and the bit string transmitted from the laser sensor.

According to the size of the container, the laser sensor is composed of two pairs in four places, and each data is transmitted to the PLC.

The display means,

A status display unit that dynamically displays the entry of the trailer;

An indicator indicating the stop position of the trailer,

It consists of a marking unit indicating the operating state of the sensing means.

The port trailer automatic stop position monitoring system provides the following effects.

First, even if a container enters from a lane other than the working lane, the laser sensor and the CCD camera simultaneously detect it, so it has the advantage that no malfunction occurs at all.

Second, the display type continuously shows the dynamic situation according to the movement of the trailer and informs the stop position, so that the driver can predictive driving, thus significantly reducing the burden on sudden stops.

Third, it provides more accurate location information by using two pairs of laser sensors in each of the four locations, and the degree of error can be provided to the driver even if the trailer passes the stopping point.

Fourth, it can be used in both loading and unloading because it detects the container vehicle by using a CCD camera with improved data processing speed and high resolution other than the laser sensor.

Fifth, the laser sensor and the CCD camera assist each other to display the stop position, thereby providing more reliable stop position information.

As a related prior art 2, a "port gate OCR system device" is registered in Patent Registration No. 10-1978292.

Port gate OCR system devices are arranged sequentially in the port gate passage lane, and when the head of the container vehicle is recognized, a value of 1 is output, and the other parts output a value of 0, and three consecutive units are recognized from the front to the rear of the container. It consists of a sensor arranged at intervals, a control unit that receives the recognition value of the sensor and transmits it as a control signal, a front camera that photographs the front and right sides of the container, and a rear camera that photographs the rear and left side of the container. By immediately grasping the location and simultaneously performing various number recognition, information is acquired within a short time and can be applied to the classification and movement of containers.

As a related prior art 3, in patent registration number 10-1068681, a "port integrated security monitoring operation system" is registered.

The port integrated security monitoring operation system includes: a port security monitoring unit including a camera for photographing a subject in a port area to be monitored; An entry and exit location detection unit for detecting location information on cargo or people in the port by detecting a unique identification tag mounted or worn on cargo or people entering and exiting the port; Based on the information provided by the access and movement location detection unit, the access information in the port and the location movement information of the monitored object are created and processed, and the port security monitoring unit can track it in real time using the movement information of the monitoring object. A port integrated monitoring server for controlling the port security monitoring unit so that; A portable port information terminal for receiving and displaying port status information including geographic information in the port and logistics location information in the port requested through an input unit while communicating with the port integrated monitoring server through a wireless network. and,

The identification tag is an RFID tag, and the entry and exit location detection unit is installed to recognize the RFID tag at each set detection location including a gate through which a person or cargo enters and exits, and provides detection information to the port integrated monitoring server. Reader,

The port integrated monitoring server is connected through a marine safety information system and a network that provides information on ships entering and leaving the port, receives information on the ship, and determines whether there is an unregistered ship through the port security monitoring unit, and is not registered. Ships are processed to be monitored through the port security monitoring unit,

The integrated port monitoring server is to obtain advance information on container cargo entering and exiting the port by being connected through a network and a port operation information system that manages logistics movement information from the production place of container cargo to the destination.

The port security monitoring unit

A central camera installed to capture a broadband image of a port area to be monitored;

A plurality of local cameras installed to capture an image of a local area allocated to the surveillance area;

A location information providing unit installed to correspond to the central camera and detecting camera location information on a change in an imaging direction of the central camera;

A distance measuring device installed corresponding to the central camera to measure a distance of an object in front of the central camera in the imaging direction;

An intrusion detection sensor installed to detect whether an intrusion into a set non-permitted area among the port areas to be monitored;

An intrusion detection unit that determines whether or not a chip is inserted from the output signal of the intrusion detection sensor;

A radar for detecting the presence or absence of a moving object in the port monitoring area; and

The integrated port monitoring server receives and stores information transmitted from the central camera, the local camera, the location information providing unit, the distance measuring device, the chip presence detection unit and the radar, and capturing the central camera and the local camera. Control the direction remotely.

According to the port integrated security monitoring and operation system, access control according to the security level is possible because the current location of incoming and outgoing vehicles, cargo, and people can be identified, and abnormal logistics activities and normal logistics in the port using prior information on moving subjects. It provides the advantage of real-time detection and tracking by distinguishing activities.

As a related prior art 4, in patent registration number 10-1071379, a "port integrated security monitoring system" is registered.

The port integrated security surveillance system includes: a central camera installed to capture a broadband image of a port area to be monitored; A plurality of local cameras installed to capture an image of a local area allocated to the surveillance area; A location information providing unit installed to correspond to the central camera and detecting location information on a change in the imaging direction of the central camera; A distance measuring device installed corresponding to the central camera to measure a distance of an object in front of the central camera in the imaging direction; An intrusion detection sensor installed to detect whether an intrusion has occurred in the port area to be monitored; An intrusion detection unit that determines whether or not a chip is inserted from the output signal of the intrusion detection sensor; A radar detecting the presence or absence of a moving object in the port monitoring area; Receiving and storing information transmitted from the central camera, the local camera, the location information providing unit, the range finder, the chip presence detection unit, and the radar, and remotely controlling the imaging direction of the central camera and the local camera A management server; And a sound wave detector that transmits ultrasonic waves underwater to monitor the presence of intrusion through the seabed area of the port, and provides intrusion information to the management server from the received ultrasonic waves,

The management server includes: a reception information processing unit that stores the image transmitted from the central camera in a database and compares frames of the image stored in the database to determine the presence or absence of a moving object; When information on a moving object is provided from the reception information processing unit, the central camera or the local camera controls the central camera or the local camera so that it can capture an area where the animal object is detected, and A remote control processing unit for transmitting alarm information to a communication address; When requesting to view the image information stored in the database, and having a data viewing providing unit that provides the information,

The location information providing unit includes a GPS receiver installed to provide location information of the central camera; And a digital magnetic compass that provides posture information of the central camera. The port integrated security monitoring system can monitor and track the intrusion into vast port facilities through water, water, air and ground.

As a related prior art 5, in Patent Registration No. 10-1045323, a panoramic image of a port is generated using a multi-view camera system, and 360-degree real-time port control is possible in connection with AIS, PORT-MIS or radar signals. "A real-time port video control system using a multi-view camera system and its method" has been registered.

A real-time port video control system using a multi-view camera system

A multi-view camera system for dividing the entire port into a plurality of cameras to simultaneously take pictures, and generating panoramic image data from the captured images;

A panoramic coordinate unit for generating a panoramic spatial coordinate system from the panoramic image data;

A ship information input unit that provides ship information by receiving at least one of a ship's AIS signal, PORT-MIS, and radar;

A topographic information input unit for inputting topographic information of a port in the electronic chart;

Provides a real-time port video control system using a multi-view camera system composed of an image control display unit that matches and displays the ship information of the ship information input unit and the terrain information of the terrain information input unit with the panoramic image data of the multi-view camera system. Accordingly, since the entire port can be displayed on one screen, 360-degree real-time port video control is possible. Ship information analyzed from AIS, PORT-MIS or radar signals is displayed on this panoramic image, and wireless communication between the existing ship and the controller The information that was exchanged by using is displayed on a single screen as a panoramic image, and the effect of controlling the video can be obtained as if the controller was watching the site directly.

As a prior art 6 related to this, in Patent Registration No. 10-1313025, "a system for recognizing a stop position of an unmanned transport equipment in a port facility using wireless recognition" is disclosed.

The stationary position recognition system of unmanned transport equipment in the port facility using wireless recognition

In the stationary position recognition system of unmanned transportation equipment,

A radio identification (RFID) tag 100 installed and configured in a designated area within the port terminal equipment field;

A wireless recognition (RFID) reader 200 installed and configured in a gantry crane 10, a tower crane 20 in a port terminal equipment site;

A wireless LAN module 300 for wireless communication with a central control means installed and configured in a gantry crane and a tower crane in the port terminal equipment;

An unmanned vehicle 400 in which a wireless identification (RFID) reader and a wireless LAN module are installed and configured with two directional antennas;

Sending location information to perform unloading work to the Gentry Crane, sending a call message to move to the work area to an unmanned vehicle, and a call message to move to the work area to the Tower Crane It is configured to include a central control means 500 for sending.

As a prior art 7 related to this, in Patent Registration No. 10-1042343, "a system for recognizing the location of vehicles in a port using a UWB sensor" is registered.

The technology for recognizing the location of a vehicle running in a port provides a method of recognizing the location of a vehicle in a port using a UWB sensor rather than an existing GPS or RFID. The position of the vehicle within the port of the existing GPS-based vehicle location recognition system showed a difference of more than 5m, and the vehicle location recognition system using RFID has a location error of more than 1~5M in the water and container human body error, which is a special environmental factor of the port. Showed. However, it is possible to have a position error value within 30cm by using the UWB signal, and to recognize the position of the vehicle more accurately by reducing the error value by using a speed filter that considers the speed of a moving vehicle with a 30cm error.

As a first aspect, in the method for reducing a position recognition error using a speed filter of a vehicle position recognition system in a port using a UWB sensor, 1 for transmitting a UWB signal from the transmitter 100 to the receiver (UWB sensor) 200 Step (S110) and; A second step (S120) of calculating the location of the sending device 100 in the receiving device 200; A third step (S130) of transmitting a signal including a location from the receiving device 200 to the location recognition server 300; A fourth step (S140) of filtering the position value using a speed filter in consideration of the speed value of the vehicle in the position recognition server 300; A method for reducing a position recognition error using a speed filter of a position recognition system of a vehicle in a port using a UWB sensor including a fifth step (S150) of displaying the filtered position value on a graphic user interface is presented.

As a second aspect, a system for recognizing a position of a vehicle in a port using a UWB sensor, comprising: a transmission device 100 installed in a vehicle in the port to transmit a specific signal; A receiving device 200 for receiving a signal transmitted from the transmitting device 100, calculating a location of the transmitting device 100 using TDOA, and transmitting the received signal to a location recognition server; Port using a UWB sensor including a location recognition server 300 that receives the signal transmitted from the receiving device 200 to calculate the speed of the sending device 100 and reduces the position error using the speed filter 310 The location recognition system of my vehicle is presented.

1. Necessity of developing non-GPU-based deep learning image processing technology

Recently, the port logistics industry in Korea, the world's third largest container port owner, overlaps Shanghai with the rapid growth of neighboring ports in neighboring countries, knowledge-based port logistics technology industry, and the downturn in shipbuilding and shipping industries, but advanced intelligent ports based on ICT technology. A system was needed.

The Ministry of Maritime Affairs and Fisheries promoted research and development of a smart automated port system (Overhead Shuttle System, OSS) to improve efficiency and productivity compared to existing commercialized overseas automated port technology (2013-2017). -Extra-large conveyor (25,000 TEU class) 22 hours unloading service (conventional: 40 hours, overseas automation: more than 28 hours)-

Until now, in domestic port container terminals, many human resources such as underman, work manager, signal officer, and combined staff have been mobilized and passively carried out during loading. This is the result of labor costs, risk of safety accidents, work efficiency, and the naked eye of workers when loading STS. The human error of the operator is occurring due to recognition. In addition, some automated port container terminals use 900MHz RFID to increase the productivity and efficiency of loading and unloading port container terminals compared to manual terminals. PNC (Busan New Port Co., Ltd.) is the only port container terminal to which GPS is applied.

1B is a Yard map of a port container terminal equipped with a port gate, Yard's Block, Block Entrance, ARMGC, QC, TC, and STS (Ship To Shore Crane). When passing through the port gate and customs inspection site (CS), the relevant vehicle's RFID is detected, and the operator receives work orders and manages port logistics information, and Yard's Blocks (A1, B1, C1, D1, E1, F1, G1) , A2,B2,C2,D2,E2,F2,G2) each block entrance is equipped with a camera and an RFID reader on the left and right, and a QC (Quay Crane) that loads and loads containers on the ship. . XT and YT vehicles have one way only on the sea side.

TC is Transfer Crane, ATC is Automated Transfer Crane (ARMGC), YT is Yard Tractor, XT is external truck, STS is Ship To Shore Shore, and QC is Quay Crane.

TC (Transfer Crane) is equipped with RTGC (Rubber Tired Gantry Crane) and RMGC (Rail Mounted Gantry Crane). RTGC is not yet unmanned (fully automatic, semi-automatic), and RMGC is operating unmanned (fully automatic, semi-automatic). Unmanned RMGC is called ARMGC (Automated Rail Mounted Gantry Crane). ARMGC moves 2-3m per second and is called ATC (Automated Transfer Crane) for each operator.

The existing port container terminal RFID system is a port mule type RFID system, equipped with a port gate entrance RFID system, Yard RFID system, ATC RFID system, and QC RFID system, and is connected to the terminal operation system (TOS), and is connected to the port gate entrance and vehicle 900MHz. Vehicle access control and work orders using RFID tags are issued, port logistics management information and customs information are managed, vehicle number recognition, and RFID readers are installed at each block entrance of Yard, and ships at QC (Quay Crane) It is equipped in ATC (Automated Transfer Crane, ARMGC), which recognizes the number of containers being loaded and unloaded. The RFID system includes a 900MHz RFID reader, middleware, and PC that detects 900MHz RFID tags attached to a vehicle.

To date, port container terminals have developed barcode recognition, 900MHz RFID recognition, and GPS+RTLS (2.4GHz beacon) for automation, but they show a large error due to GPS positioning errors and environmental variables of the port container terminal. You are not receiving information.

RTLS technology provides various solutions using RFID, GPS, and 2.4GHz beacon technology, but reliable location tracking of port container vehicles is required due to environmental effects and radio interference of the ISM band of the port terminal.

In addition, LPR (License Plate Recognition) (10 million won per lane) and OCR (Optical character recognition) (70 million won per lane for port gates) are used as vehicle detectors that recognize vehicle numbers equipped with infrared emitters used in port container terminals. ), high-end GPU (graphic processing unit) based image recognition system requires complex hardware configuration and expensive system construction cost for high-performance graphic processing, and it is difficult to be compatible with each reading software, and the recognition rate is different from each other (OCR Is not read other than characters).

For this, it is necessary to develop image recognition technology for a small FPGA-based embedded vision system equipped with an artificial intelligence deep learning module, and a non-GPU-based deep learning image recognition module and a GPS system capable of precise position correction are required. The efficiency of loading and unloading of port container terminals should be improved and reliability should be improved.

Since its establishment in 2012, the company has developed/built an automation system for domestic and overseas port container terminals, and has a track record of delivering HW/SW to terminals, access control, and port integrated logistics management systems.

In the case of overseas port container terminals, the multi-pass phenomenon does not occur because containers are not loaded high due to the large site.

In the case of domestic port container terminals, diffuse reflection is severe due to the loading of 4-6 containers, and errors due to earth rotation, shading and diffuse reflection due to loaded containers exist.

Patent Registration No. 10-0794410 (Registration Date January 7, 2008), "Port Trailer Automatic Stop Position Monitoring System", Kwon Young-soo, Lee Kwon-soon Patent Registration No. 10-1978292 (Registration Date May 8, 2019), "Port Gate OCR System Device", KNS Co., Ltd. Patent registration number 10-1068681 (Registration date September 22, 2011), "Port Integrated Security Monitoring System", Korea Ocean Research & Development Institute Patent registration number 10-1071379 (Registration date September 30, 2011), "Port Integrated Security Monitoring System", Korea Ocean Research & Development Institute Patent Registration No. 10-1045323 (Registration Date June 23, 2011), "Real-time port video control system and method using multi-view camera system", Ednet Co., Ltd. Patent Registration No. 10-1313025 (Registration Date September 24, 2013), "Stop Position Recognition System of Unmanned Transport Equipment in Harbor Equipment Using Wireless Recognition", Dongmyeong University Industry-Academic Cooperation Foundation Patent Registration No. 10-1042343 (Registration Date June 10, 2011), "Location Recognition System of Vehicles in Harbor Using UWB Sensor", Dongmyeong University Industry-Academic Cooperation Foundation

An object of the present invention for solving the above problem is to control vehicle access in a port container terminal and to control objects of camera images according to deep learning learning data by a deep learning module of an FPGA-based embedded vision system (TLEM) linked with a plurality of cameras. Detects container vehicles and people through feature extraction and classification, block/block entrance of port gate/Yard, ARMGC vehicle detection, lane recognition, vehicle number recognition, container number recognition, hazardous area worker and vehicle detection, reverse driving vehicle detection , Detects whether workers are wearing safety protective equipment/safety vests, extracts the container number (ISO code) that is being loaded onto the ship from QC, and extracts the object learned as middleware from the FPGA-based embedded vision system (TLEM), event (text) and square box It transmits the object extraction image data marked with, and transmits the location information (X,Y coordinates) of the vehicle equipped with DR-GPS to the terminal operation system (TOS), and TLEM or middleware is provided in the Yard of the port container terminal. It provides a vision camera system for vehicle access management, location tracking, and object recognition at the port container terminal, which provides safety management of dangerous areas for workers and vehicles through speaker broadcasts in the corresponding area.

Another object of the present invention is that an FPGA-based embedded vision system (TLEM) equipped with a deep learning module is connected to a terminal operation system (TOS) through a middleware, and an object of vehicle access management and location tracking and vision camera system in a port container terminal Provides a way to recognize.

The vision system for improving the productivity and efficiency of loading and unloading at the port container terminal uses the deep learning module of the non-GPU-based FPGA-based embedded vision system to identify objects of camera images and recognize characters.YT vehicles and XT vehicles It is equipped with a DR-GPS and a position correction DR-GPS terminal equipped with a 6-axis sensor or a 9-axis sensor.

First, in the port container terminal, the port gate entrance, intersection, Yard's block entrance, STS (Ship To Shore Crane) / TC (Transfer Crane), and network cameras are installed in hazardous areas to provide vehicle number, container size and container. Developing deep learning S/W to detect ISO Code, YT number recognition, YT (Yard Tractor), worker's hard hat/safety vest, detect vehicles and workers in hazardous areas and the number of people, and detect vehicle reverse running/safe speed. , Second, text recognition of the object of the image recognized YT number, container number, and vehicle number, and location recognition of XT (external truck) equipped with a DR-GPS terminal, and GPS location information corrected for the location of the vehicle is TOS (Terminal Operation System) ) To secure basic technology for commercialization in the port container terminal automation field.

In order to achieve the object of the present invention, a vision camera system for managing vehicle access and recognizing objects in a port container terminal includes a plurality of cameras installed in each area of the port gate, Yard/Block, Block Entrance, ARMGC, and QC in the port container terminal. ; And receiving the plurality of camera image data and detecting objects in the camera image according to the learning data by a deep learning module and recognizing characters, lane recognition, vehicle number recognition, container number (ISO code) character recognition, container damage recognition , Recognition of vehicles entering the block entrance, vehicles and workers entering dangerous areas, vehicles and workers entering a dangerous area, detection of whether or not wearing safety protection/safety vests for yard workers, loading/unloading equipment locations are detected, and learned object extraction event (text) and marking with square boxes It includes an FPGA-based embedded vision system (TLEM) that transmits the extracted object extracted image data to the middleware,
The FPGA-based embedded vision system (TLEM) includes a deep learning module that detects non-GPU-based image objects and recognizes characters of vehicle numbers and container numbers (ISO codes).

The vision camera system for recognizing an object in a port container terminal of the present invention includes vehicle access management and a DR-GPS terminal (DR-GPS, 6-axis sensor, or 9-axis sensor) at the port container terminal. Container number, vehicle number, etc.), container, vehicle, and people through feature extraction and classification of objects in camera image according to deep learning learning data by deep learning module of FPGA-based embedded vision system (TLEM) linked with multiple cameras. , Detecting loading and unloading equipment, Block/Block Entrance of Port Gate/Yard, ARMGC vehicle detection, lane recognition, vehicle number recognition, container number text recognition, hazardous area worker and vehicle detection, reverse driving vehicle detection, worker's safety protection /Detecting whether the safety vest is worn, extracting the container number (ISO code) loaded onto the ship from QC, extracting the object learned as middleware from the FPGA-based embedded vision system (TLEM), the event (text) and the object extracted image marked with a square box It transmits data and transmits it to the terminal operation system (TOS), and TLEM or middleware provides safety management of hazardous areas of workers and vehicles through the local speaker broadcast provided in the yard of the port container terminal. It prevents safety accidents, recognizes the vehicle number at the container terminal, recognizes the container number (ISO number) that STS Crane works on, and provides safety management for workers under the crane. The location information (X,Y coordinates) of container vehicles (XT vehicles, YT vehicles) equipped with DR-GPS are transmitted to the terminal operation system (TOS) through middleware, and the terminal operation system (TOS) is DR-GPS in real time. By providing the location information (X, Y coordinates) of the container vehicle (XT vehicle, YT vehicle) equipped with, in real time, the container vehicle is provided in the number of blocks of which yard. Through this, the automation system of the port container terminal was established to improve the unloading and unloading productivity.

In addition, in the port container terminal, in the environment where GPS signals are not received under the position error due to multi-path and the TC (Transfer Crane) or QC (Quay Crane) caused by the containers stacked high by loading containers, DR-GPS terminal (using antenna, DR-GPS, 6-axis sensor or 9-axis sensor) that can precisely position the vehicle in the port using the 9-axis sensor, the vehicle's CAN signal, and the vehicle's speed signal. Accurate location tracking (YT number, container number, vehicle number recognition) in the port of vehicles using CAN communication with GPS terminals became possible.

This product (TLEM) has a price competitiveness compared to RFID, OCR, and LPR in port container terminals, and has a performance advantage in object recognition of images from deep learning modules, and minimizes and maintains the number of equipment by integrating various functions into one function. It reduces maintenance costs and can be supplied at a price of less than 50% of the existing port access security system, container terminal, and transport equipment (Quay Crane, Yard Crane Rail Mounted Gantry Crane, Yard Tractor, Yard Chassis, Yard Chassis, Forklift Fork Lift, Reach Stacker Reach Stacker, Ampri Handler Empty Handler), entry lane recognition, port container terminal access management/control, and Yard worker's hard hat/safety vest detection to provide a multifunctional vision system specialized for port safety/security. Applicable to port automation system. In addition, it is possible to enter the overseas market of port automation terminals through joint sales through overseas TOS developers.

1A is a block diagram of a conventional port trailer automatic stop position monitoring system.
1B is a Yard map of a port container terminal equipped with a port gate, Yard's Block, Block Entrance, ARMGC, QC, TC, and STS (Ship To Shore Crane).
FIG. 2A is a diagram showing the necessity of developing technology for improving the productivity, safety and security of a port container terminal using a camera imaging device.
Figure 2b is a technical necessity for increasing productivity and cost reduction by minimizing the distribution time of port logistics by applying GPS position correction of container vehicles in port logistics and IoT technology based on FPGA-based embedded vision system (Smart Vision System), It is a diagram showing the economic effect of productivity improvement.
2C is a photograph of a port terminal Real-Time Location System (RTLS) system, a port terminal OCR system, and a port terminal RFID system.
Figure 3 is an overview of a vision camera system for vehicle access management and location tracking and object recognition equipped with a DR-GPS terminal in a port container terminal: 1) Real-time location information for the equipment site, 2) TC/ATC (YT, XT ↔ Yard) container using vision system, Shuttle, YT number, vehicle number, character recognition of container number, and vehicle correct position confirmation, 3) QC (ship ↔ XT, shuttle) container, Shuttle, YT, character recognition, and Provides vehicle correct position check and lane position check function.
4 is a diagram illustrating a comparison between an existing LPR and OCR recognition system and a GPU-based vision system in a port container terminal.
5 is a port container terminal, an existing GPS/GNSS, DGPS, 2.4GHz BLE beacon/Wi-Fi gateway port location tracking system, and a location correction system used with a developed DR-GPS terminal and a 6-axis sensor/9-axis sensor. It is a comparison drawing.
6 is a diagram showing a technology roadmap for the application field of Gate system/TC system/QC system/Block Entrance/Safety for next-generation future smart ports.
7A is a diagram illustrating a real-time object recognition of a camera image using deep learning: the concept of convolutional neural networks (CNN), recurrent neural networks (RNN), and long-short term memory (LSTM) applications.
7B is a diagram illustrating a real-time object recognition: You Only Look Once (YOLO) model of a camera image using deep learning.
8 is a diagram showing the final development goal of linking with the terminal operation system (TOS) through the middleware of a vision camera system and a monitoring PC in a port container terminal.
9A is a diagram showing a framework of (1) a non-GPU-based object recognition deep learning module SW and an FPGA-based embedded vision system (TLEM) in a port container terminal.
9B is a view showing an integrated middleware for (2) developing a high-precision dead reckoning applied DR-GPS module using a 9-axis sensor, and (3) linking with a terminal operating system (TOS) in a port container terminal. .
9C is a block diagram of an integrated middleware for linking with a terminal operating system (TOS).
10 is a block diagram of a vision camera system for vehicle access management, location tracking, and object recognition including a DR-GPS terminal (DR-GPS and 6-axis sensor/9-axis sensor) in a port container terminal according to the present invention.
11 is a block diagram of a vision camera system for managing vehicle access and recognizing an object of a camera image in a port container terminal according to an embodiment of the present invention.
12 is a block diagram of an FPGA-based embedded vision system (TLEM) equipped with a deep learning module (YOLO, SSD) that detects an object in a camera image and recognizes characters in an image including a vehicle number and a container number (ISO code). to be.
13 is a diagram showing quantitative evaluation items of an FPGA-based embedded vision system equipped with a non-GPU-based deep learning module to improve port safety and productivity and efficiency of loading and unloading container terminals.
14 is a real-time camera image data and location information service screen in a port container terminal.
Figure 15 is a port container terminal test image: (1) GPS only, (2) GPS + 9-axis sensor + location correction GPS test that can distinguish lanes at the port container terminal using a position correction algorithm-The distance between blocks of the horizontal structure Yard is It is 20~24m interval.
FIG. 16 is a chart showing the commercialization performance of Port GPS system (vehicle tracking), Port RFID system (vehicle recognition), OTR device (vehicle location recognition), and BLE Beacon System (IoT) by year in a port container terminal.
17 is a diagram showing a commercialization model business strategy and plans to enter overseas markets.
18 shows the current status of the loading/unloading equipment of a domestic container terminal and the trend of global shipments.
19 shows the annual demand forecast of domestic and overseas automated container terminals.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the description of the present invention, when it is determined that a detailed description of a related known function or a known configuration may unnecessarily obscure the subject matter of the present invention, a detailed description thereof will be omitted. In addition, when the attached reference numbers indicate the same configuration, the same reference numbers are assigned to other drawings.

The vision camera system for vehicle access management, location tracking, and object recognition in a port container terminal of the present invention includes vehicle access management and a DR-GPS terminal (with DR-GPS, 6-axis sensor or 9-axis sensor) in the port container terminal. Equipped vehicle location tracking (YT number, container number, vehicle number, etc.), deep learning module of FPGA-based embedded vision system (Tiny Local Embedded Module, TLEM) linked with multiple cameras, through intelligent image analysis, results of deep learning learning Detects container vehicles and people through feature extraction and classification of objects (containers, vehicles, people, loading/unloading equipment) in the camera image according to, and detecting block/block entrance vehicles of port gate/yard, lane recognition, vehicle Number recognition, container number (ISO code) character recognition, hazardous area worker and vehicle detection, reverse driving vehicle detection, detection of whether the worker is wearing safety protective gear/safety vest, extraction of container number (ISO code) characters quantitatively unloaded from QC to the ship , Transmits the object extraction event (text) learned from the FPGA-based embedded vision system (TLEM) to the middleware and the object extraction image data marked with a square box, and transmits it to the terminal operating system (TOS), and an FPGA-based embedded vision system (TLEM) or middleware provides safety management of dangerous areas for workers and vehicles through speaker broadcasting in the area equipped on the Yard block of the port container terminal, and the vehicle number recognition, lane recognition, and STS Crane work in the port container terminal. It provides the container number (ISO number) character recognition and safety management for workers under the crane. The FPGA-based embedded vision system (TLEM) is interlocked with the terminal operating system (TOS) through middleware. The location (X,Y coordinates) of a vehicle equipped with a GPS antenna and a DR-GPS terminal is transmitted to the terminal operation system (TOS) through middleware in real time, and the terminal operation system (TOS)

All. It is to improve the productivity of loading and unloading by establishing an automated system for port container terminals.

The vision system does not use the existing GPU-based vision system,

In the embodiment, an FPGA-based embedded vision system (TLEM, IoT device) equipped with a non-GPU-based deep learning module connected to a plurality of cameras is used.

This task detects objects of camera images (containers, vehicles, people, loading and unloading equipment, etc.) using the deep learning module of the non-GPU-based FPGA-based embedded vision system (TLEM), which was not applied in the port container terminal. By applying the data obtained by learning the object in advance to the embedded vision system, and processing the object recognition in the camera image according to the learning data by the deep learning module of TLEM, high-speed recognition and low specification of hardware can be applied. In addition, by developing a DR-GPS terminal with high-precision dead reckoning using DR-GPS and 9-axis sensors, a new technology was introduced to provide integrated port logistics services.

The deep learning module of the non-GPU-based FPGA-based embedded vision system (TLEM), in which the entrance camera image data is input at the port gate of the port container terminal, recognizes lanes, vehicle numbers, and characters in images containing container numbers to enter and exit vehicles. Management information is transmitted to the terminal operating system (TOS) through middleware, and the terminal operating system (TOS) verifies this and issues a work order.

In the port container terminal, the FPGA-based embedded vision system (TLEM), equipped with a non-GPU-based deep learning module linked to a number of cameras, uses a deep learning module to improve the productivity and efficiency of loading and unloading. Character recognition for identification, container recognition, and vehicle number recognition. The container vehicle is equipped with a DR-GPS terminal equipped with a DR-GPS, 6-axis sensor, or 9-axis sensor, and the DR-GPS position of the container vehicle is transmitted to the terminal operation system (TOS) through middleware.

2A is a diagram showing the necessity of developing technology for improving the unloading and unloading productivity, safety and security of a port container terminal using video equipment.

The vision system to improve the productivity and efficiency of loading and unloading at the port container terminal uses the deep learning module of the non-GPU-based FPGA-based embedded vision system to detect objects and recognize characters (YT number, container number, vehicle number) ), and YT and XT vehicles are equipped with a DR-GPS system for position correction using DR-GPS and 6-axis sensors or 9-axis sensors.

This task is, first, from the port container terminal to the port gate entrance, intersection, Yard's block entrance, STS (Ship To Shore Crane) / TC (Transfer Crane), network cameras installed in hazardous areas, and non-GPU based FPGA Object detection and character recognition (YT number, container number, vehicle number) of camera image using deep learning module of the based embedded vision system (TLEM) and quantitative loading equipment location and ID detection/intruder detection through image analysis, hazardous area It builds an intelligent video system equipped with a deep learning module S/W that detects vehicles and workers and detects the number of people, notifies vehicles running backwards, detects safety speed, and detects whether or not the worker's hard hat/safety vest is worn.

Second, the deep learning module of the non-GPU-based FPGA-based embedded vision system (TLEM) detects vehicles and workers in dangerous areas and the number of people, detects vehicles in reverse driving, and detects safety and safety. It provides smart video surveillance that detects whether the vest is worn and provides worker safety prevention, and provides character recognition of YT numbers, container numbers (ISO codes), and vehicle numbers recognized by the deep learning module. The FPGA-based Embedded Vision System (TLEM) is connected to the Terminal Operation System (TOS) through middleware, and the Terminal Operation System (TOS) is an XT (external truck) or YT equipped with a DR-GPS terminal through the middleware. It receives the vehicle's vehicle number and DR-GPS location information, and displays the vehicle number and the corrected DR-GPS location of the vehicle.

Terminal Operation System (TOS) manages container vehicle/worker's work order issuance, port logistics management, container location management, container loading and unloading information, and customs information.

Figure 2b is a technical necessity for increasing productivity and cost reduction by minimizing the distribution time of port logistics by applying GPS position correction of container vehicles in port logistics and IoT technology based on FPGA-based embedded vision system (Smart Vision System), It is a diagram showing the economic effect of productivity improvement.

2C is a photograph of a port terminal Real-Time Location System (RTLS) system, a port terminal OCR system, and a port terminal RFID system.

3 is an overview of a vision camera system for vehicle access management, location tracking, and object recognition equipped with a DR-GPS terminal in a port container terminal according to the present invention: 1) Provision of real-time location information for device heads, 2) TC/ATC Container using (YT,XT ↔ Yard) vision system, Recognition of characters of Shuttle, YT number, vehicle number, container number, and vehicle correct position, 3) QC (ship ↔ XT, shuttle) container, Shuttle, YT, It provides text recognition, vehicle positioning, and lane positioning functions.

A vision camera system that recognizes vehicle access management and location tracking and objects (containers, vehicles, people, loading and unloading equipment) equipped with DR-GPS terminals at the port container terminal enables precise GPS location tracking and error correction at the port container terminal. DR-GPS terminal using 6-axis sensor/9-axis sensor with dead reckoning function, vehicle location tracking using CAN communication (container number, vehicle number, YT number, etc.), using deep learning of GPU-based vision camera system Detects objects in images by intelligent image analysis, recognizes containers, vehicles and people, and objects of loading/unloading equipment through feature extraction and classification of each object, detection of location and ID of loading/unloading equipment (intruder detection), and danger Through intelligent video analysis of local safety management, detection/notification of dangerous area workers and vehicles entering detection/notification and reverse driving vehicle notification, and detection/notification of whether the worker is wearing safety protective equipment, a notification message is transmitted to the terminal operation system (TOS) whenever an event occurs. Through the port speaker broadcasting of the terminal operation system (TOS), it transmits a voice signal to prevent the safety of port workers, prevents safety accidents between workers and vehicles, and recognizes containers and cranes in which STS Cranes in the container terminal work. It can be applied to lower worker safety management, etc., and is to improve unloading and unloading productivity in port container terminals.

In addition, by developing an existing GPS position error correction DR-GPS terminal, the speed of the vehicle using CAN communication with an antenna, DR-GPS and 6-axis sensor or 9-axis sensor at the port container terminal, and vehicle ECU and CAN communication. By correcting the position of the vehicle by referring to the signal or the acceleration signal of the vehicle, it is possible to accurately track the vehicle position.

At the port container terminal, by using a deep learning module installed in a camera or video storage server (NVR or DVR), it recognizes the objects of workers and vehicles in a hazardous area (work area YT), and whether or not the worker wears safety protection and enters the hazardous area worker and vehicle. The notification on whether or not is transmitted to the terminal operation system (TOS) of the real-time STS control room and operation room. In addition, at the port container terminal, it is possible to notify the operator to provide a warning broadcast when entering the work area (STS /TC /GATE) and dangerous areas without wearing safety gear, and it is possible to determine the vehicle reverse driving and driving speed. Port security and safety monitoring are possible.

4 is a diagram illustrating a comparison between an existing LPR and OCR recognition system and a GPU-based vision system in a port container terminal.

In addition, LPR (License Plate Recognition) (10 million won per lane), OCR (Optical character recognition) (70 million won per lane for port gate), and high-end GPUs used as vehicle detectors equipped with infrared emitters used in port container terminals The image recognition system based on (graphic processing unit) requires a complicated hardware configuration and expensive system construction cost, is difficult to be compatible with each reading software, and has different recognition rates (OCR cannot read other than text).

Table 3 compares the vision system of LPR, OCR, and the proposed deep learning technology.

5 is a port container terminal, an existing GPS/GNSS, DGPS, 2.4GHz BLE beacon/Wi-Fi gateway port location tracking system, and a location correction system used with a developed DR-GPS terminal and a 6-axis sensor/9-axis sensor. It is a comparison drawing.

Table 4 compares the vehicle location tracking system of the proposed technology using GPS/GNSS, DGPS, 2.4GHz Beacon, and a DR-GPS terminal (DR-GPS, 6-axis sensor or 9-axis sensor) equipped in the vehicle.

In the case of a port container terminal, when containers are loaded high, the GPS signal is cut off or a large error occurs in the corresponding GPS location due to the radio wave interference environment that causes total reflection compared to a tall building. To this end, a separate location correction hardware (HW ) And an algorithm are required.

In the port container terminal, the port YT vehicle uses a DR-GPS terminal equipped with a DR-GPS and a 6-axis sensor or a 9-axis sensor, and a CAN communication of a vehicle ECU, a CAN controller, and a CAN tranceiver. Vehicle ECU and CAN controller-Connected to the DR-GPS terminal through the CAN tranceiver. Even if the GPS signal of the DR-GPS terminal is cut off, the vehicle speed signal according to the time provided from the vehicle ECU is used by the Dead Reckoning function to position the vehicle. It corrects the GPS location of the vehicle more precisely than the existing GPS terminal, and provides a precise GPS location service with corrected location within the port.

The 6-axis sensor uses an acceleration sensor and a Gyro sensor.

The 9-axis sensor uses an acceleration sensor, a Gyro sensor, and a geomagnetic field.

In the port container terminal, position correction using 9-axis sensors and vehicle CAN signals (vehicle speed and vehicle acceleration signals) is performed using image recognition-based object identification (container, vehicle, and vehicle) using a deep learning module for smart port automation. Provides a solution for improving the efficiency and safety of container terminals with TOS system through human, loading and unloading equipment) and character recognition (YT number, container number, vehicle number) and position correction DR-GPS terminal. .

For reference, domestic port facilities are designated as security zones (detailed security measures for each security level of international voyage ship owners and port facility owners (related to Article 3, Paragraph 3)) Difficulty in securing image data samples for port container terminal due to various Yard structures (vertical, horizontal) and transport equipment (STS, TC, YT) in selecting a camera location.

6 is a diagram showing a technology roadmap for the application field of Gate system/TC system/QC system/Block Entrance/Safety for next-generation future smart ports.

(1) Port gate system:

(As Is) Recognition of entering vehicle using LPR, vehicle recognition using RFID, Barcode + slab -> (To Be) Deep learning based AI object and character recognition Vision System (vehicle number recognition, container number recognition, block entering vehicle recognition, Yard worker safety equipment detection, accidental and dangerous situations detection)

(2) TC (Transfer Crane) system: (As Is) RFID vehicle detection -> (To Be) Deep learning based AI object and character recognition Vision System (vehicle number recognition, container number recognition, block entry vehicle recognition, yard worker safety Detecting whether or not the sphere is worn, detecting unexpected and dangerous situations)

(3) QC(Quay Crane) system: (As Is) Under Man's vehicle and container number visual recognition -> (To Be) Deep learning based AI object and character recognition Vision System (vehicle number recognition, container number recognition, block entry) Vehicle recognition, detection of wearing safety gear for yard workers, detection of unexpected and dangerous situations)

(4) Block Entrance: (As Is) Block Enter by RFID -> (To Be) Deep Learning based AI object and character recognition Vision System (Vehicle number recognition, container number recognition, block entry vehicle recognition, Yard worker wearing safety equipment) Detection, detection of unexpected and dangerous situations)

A region of interest in the container vehicle license plate or a still image of the container number attached to the container or the container vehicle license plate taken by the camera by the character recognition program of the image after saving the image of the container number or the container number attached to the container Container vehicle using vehicle number/container number recognition technology by image processing by extracting a container vehicle license plate of (ROI, Region Of Interest) rectangular structure or a still image (blob) of the container number attached to the container. A vehicle number consisting of letters and numbers is extracted from a license plate or a still image of a container number attached to a container to recognize characters.

(5) Safety: (As Is) patrol by safety manager, intuitive judgment of workers

-> (To Be) DR-GPS terminal equipped with precision DR-GPS and 6-axis sensor (acceleration sensor, Gyro sensor) or 9-axis sensor (gyro sensor/acceleration sensor/geomagnetic field) and CAN communication with ECU of automobile Real-time location tracking of provided vehicles

7A is a diagram illustrating a real-time object recognition of a camera image using deep learning: the concept of convolutional neural networks (CNN), recurrent neural networks (RNN), and long-short term memory (LSTM) applications.

Deep learning modules include CNN (Convolutional Neural Network), RNN (Recurrent Neural Networks), R-CNN (Recurrent Convolutional Neural Network), Fast RCNN, and Faster RCNN (Region based Convolutional Neural Network), YOLO (You Only Look Once), Objects in the camera image are detected using any one of SSD (Single Shot Detector) algorithms.

○ Representative CNN (Convolutional Neural Networks) of image recognition detects objects (vehicles, containers, loading and unloading equipment) in images of cameras, or character recognition (YT number recognition, vehicle number recognition, container number recognition) that extracts numbers/letters. ) It is used as a technology, and if sufficient learning data is secured, it shows a high level of recognition accuracy.

CNN is composed of one or several convolutional layers and artificial neural network layers on top of it, and additionally utilizes weights and pooling layers.Thanks to this structure, CNNs can fully utilize the input data of a two-dimensional structure. Use.

The CNN algorithm includes an input layer/L-1 hidden layer (Layer 1, Layer 2, Layer3.. )/output layer from the camera input image, among AlexNet, VGGNet, and ResNet. Any one CNN deep learning algorithm can be used.

Objects (containers, vehicles, people, loading/unloading equipment, etc.) are detected by feature extraction of camera images and classification of objects through a multilayer perceptron using a multilayered convolutional neural network (CNN). .

In the convolutional neural network (CNN), three layers including a convolutional layer, a pooling layer, and a fully connected layer (FC) are used.

The CNN algorithm reduces the amount of data in the image by repeating convolution and subsampling by moving a weighted mask (e.g., 3x3 window) and by convolutional and pooling layers, respectively. Features strong against distortion of images (feature extraction), feature maps are extracted by convolution, and objects (containers, vehicles, people, loading and unloading equipment, etc.) are extracted by neural networks. Classification.

In image processing using the CNN algorithm, convolution is image processing using a weighted mask (e.g., 3x3 window), and a weighted mask (e.g., 3x3 window, 5x5 window) is applied to the input image. Moving on, a mask having a weight is applied to the current input image, and the pixel value of the input image is multiplied by the weight of the mask, and the sum is determined as the pixel value of the output image.

Subsampling is the process of reducing the screen size, and max pooling is used to select the maximum value of the area.

The FC layer (fully connected layer) is connected to the input terminal of the neural network and classifies the objects of the image (containers, vehicles, people, and loading equipment) according to learning (learninf).

For example, it currently consists of a convolution layer of 5 layers and a fully_connected layer of 3 layers.

The output of the convolutional layer is subsampling by the Max-Pooling Layer to reduce the size of the image, and the output of the Max-Pooling classifies the object class in the FC layer (Fully Connected Layer).

As a result, the port container terminal detects the object of the image of a plurality of cameras and extracts a feature map including object location region and type information from several convolutional layers in the middle of the CNN structure, As it passes through the pooling layer, the size of the feature map decreases, and objects are detected by extracting object location area information from feature maps of different sizes, and objects are detected by the FC layer (Fully Connected Layer). Classify them.

Additionally, in order to learn object detection and object tracking of the camera image of the port container terminal, it may be learned using a database for object detection and object tracking in the image.

Additionally, the image object tracking algorithm may use a Kalman filter algorithm when processing camera images. The Kalman filter algorithm is used in computer vision and robot vision, and corrects errors from past and present data from measurement models and process models in a linear system to determine the state of future movements. It is used as an in-video object tracking algorithm that is predicted and estimated.

○ Representative technologies for video recognition are RNN (Recurrent Neural Networks) and LSTM (Long-Short Term Memory), because the camera image is a series of images with a time axis, the CNN output value is connected to the LSTM to generate a sentence describing the image. Video can be analyzed.

7B is a diagram illustrating a real-time object recognition: You Only Look Once (YOLO) model of a camera image using deep learning.

The deep learning algorithm uses artificial intelligence/deep learning vision technologies such as YOLO (You Only Look Once), SSD (Single Shot Detector), etc., which detect objects in images, using a framework development, local embedded image processing system development, 9 axis By developing a precise DR-GPS position correction technology using a sensor, it is used as an application technology for the automation of quantification and optimization of a port container terminal for a future smart port using an AI-based vision system in the port container terminal.

○ YOLO (You Only Look Once) model for real-time object recognition of camera images using deep learning

-YOLO divides each image into SxS grids (grid, bounding box), calculates the reliability of each grid, reflects the accuracy when recognizing objects in the grid, and classifies the class by viewing the entire image at once. It has twice the performance of other models by processing. To calculate whether an object is included in the grid, the object class score is calculated. As a result of this, a total of S x S x N objects is predicted.

-The SSD (Single Shot Detector) model, which is similar to YOLO and shows better performance, has a unique advantage of a balance between the speed and accuracy of object detection in the image. SSD is a feature map even after executing the CNN for the input image only once. map) calculation is possible to detect objects of various scales.

SSD is an artificial intelligence-based object detection algorithm that has a balance between object detection speed and accuracy in which a grid for object detection in a camera image is displayed. The SSD executes a convolutional neural network (CNN) for the input image only once and calculates a feature map. In order to predict the probability of grid and object classification, CNN is performed on the feature map with a 3 × 3 filter size. The SSD predicts a grid after CNN processing. This method can detect objects of various scales.

The FPGA-based embedded vision system (TLEM) is equipped with a deep learning module that detects non-GPU-based image objects and recognizes characters of vehicle numbers and container numbers.

It can be used as a portable location tracking device combining object and character recognition of camera image data of deep learning module of FPGA-based embedded vision system (TLEM), DR-GPS module + Wi-Fi communication unit of container vehicle.

(Example)

FIG. 8 is a diagram showing a final development goal of interlocking with a terminal operating system (TOS) through an FPGA-based vision system (TLEM) and middleware in a port container terminal.

9A is a diagram showing a framework of (1) a non-GPU-based object recognition deep learning module SW and an FPGA-based embedded vision system (TLEM) in a port container terminal.

FIG. 9B is a view showing an integrated middleware for connection with (2) a DR-GPS module applying high-precision dead reckoning using a 9-axis sensor, and (3) a terminal operating system (TOS) in a port container terminal to be.

9C is a block diagram of an integrated middleware for linking with a terminal operating system (TOS).

FIG. 10 is a configuration diagram of a vision camera system for vehicle access management, location tracking, and object recognition of a camera image equipped with a DR-GPS terminal (DR-GPS and 6-axis sensor/9-axis sensor) in a port container terminal according to the present invention to be.

11 is a block diagram of a vision camera system for managing vehicle access, tracking location, and recognizing an object of a camera image in a port container terminal according to an embodiment of the present invention.

The vision camera system for vehicle access management, location tracking, and object recognition in a port container terminal of the present invention

Multiple cameras installed in each area of Port Gate, Yard/Block, Block Entrance, ARMGC, and QC at the Port Container Terminal;

In the port container terminal, a deep learning module using AI algorithm is installed, and based on the pre-learning data of container/vehicle/person/loading equipment, the port gate, yard/block, block entrance, ARMGC, and QC camera image data are Detects image objects (containers, vehicles, people, loading/unloading equipment) through object feature extraction and classification, lane recognition, vehicle number recognition, container number (ISO code) character recognition, container damage recognition, block entrance Recognition of entering vehicles, detection of vehicles and workers entering dangerous areas, detection of vehicles traveling in reverse, detection of wearing safety protective equipment/safety vests for yard workers, detection of the location of loading and unloading equipment, and transmission to middleware whenever an event is detected, or directly in the port through GPIO FPGA-based embedded vision system (TLEM) 100 for transmitting a warning broadcast to the speaker (IP speaker); And

A monitoring PC that interlocks with heterogeneous hardware and software and receives the object extraction event (text) learned through TLEM deep learning middleware from the FPGA-based embedded vision system (TLEM) 100 and the object extraction image data marked with a square box. It is to be output to, and when an event is detected, it broadcasts a warning to the corresponding speaker (IP speaker) in the port through TCP/IP, and includes a middleware 200 having edge-middleware and integration-middleware,
The FPGA-based embedded vision system (TLEM) includes a deep learning module that detects non-GPU-based image objects and recognizes characters of vehicle numbers and container numbers (ISO codes).

The FPGA-based embedded vision system (TLEM) 100, which is connected to a plurality of cameras, is interlocked with the terminal operating system (TOS) 300 through the middleware 200, and the terminal operating system of the STS control room/operation room ( TOS) (300) is connected to the monitoring PC (310).

Multiple cameras (C)(101) are used at the port gate entrance, Yard's block entrance, ATC (ARMGC), intersection, STS (Ship To Shore Crane)/TC (Transfer Crane), and various port container terminals such as QC hazardous areas. It is installed in the area.

The image storage server 102 uses an NVR or DVR.

The NVR (Network Viedo Recorder) supports ONVIF and RTSP standard protocols that connect multiple cameras (C) at the same time. In addition, when an image storage server (NVR or DVR) is not used, the camera interface that connects one camera that separately monitors a specific area and the control PC is a frame grabber, Gigabit Ethernet (GigE), IEEE 1394a, Any one of IEEE 1394b, Camera Link, or USB 3.0 module can be used.

The FPGA-based embedded vision system (TLEM) 100 is equipped with a license plate/container number character recognition program and a deep learning module 110, and is directly connected to multiple cameras (C) 101 and via an HDMI interface, or It is connected to a video storage server (NVR or DVR) 102 through a plurality of cameras 101 installed in the device and TCP/IP, connected to a monitor through an HDMI interface, and connected to the middleware 200 through TCP/IP. .

The deep learning module 110 is a CNN (Convolutional Neural Network), RNN (Recurrent Neural Networks), R-CNN (Recurrent Convolutional Neural Network), Fast RCNN, and Faster RCNN for real-time object recognition based on deep learning of camera image data. Region based Convolutional Neural Network), YOLO (You Only Look Once), or SSD (Single Shot Detector) algorithms, using any one of deep learning algorithms to detect objects through feature extraction and fractionation of objects in the camera image do.

The deep learning module 100 detects objects in the camera image using the deep learning algorithm, and extracts and classifies features of the objects marked as a grid, so that the objects of containers, vehicles and people, and loading and unloading equipment are detected. Recognition, lane recognition, vehicle number recognition, container number (ISO code) text recognition, container damage recognition, block entrance vehicle recognition, hazardous area entry vehicle and worker detection, reverse driving vehicle detection, yard worker safety protection/safety vest It detects the location of detection, loading and unloading equipment, and transmits it to the middleware 200 whenever an event occurs, and transmits a remote warning broadcast to the speaker through the GPIO interface of the FPGA-based embedded vision system (TLEM) 100 whenever an event occurs. .

The edge middleware includes a TLEM deep learning middleware that receives an object extraction event (text) learned from the FPGA-based embedded vision system (TLEM) 100 and object extraction image data marked with a square box; DR-GPS middleware for receiving vehicle number and DR-GPS location of XT and YT vehicles; And a speaker broadcasting middleware that transmits a remote warning broadcast to the corresponding speaker (IP speaker) through TCP/IP whenever an event occurs.

The edge middleware may have a 900MHz RFID reader installed at each block entrance of the Yard according to the required location, and further include RFID middleware that receives 900MHz RFID tag information of the vehicle detected by each RFID reader. have.

Referring to FIG. 9C, the middleware 200 includes edge middleware and integrated middleware for linking the terminal operating system (TOS) 300.

One side of the middleware 200 is connected to the FPGA-based embedded vision system (TLEM) 100, and the other side is connected to the terminal operating system (TOS) 300.

Middleware 200 is

Edge providing container number, vehicle number, YT number, lane location, sub-device connection, near field event response (speaker warning signal transmission), and camera image data compression collected by the FPGA-based embedded vision system (TLEM) 100 -Middleware (Edge-Middleware); And

It collects geographically distributed Edge-Middleware data from the port container terminal, and includes integration-middleware that connects with logs, message routers, and terminal operating systems (TOS).

Middleware 200 is

Adapters and TOS linkage modules for heterogeneous software and hardware equipment used in the port container terminal system are provided.

Edge middleware management and control module (Edge M/W daemon management, Edge M/W status management, Edge Registry), log management module, recognition information location information module (Proxy), module and control module (module information management, module status management) , Module command processing), an ALE service engine (ALE Interface, event scheduler, Filtering & Reporting, location information management, and Event Cycle management); And

Connected to the edge-middleware, administrator browser, system monitoring and management module (log management, system monitoring, M/W management), system resource protection module (security setting and key management, Secure Notification, Edge M/W access control) , Including an integration-middleware including a management channel module in TCP/IP communication with the DB server of the terminal operating system (TOS).

The terminal operating system (TOS) 300 includes a middleware UI, an event viewer, and a real-time vehicle monitoring program.

For access control of port container terminals, XT and YT vehicles use DR-GPS terminals equipped with antennas, DR-GPS and 6-axis sensors or 9-axis sensors. Even if the GPS signal is cut off, the DR-GPS terminal is a CAN tranceiver. -It provides CAN communication to receive vehicle speed signal or vehicle acceleration signal according to time between CAN controller and vehicle ECU, and corrects the position of the vehicle using 6-axis sensor/9-axis sensor.

The container-mounted vehicle is equipped with a GPS antenna, a DR-GPS terminal (DC 12~24V), a speed indicator and an alarm connected to the DR-GPS terminal and signal cable, and the DR-GPS terminal communicates with the P-LTE router. It is connected to the location location server of the Terminal Operating System (TOS).

The vehicle uses an antenna, a DR-GPS, and a DR-GPS terminal having a 6-axis sensor or a 9-axis sensor, and the 6-axis sensor uses an acceleration sensor and a Gyro sensor, or the 9-axis sensor is an acceleration sensor and a Gyro sensor. Using a sensor and a geomagnetic field,

The vehicle's ECU is connected to the DR-GPS terminal through a CAN controller-CAN tranceiver, and provides CAN communication with the DR-GPS terminal, and the DR-GPS terminal communicates with the vehicle ECU by a dead reckoning function even when the GPS signal is cut off. Using the vehicle speed signal or vehicle acceleration signal over time through CAN communication, the corrected GPS position of the vehicle in the port container terminal is corrected by using a 6-axis sensor/9-axis sensor, and the terminal operating system (via DR-GPS middleware) TOS).

The port terminal operation system (TOS) 300 is interlocked through the FPGA-based embedded vision system (TLEM) 100 and the middleware 200, and objects (container, Video data including vehicles, people, loading/unloading equipment, etc.) are output on the screen of the monitoring PC 310, and vehicle numbers and DR-GPS locations of XT and YT vehicles are displayed on the Yard map of the port container terminal.

In addition, a vehicle equipped with an antenna, a DR-GPS, and a DR-GPS terminal using a 6-axis sensor or a 9-axis sensor can display the location of the vehicle on a 2D or 3D GIS map of the port container terminal.

The port terminal operation system (TOS) 300 is provided with a port logistics information system, and may be interlocked with a port speaker broadcasting system.

For example, when detecting a hazardous area worker or vehicle at a port container terminal, the TLEM or middleware sends a remote warning broadcast ("This is a dangerous area where access is prohibited, please leave the dangerous area") to the corresponding speaker (IP speaker). do.

For example, upon detection of a reverse driving vehicle, the TLEM or middleware sends a remote warning broadcast ("this is an area where reverse driving is prohibited") to the corresponding speaker.

For example, when detecting that the worker's safety gear is not worn, the TLEM or middleware sends out an announcement ("Please wear safety gear in the work area") to the speaker.

12 is an FPGA-based embedded vision system (TLEM) equipped with a deep learning module (YOLO or SSD) that detects an object of a camera image and recognizes characters of an image of an image including a vehicle number and a container number (ISO code). It is a block diagram of.

The FPGA-based embedded vision system (TLEM) 100 equipped with a deep learning module that detects objects in camera images and recognizes characters

A deep learning module for recognizing an object of camera image data and an FPGA equipped with an MCU (CPU);

It is connected to the deep learning module and the FPGA on which the MCU is mounted, and the deep learning module processes the image by character recognition of the image of the image including the object detection, vehicle number, container number (ISO code), and YT number. DSP to do;

A video interface unit including an encoder/decoder of a video codec, which is connected to the deep learning module and the FPGA on which the MCU is mounted, and receives and outputs image data from the camera C through HDMI 2.0 Rx/Tx;

A flash memory and SDRAM connected to an FPGA on which the deep learning module and MCU are mounted;

A USB-UART terminal unit connected to the deep learning module and the FPGA on which the MCU is mounted, and connecting USB;

A GPIO interface unit connected to the deep learning module and the FPGA on which the MCU is mounted, and connecting an external speaker (IP speaker) through a GPIO interface;

An Ethernet connection part connected to the deep learning module and the FPGA on which the MCU is mounted;

A Wi-Fi communication unit connected to the deep learning module and the FPGA on which the MCU is mounted; And

It includes a DC power supply that supplies DC 12V ~ DC 24V power.

The FPGA-based embedded system (TLEM) 100 includes an input unit, a display unit, a control unit (MCU) equipped with a deep learning module (eg, YOLO, SSD), a storage unit, and a wired/wireless communication unit (Giga LAN Ethernet connection, USB cable connection, Wi- It is manufactured as a small IoT device equipped with a Fi communication unit).

The FPGA-based embedded vision system (TLEM)

It is connected to the deep learning module and the FPGA on which the MCU is mounted, and further includes an SD card connection to which the SD card is detachably mounted/separated.

The FPGA-based embedded vision system (TLEM) is connected to a speaker through a GPIO interface, and the speaker stores pre-stored voice data, and the corresponding voice data is output according to a remote broadcast signal of the FPGA-based embedded vision system (TLEM). It is characterized by being an IP speaker.

* Deep Learning Module: Deep Learning Object Recognition SW (YOLO V3, SSD)

-Create deep learning files for each port gate, block entrance, TC, and QC

-Comparison and test of YOLO V2/V3, SSD performance according to hardware calculation

-Deep learning module SW for Port Gate, Block Entrance, TC, QC

-Port Gate, Yard Block, Block Entrance, ARMGC vehicle and worker detection, container number (ISO code) recognition, container damage, YT recognition, vehicle number recognition, line classification, yard worker's hard hat/ Deep learning module to detect whether a safety vest is worn

-Creation of deep learning learning files for loading small local devices according to vehicle, human, container, and loading equipment application factors and recognition rates

* Small framework running on FPGA-based embedded vision system (TLEM)

-Development of a framework for object detection and image processing of character recognition using deep learning of camera image data through storage media and real-time video streams transmitted from image storage devices (NVR, DVR) (OpenCV, DirectShow)

-Implementation of ONVIF, RTSP standard protocol for simultaneous connection of multiple cameras (C) to NVR (Network Viedo Recorder)

* Edge middleware and API (Application Programming Interface) design

-Development of edge-middlewre for connecting heterogeneous devices (HW/SW)

(TLEM deep learning middleware, RFID middleware, DR-GPS middleware that receives vehicle number and DR-GPS location, speaker broadcasting middleware)

-Developed API for linkage between DR-GPS terminal and TLEM (Tiny Local Embedded Module)

-IoT protocol definition and development for MQTT communication with Integration middleware

* Design and weight reduction of CNN (Convolutional Neural Network) algorithm

* Research and development of miniaturization of the network layer for real-time video recognition of specific objects

-By classifying only the objects specified in the port, the learning time is reduced and the HW specification is reduced.

Research to drive

* Development of DR-GPS module with high precision dead reckoning using 9-axis sensors (acceleration sensor/Gyro sensor/ geomagnetic field) mounted on a vehicle

* Development of integrated middleware and API for interworking with terminal operation system (TOS)

* Event viewer and monitoring SW of Terminal Operation System (TOS)

(1) Deep learning module object recognition and image processing SW and small FPGA-based embedded vision system (Tiny Local Embedded module, TLEM) development

-Low-cost/small FPGA-based embedded vision hardware (IoT device) development

-Development of object recognition software for real-time deep learning module of camera image data

-Development of optimal learning data through prior learning of CNN algorithm objects

-Develop middleware that receives the learned object extraction event (text) from the FPGA-based embedded vision system (TLEM) and object extraction image data marked with a square box and outputs it to a monitoring PC, and broadcasts a warning to the speaker when an event occurs.

-Development of an algorithm to determine whether the worker is safe or not (whether wearing a helmet/fluorescent vest) in real-time video

-Vehicle reverse driving, detection of worker collision according to driving direction, vehicle number recognition of YT/XT, lane recognition, container number (ISO code) recognition, container damage recognition

(2) Development of DR-GPS module applying high-precision dead reckoning using 9-axis sensor

-In an environment where the GPS signal is not received under the TC (Transfer Crane) or QC (Quay Crane) and the location error due to multi-path caused by containers stacked high because containers are loaded on the Yard block in the port. Development of a DR-GPS terminal capable of precise location positioning of vehicles in ports by using axis sensors, vehicle CAN signals, and vehicle speed signals

-Development of a quick and fast return algorithm when GPS reception is unavailable during initial operation

(3) Development of integrated middleware for connection with Terminal Operating System

-After object recognition and image processing of the image of the deep learning module of the FPGA-based embedded vision system interlocked with multiple cameras (C) or image storage devices (NVR or DVR),

-Development of DR-GPS middleware that transmits the location of a vehicle equipped with a DR-GPS terminal to the terminal operation system (TOS) through the middleware

-Development of monitoring program and middleware UI for container terminal service

④ Construction of a container terminal demonstration test bed in Busan New Port (Example)

-TOS system interlocking module development

-Vision system installed in QC 1 unit, DR GPS installed on 5 YT units, and then interlocked with TOS system

o Final product

(H/W)-for deep learning module's image processing framework

FPGA-based embedded vision system (Tiny Local Embedded Module, TLEM)

-Development of DR-GPS terminal module mounted on container vehicles (XT, YT vehicles)

(S/W)-Object recognition SW of camera image data of deep learning module

-Deep learning learning program for object recognition of camera image data

-Integrated middleware and API for interworking with Terminal Operation System (TOS)

-Event viewer, monitoring SW

[Detailed development goals]

(1) Deep learning-based object recognition software and small FPGA-based embedded system (Tiny Local Embedded module, TLEM) development

a. Image processing technology development

-Development of a multimedia framework for working with video files acquired from port container terminals or real-time video streams transmitted from servers or video devices [VMF (Video for windows), OpenCV, DirectShow]

-Development of technology to determine whether or not to match by comparing the stored pattern and the input image, pre-processing, post-processing, object feature point extraction, pattern recognition and camera image data compression and storage, from camera images

-Vehicle number, container number character extraction and container damage image extraction

b. Deep learning technology development

-CNN learning modeling selection and module development for extracting and classifying image feature points of camera images using convolution operation

-AlexNET, VGGNet, GoogLeNet, RestNet, Yolo, SSD module development

-Deep learning modeling comparison/selection and learning module development capable of real-time recognition for character recognition and lane classification in the character image of container number and vehicle number

-Generation of learning data for the worker's hard hat and safety vest wearer/non-wearer by deep learning algorithm

-Improvement of recognition rate of objects of camera image data and generation of learning data for structured and unstructured data for character recognition

c. Multimedia framework, FPGA-based embedded vision system (TLEM) deep learning module development

(2) Development of DR-GPS module applying high-precision dead reckoning using 9-axis sensor

-Development of a position correction algorithm using GPS and GNSS satellite signals, 3-axis gyro sensor, 6-axis gyro sensor + acceleration sensor, 9-axis gyro sensor + acceleration sensor + geomagnetic sensor

-The vehicle's shift lever and the vehicle speed signal over time are transmitted through CAN communication.

Received and developed position correction algorithm

-Development of low power algorithm for built-in battery and WARM & HOT maintenance algorithm of GPS satellite reception when vehicle power supply is not supported

(3) For interworking with Terminal Operating System

Integrated middleware development [Fig. 9c]

a. Edge-Middleware development

-Edge-Middleware (Edge M/W): TLEM deep learning middleware, RFID middleware, DR-GPS middleware receiving vehicle number and DR-GPS location, speaker broadcasting middleware

-Container number, vehicle number, YT number, lane location, sub-device connection detected by FPGA-based embedded vision system (TLEM), warning signal speaker transmission for short-range field events, camera image data compression.

b. Integration-Middleware development

-Collects distributed Edge-Middleware data and connects logs, message routers, and TOS

c. Developed adapters for heterogeneous equipment (HW/SW) used in port container terminal systems and developed TOS linkage modules

d. Middleware UI, event viewer, real-time vehicle monitoring program development

(4) Construction of a container terminal demonstration test bed in Busan New Port (Example)

-TOS system interlocking module development

-Vision system installed in QC 1 unit, DR GPS installed on 5 YT units, and then interlocked with TOS system

o Main performance values

1) Container number, vehicle number, YT number recognition rate

2) recognition speed

3) Position correction rate when GPS shaded area or Dead Reckoning is applied

4) Yard worker's hard hat/safety vest detection,

Vehicle/worker collision risk detection rate

5) Efficiency compared to previous times when building a test bed

In the embodiment, the new port container demonstration test bed was interlocked with the TOS system after the development of the TOS system interlocking middleware, the vision system installation in the 1st QC, GPS antenna and DR-GPS in 5 YTs.

QC internal container by cameras installed on the left and right for each lane such as LANE1 and LANE2 according to the cameras installed to photograph the front, rear, top and side of the vehicle entering/exiting each lane such as LANE1 and LANE2 at the port gate Side ID recognition, container rear door ID recognition and vehicle number recognition, container side left ID recognition, top ID and vehicle number recognition. In addition, ISO code characters indicated in container numbers of vehicles entering each QC lane are recognized by image processing from images captured by cameras for each camera installation location.

The vision recognition method of the vision camera system that recognizes vehicle access management and camera image objects at the port container terminal

(a) In the port container terminal, the FPGA-based embedded vision system (TLEM) interlocks with a number of cameras (C) installed in each area of the port gate, yard/block, block entrance, ARMGC, and QC for each camera image data. By deep learning module (YOLO, SSD, etc.), according to the pre-learning data of the objects, it is marked as a grid through feature extraction and classification of the objects of the port gate, Yard/Block, Block Entrance, ARMGC, and QC camera image data. Detecting the objects (containers and vehicles and people, loading and unloading equipment);

(b) The FPGA-based embedded vision system (TLEM) recognizes lanes, vehicle number recognition, character recognition of image images including container number (ISO code), container damage recognition, vehicle entering block entrance, vehicle entering dangerous area and Detecting a worker, detecting a reverse driving vehicle, detecting whether a yard worker is wearing a safety protective device/safety vest, and detecting a position of a loading and unloading equipment; And

(c) whenever an event occurs, transmitting a warning broadcast signal directly from the FPGA-based embedded vision system (TLEM) to the corresponding speaker through the GPIO interface.

The method includes (d) transmitting the object extraction event (text) learned from the FPGA-based embedded vision system (TLEM) and the object extraction image data marked as a grid to middleware (TLEM deep learning middleware) whenever an event occurs. And transmitting, by the middleware, a warning broadcast signal to a corresponding speaker using TCP/IP.

The method is (e) The FPGA-based embedded vision system (TLEM) is interlocked with the terminal operating system (TOS) through middleware (edge middleware, integrated middleware), and a GPS antenna and DR-GPS, 6-axis sensor or 9-axis sensor It further comprises the step of transmitting and outputting the location of the vehicle equipped with the DR-GPS terminal using the DR-GPS middleware to the port terminal operating system (TOS).

The terminal operation system (TOS) 300 is linked with a middleware UI, an event viewer, a real-time vehicle monitoring program, and a port logistics management system.

For access control of port container terminals, XT and YT vehicles are equipped with a GPS antenna, DR-GPS, and a DR-GPS terminal using a 6-axis sensor or a 9-axis sensor. The GPS terminal provides CAN communication for receiving vehicle speed signals or vehicle acceleration signals between CAN controller and vehicle ECU.

The container-mounted vehicle is equipped with a GPS antenna, a DR-GPS terminal (DC 12~24V), a speed indicator and an alarm connected to the DR-GPS terminal and signal cable, and the DR-GPS terminal communicates with the P-LTE router. It is connected to the location location server of the Terminal Operating System (TOS).

The vehicle uses a GPS antenna, a DR-GPS, and a DR-GPS terminal using a 6-axis sensor or a 9-axis sensor, and the 6-axis sensor uses an acceleration sensor and a Gyro sensor, or the 9-axis sensor is an acceleration sensor and a Gyro sensor and geomagnetic field are used,

The vehicle's ECU is connected to the DR-GPS terminal through a CAN controller-CAN tranceiver, provides CAN communication with a DR-GPS terminal using a 6-axis sensor or a 9-axis sensor, and the GPS signal of the DR-GPS terminal is cut off. Also, the dead reckoning function corrects the precise GPS position in the port using the vehicle speed or vehicle acceleration signal through CAN communication with the vehicle ECU, and transmits it to the terminal operating system (TOS) through the DR-GPS middleware.

The port terminal operation system (TOS) 300 is interlocked through the FPGA-based embedded vision system (TLEM) 100 and the middleware 200, and objects (container, Video data including vehicles, people, loading/unloading equipment, etc.) are output on the screen of the monitoring PC 310, and vehicle numbers and DR-GPS locations of XT and YT vehicles are displayed on the Yard map of the port container terminal.

In addition, a vehicle equipped with a GPS antenna, a DR-GPS, and a DR-GPS terminal using a 6-axis sensor or a 9-axis sensor can display the location of the vehicle on a 2D or 3D GIS map of the port container terminal. In addition, it is possible to detect the position of the container value by the camera image captured in real time.

13 is a view showing quantitative evaluation items of an FPGA-based embedded vision system (TLEM) equipped with a non-GPU-based deep learning module to improve port safety and productivity and efficiency of loading and unloading container terminals.

Quantitative evaluation items include deep learning recognition performance (1. number of recognized objects, 2. recognition success rate, 3. maximum recognition distance, 4. recognition speed 5. camera recognition resolution), deep learning recognition module environmental test (6. operation temperature, 7. Thermal Shock, 8. Waterproof Discharge, 9. Radio Wave Certification (KC)), Precision DR-GPS Module (10. Operating Temperature, 11. Data Update Time, 12. Position Error), TOS Linked Middleware (13. Middleware Software Evaluation) , 14. Host & IoT Interface compatible type), and is evaluated with an official test report or radio wave certification report.

Since its establishment in 2012, the company has developed/built an automation system for domestic and overseas port container terminals, and has a track record of delivering HW/SW to terminals, access control, and port integrated logistics management systems.

In the case of overseas port container terminals, the multi-pass phenomenon does not occur because containers are not loaded high due to the large site.

In the case of domestic port container terminals, diffuse reflection is severe due to the loading of 4-6 containers, and errors due to earth rotation, shading and diffuse reflection due to loaded containers exist.

14 is a real-time camera image data and location information service screen in a port container terminal.

Figure 15 is a port container terminal test image: (1) GPS only, (2) GPS + 9-axis sensor + location correction GPS test that can distinguish lanes at the port container terminal using a position correction algorithm-The distance between blocks of the horizontal structure Yard is It is 20~24m interval.

At the port container terminal, a location correction GPS test was performed that enables the terminal operation system (TOS) and the lane division of the horizontal structure yard to be distinguished.

In Incheon New Port Container Terminal, when only GPS terminal is used (GPS only), the distance between blocks of horizontal Yard is divided into 20 to 24m intervals, two lanes on the sea side and two lanes on the land side. For this, a location positioning system capable of distinguishing lanes was needed.

In the case of using DR-GPS + 9-axis sensor + position correction algorithm, GPS test for position correction of vehicles in port using EV KIT equipped with Ublock chipset was conducted. / There was a maximum of approximately 43m coordinate difference in accuracy.

16 is a chart showing the commercialization performance of the Port GPS system (vehicle tracking), Port RFID system (vehicle recognition), OTR device (vehicle location recognition), and BLE Beacon System (IoT device) by year in a port container terminal.

17 is a diagram showing a commercialization model business strategy and plans to enter overseas markets.

18 is a view showing the current status of the loading/unloading equipment of the domestic container terminal and the trend of global cargo volume.

19 is a diagram showing the annual demand forecast of domestic and overseas automated container terminals.

The vision camera system for vehicle access management, location tracking, and object recognition in a port container terminal of the present invention includes vehicle access management and a DR-GPS terminal (DR-GPS, 6-axis sensor or 9-axis sensor) in the port container terminal. Vehicle location tracking (YT number, container number, vehicle number, etc.), feature extraction and classification of objects in camera images according to deep learning learning results by deep learning module of FPGA-based embedded vision system (TLEM) linked with multiple cameras It detects container vehicles and people through the port gate/Yard Block/Block Entrance, ARMGC vehicle detection, hazardous area worker and vehicle detection, reverse driving vehicle, whether the worker wears safety protective equipment/safety vest, and is quantitatively unloaded by QC. Container number (ISO code) extraction, recognition of vehicle location information (X,Y coordinates) equipped with DR-GPS, object extraction event (text) and square box learned as middleware from FPGA-based embedded vision system (TLEM) It transmits the object extraction image data marked with and transmits it to the terminal operation system (TOS), and the TLEM or middleware reports safety accidents between workers and vehicles through the local speaker broadcast provided in the Yard block of the port container terminal. It provides safety management in hazardous areas to prevent, and provides vehicle number recognition in container terminals, container number (ISO number) character recognition that STS Crane works on, and safety management of workers under the crane. The location information (X,Y coordinates) of container vehicles (XT vehicles, YT vehicles) equipped with DR-GPS are transmitted to the terminal operation system (TOS) through middleware, and the terminal operation system (TOS) is DR-GPS in real time. By providing the location information (X, Y coordinates) of the container vehicle (XT vehicle, YT vehicle) equipped with, in real time, the container vehicle is provided in the number of blocks of which yard. Through this, the automation system of the port container terminal was established to improve the unloading and unloading productivity.

In addition, in the port container terminal, in the environment where GPS signals are not received under the position error due to multi-path and the TC (Transfer Crane) or QC (Quay Crane) caused by the containers stacked high by loading containers, DR-GPS terminal by developing a DR-GPS terminal (using DR-GPS, 6-axis sensor or 9-axis sensor) that can precisely position the vehicle in the port by using the 9-axis sensor, the vehicle's CAN signal and the vehicle's speed signal. Precision location tracking (YT number, container number, vehicle number recognition) in the port of vehicles using and CAN communication became possible.

This product (TLEM) has a price competitiveness compared to RFID, OCR, and LPR in port container terminals, and has a performance advantage in object recognition of images from deep learning modules, and minimizes and maintains the number of equipment by integrating various functions into one function. It reduces maintenance costs and can be supplied at a price of less than 50% of the existing port access security system, container terminal, and transport equipment (Quay Crane, Yard Crane Rail Mounted Gantry Crane, Yard Tractor, Yard Chassis, Yard Chassis, Forklift Fork Lift, Reach Stacker Reach Stacker, Ampri Handler Empty Handler), entry lane recognition, port container terminal access management/control, and Yard worker's hard hat/safety vest detection to provide a multifunctional vision system specialized for port safety/security. Applicable to port automation system. In addition, it is possible to enter the overseas market of port automation terminals through joint sales through overseas TOS developers.

The embodiments according to the present invention may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable recording medium. The computer-readable recording medium may include program instructions, data files, and data structures alone or in combination. Computer-readable recording media include storage, magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. -A hardware device configured to store and execute program commands in a storage medium such as magneto-optical media, and ROM, RAM, flash memory, etc. may be included. Examples of program instructions may include those produced by a compiler, and high-level language codes that can be executed by a computer using an interpreter as well as machine language codes. The hardware device may be configured to operate as one or more software modules to perform the operation of the present invention.

As described above, the method of the present invention is implemented as a program and can be read using software of a computer, and a recording medium (CD-ROM, RAM, ROM, memory card, hard disk, magneto-optical disk, storage device, etc.) ) Can be stored.

Although described with reference to specific embodiments of the present invention, the present invention is not limited to the same configuration and operation as the specific embodiments in order to illustrate the technical idea as described above, and within the limits not departing from the technical spirit and scope of the present invention. It can be implemented by various modifications. Therefore, such modifications should be regarded as belonging to the scope of the present invention, and the scope of the present invention should be determined by the claims to be described later.

101: camera
102: video storage server (NVR/DVR)
100: FPGA-based embedded vision system (TLEM)
110: deep learning module
200: middleware (edge middleware + integrated middleware)
210: Edge middleware (Edge M/W)
230: integrated middleware
300: Terminal Operating System (TOS)
310: Monitoring PC
YT: Yard Tractor
TC: Transfer Crane
XT: external truck
STS: Ship To Shore Shore
QC: Quay Crane
ATC: Automated Transfer Crane (ARMGC)
TOS: Terminal Operating System

Claims (7)

Multiple cameras installed in each area of Port Gate, Yard/Block, Block Entrance, ARMGC, and QC at the Port Container Terminal; And
Receive the multiple camera image data and detect objects in the camera image according to the learning data by a deep learning module and recognize characters, lane recognition, vehicle number recognition, container number (ISO code) character recognition, container damage recognition, Recognition of vehicles entering the block entrance, vehicles and workers entering dangerous areas, vehicles running in reverse, detection of wearing safety protective equipment/safety vests for yard workers, detection of the location of loading and unloading equipment, and the learned object extraction event (text) and marked with a square box. It includes an FPGA-based embedded vision system (TLEM) that transmits object extraction image data to middleware,
The FPGA-based embedded vision system (TLEM) is equipped with a deep learning module that detects non-GPU-based image objects and recognizes characters of vehicle number and container number (ISO code). Vision camera system that recognizes objects. delete The method of claim 1,
The deep learning module uses a deep learning algorithm of any one of CNN, RNN, R-CNN, Fast RCNN, Faster RCNN, YOLO, or SSD model for object recognition of camera images, Vision camera system that recognizes objects. The method of claim 1,
The FPGA-based embedded vision system (TLEM)
A deep learning module for recognizing an object of camera image data and an FPGA equipped with an MCU (CPU);
It is connected to the deep learning module and the FPGA on which the MCU is mounted, and the deep learning module detects the object of the image and recognizes the text of the image including the vehicle number, container number (ISO code), and YT number to display the image. DSP processing;
A video interface unit including an encoder/decoder of a video codec, which is connected to the deep learning module and the FPGA on which the MCU is mounted, and receives and outputs image data from the camera C through HDMI Rx/Tx;
A flash memory and SDRAM connected to an FPGA on which the deep learning module and MCU are mounted;
A USB-UART terminal unit connected to the deep learning module and the FPGA on which the MCU is mounted;
A GPIO interface unit connected to the deep learning module and the FPGA on which the MCU is mounted, and for connecting an external speaker through a GPIO interface;
An Ethernet connection part connected to the deep learning module and the FPGA on which the MCU is mounted;
A Wi-Fi communication unit connected to the deep learning module and the FPGA on which the MCU is mounted; And
A vision camera system for vehicle access management and object recognition in a port container terminal, including a DC power supply. The method of claim 1,
The FPGA-based embedded vision system (TLEM)
A vision camera system for managing vehicle access and recognizing objects in a port container terminal, further comprising an SD card connection unit connected to the deep learning module and the FPGA on which the MCU is mounted, and the SD card is detachably mounted/separated. The method of claim 1,
The FPGA-based embedded vision system (TLEM) is connected to a speaker through a GPIO interface, and the speaker stores pre-stored voice data, and the corresponding voice data is output according to a remote broadcast signal of the FPGA-based embedded vision system (TLEM). A vision camera system for vehicle access management and object recognition in a port container terminal, which is an IP speaker. The method of claim 1,
The FPGA-based embedded vision system (TLEM) is interlocked with middleware, the middleware is connected to a terminal operating system (TOS), and the middleware is provided with an adapter and a TOS linkage module for heterogeneous software and hardware equipment in use,
The middleware receives the learned object extraction event (text) from the FPGA-based embedded vision system (TLEM) each time an event occurs, and object extraction image data marked as a square box (grid), and when an event occurs, it broadcasts a warning to a corresponding speaker. , A vision camera system for vehicle access management and object recognition in a port container terminal, which is linked with the terminal operating system (TOS) that receives DR-GPS location information of a vehicle equipped with a DR-GPS terminal through an API.

Images