CN111723708A

CN111723708A - Van-type cargo vehicle carriage door state recognition device and system based on deep learning

Info

Publication number: CN111723708A
Application number: CN202010518732.5A
Authority: CN
Inventors: 邹细勇; 黄昌清; 花江峰
Original assignee: Hangzhou Goodmicro Robot Co ltd
Current assignee: Hangzhou Goodmicro Robot Co ltd
Priority date: 2020-06-09
Filing date: 2020-06-09
Publication date: 2020-09-29

Abstract

The invention provides a van-type freight car carriage door state recognition device and a system based on deep learning, wherein side-looking, rear-looking and overlooking multi-view images acquired by an image acquisition unit are fused into image samples input into a deep learning network, the deep learning network is established in a recognition controller, the feature convolution extraction depth is increased, a feature sharing layer is added in front of an output layer to realize voting judgment of the carriage door state, and the multi-view carriage door region feature map fusion and the sharing layer multi-feature fusion are mutually matched, so that the robustness and the accuracy of the network for carriage door state recognition are improved. The invention realizes the automatic identification of the state of the carriage door of the goods van, can effectively prevent hidden troubles caused by opening the carriage door in transportation, and improves the management level of logistics transportation. The number of recognition networks is reduced and the recognition efficiency is improved by realizing the recognition of the license plate area and the vehicle type in the same network; and the marking workload is saved by automatically and primarily selecting the compartment door area in the training sample image.

Description

Van-type cargo vehicle carriage door state recognition device and system based on deep learning

Technical Field

The invention belongs to the field of logistics transportation, and particularly relates to a van body door state recognition device and system based on deep learning.

Background

Logistics transportation is an important industry of pillars for economic society development, and for enterprises, establishment and improvement of logistics capacity are important driving forces for development. Currently, as logistics parks have obvious scale advantages and aggregation effects in the aspects of economic scale, geographical distribution, construction and operation modes and the like, China has formed the situation of construction and development of the logistics parks from south to north and from east to west in China. The logistics park occupies a large area, and comprises supporting services such as information, consultation, maintenance, comprehensive service and the like besides storage, transportation, processing and the like. After the logistics park appears, the pressure of logistics on urban traffic is reduced, the scale benefit of logistics operation is improved, and the requirements of large-scale development trend of warehouse construction and development of goods intermodal transportation are met.

The logistics park in China develops rapidly, but a lot of new technical requirements are continuously generated. If the logistics park service items are not sufficient, the informatization and networking levels are not high. Means of data acquisition and interaction in each link of logistics transportation are not abundant enough, and monitoring measures are not enough. The monitoring and standard management requirements of the truck for access and passage in the logistics park are more and more prominent.

Compared with other common trucks, the van truck has the advantages of flexibility, high working efficiency, large transportation amount, safety, reliability and the like. Therefore, the van is widely used in logistics transportation. The van is all-weather, and is safer and tidier than ordinary trucks, and can not wet goods when raining. The van must set up the back door, and the setting of door provides the condition for the loading and unloading goods, and the opening angle of door must reach certain design requirement, just can accomplish the safe loading and unloading of goods completely and give full play to van's service function. As shown in figure 3 in the attached drawings of the specification, the rear door opening angle of the van is 270 degrees according to the specification of ZB T52006 general technical conditions of vans.

If the compartment door of a van in logistics transportation is not closed when the van is driven at a high speed, the goods can fall off to cause economic loss; the car door swaying left and right can hurt the pedestrians, and traffic accidents are caused. There are many reports about such accidents, such as in Ningbo, a heavy van is not firm because of the door, and the door is opened to damage the height limiting rod of the toll station; in Wenzhou, a truck has a car door which is not closed, so that a walking old man is hit by the car door; in addition, in Shanxi, the rear compartment door of the vehicle is not closed, and the vehicle is thrown away in the driving process, so that the passing vehicles are threatened to avoid, and serious traffic accidents are caused by the difference.

At present, some technical schemes related to car door detection exist, such as the chinese patent application with application number 2018102987891, whether a car door is closed is judged through on-off of a contact block, the chinese patent application with application number 2016200489718 acquires opening and closing information of the car door through a contact type mechanical door lock, and the chinese patent application with application number 2017109280143 realizes sensing of state detection through a photoelectric emitting and receiving module so as to realize recognition of illegal opening of a private car. The above schemes are all used for detecting the state of the vehicle door from the inside of the vehicle, and external supervision cannot be realized.

The freight car brings great potential safety hazard for road transportation because the compartment door is not closed, therefore, in the management of the access of a logistics park, equipment and a system which can automatically identify whether the compartment door of the freight car is closed are urgently needed, and the automatic detection, reminding and control of the state of the compartment door of the compartment type freight car can timely prevent the occurrence of cargo loss and personnel injury accidents.

In the field of embedded systems, raspberry pies have been introduced as a single board computer for many years, and although the raspberry pies are initially introduced for the education field, due to the open source hardware characteristics, the raspberry pies are greatly supported by open source communities, the related software resources are abundant, and the development of new applications can be faster. At present, a bramble pie which is widely applied is provided with a plurality of camera interfaces. Only the credit card-sized raspberry pie has strong processing capability and abundant interfaces, is widely used in various mobile or embedded solutions, and has high cost performance.

Disclosure of Invention

In view of the above requirements, the present invention provides a deep learning-based device, system and controller for identifying the door state of a van truck, which can identify the door state of the truck at the positions of a loading port, a road, an entrance, and the like in a cargo yard, limit the travel when a dangerous state is detected, and send out a warning in time.

The van body door state recognition device and the van body door state recognition system provided by the invention take the embedded controller as a processing platform, acquire a data sample set by carrying out multi-view image acquisition on a van from the outside, and train by using a deep learning network to obtain a body door state recognition model, so that the state of the van body door on a road is recognized in real time by using the model based on the image acquisition of a road side image acquisition unit, and the state of the van body door on the road is judged by matching with a distance sensor to detect the speed of the van, and alarm feedback is carried out when a potential risk exists. The truck with the unclosed van door can be dammed and managed at the outlet, and timely treatment and hidden danger elimination are required.

In order to obtain a network model which is suitable for application of a raspberry-type embedded platform and has strong recognition robustness and high accuracy, the yolo-v3-tiny network is improved, a feature processing layer is supplemented in a specific area of the network to increase the depth of the network, and a feature sharing layer which is adaptive to the characteristics of an input sample image is added before an output layer to improve the recognition capability of the network. Meanwhile, correspondingly, the characteristic expression level of the input image is improved by acquiring images of the truck at three visual angles of lateral direction, rear lateral direction and overlooking and fusing the images at the three visual angles into one image.

In order to identify the vehicle ID, an image of a marked license plate area is added in an input sample based on the same identification network, so that a license plate area image is obtained through a license plate camera, a license plate area anchor frame is extracted based on the trained network, and then a license plate number is extracted from the anchor frame area through character extraction.

In order to reduce the labeling workload before off-line training, the preliminary labeling of the door area in the image is also obtained through image differential processing and heuristic rule-based searching. In order to ensure the effectiveness and the characteristic consistency of image acquisition, automatic acquisition of images at various visual angles based on the triggering of a distance sensing unit is designed.

And the recognition result of the carriage state of the truck is associated with the acousto-optic unit, the vehicle-mounted alarm and the gateway controller through the communication interface, so that the automatic recognition and management of unsafe factors of the carriage door are realized.

The technical solution of the invention is to provide a deep learning-based van body door state recognition system, which comprises a recognition controller, a user interface unit, an acousto-optic unit, a distance sensing unit, an image acquisition unit and a communication interface, wherein the communication interface is also connected with a barrier gate controller, a vehicle-mounted alarm and a server,

the acousto-optic unit carries out information prompt on a truck driver and an operator through sound and/or light,

the user interface unit comprises an operation panel and a display screen, is used for entering parameters, initiating operation and carrying out information interaction,

based on the detection of the distance sensing unit to the vehicle, the image acquisition unit performs multi-angle image acquisition to the vehicle after being triggered,

the recognition controller comprises an input module, a main processing module, an image preprocessing module, an image recognition module, an image fusion module, a storage module and an output module, and is configured to:

a network model for improving the recognition of the state of the freight car door of the network based on yolo-v3-tiny is established in an image recognition module,

the network sequentially adds two layers of convolution layers with convolution kernel sizes of 3 multiplied by 3 and 1 multiplied by 1 before the original 8 th layer, and the number of filters of the two layers of convolution layers is respectively 256 and 128; adding a common sharing layer before the two yolo layers, wherein the sharing layer takes the weighted average of the output values of each class in the original input feature maps of the two yolo layers as the input of the two yolo layers,

three pictures collected by a side rear camera positioned at the side rear part of the vehicle, a top view camera positioned above a transportation channel and a side view camera are fused into a sample picture, a training data set is formed after the door state of the sample picture is labeled, the network model is trained off line by the data set to obtain a freight car door state recognition model,

when the truck door state identification module is in online operation, the image fusion module fuses three pictures of a vehicle collected by the side rear camera, the overlook camera and the side view camera into a picture to be detected and inputs the picture to the image identification module, the image identification module processes the picture to be detected based on the truck door state identification model to obtain the truck door state information, and sends out an alarm signal to the vehicle-mounted alarm and the acousto-optic unit and/or sends out a brake falling instruction to the barrier controller through the output module when the opening of the truck door is detected.

Preferably, the identification controller further stores the fused to-be-detected picture, the truck identification number, the vehicle type and other information in a database of the server.

Preferably, the server is provided with a logistics transportation information database in the ERP system.

In another embodiment of the invention, a deep learning-based van body door state recognition device is also provided, which comprises a recognition controller, a user interface unit, an acousto-optic unit, a distance sensing unit, an image acquisition unit and a communication interface,

according to the detection of the distance sensing unit to the vehicle, the image acquisition unit is triggered to acquire the multi-angle image of the vehicle,

three pictures collected by a side rear camera positioned at the side rear part of the vehicle, a top view camera positioned above a transportation channel and a side view camera are fused into a sample picture, the sample picture is labeled in a compartment door state to form a training data set,

the network model is trained off line by the data set to obtain a freight car door state identification model,

when the system runs online, the image fusion module fuses three pictures of the vehicle collected by the side rear camera, the overlook camera and the side view camera into a picture to be detected and inputs the picture to the image recognition module, the image recognition module processes the picture to be detected based on the freight car door state recognition model to obtain freight car door state information, and the information is output through the output module.

Preferably, the main processing module further calculates a running speed V of the truck according to a trigger time difference of the distance sensing unit, and when the speed V is greater than a set value and the door is opened, an alarm signal is sent to the vehicle-mounted alarm and the acousto-optic unit through the output module, and/or a gate dropping instruction is sent to the gateway controller.

Preferably, the recognition controller is further configured to,

if the time interval between the times of the vehicle detected by the first and second detection modules is recorded as Δ t1 and the time interval between the times of the vehicle leaving the first and second detection modules is recorded as Δ t2, the calculation formula of the speed V is:

wherein L is the distance between the first and second detection modules.

Preferably, the compartment door state labeling comprises door closing and door opening two kinds of output, in the sample picture and the picture to be detected, the three pictures are fused into one sample picture in an inverted triangle or triangle mode, the side view picture occupies about half of the space of the picture, and the included angle between the optical axis and the rear end face of the cargo compartment when the side rear camera collects the image is 15-45 degrees.

Preferably, the compartment door state labeling comprises four kinds of output of the respective opening and closing states of the left and right compartment doors, the three pictures are fused into one sample picture in an inverted triangle or triangle mode in the sample picture and the picture to be detected, the side view picture occupies about half of the space of the picture, and the included angle between the optical axis and the rear end face of the cargo compartment when the side rear camera collects the image is 15-45 degrees.

Preferably, the four category outputs are four carriage door states of left door open and right door closed, double door open, double door closed and left door closed and right door open.

Preferably, the photographing angle of the camera may be adjusted such that the body of the target van occupies 40% to 70% of the picture space when photographed.

Preferably, the image capturing unit includes a license plate camera located in front of the tunnel side for taking an image of a license plate region, which is used to acquire a license plate image,

the training data set also comprises a license plate image sample marked with a license plate frame, when the training data set runs on line, the network model trained off line identifies the acquired license plate image to obtain a license plate area anchor frame,

the license plate detection module firstly analyzes and constructs a binarization characteristic template library of characters according to the characters possibly appearing in the anchor frame area; then, aiming at the anchor frame area, detecting and separating single characters, extracting the characteristics of each character, then performing template matching, and then identifying phrases to obtain license plate numbers.

Preferably, the plurality of cameras connected with the switching array in the image acquisition unit are respectively deployed on different road sections of a vehicle driving channel;

the distance sensing unit adopts a linear array type detection module, and comprises a first detection module, a second detection module and a third detection module, wherein the first detection module and the second detection module are sequentially arranged on the side of a road along the advancing direction of a vehicle, and the third detection module is positioned between the two detection modules and corresponds to the side-looking camera;

when a vehicle leaves the first detection module, the side-view camera and the top-view camera are triggered when the following conditions are met: the third detection module detects the vehicle, and the number ratio of the detection modules which are positioned in the forward direction and the backward direction of the third detection module and used for detecting the vehicle is a set value.

Preferably, the number ratio is 5:3, and the detection module points in the linear array type detection module are uniformly arranged at intervals.

Preferably, the rear-side camera in the image acquisition unit is triggered a set time after the vehicle leaves the first detection module.

Preferably, the identification device further comprises an illuminance sensing unit and a lighting unit, the controller controls the lighting unit to supplement light based on the detection of the illuminance sensing unit on the environment,

the acousto-optic unit carries out information prompt on a truck driver and an operator through sound and/or light; the user interface unit comprises an operation panel and a display screen and is used for entering parameters, initiating operation and carrying out information interaction.

In yet another embodiment of the present invention, there is also provided a deep learning based van door status recognition controller, which includes an input module, a main processing module, an image preprocessing module, an image recognition module, an image fusion module, a storage module and an output module,

the input module respectively obtains setting parameters, user operation instructions and vehicle position signals collected by the vehicle detection modules through the user interface unit and the distance sensing unit,

the storage module is used for storing intermediate data, archive files and the like in the information processing process of each module,

and, the recognition controller is configured to:

controlling the image acquisition unit to acquire multi-angle images of the vehicle according to the detection of the distance sensing unit on the vehicle,

Preferably, the annotation information of the sample picture input into the image recognition module and the category output value both comprise vehicle type category information; and when the vehicle runs on line, the image recognition module further recognizes the picture to be detected by the trained recognition model to obtain the vehicle type information of the vehicle.

Preferably, the identification controller is further configured to automatically label the door region anchor frame in the sample picture through image difference processing, obtain preliminary data of the training set and the testing set,

the method comprises the following steps of performing differential calculation on three visual angle images of a truck collected in an image collection unit by taking road surface images of three visual angles as background images to obtain three differential images, and processing each differential image:

searching a first transverse line and a second transverse line which have the length exceeding a set value and the row number being respectively minimum and maximum in the differential image, searching a second column line which has the length exceeding the set value and the column number being maximum in the differential image,

and taking the first transverse line and the second transverse line as upper and lower side references, taking the second alignment line as a right reference, determining the left side by using a preset length, taking the four sides as references, outwards expanding a certain range to be used as an ROI of a truck tail image, and using the ROI as a primary region for anchor frame marking.

Preferably, a first row line with the length exceeding a set value and the smallest row number in the differential image is searched, and the vehicle type is judged after the distance between the first row line and the second row line and the linear array type detection module perform data fusion on the detection values of the length and the height of the vehicle; and selecting the preset length, namely the rectangular length of the ROI according to the vehicle type.

Preferably, the detected value of the vehicle length is a converted distance corresponding to a distance between points or a number of points of the detection module at the forefront and the rearmost of the detection modules at which the vehicle is detected in the linear array detection module.

Preferably, the images of the side view, the rear view and the top view are collected twice after being triggered, the difference operation of the collected images of the two times is carried out on the images of the view respectively, and the column line with the length exceeding the set value searched from the tail of the vehicle is used as the second column line.

Compared with the prior art, the scheme of the invention has the following advantages: the method synchronously acquires side-looking, side-looking and overlooking multi-view images of the trucks running in the logistics park, and fuses the multi-view images into one image after framing and marking the door areas at the tail parts of the trucks in the images, so as to correspondingly generate a data sample set; an improved yolo-v3-tiny network is adopted as a freight car door state recognition model, according to the characteristics of a sample image, a convolutional layer is added to deepen the feature extraction depth, and a feature sharing layer is added in front of an output layer to realize voting judgment of the car door state; the fusion of the multi-view door area characteristic maps and the mutual cooperation of the voting judgment of the multi-features of the sharing layer improve the robustness and the accuracy of the network for the door state identification. Based on the state of the identified freight car door, risk reminding can be timely carried out through control such as alarming and a barrier gate, the hidden danger that loss and injury are possibly caused due to the fact that the car door is opened in transportation is eliminated, and the management and service level of a freight yard or a park is improved. By adding the license plate region feature recognition function in the same image recognition model, the number of recognition networks is reduced, and the recognition efficiency is improved. Through the automatic preliminary frame selection of the compartment door area in the training sample image, the labeling workload is saved.

Drawings

Fig. 1 is a composition structure diagram of a van door state recognition device and system based on deep learning;

FIG. 2 is a block diagram of a deep learning based van door status identification controller;

FIG. 3 is a schematic view of a door area of a van;

FIG. 4 is a schematic view of a logistics park road;

FIG. 5 is a fragmentary schematic view of a truck haul road;

FIG. 6 is a partial schematic view of a transport corridor in front of a barrier;

FIG. 7 is a partial schematic view of a loading and unloading port;

FIG. 8 is a diagram of an image recognition module deep learning network architecture;

FIG. 9 is a schematic diagram of a van anisotropic image fusion partition;

FIG. 10 is a flowchart of template matching for a license plate detection module.

Wherein:

10000 wagon compartment door state identification system; 1000 van body door state recognition device, 2000 server, 3000 vehicle mounted alarm, 4000 gateway controller;

100 identification controller, 200 user interface unit, 300 acousto-optic unit, 400 illumination sensing unit, 500 lighting unit, 600 distance sensing unit, 700 image acquisition unit, 800 communication interface;

710 switching array, 720 camera, 721 side rear camera, 722 top view camera, 723 side view camera; 601 a first detection module, 602 a second detection module, 603 a third detection module;

the system comprises a 110 input module, a 120 main processing module, a 130 image preprocessing module, a 140 image identification module, a 150 image fusion module, a 160 output module and a 170 storage module;

the system comprises a van 10, a vehicle channel 11, a pedestrian channel 12 and a loading and unloading port 13;

21 upright posts, 22 lifting and releasing rotating shafts and 23 brake levers;

31 side view image section, 32 top view image section, 33 side back image section, 34 door area anchor frame.

Detailed Description

Preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings, but the present invention is not limited to only these embodiments. The invention is intended to cover alternatives, modifications, equivalents and alternatives which may be included within the spirit and scope of the invention.

In the following description of the preferred embodiments of the present invention, specific details are set forth in order to provide a thorough understanding of the present invention, and it will be apparent to those skilled in the art that the present invention may be practiced without these specific details.

The invention is described in more detail in the following paragraphs by way of example with reference to the accompanying drawings. It should be noted that the drawings are in simplified form and are not to precise scale, which is only used for convenience and clarity to assist in describing the embodiments of the present invention. The front and rear of the invention are relative, according to the advancing direction of the vehicle.

Example 1

At present, the logistics park in China develops rapidly, so that a plurality of technical requirements are brought forward, such as the management and service capacity, the informatization level and the like of the logistics park are all to be improved. In order to perfect data acquisition and automatic management capacity in each link of logistics transportation, aiming at the problem that the compartment of a van in a logistics park is not closed tightly and is lack of recognition and monitoring at present, the embodiment provides a compartment door state recognition system, a compartment door state recognition device and a compartment door state recognition controller based on deep learning by combining the requirements of logistics transportation services and the characteristics of an embedded raspberry dispatching processing platform.

As shown in fig. 1, a deep learning based van body door state identification system 10000 includes an identification controller 100, a user interface unit 200, an acousto-optic unit 300, a distance sensing unit 600, an image acquisition unit 700 and a communication interface 800, wherein the communication interface 800 is further connected with a gateway controller 4000 and a server 2000.

The acousto-optic unit 300 performs information prompt on a truck driver and an operator through sound and/or light, for example, normal/abnormal states are represented by different colors of light and twinkling, and state prompt is performed through voice information; the user interface unit 200 includes an operation panel and a display screen for entering parameters, initiating operations, and performing information interaction; according to the detection of the distance sensing unit 600 on the vehicle, the image capturing unit 700 performs multi-view image capturing on the vehicle after being triggered.

In logistics transportation, van trucks have the advantage of being all-weather, and are therefore heavily used. As shown in fig. 3, the van has a rear door, and the cargo door is provided to facilitate loading and unloading of cargo. However, the loading and unloading time periods of the trucks are all over the day, the labor intensity of the operation is high, particularly, workers are easy to get tired at night, and the drivers and the loaders are often separated from each other, so that the drivers and the loaders are not smooth to hand over after the loading and unloading operation is finished, and the situation that the inspection of whether the door of the truck compartment is closed is lack of main responsibility is easily caused, so that accidents of personnel injury and material loss caused by opening of the door of the truck compartment are frequently caused in news reports. Therefore, in the management of the logistics park, an apparatus and a system capable of automatically recognizing the state of the door of the incoming/outgoing and moving truck from the outside are required. Therefore, the invention identifies the opening and closing states of the van door of the van by combining the operation characteristics of the logistics park, finds abnormality in time and gives an alarm and intervenes.

Referring to fig. 1, in the identification system, an embedded processor such as a raspberry pi is used as a main control processing platform of a deep learning-based van door state identification device 1000, which includes the identification controller 100, a user interface unit 200, an acousto-optic unit 300, a distance sensing unit 600, an image acquisition unit 700, and a communication interface 800.

The image acquisition unit 700 in turn comprises a switching array 710 and a plurality of cameras 720, the recognition controller enabling acquisition of image information of the vehicle from one or more of the plurality of cameras by switching the array 710. As shown in fig. 5 and 6, the cameras 720 include a rear-side camera 721 positioned behind the tunnel side to capture a stereoscopic image of the vehicle, a top-view camera 722 to capture an image of the cargo compartment of the vehicle from above the roof, and a side-view camera 723 to capture an image of the side of the vehicle. Through these cameras that set up in loading and unloading goods mouth and banister region, carry out the image acquisition of multi-angle to the freight train, the collection of image is triggered through the chronogenesis detection of distance sensing unit 600, promptly catches the vehicle and just obtains the image in specific position when moving to preset the position.

The multi-angle image acquisition of the van is carried out because the research finds that the door state characteristics which can be expressed by the single-angle image are not rich enough and are not enough to be used for recognizing the door state through a deep learning network all the day.

Based on the acquired image information, a deep learning-based van door state recognition controller 100 extracts image features of a van door and recognizes the state thereof. As shown in fig. 1 and 2, the recognition controller 100 preferably includes an input module 110, a main processing module 120, an image preprocessing module 130, an image recognition module 140, an image fusion module 150, an output module 160, and a storage module 170. The input module 110 obtains setting parameters and user operation instructions, and vehicle position signals collected by each vehicle detection module through the user interface unit 200 and the distance sensing unit 600, respectively. The storage module 170 is used for storing intermediate data, archived files and the like in the information processing process of each module; the output module 160 transmits control information of the main processing module 120 to the gateway controller 4000, the vehicle-mounted alarm 3000, and the like through the external communication interface 800, and the input and output modules also perform information interaction with the external server 2000 through the communication interface.

As shown in fig. 4, after a truck enters a park in a logistics park, the truck passes through a vehicle passage 11 to pass through various warehouses or cargo handling points A, B, and in order to improve road safety, some parks divert the flow of pedestrians and vehicles, and pedestrians and non-motor vehicles move through a pedestrian passage 12. With reference to fig. 4, 5, 6 and 7, in order to realize automatic recognition of the state of the truck compartment door, the truck tail image is collected by the side rear camera 721, the overlook camera 722 and the side view camera 723 on the inverted L-shaped travel rod, the images of three viewing angles are fused into one image and input into the deep learning network, and the network is used for extracting multi-viewing-angle and multi-scale features of the truck compartment door, so that the state of the compartment door can be accurately recognized based on the learned features and the mutual correlation among the features.

Specifically, as shown in fig. 1, 5 and 6, the vehicle will arrive at the barrier gate area shown in fig. 6 when going forward from the road shown in fig. 5, and the plurality of cameras in the image capturing unit are respectively disposed at different positions of the vehicle driving channel, and the distance sensing unit 600 is in a linear array structure and includes distance detection modules disposed at the side of the road. Preferably, the distance detection module may employ a photoelectric sensor. As shown in fig. 5, the linear array detection modules include a first detection module 601 at the rearmost position in the forward direction of the vehicle, a second detection module 602 at the frontmost position, and a third detection module 603 in the middle. The distance between the first detection module 601 and the second detection module 602 in the linear array detection module is far enough, and preferably, the distance can be set to be 1.1 to 1.3 times of the longest vehicle type length.

In a plurality of visual angles for acquiring images of the truck, the side-looking camera can acquire the front images of the door plate of the compartment door opened by the truck, but the characteristics are not obvious when the opening degree of the compartment door is very small, and the compartment doors on two sides are difficult to distinguish; the overlooking camera can acquire the thickness side image of the door plate of the compartment door opened by the truck, but the characteristic dimension of the compartment door in the direction is small; and the three-dimensional characteristics of the compartment door area when the compartment door is opened by the truck can be supplemented through the image acquisition of the side rear camera.

In order to automatically capture the image characteristics of the door area from a plurality of visual angles, the invention triggers the image acquisition by acquiring signals of each distance detection module. Specifically, an included angle formed by the optical axis of the side rear camera and the rear end face of the cargo compartment, namely the transverse direction of the road, is 15-45 degrees; then, an intersection point of the optical axis of the side rear camera and the road center line is made, and the first detection module is arranged by taking the position of the intersection point in the longitudinal direction of the road as a base point. Therefore, when the vehicle leaves the first detection module, the image acquisition of the side rear camera is triggered, and the three-dimensional characteristics of the compartment door area can be better obtained.

Preferably, the triggering time of the rear side camera can be finely adjusted based on the recognized truck type, and the truck type can be preliminarily recognized according to detection signals of different point modules in the whole linear array type detection module.

Preferably, the side rear camera is triggered a set time after the vehicle leaves the first detection module.

In order to realize the automatic triggering of the top view camera and the side view camera, a third detection module 603 is selected from the line array detection modules, the top view camera 722 and the side view camera 723 are arranged on the same horizontal line or adjacent positions, and a corresponding third detection module 603 is selected according to the positions of the two cameras in the longitudinal direction of the road. Thus, the acquisition trigger conditions for the side view camera and the top view camera are set as: the third detecting module 603 detects the vehicle, and the number ratio of the detecting modules in the forward direction and the backward direction of the vehicle is a set value based on the third detecting module.

Preferably, the number ratio is 5:3 or 7:4 or is preferably selected according to the vehicle type, and the detection module points in the linear array type detection module are uniformly arranged at intervals.

The load port 13 shown in fig. 7 shows a refrigerated cargo loading area, and a linear array type detection module in the distance sensing unit can be arranged in front of a loading and unloading point in a similar manner to a square partition manner adopted in ordinary cargo loading and unloading, and only one inverted L-shaped upright post and an overhead camera are illustrated in fig. 7.

It is easy for people to extract the characteristics of the carriage door from the image of the tail of the truck and then judge the opening and closing state of the carriage door, but the method is very difficult for machines. The invention adopts a deep learning network to identify the state of the freight car door based on the vehicle images shot at a plurality of specific angles.

The traditional vehicle image feature detection method comprises a background difference method, a target vehicle feature-based detection method and the like. The background difference method divides the image into foreground and background, firstly, the background is modeled, then the current frame image and the background model are compared pixel by pixel, and the area inconsistent with the background model is regarded as a motion area. The target feature-based detection method is to achieve the purpose of detecting a target by using pattern recognition through training and learning of basic features of a vehicle target, such as Haar features, HOG features, LBP features and the like, and common machine learning algorithms include an AdaBoost algorithm, an SVM support vector machine, a K-means algorithm and the like.

In recent years, a target detection algorithm makes a great breakthrough, and with the development of artificial intelligence and deep learning technology, a method for making a classifier by utilizing a convolutional neural network to perform an image understanding task to gradually replace manual feature extraction is adopted. The popular algorithms can be divided into two types, one type is an R-CNN algorithm based on a candidate region, the algorithms are divided into two stages, a heuristic method or a CNN network is required to generate the candidate region, and then classification and regression are carried out on the candidate region. And the other is a one-stage completion algorithm such as Yolo (abbreviation for young only look once), SSD, which uses only one CNN network to directly predict the categories and locations of different objects. Because the second type of algorithm has the characteristic of high speed, the algorithm has more potential to be used on an embedded processing platform compared with the former algorithm.

Many deep learning algorithms can only satisfy both the detection accuracy and the detection speed on the GPU. However, since the GPU is expensive and has high heat generation, it is difficult to load the GPU on a portable platform, and the embedded platform with low cost cannot achieve the effect of real-time detection due to the lack of a large-capacity GPU. Due to the end-to-end design of the Yolo model, the implementation process is simple, the characteristics of the picture are extracted only once, the speed is high, and the Yolo model becomes one of the classical target detection models.

Based on the force calculation characteristic of an embedded platform raspberry pi, the yolo-v3-tiny network is selected as a cargo compartment door state recognition deep learning network model. The yolo-v3-tiny network is a lightweight model of the latest optimized version of the yolo network, and has the characteristics of strong generalization capability, relatively low computational complexity and high recognition processing efficiency. In the yolo-v3-tiny network, each grid unit feature map predicts 3 candidate frames, and each candidate frame needs four coordinates and five basic parameters including a confidence coefficient, so the number of convolution kernels is B x (M +5), where B is the number of candidate frames and M is the number of categories. Compared with yolo-v3, the predicted output branch number of the yolo-v3-tiny network is reduced from 3 to 2, namely, the feature diagram adopts two types of 13 x 13 and 26 x 26, and the calculation amount is reduced. For the present invention, since the image sample collection is performed under a specific angle and triggered based on specific conditions, the target cargo compartment door has significant geometric size characteristics, and therefore, two characteristic maps of 13 × 13 and 26 × 26 are very suitable for being adopted; however, the testing experiment still finds that the identification processing capability of the original network does not meet the requirement. Thus, after in-depth testing and analysis, an improved network architecture is proposed and employed in conjunction with the image acquisition and fusion process of the present invention.

Specifically, referring to fig. 8, the original network is modified and optimized: sequentially adding two convolutional layers with convolutional cores of 3 multiplied by 3 and 1 multiplied by 1 in front of the original 8 th layer, wherein the number of filters of the two convolutional layers is N and 128 respectively; and a common sharing layer is added before the two yolo layers, the sharing layer takes the weighted average of the output values of each category in the original input feature maps of the two yolo layers as the input of the two yolo layers,

where N takes an even number between 128 and 512. Preferably, N has a value of 256. Repeated tests show that the recognition effect is better when N is selected to be 256, and the method can better link the extraction of the characteristics of the front layer and the rear layer.

Preferably, the compartment door state label includes two types of output of door closing and door opening. At this time, since the number of yolo output layer convolution kernels is B × (M +5) and M is 2, the number of convolution kernels is 21, and as shown in fig. 6, the yolo output layer outputs 13 × 13 × 21 and 26 × 26 × 21 candidate frames and corresponding identification types, i.e., gate-on or gate-off.

In order to improve the car door feature expression of the input network model image as much as possible, referring to fig. 9, the invention fuses each picture of three visual angles into a sample picture in an inverted delta-shaped or delta-shaped manner; also, the side view takes up about half of the space of the picture, because the door status feature of the side view is relatively more pronounced. In fig. 9, the top image subarea 32, the side rear image subarea 33, and the side image subarea 31 are arranged in an inverted triangle, and 34 is an anchor frame selected for the door area in each subarea image.

Based on in-depth testing, the image fusion method can make full use of two different scales of feature maps in the network model, so that the side-view image with large scale can make full use of the deep 13 × 13 features, and the other two view-angle images correspond to the 26 × 26 features more.

Preferably, the photographing angle of the camera may be adjusted such that the body of the target van occupies 40% to 70% of the picture space when photographed, to fully utilize the image space, and to normalize the size of the obtained picture to 416 × 416.

Through image acquisition in the process that various vehicles pass through transportation channels such as an entrance and an exit, images acquired based on a side rear camera, a top view camera and a side view camera are fused into a sample picture, the vehicles in the picture are subjected to frame selection and compartment door state category marking, samples under various natural conditions such as different illumination and meteorological conditions are acquired, the samples are abundant enough, and then offline training is carried out on the modified network. In order to simplify image acquisition, video shooting can be carried out on road traffic, then conversion from a video image to one frame of picture is realized through a video processing system, and the converted picture is screened and then manually marked to form a training sample.

And performing off-line training on the network model by using a training set to obtain a compartment door state recognition model in the image recognition module. When the system runs on line, an image preprocessing module in the recognition controller carries out image denoising, filtering, size normalization and other processing on vehicle pictures collected by each camera, an image fusion module fuses three pictures collected by a side rear camera, a top view camera and a side view camera into pictures to be detected in the same mode and then inputs the pictures to the image recognition module, the image recognition module processes the pictures to be detected based on the freight car door state recognition model to obtain freight car door state information, and the information is output through an output module.

In the established network model, the category output value is obtained after weighted averaging before the yolo layer, namely the output value is the weighted average value of the compartment door state output values in the three pictures, so that the judgment of the compartment door state is determined based on the voting of the multiple pictures, the processing avoids the misjudgment of the single-view-angle image when the GT category characteristics are not particularly superior, and the robustness of network identification is increased.

Preferably, in the weighted average, the weights corresponding to the side-view, side-rear-view and top-view images decrease in sequence, for example, by 0.45, 032 and 0.23, respectively.

The method comprises the steps of taking images of van trucks on a transport channel collected in a logistics park as a data set, dividing the images into a training set and a testing set, and performing frame selection and marking on door areas of various trucks according to types. The van in the samples comprises six types of vehicles, namely 3T, 5T, 8T, 10T, 25T and 30T, and the comparison test shows that the identification accuracy of the modified network-based multi-view fusion image sample is improved by 3 to 6 percent compared with that of a side-view single-view image sample; under the condition of multi-view fusion image samples, the recognition accuracy of the modified network is improved by 4% to 11% compared with the original network.

And the states of the carriage doors on the two sides of the truck can be respectively identified through the fusion of the three visual angle images. Preferably, the four category outputs are four carriage door states of left door open and right door closed, double door open, double door closed and left door closed and right door open. By subdividing the types of the states of the doors, the states of the doors on all sides can be more accurately identified.

Referring to fig. 1 and 6, in order to alarm a truck in time when an abnormal state is recognized, the alarm is performed through the acousto-optic unit 300, and an alarm signal may be sent to the vehicle alarm 3000 on the truck through the wireless communication interface.

Preferably, a license plate camera positioned in front of the channel side is arranged in the image sensing unit, a license plate photographing triggering detection module is arranged in the distance sensing unit, license plate photographing and identification are carried out on a truck passing through the module, and warning signals are sent to a vehicle-mounted warning device of the truck according to a wireless communication address corresponding to the license plate in the database.

In order to extract the license plate number of the truck, the license plate number is often extracted in two stages, namely, the license plate area is firstly positioned, and then characters are extracted from the area. In order to meet the requirement of improving the processing efficiency, the license plate region is extracted and merged into the freight car door state recognition for processing.

Referring to fig. 6, a license plate camera is arranged on a front upright post of a barrier to acquire a license plate area image, and functions of license plate frame area extraction are fused in a yolo-v3-tiny network of an image recognition module in addition to car door state recognition. And adding a license plate image sample marked with a license plate frame into the training data set of the network, correspondingly modifying the number of filters of the convolution layer before the yolo layer, and identifying the acquired license plate image through the offline-trained network model during online operation to obtain a license plate area anchor frame.

Then, a license plate detection module is arranged in the recognition controller, and the module firstly analyzes and constructs a binarization characteristic template library of characters according to the characters possibly appearing in the anchor frame area, wherein the binarization characteristic template library comprises ten Arabic numerals of 0-9, twenty-six capital English letters A-Z and license plate Chinese characters of provinces, direct cities and autonomous regions; then, aiming at the anchor frame area, detecting and separating single characters, extracting the characteristics of each character, then performing template matching, and then identifying phrases to obtain license plate numbers.

Specifically, as shown in fig. 10, the license plate detection module adopts the following processing steps:

p1) preprocessing the license plate region image, and acquiring an ROI (region of interest) containing target characters according to the license plate region anchor frame;

p2) constructing a binary feature template library of the characters according to the character analysis which can appear in the ROI area;

p3) detecting and separating single characters aiming at the obtained ROI, extracting the characteristics of each character, and then performing template matching to identify the single character;

p4) combine all single characters into a license plate number in order.

In order to obtain a single character, when the character outlines are adhered, because the width of each character is the same, firstly, the edge detection is carried out on the character connected domain to locate the initial position of the character, the upper height and the lower height of a single adhered character image are determined in the width range of each character, and the single character image is sequentially and circularly searched and segmented.

After the single character image is divided, the binarization processing and other processing are carried out on the single character image, and the size of the corresponding character is normalized to be the same as the size of the character in a matching template established in advance. Extracting character feature vectors based on a pixel-by-pixel feature method, as shown in fig. 10, dividing a character image into small blocks of 3 × 3 to 9, counting the number of non-0 pixels in a range, calculating features at intersections of 3 bisectors in the horizontal and vertical directions, totaling 13 feature values, and recording the feature values in an array. Counting non-0 pixel points in a single character for the normalized single character image, and storing the non-0 pixel points in a defined matrix; and then, extracting corresponding characteristic vectors, comparing the characteristic vectors with characteristic values of all regions of the character template, and matching and identifying the character to be detected.

As shown in fig. 6, at the exit of the logistics park or the goods yard, when it is recognized that the freight car to be discharged is not tightly closed, the recognition controller sends a gate falling instruction to the gateway controller 4000 to prevent the freight car from being discharged; meanwhile, besides the signal of the vehicle-mounted alarm, the acousto-optic unit 300 is used for reminding workers of timely intervention and intervention, the truck is required to be rectified and changed, the compartment door is tightly closed, and accidents are prevented. Specifically, as shown in fig. 6, after receiving a brake drop command, the barrier controller 4000 controls the lifting and releasing of the brake lever 23 mounted on the pillar 21 through the lifting and releasing shaft 22. The identification controller interacts the truck transportation information with a database in the server 2000 through a communication interface, after the verification of the compartment door and other states and information is passed, the gate rod is controlled to be erected through the aisle gate controller, a truck can leave, and the rod is dropped after the truck is detected to leave; when the state verification fails, such as the abnormal state of the car door, the information prompt and alarm are carried out through the acousto-optic unit and the display module in the user interface unit, and the brake lever is in a brake closing state.

The vehicle access information is recorded correctly while the vehicle is allowed to access. Preferably, the identification controller stores the door image, the door state, the truck identification number, the vehicle type, the access time and other information in a database, and takes the door state as one of the KPI examination bases of the driver. Preferably, the server 2000 is provided with a logistics transportation information database of the ERP system.

Preferably, the van door state recognition device further comprises an illumination sensing unit and a lighting unit, and the recognition controller controls the lighting unit to supplement light based on the detection of the illumination sensing unit on the environment.

Example 2:

different from embodiment 1, the present embodiment further identifies the truck type information through an image identification module. In order to record and compare the vehicle type information in the cargo yard access management, as shown in fig. 1 and 3, the invention also extracts the vehicle type category of the vehicle after being processed by the image recognition module based on the collected images.

The composition of the transport vehicles in the logistics park is complex, and vehicles of cooperative transport enterprises and the like exist in individual transport. Therefore, in the ERP system of the logistics park, vehicle information needs to be updated and maintained, and the vehicle information is input into a vehicle management subsystem, wherein the vehicle information comprises vehicle types, rated carrying capacity, license plate numbers, colors, empty vehicle weight, owners, drivers, contacts, telephones and the like. The vehicle type information is important, and the vehicle type can directly correspond to the load capacity generally and is relatively intuitive; secondly, different vehicle models can be arranged in the inlet and outlet pipeline to run different channels.

Therefore, in addition to the marking of the door state, the vehicle type information of the vehicle body in the image is marked in the marking information of the sample picture input into the image recognition module; meanwhile, the number of filters of the convolution layer before the yolo layer is correspondingly modified, namely the vehicle type data is correspondingly added into the output value of the identification network. When the vehicle runs online, the image recognition module is used for recognizing the picture to be detected through the trained recognition model, and then the vehicle type information of the vehicle can be obtained.

Similar to the calculation chart of the output values of the door state categories, the output values of the vehicle type categories are weighted and averaged in the sharing layer based on the output values of the categories and then serve as the input of two yolo layers. The test result shows that the images of three visual angles collected from the side, the upper space and the lateral rear part can supplement the characteristics of the vehicle body, thereby improving the accuracy rate of vehicle type identification.

Because the deep learning network has strong generalization and knowledge self-learning characteristics, the invention obtains the identification information of the door state and the vehicle type based on the single same yolo-v3-tiny network. By sharing the same network in a plurality of identification tasks, the model complexity is reduced, and the identification capability and the management efficiency of logistics transportation are improved.

In addition, when loading and unloading are carried out in a cargo yard, short-range displacement such as multipoint loading and unloading is sometimes required, and in this case, when the vehicle speed is detected to be greater than a set value, the state of the door is recognized, and when a danger is recognized, an alarm signal is sent to the vehicle-mounted alarm and the acousto-optic unit through the output module via the communication interface in a wireless communication mode.

For this purpose, the main processing module in the recognition controller also calculates the truck driving speed V according to the trigger time difference of the distance sensing unit. The speed detection is based on a first detection module and a second detection module which are positioned beside a road, and the main processing module calculates the running speed V of the vehicle by recording the interval delta t1 of the time when the vehicle is detected by the first detection module and the second detection module and the interval delta t2 of the time when the vehicle leaves the first detection module and the second detection module:

wherein L is the distance between the first and second detection modules.

Example 3

Different from other embodiments, in the embodiment, in the offline processing of the sample, the identification controller 100 further automatically labels the door area anchor frame in the sample picture through image difference processing, and obtains the preliminary data of the training set and the test set.

Specifically, as shown in fig. 5 and 9, the difference calculation is performed on the three perspective images of the truck collected by the image collection unit by using the road surface images of the three perspectives as background images, so as to obtain three difference images, and each difference image is processed:

and taking the first transverse line and the second transverse line as upper and lower side references, taking the second alignment line as a right reference, determining the left side by using a preset length, taking the four sides as references, outwards expanding a certain range to be used as an ROI of a truck tail image, and using the ROI as a primary region for anchor frame marking. Therefore, during subsequent manual marking, the anchor frame area can be not adjusted or only needs to be finely adjusted.

Preferably, in the linear array detection module, the modules are disposed in matrix type at multiple points in the height direction perpendicular to the road surface. Meanwhile, a first row line with the length exceeding a set value and the smallest row number in the differential image is searched, the distance between the first row line and the second row line is used for carrying out data fusion on the detection values of the length and the height of the vehicle with the linear array type detection module, and then the vehicle type is judged; and selecting the preset length, namely the rectangular length and the width of the ROI according to the vehicle type.

Preferably, the data fusion adopts a combination rule in the DS evidence theory to perform multi-attribute fusion or is determined through majority voting. If the truck is judged to be a large vehicle through the distance between the first row line and the second row line, the specific vehicle type cannot be determined, and the characteristics can be supplemented through sensing signals of detection points of the linear array type detection module in different height directions, such as distinguishing a common dry van truck or a refrigerator truck. Through data fusion, inconsistent parts in the data set can be eliminated,

While the embodiments of the present invention have been described above, these embodiments are presented as examples and do not limit the scope of the invention. These embodiments may be implemented in other various ways, and various omissions, substitutions, combinations, and changes may be made without departing from the spirit of the invention. These embodiments and modifications are included in the scope and gist of the invention, and are also included in the invention described in the claims and the equivalent scope thereof.

Claims

1. A van body door state recognition device based on deep learning comprises a recognition controller, a user interface unit, an acousto-optic unit, a distance sensing unit, an image acquisition unit and a communication interface,

when the vehicle is in online operation, the image fusion module fuses three pictures of the vehicle, which are acquired by the side rear camera, the top view camera and the side view camera, into a picture to be detected and then inputs the picture to the image recognition module,

and the image recognition module processes the picture to be detected based on the freight car door state recognition model to obtain freight car door state information, and outputs the information through an output module.

2. The deep learning-based van body door state recognition device according to claim 1, wherein the main processing module further calculates a driving speed V of the van according to a trigger time difference of the distance sensing unit, and when the speed V is greater than a set value and the door state is open, an alarm signal is sent to a vehicle-mounted alarm and an acousto-optic unit through the output module, and/or a gate-off instruction is sent to a barrier gate controller.

3. A deep learning based van door status recognition device according to claim 2, wherein the recognition controller is further configured to,

wherein L is the distance between the first and second detection modules.

4. The deep learning-based van body door state recognition device is characterized in that the state labeling of the van body door comprises door closing and door opening output, in the sample picture and the picture to be detected, the three pictures are fused into one sample picture in an inverted delta-shaped or delta-shaped mode, a side view picture occupies about half of the space of the picture, and the included angle between the optical axis and the rear end face of the van body is 15-45 degrees when the side rear camera collects the image.

5. The deep learning-based van body door state recognition device is characterized in that the door state labels comprise four types of output of the opening and closing states of a left door and a right door, in the sample picture and the picture to be detected, the three pictures are fused into one sample picture in an inverted delta-shaped or delta-shaped mode, a side-view picture occupies about half of the space of the picture, and the included angle between the optical axis and the rear end face of the van body is 15-45 degrees when the side-rear camera collects the picture.

6. The deep learning-based van body door state recognition device according to claim 1, wherein the image acquisition unit comprises a license plate camera located in front of the side of the passageway for capturing an image of a license plate region, which is used to acquire a license plate image,

7. A deep learning based van compartment door state recognition device according to claim 1,

a plurality of cameras connected with the switching array in the image acquisition unit are respectively deployed on different road sections of a vehicle driving channel;

8. The deep learning-based van body door state recognition device according to claim 1, further comprising an illumination sensing unit and a lighting unit, wherein the controller controls the lighting unit to supplement light based on the detection of the illumination sensing unit on the environment,

9. A van body door state recognition system based on deep learning is characterized by comprising a recognition controller, a user interface unit, an acousto-optic unit, a distance sensing unit, an image acquisition unit and a communication interface, wherein the communication interface is also connected with a gateway controller, a vehicle-mounted alarm and a server,

10. The deep learning-based van body door state identification system according to claim 9, wherein the identification controller further stores the fused information such as the to-be-detected picture and the van identification number in a database of the server.