CN112232259B

CN112232259B - Method, device and equipment for monitoring behaviors of taxi appointment drivers

Info

Publication number: CN112232259B
Application number: CN202011163532.9A
Authority: CN
Inventors: 王小刚; 余程鹏
Original assignee: Nanjing Leading Technology Co Ltd
Current assignee: Nanjing Leading Technology Co Ltd
Priority date: 2020-10-27
Filing date: 2020-10-27
Publication date: 2022-06-07
Anticipated expiration: 2040-10-27
Also published as: CN112232259A

Abstract

The invention provides a method, a device and equipment for monitoring behaviors of a driver of a network appointment vehicle, wherein the method comprises the following steps: simultaneously acquiring first monitoring data and second monitoring data acquired by a video monitoring system and a driver detection system, and respectively labeling data of an overlapping area in the data; respectively extracting the characteristics of the first monitoring data and the second monitoring data to obtain first characteristic data and second characteristic data corresponding to the data of the non-overlapping area and third characteristic data and fourth characteristic data corresponding to the data of the overlapping area; respectively performing multi-channel feature fusion on the first feature data and the second feature data, and performing feature fusion in a spatial domain on the third feature data and the fourth feature data; and determining whether the driver is in an abnormal behavior state or not according to the obtained fusion data. By utilizing the method provided by the invention, the characteristic level fusion of the data acquired by the plurality of cameras can be realized, the processing efficiency is improved, and the monitoring accuracy is improved.

Description

Method, device and equipment for monitoring behaviors of taxi appointment drivers

Technical Field

The invention relates to the technical field of communication, in particular to a method, a device and equipment for monitoring behaviors of a network car booking driver.

Background

In recent years, with the vigorous development of the automobile industry and the internet industry, a network car-booking service (called network car booking for short) becomes an important way for users to go out. The online taxi reservation system can meet the use requirements of users in different travel scenes, rapidly occupies a large amount of user markets in a short time, and brings great convenience for the travel of the users.

In the operation process of the network car booking, abnormal states of some network car booking drivers, such as smoking, calling, unbuckling a safety belt, yawning and the like, occur occasionally, so that the travel experience of passengers can be reduced, and the personal safety of the passengers can be violated. For example, some drivers are tired of driving and have yawning connection, which increases the possibility of traffic accidents, and the above-mentioned irregular driving behaviors are not monitored and solved in time.

In the prior art, the behavior of a driver is monitored by a single camera, but the single camera has a blind area, so that the behavior of the driver cannot be completely reflected, and the monitoring result is incomplete; the prior art also provides a scheme for separately processing data of two cameras, but the processing efficiency of the scheme is low.

Disclosure of Invention

The invention provides a method, a device and equipment for monitoring behaviors of a vehicle booking driver, which solve the problems of poor detection effect and low processing efficiency of the current scheme for monitoring the behaviors of the vehicle booking driver.

In a first aspect, the present invention provides a method for monitoring behavior of a networked car booking driver, the method comprising:

simultaneously acquiring first monitoring data acquired by a video monitoring system DVR and second monitoring data acquired by a driver detection system DMS, and respectively labeling data of overlapping areas belonging to the same shooting area in the first monitoring data and the second monitoring data;

performing feature extraction on the first monitoring data to obtain first low-level feature data corresponding to data in a non-overlapping area and third low-level feature data corresponding to data in an overlapping area in the first monitoring data;

performing feature extraction on the second monitoring data to obtain second low-level feature data corresponding to data in a non-overlapping area and fourth low-level feature data corresponding to data in an overlapping area in the second monitoring data;

respectively performing multi-channel feature fusion on the first low-level feature data and the second low-level feature data, and performing feature fusion in a spatial domain on the third low-level feature data and the fourth low-level feature data;

and determining whether the driver is in an abnormal behavior state or not according to fusion data obtained after the multi-channel feature fusion and the spatial domain feature fusion.

Optionally, determining whether the driver is in an abnormal behavior state according to fusion data obtained after the multi-channel feature fusion and the feature fusion of the spatial domain, including:

performing multi-channel feature fusion on the first low-level feature data by using a deep convolutional neural network to obtain fusion data, and performing deep feature extraction to obtain first deep-level feature data;

performing multi-channel feature fusion on the second low-level feature data by using a deep convolutional neural network to obtain fusion data, and performing deep feature extraction to obtain second deep-level feature data;

performing spatial domain feature fusion on the third low-level feature data and the fourth low-level feature data by using a deep convolutional neural network to obtain fusion data, and performing deep feature extraction to obtain comprehensive deep feature data;

carrying out weighted feature fusion on the first deep feature data, the second deep feature data and the comprehensive deep feature data to obtain feature expression;

and determining whether the driver is in an abnormal behavior state or not according to the characteristic expression.

Optionally, performing multi-channel feature fusion on the first low-level feature data, including:

wherein, FeataA (c)_kI, j) is the c-th data of the first fusion data obtained by multi-channel feature fusion of the first low-layer feature data_kData in lane i, row and column j, FeatA (c)_kI, j) is the c_kThe first low-level feature data of row j column of channel i,

ch is a fixed value and Ch is the number of channels.

Optionally, performing multi-channel feature fusion on the second low-level feature data, including:

wherein, FeatBB (c)_kI, j) is the c-th data of the second fusion data obtained by multi-channel feature fusion of the second low-level feature data_kChannel i row j column data, FeatB (c)_kI, j) is the c_kThe second low-level feature data of row j column of channel i,

ch is a fixed value and Ch is the number of channels.

Optionally, performing feature fusion of a spatial domain on the third low-level feature data and the fourth low-level feature data, including:

labeling any four points in the data belonging to the overlapping region in the first monitoring data, and determining coordinates of the labeled four points;

determining four coincident points corresponding to the four marked points in the data belonging to the overlapping area in the second monitoring data, and determining coordinates of the four coincident points;

calculating a homography matrix H reflecting the mapping relation of the third low-level feature data and the fourth low-level feature data according to the coordinates of the marked four points and the coordinates of the corresponding four coincident points;

and performing spatial domain feature fusion on the third low-level feature data and the fourth low-level feature data according to the homography matrix H.

Optionally, performing feature fusion of a spatial domain on the third low-level feature data and the fourth low-level feature data according to the homography matrix H, including:

according to the formula FeataB (c)_k，i，j)＝α*FeatA(c_k，i，j)+β*FeatB(c_kM, n) for the third and fourth low-level feature dataPerforming feature fusion of a spatial domain;

wherein the content of the first and second substances,

α＝|FeatA(c_k，i，j)|/(λ+|FeatA(c_k，i，j)|+|FeatB(c_k，m，n)|)，

β＝|FeatB(c_k，i，j)|/(λ+|FeatA(c_k，i，j)|+|FeatB(c_k，m，n)|)，

the feature fusion method includes the steps that FeataB is comprehensive fusion data obtained by performing feature fusion of a space domain on third low-level feature data and fourth low-level feature data, alpha is a feature weight value of the third low-level feature data, beta is a feature weight value of the fourth low-level feature data, coordinates (m, n) are coordinates of a point, on the third low-level feature data FeataA, of which the coordinates are (i, j) and which corresponds to the fourth low-level feature data FeatB, calculated according to the homography matrix H, and lambda is a fixed value.

Optionally, performing feature extraction on the first monitoring data to obtain first low-level feature data corresponding to data in a non-overlapping region and third low-level feature data corresponding to data in an overlapping region in the first monitoring data, including:

performing face recognition on the first monitoring data to obtain face data, and expanding the range of the face data by a preset proportion to obtain first intermediate data;

on a spatial domain, adjusting the first intermediate data to a preset size to obtain a first data block;

and extracting the low-level characteristics of the first data block by using a first convolutional neural network established by taking a residual error network structure resnet as a base network, and obtaining the first low-level characteristic data and the third low-level characteristic data.

Optionally, performing feature extraction on the second monitoring data to obtain second low-level feature data corresponding to data in a non-overlapping region and fourth low-level feature data corresponding to data in an overlapping region in the second monitoring data, including:

performing face recognition on the second monitoring data to obtain face data, and expanding the range of the face data by a preset proportion to obtain second intermediate data;

on the spatial domain, adjusting the second intermediate data to a preset size to obtain a second data block;

and extracting the low-level features of the second data block by using a second convolutional neural network established by taking a residual error network structure resnet as a base network, so as to obtain second low-level feature data and fourth low-level feature data.

Optionally, determining whether the driver is in the abnormal behavior state before determining according to the feature expression further includes:

adding a full connection layer after the characteristic expression, and constructing an abnormal behavior detection model;

and taking the first monitoring data and the second monitoring data as input, taking whether a driver is in an abnormal behavior state as output, and performing model training on the abnormal behavior detection model by using a loss function.

Optionally, after determining whether the driver is in an abnormal behavior state according to fusion data obtained after the multi-channel feature fusion and the feature fusion of the spatial domain, the method further includes:

and determining that the driver is in an abnormal behavior state, uploading the video or image for recording the abnormal behavior of the driver to the IOV and giving an alarm.

In a second aspect, the present invention provides a network appointment driver behavior monitoring device comprising a memory and a processor, wherein:

the memory is used for storing a computer program;

the processor is used for reading the program in the memory and executing the following steps:

Optionally, the determining, by the processor, whether the driver is in the abnormal behavior state according to fusion data obtained after the multi-channel feature fusion and the feature fusion of the spatial domain includes:

Optionally, the processor performs multi-channel feature fusion on the first low-level feature data, including:

wherein, FeataA (c)_kI, j) is the c-th data of the first fusion data obtained by multi-channel feature fusion of the first low-layer feature data_kChannel i row j column data, FeatA (c)_kI, j) is the c_kThe first low-level feature data of row j column of channel i,

ch is a fixed value, and Ch is the number of channels.

Optionally, the processor performs multi-channel feature fusion on the second low-level feature data, including:

wherein, FeatBB (c)_kI, j) is the c-th data of the second fusion data obtained by multi-channel feature fusion of the second low-level feature data_kChannel i row j column data, FeatB (c)_kI, j) is the c_kThe second lower-level feature data of row j column of channel i,

ch is a fixed value and Ch is the number of channels.

Optionally, the performing, by the processor, feature fusion in a spatial domain on the third low-level feature data and the fourth low-level feature data includes:

Optionally, the performing, by the processor, feature fusion in a spatial domain on the third low-level feature data and the fourth low-level feature data according to the homography matrix H includes:

according to the formula FeataB (c)_k，i，j)＝α*FeatA(c_k，i，j)+β*FeatB(c_kM, n), performing feature fusion of a spatial domain on the third low-level feature data and the fourth low-level feature data;

wherein the content of the first and second substances,

α＝|FeatA(c_k，i，j)|/(λ+|FeatA(c_k，i，j)|+|FeatB(c_k，m，n)|)，

β＝|FeatB(c_k，i，j)|/(λ+|FeatA(c_k，i，j)|+|FeatB(c_k，m，n)|)，

Optionally, the performing, by the processor, feature extraction on the first monitoring data to obtain first low-level feature data corresponding to data in a non-overlapping region and third low-level feature data corresponding to data in an overlapping region in the first monitoring data includes:

Optionally, the performing, by the processor, feature extraction on the second monitoring data to obtain second low-level feature data corresponding to data in a non-overlapping region and fourth low-level feature data corresponding to data in an overlapping region in the second monitoring data includes:

Optionally, the processor determines whether the driver is in an abnormal behavior state before determining, according to the feature expression, further:

Optionally, the processor is further configured to, after determining whether the driver is in an abnormal behavior state according to fusion data obtained after the multi-channel feature fusion and the feature fusion of the spatial domain, further:

In a third aspect, the present invention provides a device for monitoring behavior of a vehicle booking driver, comprising:

the data acquisition unit is used for simultaneously acquiring first monitoring data acquired by a video monitoring system DVR and second monitoring data acquired by a driver detection system DMS and respectively marking data of overlapping areas of the same shooting area in the first monitoring data and the second monitoring data;

the first feature extraction unit is used for performing feature extraction on the first monitoring data to obtain first low-level feature data corresponding to data in a non-overlapping area and third low-level feature data corresponding to data in an overlapping area in the first monitoring data;

the second feature extraction unit is used for performing feature extraction on the second monitoring data to obtain second low-level feature data corresponding to data in a non-overlapping area and fourth low-level feature data corresponding to data in an overlapping area in the second monitoring data;

the feature fusion unit is used for respectively performing multi-channel feature fusion on the first low-layer feature data and the second low-layer feature data and performing spatial domain feature fusion on the third low-layer feature data and the fourth low-layer feature data;

and the abnormality determination unit is used for determining whether the driver is in an abnormal behavior state according to fusion data obtained after the multi-channel feature fusion and the feature fusion of the spatial domain.

Optionally, the determining unit determines whether the driver is in an abnormal behavior state according to fusion data obtained by multi-channel feature fusion and feature fusion of a spatial domain, including:

Optionally, the feature fusion unit performs multi-channel feature fusion on the first low-level feature data, including:

wherein, FeataA (c)_kI, j) is the c-th data of the first fusion data obtained by multi-channel feature fusion of the first low-level feature data_kChannel i row j column data, FeatA (c)_kI, j) is the c_kThe first low-level feature data of row j column of channel i,

ch is a fixed value and Ch is the number of channels.

Optionally, the feature fusion unit performs multi-channel feature fusion on the second low-level feature data, including:

ch is a fixed value and Ch is the number of channels.

Optionally, the feature fusion unit performs feature fusion of a spatial domain on the third low-level feature data and the fourth low-level feature data, including:

Optionally, the performing, by the feature fusion unit, feature fusion of a spatial domain on the third low-level feature data and the fourth low-level feature data according to the homography matrix H includes:

wherein the content of the first and second substances,

α＝|FeatA(c_k，i，j)|/(λ+|FeatA(c_k，i，j)|+|FeatB(c_k，m，n)|)，

β＝|FeatB(c_k，i，j)|/(λ+|FeatA(c_k，i，j)|+|FeatB(c_k，m，n)|)，

Optionally, the feature extracting unit performs feature extraction on the first monitoring data to obtain first low-level feature data corresponding to data in a non-overlapping area and third low-level feature data corresponding to data in an overlapping area in the first monitoring data, and the feature extracting unit includes:

Optionally, the feature extracting unit performs feature extraction on the second monitoring data to obtain second low-level feature data corresponding to data in a non-overlapping area and fourth low-level feature data corresponding to data in an overlapping area in the second monitoring data, and the feature extracting unit includes:

Optionally, the abnormality determination unit determines whether the driver is in an abnormal behavior state before according to the feature expression, and is further configured to:

Optionally, the abnormality determination unit is further configured to, after determining whether the driver is in an abnormal behavior state according to fusion data obtained after the multi-channel feature fusion and the feature fusion of the spatial domain, further:

In a fourth aspect, the present invention provides a computer program medium having a computer program stored thereon, which when executed by a processor, performs the steps of a method of networked car reservation driver behavior monitoring as provided in the first aspect above.

The method, the device and the equipment for monitoring the behavior of the online taxi booking driver have the following beneficial effects that:

the data in the network car booking are synchronously collected in real time through a DVR (digital video recorder) and a DMS (Driver Monitor System), so that the blind area of a camera can be reduced, the Driver of the network car booking can be subjected to large-angle behavior detection, and the monitoring performance is improved; the collected data are subjected to feature level fusion, the non-overlapping regions are subjected to multi-channel feature fusion, the overlapping regions are subjected to spatial domain feature fusion, the processing efficiency is improved, the comprehensive features are more perfect, the redundant features of the overlapping regions can be removed, and the overall effect is better.

Drawings

Fig. 1 is an application scene diagram of monitoring the behavior of a networked car booking driver according to an embodiment of the present invention;

FIG. 2 is a flow chart of a method for monitoring behavior of a networked car booking driver according to an embodiment of the invention;

FIG. 3 is a diagram illustrating data acquisition according to an embodiment of the present invention;

FIG. 4 is a diagram illustrating low-level feature extraction according to an embodiment of the present invention;

FIG. 5 is a schematic diagram of feature fusion provided by an embodiment of the present invention;

FIG. 6 is a schematic diagram of determining whether the driver is in an abnormal behavior state according to the characteristic expression provided by the embodiment of the invention;

FIG. 7 is a schematic diagram of a device for monitoring behavior of a networked car booking driver according to an embodiment of the invention;

fig. 8 is a schematic diagram of a device for monitoring behavior of a networked car reservation driver according to an embodiment of the present invention.

Detailed Description

The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only some embodiments of the present application, and not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.

It should be noted that the embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present disclosure, as detailed in the appended claims.

Hereinafter, some terms in the embodiments of the present disclosure are explained to facilitate understanding by those skilled in the art.

(1) In the embodiment of the present disclosure, the term "video monitoring system" (DVR) is a set of computer systems that stores and processes images by using computers, networks and video capture tools, and can sense the environment in the vehicle, and the system has the functions of recording images and voices for a long time, and remotely monitoring and controlling the images and voices.

(2) The term "Driver detection System" (DMS, Driver Monitor System) in this disclosed embodiment can carry out the perception to the environment in the car, mainly realizes the monitoring function to Driver's identification, Driver fatigue monitoring and dangerous driving action, can save and upload the high in the clouds to key data, promotion intelligence driving safety level that can be better, improvement vehicle safety of traveling.

(3) In the embodiment of the present disclosure, the term "secure intelligent terminal" (AI-BOX, intelligent BOX) needs to complete the learning and processing of multi-view data, receive and analyze and process related data in real time according to a built-in algorithm or model, and upload the processed result.

(4) In the embodiment of the present disclosure, the term "vehicle networking cloud platform" (IOV, Internet of Vehicles Command Center), in which a vehicle-mounted device on a vehicle collects dynamic information of all Vehicles in an information network platform through a wireless communication technology, and the vehicle networking cloud platform records and stores the collected information to be called and utilized subsequently.

(5) In the embodiment of the present disclosure, the term "homography matrix" refers to that the homography can be found in reverse when projection is performed, for example, an object can obtain two different photos by rotating a camera lens, and the contents of the two photos only need to partially correspond to each other; the homography matrix mainly refers to a plane homography matrix, and XYZ and Z are 1 in three-axis coordinates. The homography matrix is mainly used to solve two problems: representing the perspective transformation of a plane in the real world and the image corresponding to the plane; from one view to another by a perspective transformation.

(6) In the embodiment of the present disclosure, the term "residual network structure" means that, through the residual network structure, the final classification effect is still good while the network layers reach multiple layers.

To make the objects, technical solutions and advantages of the present disclosure clearer, the present disclosure will be described in further detail with reference to the accompanying drawings, and it is apparent that the described embodiments are only a part of the embodiments of the present disclosure, rather than all embodiments. All other embodiments, which can be derived by one of ordinary skill in the art from the embodiments disclosed herein without making any creative effort, shall fall within the scope of protection of the present disclosure.

Based on the above problems, embodiments of the present invention provide a method, an apparatus, and a device for monitoring behaviors of a network car booking driver, which can synchronously acquire data in the network car booking in real time through a video monitoring system DVR and a driver detection system DMS, and perform feature level fusion on the acquired data, thereby improving processing efficiency and monitoring performance. The following provides an implementation mode of a method, a device and equipment for monitoring the behavior of a network appointment vehicle driver, which are provided by the embodiment of the invention.

Example 1

As shown in fig. 1, an embodiment of the present invention provides an application scenario for monitoring behavior of a networked car booking driver, including:

the system comprises a 101 video monitoring system, a safety intelligent terminal and a data processing system, wherein the video monitoring system is installed in a network appointment car and is used for recording the behavior of a driver of the network appointment car in real time and sending the recorded data to the safety intelligent terminal in real time;

the system comprises a 102 driver detection system, a safety intelligent terminal and a data processing system, wherein the 102 driver detection system is arranged in a network appointment vehicle and is used for recording the behavior of a network appointment vehicle driver in real time and sending the recorded data to the safety intelligent terminal in real time;

and the 103 safety intelligent terminal is used for receiving the real-time data sent by the video monitoring system and the driver detection system, carrying out real-time processing and analysis on the data, judging whether the driver of the networked car has abnormal behaviors or not, uploading the video or image recording the abnormal behaviors of the driver to the internet of vehicles cloud platform and giving an alarm.

The abnormal behavior of the driver includes abnormal behaviors such as smoking, calling, unbelting, yawning, and the like.

It should be noted that the operation of uploading the video or image recording the abnormal behavior of the driver to the internet of vehicles cloud platform and giving an alarm is not limited to recording relevant information and giving an alarm, and any operation of recording an image and performing subsequent processing on abnormal state information may be implemented on the safety intelligent terminal provided by this embodiment.

And 104, the Internet of vehicles cloud platform is used for receiving and storing the video or the image for recording the abnormal behavior of the driver.

It should be noted that, when the video monitoring system and the driver detection system are installed and arranged in the online taxi appointment system, a blind area of vision is avoided as much as possible, and it is ensured that data collected by the video monitoring system and the driver detection system can completely record the conditions in the online taxi appointment system, especially the face of the driver and the area near the face.

It should be noted that the application scenario is only an example of an application scenario for monitoring the behavior of the car booking driver, and does not constitute a specific limitation, and the application scenario for monitoring the behavior of the car booking driver may add or subtract modules on the basis of the application scenario, for example, add one or more data acquisition modules.

It should be noted that, when two or more camera devices are used for data acquisition of full coverage in a vehicle, the above scheme for monitoring the behavior of the networked car reservation driver can be expanded to be used as a scheme for monitoring the behavior of passengers.

As shown in fig. 2, an embodiment of the present invention provides a flowchart of a method for monitoring behavior of a networked car booking driver, including:

step S201, simultaneously acquiring first monitoring data acquired by a video monitoring system DVR and second monitoring data acquired by a driver detection system DMS, and respectively labeling data of overlapping areas belonging to the same shooting area in the first monitoring data and the second monitoring data;

it should be noted that the first and second monitoring Data of the first monitoring Data and the second monitoring Data are only used for distinguishing the Data collected by the video monitoring system DVR and the driver detection system DMS, and there is no actual difference between the quality and the quality.

As an alternative implementation, as shown in fig. 3, an embodiment of the present invention provides a schematic diagram of data acquisition, where the shaded portion in fig. 3 is data of the marked overlapping area.

Step S202, performing feature extraction on the first monitoring data to obtain first low-level feature data corresponding to data in a non-overlapping area and third low-level feature data corresponding to data in an overlapping area in the first monitoring data;

as an optional implementation manner, performing feature extraction on the first monitoring data to obtain first low-level feature data corresponding to data in a non-overlapping region and third low-level feature data corresponding to data in an overlapping region in the first monitoring data includes:

It should be noted that, near the face of the driver, feature extraction is performed on the real-time data to obtain the first low-level feature data and the third low-level feature data FeatA.

As an optional implementation, the first convolutional neural network has a structure including: convolutional layer conv, batch normalization layer bn, activation function layer relu, pooling layer pooling.

The first intermediate data Da includes data of an overlapping area and data of a non-overlapping area mapped in synchronization with the data of the overlapping area.

It should be noted that the preset ratio can be specifically set according to a specific application scenario, and when the preset ratio of the outward expansion is 25% to 35%, the processing effect is better.

It should be noted that the preset size may be changed according to specific data conditions, and as an alternative embodiment, the preset size of the first intermediate data Da is set to be 3 × 224, and after the setting, the sizes of the first low-level feature data and the third low-level feature data FeatA obtained are 256 × 56.

Step S203, performing feature extraction on the second monitoring data to obtain second low-level feature data corresponding to data in a non-overlapping area and fourth low-level feature data corresponding to data in an overlapping area in the second monitoring data;

it should be noted that, near the face of the driver, feature extraction is performed on the real-time data to obtain second low-level feature data and fourth low-level feature data FeatB.

As an optional implementation manner, performing feature extraction on the second monitoring data to obtain second low-level feature data corresponding to data in a non-overlapping region and fourth low-level feature data corresponding to data in an overlapping region in the second monitoring data includes:

As an alternative implementation, the structure of the second convolutional neural network includes: convolutional layer conv, batch normalization layer bn, activation function layer relu, pooling layer pooling.

The second intermediate data includes data of an overlapping area and data of a non-overlapping area mapped in synchronization with the data of the overlapping area.

It should be noted that the preset size may be changed according to specific data situations, and as an alternative embodiment, the preset size of the second intermediate data Db is set to be 3 × 224, and after the setting, the sizes of the second low-level feature data and the fourth low-level feature data FeatB obtained are 256 × 56.

As an alternative implementation, as shown in fig. 4, an embodiment of the present invention provides a schematic diagram of low-level feature extraction, where the data in the overlap area in fig. 4 is the data after the low-level feature extraction, that is, the third low-level feature data and the fourth low-level feature data.

Step S204, respectively performing multi-channel feature fusion on the first low-level feature data and the second low-level feature data, and performing feature fusion in a spatial domain on the third low-level feature data and the fourth low-level feature data;

the feature data FeatA and FeatB, that is, the first low-level feature data and the second low-level feature data, are subjected to multi-channel feature fusion to obtain FeatAA and FeatBB, and the feature of the overlap region, that is, the third low-level feature data and the fourth low-level feature data, are subjected to feature fusion in the spatial domain to form FeatAB.

It should be noted that the multi-channel feature fusion functions to enhance the data fusion capability between channels, and the feature fusion in the spatial domain functions to be information fusion with weight in space.

When performing spatial domain feature fusion on third low-level feature data and fourth low-level feature data, it is necessary to calculate a homography matrix H reflecting a mapping relationship between the third low-level feature data and the fourth low-level feature data in advance, and then perform spatial domain feature fusion on the third low-level feature data and the fourth low-level feature data according to the homography matrix H.

And S205, determining whether the driver is in an abnormal behavior state according to fusion data obtained after the multi-channel feature fusion and the feature fusion of the spatial domain.

The above FeatAA, FeatBB and FeatAB are further expressed by a deep convolutional neural network decibel to obtain first deep feature data Feat1, second deep feature data Feat2 and comprehensive deep feature data Feat 3;

and (3) performing weighted feature fusion on the three features Feat1, Feat2 and Feat3 to judge the abnormal behavior of the driver.

As an optional implementation manner, determining whether the driver is in an abnormal behavior state according to fusion data obtained after multi-channel feature fusion and feature fusion in a spatial domain includes:

As an alternative embodiment, the deep convolutional neural network includes: convolutional layer conv, batch normalization layer bn, activation function layer relu, pooling layer pooling.

As an alternative implementation, as shown in fig. 5, an embodiment of the present invention provides a schematic diagram of feature fusion. The method comprises the steps of carrying out multichannel feature fusion on the acquired first low-level feature data to obtain first fusion data FeataA, carrying out spatial domain feature fusion on the third low-level feature data and the fourth low-level feature data to obtain comprehensive fusion data FeataB, carrying out multichannel feature fusion on the second low-level feature data to obtain second fusion data FeatBB, and extracting high-level features by using a deeper neural network CNNC to obtain first deep-level feature data Feat1, second deep-level feature data Feat2 and comprehensive deep-level feature data Feat 3.

As an optional implementation manner, performing weighted feature fusion on the first deep feature data, the second deep feature data, and the comprehensive deep feature data to obtain a feature expression, including:

aiming at the acquired first deep characteristic data Feat1, second deep characteristic data Feat2 and comprehensive deep characteristic data Feat3, according to the importance degree, weight coefficients w1, w2 and w3 are correspondingly set, and contact operation is carried out, so that the final characteristic expression is obtained:

Feat＝contact(w1*Feat1)+contact(w2*Feat2)+contact(w3*Feat3)。

it should be noted that, the concat is used to fuse the features, and the concat operation is the merging of the number of channels, that is, the number of features describing the image itself, that is, the number of channels is increased, but the information under each feature is not increased.

As an alternative embodiment, the final output lengths of the first deep feature data Feat1, the second deep feature data Feat2 and the integrated deep feature data Feat3 are 512 x 1, 1024 x 1, respectively, and the final feature expression Feat has a length of 2048 x 1.

As shown in fig. 6, the embodiment of the present invention provides a schematic diagram for determining whether the driver is in an abnormal behavior state according to the feature expression.

As an optional implementation manner, determining whether the driver is in the abnormal behavior state before the characteristic expression, further includes:

As an optional implementation, performing multi-channel feature fusion on the first low-level feature data includes:

wherein, FeataA (c)_kI, j) is the c-th data of the first fusion data obtained by multi-channel feature fusion of the first low-level feature data_kChannel i row j column data, FeatA (c)_kI, j) is the c_kFirst low-level feature data of i rows and j columns of channel，

Ch is a fixed value and Ch is the number of channels.

The above FeatA (c) is defined_kNumber of channels c in i, j)_kThe number of rows i and the number of columns j are the number of channels c of the first low-level feature data_kRow number i, column number j, and channel number c of third low-level feature data_kThe number of rows i and the number of columns j are different.

As an optional implementation, performing multi-channel feature fusion on the second low-level feature data includes:

wherein, FeatBB (c)_kI, j) is the c-th data of the second fusion data obtained by multi-channel feature fusion of the second low-layer feature data_kChannel i row j column data, FeatB (c)_kI, j) is the c_kThe second low-level feature data of row j column of channel i,

ch is a fixed value and Ch is the number of channels.

The above FeatB (c) is defined_kNumber of channels c in i, j)_kThe number of rows i and the number of columns j are the number of channels c of the second low-level feature data_kRow number i, column number j, and channel number c of the fourth low-level feature data_kThe number of rows i and the number of columns j are different.

As an optional implementation, performing feature fusion in a spatial domain on the third low-level feature data and the fourth low-level feature data includes:

As an alternative embodiment, the homography matrix H is a matrix with a size of 3 × 3, and is obtained by calculating the coordinates of the marked points and the coordinates of the corresponding coincident points.

As an optional implementation manner, calculating a homography matrix H reflecting a mapping relationship between the third low-level feature data and the fourth low-level feature data according to the coordinates of the labeled four points and the coordinates of the corresponding four coincident points includes:

wherein (DA)_x，DA_y) For the coordinates of the noted point, (DB)_x，DB_y) The coordinates of the corresponding coincident points.

As an optional implementation manner, performing feature fusion in a spatial domain on the third low-level feature data and the fourth low-level feature data according to the homography matrix H includes:

wherein the content of the first and second substances,

α＝|FeatA(c_k，i，j)|/(λ+|FeatA(c_k，i，j)|+|FeatB(c_k，m，n)|)，

β＝|FeatB(c_k，i，j)|/(λ+|FeatA(c_k，i，j)|+|FeatB(c_k，m，n)|)，

The above FeatA (c) is defined_kNumber of channels c in i, j)_kThe number of rows i and the number of columns j are the number of channels c of the third low-level feature data_kThe number of rows i, the number of columns j, and the number of channels c of the first low-level feature data_kThe number of rows i and the number of columns j are different.

The above FeatB (c) is defined_kNumber of channels c in i, j)_kThe number of rows i and the number of columns j are the number of channels c of the fourth low-level feature data_kThe number of rows i, the number of columns j, and the number of channels c of the second low-level feature data_kThe number of rows i and the number of columns j are different.

The feature weight value α of the third low-level feature data and the feature weight value β of the fourth low-level feature data are determined according to specific data and specific conditions of an implementation environment.

It should be noted that, the specific implementation manner for performing multi-channel feature fusion and performing feature fusion in spatial domain on data given in the foregoing embodiment is only one implementation manner given in the embodiment of the present invention, and does not specifically limit the present invention, and any implementation manner that can implement multi-channel feature fusion and feature fusion in spatial domain may be applied in the embodiment of the present invention.

Example 2

An embodiment of the present invention provides a device 700 for monitoring behavior of a networked car booking driver, including a memory 701 and a processor 702, as shown in fig. 7, wherein:

the memory is used for storing a computer program;

ch is a fixed value and Ch is the number of channels.

ch is a fixed value and Ch is the number of channels.

wherein the content of the first and second substances,

α＝|FeatA(c_k，i，j)|/(λ+|FeatA(c_k，i，j)|+|FeatB(c_k，m，n)|)，

β＝|FeatB(c_k，i，j)|/(λ+|FeatA(c_k，i，j)|+|FeatB(c_k，m，n)|)，

and determining that the driver is in an abnormal behavior state, uploading the video or image recording the abnormal behavior of the driver to an IOV (Internet of vehicles) cloud platform and giving an alarm.

The embodiment of the invention provides a device for monitoring behaviors of a networked taxi appointment driver, which comprises the following components in percentage by weight as shown in fig. 8:

a data obtaining unit 801, configured to obtain first monitoring data collected by a video monitoring system DVR and second monitoring data collected by a driver detection system DMS at the same time, and label data of overlapping areas belonging to the same shooting area in the first monitoring data and the second monitoring data, respectively;

a first feature extraction unit 802, configured to perform feature extraction on the first monitoring data, to obtain first low-level feature data corresponding to data in a non-overlapping area and third low-level feature data corresponding to data in an overlapping area in the first monitoring data;

a second feature extraction unit 803, configured to perform feature extraction on the second monitoring data to obtain second low-level feature data corresponding to data in a non-overlapping area and fourth low-level feature data corresponding to data in an overlapping area in the second monitoring data;

a feature fusion unit 804, configured to perform multi-channel feature fusion on the first low-level feature data and the second low-level feature data, respectively, and perform feature fusion in a spatial domain on the third low-level feature data and the fourth low-level feature data;

and an abnormal determination unit 805 configured to determine whether the driver is in an abnormal behavior state according to fusion data obtained by multi-channel feature fusion and feature fusion in a spatial domain.

performing spatial domain feature fusion on the third low-level feature data and the fourth low-level feature data by using a deep convolutional neural network to obtain fusion data, and performing deep level feature extraction to obtain comprehensive deep-level feature data;

ch is a fixed value and Ch is the number of channels.

ch is a fixed value and Ch is the number of channels.

wherein the content of the first and second substances,

α＝|FeatA(c_k，i，j)|/(λ+|FeatA(c_k，i，j)|+|FeatB(c_k，m，n)|)，

β＝|FeatB(c_k，i，j)|/(λ+|FeatA(c_k，i，j)|+|FeatB(c_k，m，n)|)，

and extracting the low-level features of the first data block by using a first convolutional neural network established by taking a residual error network structure resnet as a baseline network to obtain first low-level feature data and third low-level feature data.

The present invention also provides a computer program medium having a computer program stored thereon, which when executed by a processor, implements the steps of a method for monitoring behavior of a networked car booking driver as provided in embodiment 2 above.

In the several embodiments provided in the present application, it should be understood that the disclosed system, apparatus and method may be implemented in other manners. For example, the above-described apparatus embodiments are merely illustrative, and for example, the division of the modules is only one logical functional division, and other divisions may be realized in practice, for example, a plurality of modules or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or modules, and may be in an electrical, mechanical or other form.

The modules described as separate parts may or may not be physically separate, and parts displayed as modules may or may not be physical modules, may be located in one place, or may be distributed on a plurality of network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment.

In addition, functional modules in the embodiments of the present application may be integrated into one processing module, or each of the modules may exist alone physically, or two or more modules are integrated into one module. The integrated module can be realized in a hardware mode, and can also be realized in a software functional module mode. The integrated module, if implemented in the form of a software functional module and sold or used as a stand-alone product, may be stored in a computer readable storage medium.

In the above embodiments, the implementation may be wholly or partially realized by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product.

The computer program product includes one or more computer instructions. When loaded and executed on a computer, cause the processes or functions described in accordance with the embodiments of the application to occur, in whole or in part. The computer may be a general purpose computer, a special purpose computer, a network of computers, or other programmable device. The computer instructions may be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another, for example, from one website site, computer, server, or data center to another website site, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can store or a data storage device, such as a server, a data center, etc., that is integrated with one or more available media. The usable medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a Solid State Disk (SSD)), among others.

The technical solutions provided by the present application are introduced in detail, and the present application applies specific examples to explain the principles and embodiments of the present application, and the descriptions of the above examples are only used to help understand the method and the core ideas of the present application; meanwhile, for a person skilled in the art, according to the idea of the present application, there may be variations in the specific embodiments and the application scope, and in summary, the content of the present specification should not be construed as a limitation to the present application.

As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and so forth) having computer-usable program code embodied therein.

The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to the application. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

It will be apparent to those skilled in the art that various changes and modifications may be made in the present application without departing from the spirit and scope of the application. Thus, if such modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is intended to include such modifications and variations as well.

Claims

1. A method for monitoring behavior of a networked car booking driver, comprising:

2. The method according to claim 1, wherein determining whether the driver is in an abnormal behavior state according to fusion data obtained after multi-channel feature fusion and feature fusion in a spatial domain comprises:

3. The method of claim 1, wherein performing multi-pass feature fusion on the first low-level feature data comprises:

ch is a fixed value, and Ch is the number of channels.

4. The method of claim 1, wherein performing multi-pass feature fusion on the second low-level feature data comprises:

ch is a fixed value and Ch is the number of channels.

5. The method of claim 1, wherein performing feature fusion in the spatial domain on the third and fourth lower-level feature data comprises:

6. The method according to claim 5, wherein performing spatial domain feature fusion on the third and fourth low-level feature data according to the homography matrix H comprises:

wherein the content of the first and second substances,

α＝|FeatA(c_k，i，j)|/(λ+|FeatA(c_k，i，j)|+|FeatB(c_k，m，n)|)，

β＝|FeatB(c_k，i，j)|/(λ+|FeatA(c_k，i，j)|+|FeatB(c_k，m，n)|)，

7. The method according to claim 1, wherein performing feature extraction on the first monitoring data to obtain first low-level feature data corresponding to data in a non-overlapping region and third low-level feature data corresponding to data in an overlapping region in the first monitoring data comprises:

8. The method according to claim 1, wherein performing feature extraction on the second monitoring data to obtain second low-level feature data corresponding to data in a non-overlapping region and fourth low-level feature data corresponding to data in an overlapping region in the second monitoring data comprises:

9. The method of claim 2, wherein determining whether the driver is in an abnormal behavior state before determining from the characterization expression, further comprises:

10. The method according to claim 1, wherein after determining whether the driver is in the abnormal behavior state according to the fusion data obtained after the multi-channel feature fusion and the feature fusion in the spatial domain, the method further comprises:

11. A net appointment vehicle driver behavior monitoring device comprising a memory and a processor, wherein:

the memory is used for storing a computer program;

the processor is used for reading the program in the memory and executing the method for monitoring the behavior of the online taxi appointment driver as claimed in any one of claims 1 to 10.

12. A computer program medium, characterized in that a computer program is stored thereon which, when being executed by a processor, carries out the steps of a method for monitoring behaviour of a vehicle-reserving driver according to any one of claims 1-10.