CN112597892B - System and method for detecting remnants in automobile cabin - Google Patents

System and method for detecting remnants in automobile cabin Download PDF

Info

Publication number
CN112597892B
CN112597892B CN202011539422.8A CN202011539422A CN112597892B CN 112597892 B CN112597892 B CN 112597892B CN 202011539422 A CN202011539422 A CN 202011539422A CN 112597892 B CN112597892 B CN 112597892B
Authority
CN
China
Prior art keywords
image
passengers
vehicle
row
suspicious
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202011539422.8A
Other languages
Chinese (zh)
Other versions
CN112597892A (en
Inventor
米建勋
高翔
钱基业
陈涛
向菲
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Chongqing University of Post and Telecommunications
Original Assignee
Chongqing University of Post and Telecommunications
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Chongqing University of Post and Telecommunications filed Critical Chongqing University of Post and Telecommunications
Priority to CN202011539422.8A priority Critical patent/CN112597892B/en
Publication of CN112597892A publication Critical patent/CN112597892A/en
Application granted granted Critical
Publication of CN112597892B publication Critical patent/CN112597892B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/50—Context or environment of the image
    • G06V20/59—Context or environment of the image inside of a vehicle, e.g. relating to seat occupancy, driver state or inner lighting conditions
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/20—Analysis of motion
    • G06T7/246—Analysis of motion using feature-based methods, e.g. the tracking of corners or segments
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/20—Analysis of motion
    • G06T7/254—Analysis of motion involving subtraction of images
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/20—Image preprocessing
    • G06V10/25—Determination of region of interest [ROI] or a volume of interest [VOI]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/74—Image or video pattern matching; Proximity measures in feature spaces
    • G06V10/75—Organisation of the matching processes, e.g. simultaneous or sequential comparisons of image or video features; Coarse-fine approaches, e.g. multi-scale approaches; using context analysis; Selection of dictionaries
    • G06V10/751—Comparing pixel values or logical combinations thereof, or feature values having positional relevance, e.g. template matching
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/74—Image or video pattern matching; Proximity measures in feature spaces
    • G06V10/75—Organisation of the matching processes, e.g. simultaneous or sequential comparisons of image or video features; Coarse-fine approaches, e.g. multi-scale approaches; using context analysis; Selection of dictionaries
    • G06V10/758—Involving statistics of pixels or of feature values, e.g. histogram matching
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10048—Infrared image
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30248—Vehicle exterior or interior
    • G06T2207/30268—Vehicle interior

Abstract

The invention relates to the technical field of intelligent detection, in particular to a detection system and a detection method for a remnant in an automobile cabin, wherein the method comprises the steps of setting a frame number threshold value according to the characteristic that a picture of a monitoring video in the automobile cabin is stable before passengers get on the automobile, and establishing a seat background picture; the pressure sensor under the seat records the number of passengers and the sitting positions and continuously monitors the change of pressure; when the pressure quantity is changed, carrying out the carry-over detection on the front row seats, and carrying out the carry-over detection on the vacant position areas of the rear row seats; after detecting and judging the abandoned object, identifying the type of the abandoned object according to an image classification algorithm proposed by us; after the identification is successful, the vehicle-mounted system sends a short message to inform a mobile phone of a passenger or inform a driver of reminding the passenger who just gets off to pick up the lost article; the invention utilizes the target detection method to track the remnant, replaces the idea of generally using the target tracking method, realizes a detection, namely tracking method, and overcomes the difficulty of large calculation amount of the target tracking algorithm.

Description

System and method for detecting remnants in automobile cockpit
Technical Field
The invention relates to the technical field of intelligent detection, in particular to a system and a method for detecting a left object in an automobile cabin.
Background
The existing algorithms for detecting the remnants are generally divided into three types, one is a background modeling method, namely, a picture without the remnants is established as a background, and a subsequent image is subtracted from the background to detect new articles left in the picture, wherein the method depends on the quality of the background picture; secondly, a target tracking method needs to consume a large amount of resources, which brings huge calculation burden, and the requirement is difficult to meet under the commonly applied scene, so the scheme is usually not adopted at a mobile terminal, and the tracking method is generally difficult to deal with the challenge brought by object deformation and shielding; the last type is a target detection method, which has the advantages of high speed, intuition, easy operation and the like, can clearly show the position and the type of the left-over object, has relatively mature technology, but is easily influenced by factors such as shielding and illumination change which cause the unclear picture, so that the recognition rate is reduced, and the false positive rate is increased. In daily research, people generally adopt a mode of combining any two or three types of methods to invent a more excellent carry-over detection algorithm.
In the field of video surveillance, in recent years, researchers have conducted many intensive studies on the detection of carry-over, in order to detect events which are of great interest to the public, such as identification of dangerous explosives and illegal parked vehicles, etc., to reduce the possibility of accidents, and to ensure the life safety and property safety of the people. Unlike the current hot application, a wide public space, detection of carry-over in the scene in the vehicle has recently begun to receive attention from practitioners as a problem that has been ignored by the public in the past but has been left alone.
The patent with the patent application number of CN201911244307.5 uses a visual method as well as the invention, and proposes a method of using multiple cameras and multiple visual angles to track and locate the left objects in the car, the technology relates to the cooperative work of the multiple cameras, but the burden of calculation is increased; meanwhile, the patent does not relate to the serious influence of illumination on the effect, and the practical application scene is limited.
Disclosure of Invention
Aiming at the problems, the invention provides a detection system and a detection method for the left objects in an automobile cabin, wherein the system comprises front and rear seat pressure sensors, front and rear image acquisition equipment, a center console display, a memory, an embedded processor and a vehicle control unit; wherein:
the front-row and rear-row pressure sensors are arranged below each seat of the front-row and rear-row passenger seats and used for monitoring the pressure from each seat and judging the positions of passengers and the actions of getting on and off the train according to the change of the pressure;
the system comprises front and rear rows of image acquisition equipment, a front row of image acquisition equipment and a camera, wherein the front row of image acquisition equipment is arranged above an automobile center console and is aimed at a front passenger seat through a lens, and the camera which is arranged at the top of a cabin and is obliquely positioned above a rear seat and is aimed at a rear seat through the lens uses a near infrared image format; the front row image acquisition equipment acquires an image of a passenger seat; the rear-row image acquisition equipment acquires an image of a rear-row seat;
the memory is used for storing data generated by the front and rear rows of image acquisition equipment, the front and rear rows of pressure sensors and the embedded processor;
the embedded processor is a vehicle gauge-level deep learning processor, can read, write and calculate data in the memory, and temporarily stores a calculation result in the memory or sends the calculation result to the vehicle control unit;
the vehicle control unit is used for transferring and processing the data of the whole vehicle and controlling the operation of each part of the vehicle, and in the invention, the embedded processor is communicated with the center console in a display way, so that the center console is communicated with the embedded processor in a data way;
and the console display is used for outputting a carry-over detection result from the embedded processor and informing a driver of reminding the passengers getting off to take back the carry-over.
The invention also provides a method for detecting the left objects in the automobile cabin, which comprises the following steps:
s1: reading a video, setting a frame number threshold according to the characteristic that a picture of a monitoring video in an automobile cabin is stable before passengers get on the automobile, and establishing a seat background picture;
s2: after passengers get on the vehicle, the pressure sensors under the seats record the number and the sitting positions of the passengers, and continuously monitor the pressure change;
s3: when the pressure reaches a preset threshold value, carrying out remnant detection on the front row seats, and carrying out remnant detection on the vacant position areas of the rear row seats;
s4: after detecting and judging the abandoned object, identifying the type of the abandoned object according to an image classification algorithm proposed by us;
s5: after the identification is successful, the vehicle-mounted system sends a short message to inform a mobile phone of a passenger or inform a driver of the short message to remind the passenger who just gets off to pick up the lost article.
Further, if the automobile is in a flameout or non-driving state, a clear background picture of the front row and the rear row without passengers is obtained by adopting median modeling, otherwise, a clear background picture of the front row and the rear row without passengers is obtained by adopting Gaussian modeling.
Further, during background modeling, a background picture of a rear row needs to be respectively established according to positions of three seats of the rear row of the vehicle.
Further, performing carryover detection comprises:
subtracting the current frame from the background to obtain a difference image, performing morphological operation to obtain a plurality of connected regions, and removing passengers according to the areas of the connected regions;
storing the remaining areas into a suspicious object queue, and continuously tracking the objects in the queue in the subsequent frame;
counting the number of frames of each object in the suspicious object queue and the number of lost frames by calculating the distance between the centroid coordinates, the matching degree between the gray level histograms and the matching degree between the Hu moments;
when the number of frames of a certain object exceeds the set number of frames, the object is cut off as a remnant;
when the number of the lost frames of a certain object exceeds the set number of frames, the lost frames are removed from the suspicious object queue;
if a new object is detected, it is added to the suspect queue and tracking thereof is started.
Further, the centroid coordinate distance of different objects is the sum of the corresponding abscissa and ordinate differences of the centroids of different objects, and the abscissa and ordinate of the centroid of the jth object can be expressed as:
Figure BDA0002854127240000031
the matching degree between the gray level histograms is expressed as:
Figure BDA0002854127240000032
the degree of match between the Hu moments is expressed as:
Figure BDA0002854127240000041
wherein N is the number of objects, c j Is the abscissa or ordinate of the centroid of the jth object, c i Is the abscissa or ordinate of the ith pixel of the jth object, and n is the number of pixels of the jth object; p and q are the two objects being compared, L (p, q) is the Hu moment match of the p and q objects,
Figure BDA0002854127240000042
the kth component of the Hu moment of the p object,
Figure BDA0002854127240000043
the kth component of the Hu moment of q objects, D (p, q) is the gray histogram matching degrees of p and q objects,
Figure BDA0002854127240000044
the number of pixels of s gray in the gray histogram of the p object,
Figure BDA0002854127240000045
the number of s-gray pixels in the gray histogram of the q object is shown, and m is the range of the gray histogram.
Further, the acquiring the binarized image includes:
J i (x,y)＝|F i (x,y)-B(x,y)|,i＝s,s+1……n；
Figure BDA0002854127240000046
wherein, B (x, y) is the gray value of the pixel point corresponding to the background frame, F i (x, y) is the gray value of the pixel point corresponding to the current frame; j. the design is a square i (x, y) is a differential image of the background frame and the current frame; t is a threshold value for performing binarization operation, O i (x, y) is a binarized image, s is the number of frames used to create a background picture, and n is the total number of frames.
Further, continuously tracking the objects in the queue comprises:
reading a first frame image, obtaining an original suspicious object queue Q1, creating and allocating a structural body for each suspicious object in Q1, wherein the structural body members comprise: a boundary frame coordinate, a total visible frame number, a continuous invisible frame number, a gray level histogram array and a Hu moment array;
reading the next frame of image, obtaining a suspicious object queue Q2, and similarly creating and distributing a structural body for each suspicious object in Q2;
calculating centroid coordinates, a gray histogram and a Hu moment of the suspicious object in Q1 and Q2, when the centroid distance, the matching degree of the gray histogram and the matching degree of the Hu moment of the suspicious object A1 in Q1 and the suspicious object B1 in Q2 are all smaller than a threshold value, the matching is successful, the structural body of A1 is updated to the structural body of B1, and the total number of visible frames is added by 1;
when the centroid distances, the gray histogram matching degrees and the Hu moment matching degrees of all the suspicious objects A2 in the Q1 and all the suspicious objects in the Q2 are all larger than the threshold values, adding 1 to the continuous invisible frame number of the A2, and when the value reaches the maximum invisible frame number, the A2 is successfully matched, and the continuous invisible frame number returns to zero;
the doubtful objects which are not matched in the Q2 are all added into the Q1, and the Q2 is emptied;
when the total number of visible frames of a suspicious object is greater than 10, the suspicious object is judged to be left, and when the number of continuous invisible frames of a suspicious object is greater than 20, the suspicious object is discarded.
Further, identifying the carryover category includes:
collecting historical data with labels, inputting the historical data into a neural network, and outputting the data after dimension reduction and feature extraction processing;
in the neural network, an objective function is solved for each type of label according to historical data to obtain a coding coefficient x for training, and the objective function is expressed as:
Figure BDA0002854127240000051
where A is training set data, b is test set data, λ is a trade-off parameter that balances the fidelity term and the regularization term, α i Is a trade-off parameter, x i Is the coding coefficient of each class of samples; x is the coding coefficient of all class samples; II | · |) 1 To calculate the L1 norm, | 2 In order to calculate the L2 norm,
Figure BDA0002854127240000053
to calculate the square of the L2 norm.
And classifying according to the image feature vector of the remnant and the reconstructed residual error and the dispersion degree between classes, and outputting the label of the remnant.
Further, the label of the legacy is represented as:
Figure BDA0002854127240000052
wherein label (y) represents the category information of the feature vector y of the image of the remnant; x is the coding coefficient of all classes of samples.
The invention has the following advantages and beneficial effects:
(1) the invention uses the near-infrared camera as the image acquisition equipment, compared with the visible light camera, the invention can be used in the scene without light at night and can hardly be influenced by illumination change in the scene in the daytime;
(2) according to the invention, when the background image is established for the rear seats and the objects left by the rear seats are judged, the partition thought is adopted, compared with the non-partition thought, the invention fully considers various situations of the taking of the rear passengers, and the judgment for the objects left by the rear seats is more precise;
(3) the invention utilizes a target detection method to track the remnants, replaces the idea of generally using a target tracking method, realizes a detection, namely a tracking method, and overcomes the difficulty of large calculation amount of a target tracking algorithm.
Drawings
FIG. 1 is a schematic illustration of an operating environment in an embodiment of the present invention;
FIG. 2 is an overall method flow diagram of an embodiment of the present invention;
FIG. 3 is a flow chart of carryover detection and determination in an embodiment of the present invention;
FIG. 4 is a flow chart of a carryover identification in an embodiment of the present invention;
FIG. 5 is a diagram of a legacy detection hardware system architecture according to an embodiment of the present invention;
FIG. 6 is a schematic diagram illustrating the identification result of a mobile phone left behind on a rear seat using a near-infrared image in a dark environment according to an embodiment of the present invention;
FIG. 7 is a schematic diagram illustrating the identification result of a mobile phone left on a rear seat by using a visible light image in a dark environment according to an embodiment of the present invention;
510, a high-speed bus; 520. a vehicle control unit (ECU); 530. an embedded processor; 540. a memory; 550. front and rear rows of pressure sensors; 560. front row image acquisition equipment; 570. back row image acquisition equipment; 580. and displaying on a center console.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Example 1
The embodiment provides a system for detecting a remnant in an automobile cabin, which comprises front and rear seat pressure sensors, front and rear image acquisition equipment, a center console display, a memory, an embedded processor and a vehicle control unit, as shown in fig. 1; wherein:
the front-row and rear-row pressure sensors are arranged below each seat of the front-row and rear-row passenger seats and used for monitoring the pressure from each seat and judging the positions of passengers and the actions of getting on and off the train according to the change of the pressure;
the system comprises front and rear rows of image acquisition equipment, a front row of image acquisition equipment and a camera, wherein the front row of image acquisition equipment is arranged above an automobile center console and is aimed at a front passenger seat through a lens, and the camera which is arranged at the top of a cabin and is obliquely positioned above a rear seat and is aimed at a rear seat through the lens uses a near infrared image format; the front-row image acquisition equipment acquires an image of a co-driver seat; the rear-row image acquisition equipment acquires an image of a rear-row seat;
the memory is used for storing data generated by the front and rear rows of image acquisition equipment, the front and rear rows of pressure sensors and the embedded processor;
the embedded processor is a vehicle gauge-level deep learning processor, can read, write and calculate data in the memory, and temporarily stores a calculation result in the memory or sends the calculation result to the vehicle control unit;
the vehicle control unit is used for transferring and processing the data of the whole vehicle and controlling the operation of each part of the vehicle, and in the invention, the embedded processor is electrically communicated with the display of the central console, so that the central console is communicated with the embedded processor in a data way;
and the console display is used for outputting a carry-over detection result from the embedded processor and informing a driver of reminding the passengers getting off to take back the carry-over.
In a specific implementation, as shown in fig. 5, the system includes a high-speed bus 510, a vehicle control unit (ECU)520, an embedded processor 530, a memory 540, a front-back pressure sensor 550, a front-back image capturing device 560, a back-back image capturing device 570, and a console display 580, where:
the high-speed bus 510 is used for data transmission of the whole system, including transmission of acquired image data, data of the pressure sensor and data output to the console for display;
the front and rear row pressure sensors are used for monitoring the pressure change of the seats and transmitting the numerical value generated at the moment to the memory 540 through the high-speed bus 510, the embedded processor 530 reads the pressure data at each moment in the memory 540 and judges whether a passenger gets off the vehicle or not, and when the passenger gets off the vehicle, the embedded processor 530 enables the memory 540 to stop receiving the data of the front and rear row pressure sensors 550;
the front row image acquisition equipment 560 and the rear row image acquisition equipment 570 are used for acquiring near-infrared images of front and rear seats and transmitting the near-infrared images to the memory 540 through the high-speed bus 510, the embedded processor 530 reads video frames in the memory 540, the legacy detection algorithm is operated, the calculated result is transmitted to the whole vehicle controller (ECU)520, the whole vehicle controller (ECU)520 outputs the video frames to the center console through the high-speed bus 510 to be displayed 580, after receiving information, a driver can select a closing message to remind, a closing signal is reversely transmitted to the embedded processor 530, and the algorithm operation is forcibly stopped; or when the passenger takes the article away, the algorithm can not detect the left article, and the algorithm automatically stops the operation of the algorithm;
the high-speed bus 510 includes: any one or combination of a plurality of automobile buses meeting high transmission rate, such as a CAN bus, a FlexRay bus or an MOST/1394 bus;
the memory 540 includes: any one or more of various media that can be stored, such as a usb disk, a removable hard disk, a Read-only memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk.
Example 2
In this embodiment, the present invention provides a method for detecting a remnant in an automobile cabin, as shown in fig. 2, specifically including the following steps:
s1: reading a video, setting a frame number threshold according to the characteristic that a picture of a monitoring video in an automobile cabin is stable before passengers get on the automobile, and establishing a seat background picture;
s2: after passengers get on the vehicle, the pressure sensors under the seats record the number and the sitting positions of the passengers and continuously monitor the pressure change;
s3: when the pressure reaches a preset threshold value, carrying out remnant detection on the front row seats, and carrying out remnant detection on the vacant position areas of the rear row seats;
s4: after detecting and judging the remnants, identifying the types of the remnants according to an image classification algorithm proposed by people;
s5: after the identification is successful, the vehicle-mounted system sends a short message to inform a mobile phone of a passenger or inform a driver of the short message to remind the passenger who just gets off to pick up the lost article.
A plurality of image acquisition devices, pressure sensors and an on-board computer for processing data and running a carry-over detection algorithm are arranged in the automobile cabin. The pressure sensor is arranged below each seat and used for monitoring the number and the riding position of passengers getting on the vehicle, the plurality of image acquisition devices comprise a front row of image acquisition devices and a rear row of image acquisition devices and are used for acquiring images of the front row of seats and the rear row of seats, the vehicle-mounted computer supports the operation of the whole system, stores and processes data of the pressure sensor and the image acquisition devices, operates a carry-over detection algorithm and outputs a result to the whole vehicle controller. Specifically, considering that the rear seats have different riding scenes with different numbers of people and different getting-off conditions, the algorithm is used for carrying out the leave-behind algorithm detection on vacant seat areas, and the invention summarizes the following conditions:
(1) two or three passengers take a bus, one person near the bus gets off the bus, and the positions of the rest passengers are not changed;
(2) two or three passengers get on the bus, one person close to the bus gets off the bus, and the rest passengers change positions;
(3) two or three people take a bus, and more than one person gets off the bus;
(4) when three passengers get on the bus, the passengers at the sides and the passengers at the middle get off the bus, and the passengers at the sides get on the bus again.
In view of this, the rear-row image capturing device needs to capture the first 50 frames of images in advance and perform background modeling, and divide the obtained background images into three regions according to the seat positions to be stored respectively. Meanwhile, a pressure sensor is arranged under each seat, the sensor sends a signal to an on-board computer after monitoring the pressure change, and the on-board computer judges the number change and the position change before and after getting off the vehicle and determines a detection area of the left-over object; and then, the detection algorithm of the left object in the vehicle-mounted computer starts to operate, the background image of the area and the current image of the area are used as input, the algorithm determines the position and the type information of the left object and outputs the position and the type information to the vehicle control unit, and the vehicle control unit sends the information to the central console for displaying, so that a driver is told to remind the passengers getting off the vehicle to pick up the left object.
When the pressure quantity changes, carry out the thing detection of leaving behind to the front seat beginning, carry out the thing detection of leaving behind to the vacant position region of back seat, as fig. 3, specifically include the following step:
step S210: subtracting the current frame from the background to obtain a difference image, performing morphological operation to obtain a plurality of connected regions, and removing passengers according to the areas of the connected regions;
the specific implementation of this step is as follows: when the vehicle-mounted computer determines the vacancy, cutting out an image of the current frame in a vacancy area, performing frame difference with a background image of the vacancy area to obtain a difference image, and binarizing the difference image, wherein the difference image is represented as:
J i (x,y)＝|F i (x,y)-B(x,y)|,i＝s,s+1……n；
Figure BDA0002854127240000101
wherein B (x, y) is the gray value of the pixel corresponding to the background frame, F i (x, y) is the gray value of the pixel point corresponding to the current frame; j. the design is a square i (x, y) is a differential image of the background frame and the current frame; t is a threshold value for performing binarization operation, O i (x, y) is a binarized image, s is the number of frames used to create a background picture, and n is the total number of frames.
And then performing morphological operation and connected domain analysis on the difference image to obtain a plurality of connected regions, limiting the area of each connected region to be more than 450 and less than 4000, removing regions caused by noise and passengers, and obtaining a final remnant image.
Step S220: storing the remaining areas into a suspicious object queue, and continuously tracking the objects in the queue in the subsequent frame;
step S230: counting the number of frames of each object in the suspicious object queue and the number of lost frames by calculating the centroid coordinate distance, the gray level histogram matching degree and the Hu moment matching degree of different objects; the distance of the centroid coordinates of different objects is the sum of the corresponding horizontal coordinate difference value and vertical coordinate difference value of the centroids of different objects, and the horizontal coordinate and vertical coordinate of the centroid of the jth object can be expressed as:
Figure BDA0002854127240000102
the matching degree between the gray level histograms is expressed as:
Figure BDA0002854127240000103
the degree of match between the Hu moments is expressed as:
Figure BDA0002854127240000104
wherein N is the number of objects, c j Is the abscissa or ordinate of the centroid of the jth object, c i Is the abscissa or ordinate of the ith pixel of the jth object, and n is the number of pixels of the jth object; p and q are the two objects being compared, L (p, q) is the Hu moment match of the p and q objects,
Figure BDA0002854127240000105
the kth component of the Hu moment for the p object,
Figure BDA0002854127240000106
for the kth component of the Hu moment of the q object,d (p, q) is the gray level histogram matching degree of the p and q objects,
Figure BDA0002854127240000107
the number of pixels of s gray in the gray histogram of the p object,
Figure BDA0002854127240000108
the number of s-gray pixels in the gray histogram of the q object is shown, and m is the range of the gray histogram.
Step S240: when the number of the frames of a certain object exceeds a certain number, the object is judged to be a remnant. And when the number of the lost frames of a certain object exceeds a certain number, removing the lost frames from the suspicious object queue. If a new object is detected, it is added to the suspect queue and tracking thereof is started.
As an alternative embodiment, the specific implementation steps of suspicious object tracking are as follows:
(1) reading a first frame image, obtaining an original suspicious object queue Q1, creating and allocating a structural body for each suspicious object in Q1, wherein the structural body members comprise: the method comprises the following steps of (1) bounding box coordinates, total visible frame number, continuous invisible frame number, gray histogram array and Hu moment array;
(2) reading the next frame of image, obtaining a suspicious object queue Q2, and similarly creating and distributing a structural body for each suspicious object in Q2;
(3) calculating centroid coordinates, a gray histogram and a Hu moment of the suspicious objects in Q1 and Q2, when the centroid distance, the gray histogram matching degree and the Hu moment matching degree of the suspicious object A1 in Q1 and the suspicious object B1 in Q2 are all smaller than threshold values, matching is successful, the structural body of A1 is updated to the structural body of B1, and the total visible frame number is added with 1;
(4) when the centroid distances, the gray histogram matching degrees and the Hu moment matching degrees of all the suspicious objects A2 in the Q1 and all the suspicious objects in the Q2 are all larger than the threshold values, adding 1 to the continuous invisible frame number of the A2, and when the value reaches the maximum invisible frame number, the A2 is successfully matched, and the continuous invisible frame number returns to zero;
(5) the doubtful objects which are not matched in the Q2 are all added into the Q1, and the Q2 is emptied;
(6) when the total number of visible frames of a suspicious object is more than 10, the suspicious object is judged to be left. When the number of continuous invisible frames of a suspicious object is more than 20, discarding the suspicious object;
(7) if the remnants are not taken away and the driver does not stop the detection of the remnants, returning to the step (2); otherwise, ending the detection.
The above-mentioned method is a process of counting the number of frames existing in each object in the suspicious queue and the number of frames disappeared by calculating the distance between centroid coordinates, the matching degree between gray level histograms and the matching degree between Hu moments.
In this example, the preset neural network may be a category classification network trained by using any ImageNet data set, taking a pre-trained ResNet-50 neural network as an example, graying the obtained image, scaling the grayed image to 224 as the input image size needs 224 × 3, copying two same grayscale images to generate an image format of 224 × 224 3, processing the images by using ResNet-50, taking the 1 × 1000 dimensional vectors output by the last layer of the neural network as the result of dimension reduction and feature extraction, and finally saving the 1 × 1000 dimensional vectors obtained after processing each remnant image as a mat file.
The output of the neural network is used as the input of the image classification method proposed by the invention which is trained in advance, in this example, the classes of the belongings of the passengers are predefined to be five classes, namely, wallet, mobile phone, bag and umbrella, it should be clear that the classes of the belongings are not limited to the five classes, and can be arbitrarily added or deleted according to the actual use condition, as shown in fig. 4, and the method specifically includes the following steps:
(1) acquiring the images of the types of the objects left over on the Internet, wherein the quantity requirement of each type of image is the same, and the same operation is adopted for all the images, and the slightly different is that when an after-mat file is saved, a column of label information is added to the last column of a vector matrix;
(2) dividing the data in the step (1) into a training set and a test set, and training the image classification method provided by the invention by solving an objective function for each class:
Figure BDA0002854127240000121
obtaining a coding coefficient x, wherein x i Is the coding coefficient of each type of sample, A is the training set data, b is the test set data, lambda is the trade-off parameter of the balance fidelity term and the regularization term, alpha i Are trade-off parameters that balance the constraints imposed on each class. The example splits the original objective function, and obtains the optimal solution of the original problem by alternately updating each parameter until the convergence condition is satisfied. Specifically, the method comprises the following steps:
the augmented lagrange form of its scaled version is:
Figure BDA0002854127240000122
st.z-x＝0
where z is an independent variable introduced with the aim of partitioning the original non-convex problem (objective function) into terms x and x i The two convex problems of (1) are convenient to solve. n is the number of classes contained in the data set, p>0 is a penalty parameter, and the penalty parameter,
Figure BDA0002854127240000123
to obtain a scaled version of the augmented Lagrangian function, y is called the Lagrangian multiplier, | - | 1 Is L1 norm, | 2 Is a norm of L2 and,
Figure BDA0002854127240000124
is the square of the norm of L2.
The optimized iterative process is as follows:
Figure BDA0002854127240000131
Figure BDA0002854127240000132
u k+1 :＝u k +x k+1 -z k+1 ；
and further:
x k+1 :＝(A′A+ρI) -1 [A′b+ρ(z k -u k )]；
order to
Figure BDA0002854127240000133
Then:
Figure BDA0002854127240000134
wherein A' is a transposed matrix of the training sample A, and I is a unit matrix with main diagonals all being 0; the stopping conditions for the optimization of the objective function are two: 1. the maximum iteration times are reached; 2. the original problem residual and the dual residual are sufficiently small, and their formulas are:
‖p k ‖ 2 ≤∈ pri and‖d k ‖ 2 ≤∈ dual ；
wherein p is k X-z is the original problem residual, d k ＝-ρ(z k+1 -z k ) For dual problem residuals, e pri >0 and e dual >0 is the tolerance of the original problem and the dual problem.
(3) After training is finished, classifying the residual images by using the obtained coefficient x, dividing the residual image feature vectors into categories with minimum reconstructed residuals, wherein the inter-category dispersion degree is also a reference factor for decision making, and the formula is as follows:
Figure BDA0002854127240000135
wherein y is the image feature vector of the legacy, label (y) is the category information thereof, and x is the coding coefficient of all the category samples.
And classifying according to the image characteristic vector of the remnant and the reconstructed residual error and the dispersion degree among classes, and outputting the label of the remnant.
Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that changes, modifications, substitutions and alterations can be made in these embodiments without departing from the principles and spirit of the invention, the scope of which is defined in the appended claims and their equivalents.

Claims (8)

1. A method for detecting the remnant in the automobile cabin is characterized by comprising a system for detecting the remnant in the automobile cabin, which comprises the following steps:
s1: reading a video, setting a frame number threshold value according to the characteristic that a picture of a monitoring video in an automobile cabin is stable before passengers get on the automobile, and establishing a seat background picture;
s2: after passengers get on the vehicle, the pressure sensors under the seats record the number and the sitting positions of the passengers and continuously monitor the pressure change;
s3: when the pressure reaches a preset threshold value, carrying out remnant detection on the front row seats, and carrying out remnant detection on the vacant position areas of the rear row seats; performing carryover detection includes:
subtracting the current frame from the background to obtain a difference image, performing morphological operation to obtain a plurality of connected regions, and removing passengers according to the areas of the connected regions to obtain a binary image of the remnant;
storing the remaining areas into a suspicious object queue, and continuously tracking objects in the queue in a subsequent frame;
counting the number of frames of each object in the suspicious object queue and the number of lost frames by calculating the distance between the centroid coordinates, the matching degree between the gray level histograms and the matching degree between the Hu moments; the centroid coordinate distance of different objects is the sum of the corresponding abscissa and ordinate differences of the centroids of different objects, and the abscissa and ordinate of the centroid of the jth object can both be expressed as:
Figure 850992DEST_PATH_IMAGE002
the matching degree between the gray level histograms is expressed as:
Figure DEST_PATH_IMAGE003
；
the degree of match between the Hu moments is expressed as:
Figure 484099DEST_PATH_IMAGE004
；
wherein the content of the first and second substances,
Figure DEST_PATH_IMAGE005
is the number of the objects and,
Figure 414140DEST_PATH_IMAGE006
is the first
Figure DEST_PATH_IMAGE007
The abscissa or ordinate of the centroid of an individual object,
Figure 158105DEST_PATH_IMAGE008
is the first
Figure 722947DEST_PATH_IMAGE007
A first object
Figure DEST_PATH_IMAGE009
The abscissa or ordinate of an individual pixel,
Figure 261376DEST_PATH_IMAGE010
is the first
Figure 433642DEST_PATH_IMAGE007
The number of pixels of each object;
Figure DEST_PATH_IMAGE011
and
Figure 184560DEST_PATH_IMAGE012
for the two objects to be compared,
Figure DEST_PATH_IMAGE013
is composed of
Figure 869488DEST_PATH_IMAGE011
And
Figure 172294DEST_PATH_IMAGE012
the degree of matching of the Hu moments of the object,
Figure 795036DEST_PATH_IMAGE014
is composed of
Figure 834798DEST_PATH_IMAGE011
Second moment of Hu of object
Figure DEST_PATH_IMAGE015
The number of the components is such that,
Figure 390545DEST_PATH_IMAGE016
is composed of
Figure 864251DEST_PATH_IMAGE012
Second moment of Hu of object
Figure 223557DEST_PATH_IMAGE015
The number of the components is such that,
Figure DEST_PATH_IMAGE017
is composed of
Figure 581857DEST_PATH_IMAGE011
And
Figure 208754DEST_PATH_IMAGE012
the degree of matching of the gray level histogram of the object,
Figure 853362DEST_PATH_IMAGE018
is composed of
Figure 185118DEST_PATH_IMAGE011
In a grey level histogram of an object
Figure DEST_PATH_IMAGE019
The number of pixels of the gray scale is,
Figure 596376DEST_PATH_IMAGE020
is composed of
Figure 454611DEST_PATH_IMAGE012
In a grey level histogram of an object
Figure 411066DEST_PATH_IMAGE019
The number of pixels of the gray scale, and m is the range of a gray scale histogram;
when the number of frames of a certain object exceeds the set number of frames, judging the object as a remnant;
when the number of continuous disappearing frames of a certain object exceeds the set number of frames, removing the object from the suspicious object queue;
if a new object is detected, adding the new object into a suspicious object queue and starting to track the suspicious object;
s4: after detecting and judging the abandoned object, identifying the type of the abandoned object according to an image classification algorithm;
s5: after the identification is successful, the vehicle-mounted system sends a short message to inform a mobile phone of a passenger or inform a driver of the short message to remind the passenger who just gets off to pick up the lost article.
2. The method of claim 1, wherein during the background modeling, if the vehicle is in a flameout or non-driving state, the median modeling is used to obtain a clean background picture of the front and back rows without passengers, otherwise the Gaussian modeling is used to obtain a clean background picture of the front and back rows without passengers.
3. The method according to claim 1 or 2, wherein in the background modeling, the background pictures are respectively established according to the positions of three seats in the back row of the vehicle.
4. The method according to claim 1, wherein the step of obtaining the binarized image of the carry-over comprises:
Figure 512008DEST_PATH_IMAGE022
Figure 946532DEST_PATH_IMAGE024
wherein the content of the first and second substances,
Figure DEST_PATH_IMAGE025
the gray value of the corresponding pixel point of the background frame,
Figure 315065DEST_PATH_IMAGE026
the gray value of the pixel point corresponding to the current frame is obtained;
Figure DEST_PATH_IMAGE027
is a differential image of a background frame and a current frame;Tin order to perform the threshold value for the binarization operation,
Figure 442421DEST_PATH_IMAGE028
in order to binarize the image, the image is processed,sthe number of frames used to create the background picture,nis the total number of frames.
5. The method of claim 1, wherein the continuously tracking objects in the queue comprises:
reading a first frame image, obtaining an original suspicious object queue Q1, creating and allocating a structural body for each suspicious object in Q1, wherein the structural body members comprise: the method comprises the following steps of (1) bounding box coordinates, total visible frame number, continuous invisible frame number, gray histogram array and Hu moment array;
reading the next frame of image, obtaining a suspicious object queue Q2, and similarly creating and distributing a structural body for each suspicious object in Q2;
calculating centroid coordinates, a gray histogram and a Hu moment of the suspicious object in Q1 and Q2, when the centroid distance, the matching degree of the gray histogram and the matching degree of the Hu moment of the suspicious object A1 in Q1 and the suspicious object B1 in Q2 are all smaller than a threshold value, the matching is successful, the structural body of A1 is updated to the structural body of B1, and the total number of visible frames is added by 1;
when the centroid distances, the gray histogram matching degrees and the Hu moment matching degrees of all the suspicious objects A2 in the Q1 and all the suspicious objects in the Q2 are all larger than the threshold values, adding 1 to the continuous invisible frame number of the A2, and when the value reaches the maximum invisible frame number, the A2 is successfully matched, and the continuous invisible frame number returns to zero;
the doubtful objects which are not matched in the Q2 are all added into the Q1, and the Q2 is emptied;
when the total visible frame number of a suspicious object is more than 10, the suspicious object is judged to be left, and when the continuous invisible frame number of the suspicious object is more than 20, the suspicious object is discarded.
6. The method of claim 1, wherein identifying the carryover category comprises:
collecting historical data with labels, inputting the historical data into a neural network, and outputting the data after dimension reduction and feature extraction processing;
solving an objective function for each kind of label according to historical data to obtain a coding coefficient x for training, wherein the objective function is expressed as:
Figure DEST_PATH_IMAGE029
wherein the content of the first and second substances,
Figure 293309DEST_PATH_IMAGE030
is the data of the training set and is,
Figure DEST_PATH_IMAGE031
is a test set of data that is,
Figure 797102DEST_PATH_IMAGE032
is a trade-off parameter that balances the fidelity term and the regularization term,
Figure DEST_PATH_IMAGE033
is a trade-off parameter;
Figure 285721DEST_PATH_IMAGE034
is the coding coefficient for each class of samples;
Figure DEST_PATH_IMAGE035
is the coding coefficient of all classes of samples;
Figure 849558DEST_PATH_IMAGE036
in order to calculate the norm of L1,
Figure DEST_PATH_IMAGE037
in order to calculate the L2 norm,
Figure 925093DEST_PATH_IMAGE038
to calculate the square of the L2 norm;
and classifying according to the image feature vector of the remnant and the reconstructed residual error and the dispersion degree between classes, and outputting the label of the remnant.
7. The method of claim 6, wherein the tag of the carry-over is expressed as:
Figure DEST_PATH_IMAGE039
；
wherein the content of the first and second substances,
Figure 498156DEST_PATH_IMAGE040
category information representing a feature vector y of the legacy image;xthe coding coefficients for all classes of samples.
8. The method for detecting the vehicle interior residues according to claim 1, wherein a vehicle interior residue detection system comprises front and rear seat pressure sensors, front and rear image acquisition equipment, a center console display, a memory, an embedded processor and a vehicle control unit; wherein:
the front-row and rear-row pressure sensors are arranged below each seat of the front-row and rear-row passenger seats and used for monitoring the pressure from each seat and judging the positions of passengers and the actions of getting on and off the train according to the change of the pressure;
the system comprises front and rear image acquisition equipment, a front image acquisition equipment which is arranged above an automobile center console and a camera which uses a near infrared image format and is arranged on the top of a cabin and is positioned obliquely above a rear seat, and a rear image acquisition equipment which is arranged above the cabin and is provided with a lens aligned with a rear seat; the front row image acquisition equipment acquires an image of a passenger seat; the rear-row image acquisition equipment acquires an image of a rear-row seat;
the memory storage is used for storing data generated by the front and rear rows of image acquisition equipment, the front and rear rows of pressure sensors and the embedded processor;
the embedded processor is a vehicle-specification-level deep learning processor, can read, write and calculate data in the memory, and temporarily stores a calculation result in the memory or sends the calculation result to the vehicle controller;
the vehicle control unit is used for transferring and processing the data of the whole vehicle and controlling the operation of each part of the vehicle, and in the invention, the embedded processor is communicated with the center console in a display way, so that the center console is communicated with the embedded processor in a data way;
and the console display is used for outputting a carry-over detection result from the embedded processor and informing a driver of reminding the passengers getting off to take back the carry-over.
CN202011539422.8A 2020-12-23 2020-12-23 System and method for detecting remnants in automobile cabin Active CN112597892B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202011539422.8A CN112597892B (en) 2020-12-23 2020-12-23 System and method for detecting remnants in automobile cabin

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202011539422.8A CN112597892B (en) 2020-12-23 2020-12-23 System and method for detecting remnants in automobile cabin

Publications (2)

Publication Number Publication Date
CN112597892A CN112597892A (en) 2021-04-02
CN112597892B true CN112597892B (en) 2022-09-23

Family

ID=75200443

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202011539422.8A Active CN112597892B (en) 2020-12-23 2020-12-23 System and method for detecting remnants in automobile cabin

Country Status (1)

Country Link
CN (1) CN112597892B (en)

Families Citing this family (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113456039A (en) * 2021-07-30 2021-10-01 芯海科技（深圳）股份有限公司 Vehicle-mounted vital sign early warning method and device, vehicle-mounted electronic equipment and medium
CN113642454B (en) * 2021-08-11 2024-03-01 汇纳科技股份有限公司 Seat use condition identification method, system, equipment and computer storage medium
CN113705543A (en) * 2021-10-27 2021-11-26 珠海市魅族科技有限公司 In-vehicle article anti-forgetting monitoring method, device, equipment and medium
CN113997898B (en) * 2021-11-30 2022-10-14 浙江极氪智能科技有限公司 Living body detection method, apparatus, device and storage medium
CN116596958B (en) * 2023-07-18 2023-10-10 四川迪晟新达类脑智能技术有限公司 Target tracking method and device based on online sample augmentation

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102314695A (en) * 2011-08-23 2012-01-11 北京黄金视讯科技有限公司 Abandoned object detection method based on computer vision
EP3196863A1 (en) * 2014-08-01 2017-07-26 Shenzhen Cimc-tianda Airport Support Ltd. System and method for aircraft docking guidance and aircraft type identification
CN107316059A (en) * 2017-06-16 2017-11-03 陕西师范大学 Learner's gesture recognition method
CN107314816A (en) * 2017-06-08 2017-11-03 贵阳学院 The multi-level information monitoring method for early warning of crops early disease
CN107585096A (en) * 2016-07-08 2018-01-16 奥迪股份公司 Anti- forgetting system for prompting, method and vehicle

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102314695A (en) * 2011-08-23 2012-01-11 北京黄金视讯科技有限公司 Abandoned object detection method based on computer vision
EP3196863A1 (en) * 2014-08-01 2017-07-26 Shenzhen Cimc-tianda Airport Support Ltd. System and method for aircraft docking guidance and aircraft type identification
CN107585096A (en) * 2016-07-08 2018-01-16 奥迪股份公司 Anti- forgetting system for prompting, method and vehicle
CN107314816A (en) * 2017-06-08 2017-11-03 贵阳学院 The multi-level information monitoring method for early warning of crops early disease
CN107316059A (en) * 2017-06-16 2017-11-03 陕西师范大学 Learner's gesture recognition method

Non-Patent Citations (3)

* Cited by examiner, † Cited by third party
Title
Image compression using principal component neural network;Jianxun Mi等;《ICARCV 2004 8th Control, Automation, Robotics and Vision Conference, 2004》;20050725;全文 *
Regularization and variable selection via the elastic net;Zou Hui等;《J ournal of the Royal Statistical Society》;20050309;全文 *
基于图像分割与特征匹配的目标检测及跟踪;隆武林等;《信息技术》;20091125(第11期);全文 *

Also Published As

Publication number Publication date
CN112597892A (en) 2021-04-02

Similar Documents

Publication Publication Date Title
CN112597892B (en) System and method for detecting remnants in automobile cabin
US20030169906A1 (en) Method and apparatus for recognizing objects
AU2007291884B2 (en) Method and system of identifying one or more features represented in a plurality of sensor acquired data sets
US9317752B2 (en) Method for detecting large size and passenger vehicles from fixed cameras
US7650034B2 (en) Method of locating a human eye in a video image
JP2019106193A (en) Information processing device, information processing program and information processing method
CN111652114A (en) Object detection method and device, electronic equipment and storage medium
CN106022458B (en) Towards the number faster statistical approach of safety of school bus
Mathew et al. Detecting new stable objects in surveillance video
CN115346197A (en) Driver distraction behavior identification method based on bidirectional video stream
CN113408364B (en) Temporary license plate recognition method, system, device and storage medium
CN116012822B (en) Fatigue driving identification method and device and electronic equipment
JP6472504B1 (en) Information processing apparatus, information processing program, and information processing method
Xia et al. An automobile detection algorithm development for automated emergency braking system
CN111881843A (en) Taxi passenger carrying number counting method based on face detection
GUO et al. Vehicle classification with convolutional neural network on motion blurred images
Hnoohom et al. Image problem classification for dashboard cameras
CN116152790B (en) Safety belt detection method and device
US20050281461A1 (en) Motion-based image segmentor
EP3937066A1 (en) Peer vehicle center line determination system and method
CN117423059A (en) Method for detecting degree of congestion of carriage
Peruski et al. Exploring Object Detection and Image Classification Tasks for Niche Use Case in Naturalistic Driving Studies
Mohamed Asjath et al. Automated vehicle parking slot recognition and monitoring using Optimised image processing techniques
Asjath et al. Automated Vehicle Parking Slot Recognition and Monitoring using Optimised Image Processing Techniques
Kurtkaya et al. Deep Learning-Based Driver Assistance System.

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant