JP7403546B2 - Remaining object detection - Google Patents

Description

<Mutual citation of related applications>
The present invention claims priority to the Chinese patent application whose filing date is March 25, 2020 and application number 202010217625.9, and the entire content of the Chinese patent application is incorporated by reference into the present application.
TECHNICAL FIELD The present invention relates to the field of computer vision technology, and particularly to a method, apparatus, and means of transportation for detecting a residual object.

When people use transportation, they often leave behind items (e.g., wallets, keys, etc.) or even living organisms (e.g., pets, children, etc.) in the vehicle, causing property damage and even leaving items behind in the vehicle. There is also a danger to the lives of the affected organisms. Therefore, it is necessary to reduce losses and risks by detecting objects left behind in means of transportation and taking measures according to the detection results.

The present invention provides a method, device, and means of transportation for detecting a remaining object.

According to a first aspect of an embodiment of the present invention, there is provided a method for detecting a remaining object, the method comprising: obtaining a reference image inside a cabin of a means of transportation when there is no remaining object in the cabin; collecting a first image inside the cabin when the person leaves the means of transportation; and detecting a target left behind in the cabin when the person leaves the means of transportation based on the first image and a reference image. and include.

According to a second aspect of the embodiment of the present invention, there is provided a device for detecting a remaining object, and the device is configured to detect a remaining object in a cabin of a means of transportation. an acquisition module for collecting a first image of the interior of the cabin when the person leaves the vehicle, and a first image of the cabin when the person leaves the vehicle based on the first image and a reference image; and a detection module for detecting a target remaining in the cabin.

According to a third aspect of an embodiment of the invention, a computing device is provided, the computing device comprising a memory and a processor. A computer program is stored in the memory, and the computer program is executed by the processor to implement the method described in any embodiment.

According to a fourth aspect of an embodiment of the invention, there is provided a means of transportation, wherein in the cabin of the means of transportation there is provided an image acquisition device and an embodiment of the invention communicatively connected to the image acquisition device. A residual object detection apparatus as described in the examples or a computing device as described in any embodiment of the invention is provided.

According to a fifth aspect of embodiments of the invention, there is provided a computer-readable recording medium on which a computer program is stored, and when the program is executed by a processor, the method according to any embodiment is realized. .

According to a sixth aspect of embodiments of the invention, there is provided a computer program product, which when read and executed by a computer, implements the method according to any embodiment.

According to a seventh aspect of embodiments of the invention, there is provided a computer program comprising a computer readable code, wherein when the computer readable code is run on an electronic device, a processor in the electronic device performs any of the steps according to the invention. The methods described in the Examples are carried out.

According to an embodiment of the present invention, obtaining a reference image inside the cabin when there is no object left behind in the cabin of the means of transportation, and collecting a first image inside the cabin when the person leaves the means of transportation; An object left behind in the cabin when the person leaves the means of transportation is detected based on the first image and the reference image. The above method is capable of detecting not only living organisms but also non-living objects, is simple, has a wide range of application, and has a high detection accuracy rate.

It is to be understood that the above general statements and the following detailed statements are illustrative and interpretive only and are not limitations on the invention.

The drawings herein are incorporated into and constitute a part of the specification. These drawings illustrate embodiments consistent with the invention and, together with the specification, serve to explain embodiments of the invention.
3 is a flowchart of a method for detecting a remaining object according to an embodiment of the present invention. FIG. 3 is a schematic diagram of a first image of an example of the present invention. FIG. 3 is a schematic diagram of a reference image of an example of the present invention. FIG. 2 is a schematic diagram of the relationship between a machine learning model and an image collection device according to an embodiment of the present invention. FIG. 3 is a schematic diagram of a message notification interface of a communication terminal according to an embodiment of the present invention. FIG. 1 is a block diagram of a remaining object detection device according to an embodiment of the present invention. 1 is a schematic diagram of a computer device according to an embodiment of the present invention; FIG. 1 is a schematic diagram of a means of transportation according to an embodiment of the present invention. FIG. 3 is a schematic diagram of the distribution of image acquisition devices according to an embodiment of the present invention. FIG. 3 is a schematic diagram of another distribution of the image acquisition device according to the embodiment of the present invention.

Exemplary embodiments will now be described in detail, examples of which are illustrated in the drawings. Where the following description refers to drawings, the same numerals in different drawings indicate the same or similar elements, unless stated otherwise. The embodiments described in the following illustrative examples are not representative of all embodiments consistent with the present invention. On the contrary, they are merely examples of apparatus and methods consistent with certain aspects of the invention as described in the appended claims.

The terminology used in the present invention is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this invention and the appended claims, the singular forms "a", "a", "the" and the like are intended to include the plural unless the context clearly dictates otherwise. ing. It should be understood that the term "and/or" as used herein refers to include any one or all possible combinations of one or more of the associated listed items. . In addition, the term "at least one" as used herein means any one of a plurality of types or any combination of at least two of a plurality of types.

In the present invention, various information may be described using terms such as first, second, third, etc., but it should be understood that these information should not be limited by these terms. These terms are only used to distinguish information of the same type from each other. For example, first information may also be referred to as second information, and similarly, second information may also be referred to as first information, without departing from the scope of this disclosure. Depending on the context, the word "if" as used herein can be interpreted as "if...", "then", or "in response to".

Hereinafter, in order for those skilled in the art to better understand the technical solutions of the embodiments of the present invention, and to make the above objects, features and advantages of the embodiments of the present invention more clear and understandable, reference is made to the drawings. The technical solutions of the embodiments of the present invention will now be explained in more detail.

Embodiments of the present invention provide a method for detecting objects left behind in a vehicle, and as shown in FIG. 1, the method may include the following steps 101-103.

In step 101, a reference image of the inside of the cabin of the means of transportation when there is no object left behind in the cabin of the means of transportation is acquired.

In step 102, a first image of the interior of the cabin is collected when the person leaves the vehicle.

In step 103, an object left behind in the cabin when the person leaves the means of transportation is detected based on the first image and the reference image.

In embodiments of the present invention, the means of transportation may be a vehicle such as a private car, a bus, a school bus, a truck, a train, or a means used to transport people or cargo, such as a boat or an airplane. There may be. Accordingly, the cabin of the means of transportation may be a vehicle cabin, a ship cabin, an airplane cabin, etc. Hereinafter, the solutions of the embodiments of the present invention will be described by taking an example in which the means of transportation is a vehicle and the cabin is a cabin of a vehicle. The implementation of the remaining object detection technology for other types of transportation is similar, and will not be described repeatedly.

In step 101, a reference image of the inside of the vehicle cabin when there is no object remaining inside the vehicle cabin can be acquired. Here, the reference image inside the vehicle cabin includes only unique objects (for example, seats, steering wheels, vehicle trim, etc.) inside the vehicle cabin, and does not include any remaining objects. Said reference image can be collected and stored when there are no leftover objects in the vehicle cabin and can be used once or multiple times, for example when multiple leftover object detection needs to be performed. Can be used repeatedly. For example, it can be collected using an on-vehicle image acquisition device and stored in the vehicle's storage unit before the vehicle is shipped, or the user can can be collected using an image collection device on the vehicle, or can be collected using a user terminal (e.g., mobile phone, tablet computer, camera, etc.) and stored in the vehicle's storage unit. . Furthermore, by performing image compression processing on the collected original images inside the cabin and storing the compressed images as background images, storage space can be reduced and image processing efficiency can be improved. can.

A reference image can be collected once, and the collected reference image can be used as a reference image when executing residual object detection each time. Alternatively, the reference image can be updated according to a specific method. Updating the reference image according to the specific method may include updating the reference image at regular intervals (for example, one day or one week) or in response to a specific event trigger. The reference image may be updated based on the reference image. Said specific event may be that a lingering object has been detected and a reference image update command has been received, for example, the background within the vehicle cabin has changed or a person is attempting to enter the cabin of the vehicle. (e.g., the vehicle is unlocked) triggers an update of the reference image.

Said background image may be a background image of the entire vehicle cabin, or may be a background image of the entire vehicle cabin, or may be a background image of a specific or specified area within the vehicle cabin (e.g. driving area, co-pilot area, back seat, child seat, love seat area, trunk area). , a luggage area, etc.). After the image of the entire vehicle cabin is collected, the directly collected image may be used as the background image, or the image may be cut as necessary, and the cut image may be used as the background image. good. The number of background images may be one or more. For example, if an image collection device is included in the vehicle cabin, the background image may be an image collected by the image collection device. If a plurality of image acquisition devices are included in the vehicle cabin, the number of background images is greater than one, and each background image is acquired by one of the image acquisition devices.

In practical applications, after a person gets off the car, if there is no object left behind in the vehicle cabin, the next time the person leaves the vehicle by taking and storing the image inside the vehicle cabin. This image can be used as a reference image for detecting remaining objects. Alternatively, a target image within the cabin may be collected before the person enters the vehicle and used as a reference image for performing lingering object detection when the person leaves the vehicle. The objects left behind may include items such as wallets, keys, mobile phones, umbrellas, briefcases, and suitcases, and may further include living organisms such as people and pets. The person can be a child, an elderly person, a sleeping person, a disabled person, etc., who may be left behind in a variety of vehicles.

In step 102, the personnel may include any personnel within the vehicle, such as, for example, a driver, crew, passengers, etc. Determining whether the person leaves the vehicle based on at least one of an open status of a cabin door of the vehicle, an operation status of the vehicle, a movement trajectory of the person, and a specific command. can. For example, if a cabin door is detected to be open or a person's seat belt is unbuckled, it can be determined that the person has exited the vehicle. Also, for example, if it is detected that the vehicle is powered off and the cabin door is opened, it can be determined that the person has exited the vehicle. Further, for example, if it is detected that the movement trajectory of the person is from inside the cabin to the cabin, it can be determined that the person is getting out of the car. Further, for example, if a confirmation command for the end of the trip is detected, it can be determined that the person has gotten off the vehicle. Besides this, other methods can be used to determine whether a person leaves the vehicle or not, and will not be described repeatedly here.

The first image inside the cabin may include one image or a plurality of images. Each first image may be collected by one image collection device in the vehicle cabin, for example, the first image or images may be collected from a video stream collected using the image collection device. can be obtained as.

In step 103, an object left behind in the cabin when the person leaves the vehicle can be detected based on the first image and the reference image. Here, the left behind object may be a living body and/or an item carried into the cabin by the personnel and left behind in the cabin when the personnel leaves the vehicle. If the quantities of the first image and the reference images are both greater than 1, based on each first image and the corresponding one reference image, what remains in the cabin when the person leaves the means of transportation; can be detected. For example, if the i-th first image and the i-th reference image are both images corresponding to the i-th subarea in the cabin, the personnel can be identified based on the i-th first image and the i-th reference image. It is possible to detect a target remaining in an i-th sub-area within the cabin when leaving the means of transportation. Here, each time a remaining object is detected, remaining object detection may be performed for all subareas, or for some of the subareas.

The operation of detecting a remaining target may be performed all the time, or may be triggered and performed in a specific case. For example, it can be triggered by means of transportation. It can also be triggered passively, for example by a user terminal that has previously established a communication connection with the means of transportation. After the detection trigger command is sent by the user terminal and the detection trigger command is received, the operation of performing the residual object detection can be started. The detection trigger command may further include a target detection type, and can determine whether a specific type of remaining target is included in the image to be processed. For example, a user may realize that the keys are missing after leaving the car. At this time, a detection trigger command including the type of "key" can be sent via the mobile phone to trigger detection of the retained object of the type of "key".

In some embodiments, objects left behind in the cabin when the person leaves the vehicle can be detected based on a difference between the first image and a reference image. For example, at least one target object included in the first image but not included in the reference image may be determined as the remaining object. As shown in FIG. 2A, the first image includes a mobile phone, a child, a seat, and a pillow, but as shown in FIG. 2B, the reference image includes a seat and a pillow. Included in the first image but not included in the reference image are a mobile phone and a child, and the mobile phone and/or child are not included in the vehicle when the person leaves the vehicle. It can be confirmed that the person remains in the cabin. Such a method allows for more intuitive detection of remaining objects, is simple to implement, and has low implementation cost.

Here, the target object in the reference image may be obtained by a method of labeling the reference image, or may be obtained by a method of detecting the reference image. Each time the difference between the first image and the reference image is determined, the target object may be detected once from the reference image, or a previously detected target object may be directly adopted as the target object. .

In some embodiments, collecting a second image of the interior of the cabin when the person enters the vehicle, and identifying a lingering object pending detection based on a difference between the second image and the reference image. It can be confirmed. The case where the person enters the means of transportation may include the case when the person is entering the means of transportation, or the case where the person enters the means of transportation and before leaving the means of transportation. . For example, the second image may be collected when the vehicle is about to arrive at a destination. In a practical application, it is determined whether the means of transportation is about to arrive at a destination according to an application program (e.g., a map application program or a network vehicle reservation application program, etc.) operated on a smart terminal on the means of transportation. It is possible. By determining the remaining objects waiting to be detected that are related to a specific person by such a method, the remaining objects detection can be performed only for the remaining objects that are waiting to be detected and are related to the specific person. Improved detection accuracy and reduced detection resource consumption.

In some embodiments, at least one target object included in the second image but not included in the reference image may be determined as the remaining object awaiting detection. By such a method, it is possible to construct a relationship between a retained object and a person, and to detect only retained objects related to a specific person. The association may be established by assigning identity information to a person entering the means of transportation and binding the identity information of the person with a confirmed remaining object awaiting detection when the person enters the means of transportation. can. Based on this, when the person leaves the means of transportation, the remaining object is determined from only the remaining objects that are waiting to be detected related to the person, and other personnel select the remaining objects left behind in the means of transportation from the person. The probability of being confirmed as a survivor has been reduced.

For example, a second image taken when Person A gets into a car includes a cell phone, and since Person A has a set of keys in the vehicle before Person A gets into the car, only the cell phone can be seen by Person A. Confirm as a remaining target awaiting detection corresponding to. If a mobile phone is included in the first image taken when the person gets off Car A, it is determined that there is a subject left inside the car when the person gets off Car A. If the first image taken when Person A exits the vehicle does not include a mobile phone, it is determined that there is no object left behind in the vehicle when Person A exits the vehicle.

If there are multiple persons entering the cabin, collect second images of the cabin when each person enters the means of transportation, and calculate the difference between the second image corresponding to each person and the reference image. Based on this, it is possible to determine the remaining targets of each person who are waiting to be detected.

In some other embodiments, a second image of the interior of the cabin when the person enters the vehicle is collected, and based on the reference image and the second image, the first image of the remaining object awaiting detection is collected. The position in the image is determined, and the remaining object can be detected from among the remaining objects waiting to be detected in the first image based on the position of the remaining object waiting to be detected in the first image. In this way, the detection efficiency is improved by first roughly determining the position of the remaining object waiting to be detected, and then detecting the remaining object based on the position.

Here, the second image and the reference image are input to a first machine learning model that has been trained in advance, and based on the results of the first machine learning model, the residual object (referred to as a pseudo residual object) waiting for detection and the pseudo residual object are detected. The position of the remaining object in the second image may be determined, and then the position of the pseudo remaining object in the first image may be determined based on the position of the pseudo remaining object in the second image. The machine learning model may employ a neural network, or the machine learning model may employ a neural network and a traditional vision algorithm (e.g., an optical flow method, an image sharpening method, an image difference algorithm, or a Carter tracking algorithm). ) can be adopted. Neural networks according to embodiments of the invention may include an input layer, at least one hidden layer, and an output layer, each of which has one or more contains neurons. Here, the intermediate layer usually refers to a layer located between an input layer and an output layer, such as a hidden layer. In one optional example, the intermediate layers of the neural network may include, but are not limited to, at least one of a convolution layer, a Rectified Linear Units (ReLU) layer, and the like. The greater the number of hidden layers included in the neural network, the deeper the network. The neural network may specifically be a deep neural network or a convolutional neural network.

For example, target tracking is performed on the remaining object waiting to be detected based on the reference image and the second image of a plurality of frames, and the position of the remaining object waiting to be detected in the first image is determined based on the result of target tracking. can be determined. By this method, we first determine objects that may be left behind in the cabin (i.e., pseudo-remaining objects), determine the location of the pseudo-remaining objects more accurately, and then determine the location of the pseudo-remaining objects based on the location of the pseudo-remaining objects. By detecting the object, the detection efficiency and accuracy were improved.

As shown in Figure 3, when the second image is collected using multiple image acquisition devices in the cabin, each image acquisition device corresponds to one machine learning model, and each machine learning model corresponds to one image acquisition device. It is used to detect the second image collected by the collection device. For example, if a cabin contains N image collection devices, input the second image and background image collected by image collection device 1 to machine learning model 1, and input the second image and background image collected by image collection device 1 to Detect a pseudo-remaining target from the second image, input the second image and background image collected by the image collecting device 2 to the machine learning model 2, and detect the pseudo-remaining target from the second image collected by the image collecting device 2. Detect, and make an analogy like this. Machine learning models may be shared to detect images collected by multiple different image collection devices, and the invention is not limited thereto.

In some embodiments, before detecting lingering objects in the cabin when the person leaves the vehicle based on the first image and a reference image, non-remaining objects are first excluded from the first image. can do. The non-remaining objects are items carried into the cabin by personnel and expected to remain in the cabin, and may include, for example, pillows, car accessories, etc. For example, it is possible to receive a remaining object confirmation command and exclude non-remaining objects from the first image according to the remaining object confirmation command. One implementation is to take a third image of the interior of the vehicle's cabin before the person leaves the vehicle, and to display the third image on a display device (e.g., a central control screen of a vehicle or a display interface of a user terminal). ) and display the command, and personnel can send a command to confirm the remaining subjects using a user terminal or a central control screen. Furthermore, for example, past processing results of the retention target can be acquired. If a specific object has been determined as a remaining object in the past detection process, but has not been processed for a long time or many times (for example, it was not taken out from the cabin), the object is determined as a non-remaining object. By such a method, it is possible to reduce the probability that a non-remaining object, such as an item that is carried into the cabin by the above-mentioned personnel and is expected to remain in the cabin, is erroneously determined as a leftover object, and to reduce false detection. can.

When the remaining object is detected, the position of the remaining object in the cabin and/or the type of the remaining object can be determined. The position may be a general position such as a co-pilot seat or a rear seat, or may be more precise position information, for example, coordinates in the cabin of the person to be left behind. The types of the remaining objects can be easily classified into biological types and article types, and each type can be further classified into more detailed sub-types. For example, biological types can be classified into pet types and child types, and item types can be classified into key types, wallet types, and so on. It can be classified by mobile phone type, etc. By determining the location and/or type of the remaining object, subsequent operations can be easily performed based on the location and/or type of the remaining object, for example, by sending notification messages to By controlling the environmental parameters, the probability of security problems for the remaining target can be reduced.

The first image may be input to a pre-trained second machine learning model, and the location and/or type of the retained object may be obtained based on the output result of the second machine learning model. The second machine learning model and the first machine learning model may be the same model or different models. Additionally, the second machine learning model may include a first sub-model and a second sub-model, where the first sub-model detects a biological object and the second sub-model detects an article object. . Here, the first sub-model may be pre-trained with a sample image of a biological object, and the second sub-model may be pre-trained with a sample image of an article object. The sample images may include images taken at different light intensities and different scenes to improve the accuracy of the trained object recognition model.

In some embodiments, a first notification message may be sent to the means of transportation and/or a preconfigured communication terminal when the remaining object is detected. Sending the first notification message helps personnel locate the retained object and quickly retrieve the retained object. The first notification message may include prompt information to indicate that a remaining target currently exists. Further, the first notification message further includes a time when the retained object was left behind, a type and/or a location of the retained object. After sending the first notification message to the vehicle, the vehicle may output prompt information, the prompt information being sound prompt information output via car audio or horn and/or vehicle lights. Contains light prompt information output via. Furthermore, different positions of the remaining object can be indicated by outputting different audio prompt information or outputting light prompt information through light emitting devices located at different positions.

The communication terminal can establish a communication connection with a means of transportation via any connection method such as a mobile data connection, a Bluetooth connection, a WiFi connection, and the communication terminal can be a mobile phone, a tablet computer, It can be a smart terminal such as a smart watch and a laptop computer. After sending the first notification message to the communication terminal, the communication terminal can output prompt information, and as shown in FIG. 4, the prompt information includes at least one of text prompt information and image prompt information. Contains one. The text prompt information may be text content in the form of "There is an item left behind in the back seat of the car" or "There is a child in the car." It may further include the time when the remaining object was detected, and may be, for example, text content such as "Time: February 13, 2020 18:35; Location: Backseat of car; Remaining object type: Wallet." The image prompt information may include the first image, or may include only the image to be retained that is cut from the first image. By transmitting the cut image of the target to be retained, the amount of data transmission can be reduced.

If the remaining object is detected, based on the type of the remaining object, transmitting first control information for adjusting environmental parameters in the cabin to the means of transportation. A comfortable cabin environment can be provided. For example, if the type of the remaining object is a biological type, first control information for adjusting environmental parameters in the cabin may be transmitted to the means of transportation, such as window opening control information and/or air conditioner. By transmitting operational control information, it is possible to reduce the probability that a hazard will occur to the living object due to adverse conditions in the cabin (e.g., too high temperature or low oxygen content). . Specifically, when the type of the object to be left behind is a biological type, window opening control information is transmitted to the means of transportation to open the window of the means of transportation.

The window opening control information includes, but is not limited to, quantity information of windows waiting to be opened, position information, and/or opening degree information. Optionally, the window opening control information may be generated based on at least one of a position of the remaining object, an environmental parameter inside the cabin, and an environmental parameter outside the cabin. For example, if the subject is located in the rear seat of the vehicle, one window at the rear of the vehicle is controlled to be opened. Further, for example, when the oxygen content in the cabin is lower than a predetermined threshold value, the two windows at the rear of the vehicle are controlled to be opened. The degree of opening can be set in advance, for example, the moving distance of the window opening can be fixedly set to 5 cm, so that the oxygen content in the cabin is within the required range. At the same time, it is possible to ensure the safety of the remaining subjects by preventing personnel outside the cabin from causing harm to the remaining subjects or from going out of the window. . The degree of opening can be dynamically set according to environmental parameters outside the cabin, for example, if the environmental temperature outside the cabin is outside a predetermined range, the degree of opening can be set smaller (for example, 5 cm), and vice versa. In this case, the degree of opening can be set larger (for example, 8 cm). By doing so, the influence of the environment outside the cabin on the remaining object can be reduced.

If the type of the object to be left behind is a biological type, transmitting air conditioner operation control information to the means of transportation, and further controlling the temperature and/or operation mode (for example, cooling or heating) of the air conditioner. can be controlled. By controlling windows and/or air conditioning, the probability of too high a temperature or insufficient oxygen content in the cabin can be reduced, reducing the probability of creating a hazard to biological subjects. .

In one example, when a remaining subject is detected, the first control information can be directly transmitted, for example, by transmitting air conditioner operation control information to the means of transportation, the air conditioner can be set at a temperature appropriate for the living body. The vehicle may be operated in a humidity control mode, or the degree of window opening may be controlled by transmitting window opening control information to the means of transportation. For example, windows can be controlled to have gaps rather than all open to improve the air environment within the vehicle, while at the same time preventing living organisms from exiting the vehicle cabin or threats from outside the vehicle cabin. Try to avoid being exposed to.

In another example, an environmental parameter within the cabin may be detected first, and the first control information may be transmitted only if the environmental parameter exceeds a predetermined range. For example, if it is detected that the temperature in the cabin is too high or too low, air conditioning operation control information can be sent to the means of transportation. The air conditioner can be controlled to switch off again after the cabin temperature is suitable. Also, for example, window opening control information can be sent to the vehicle to control the window to open a gap when it is detected that the oxygen content in the cabin is too low.

In some embodiments, if the retained object is detected, it may be determined whether the retained object has been removed from within the cabin. If the retained object is determined to have been removed from within the cabin, a specific operation may be performed, and the specific operation may be performed based on the time when the retained object was removed from within the cabin and/or the retained object. recording the identity information of the person who took out the object; transmitting second control information for adjusting environmental parameters in the cabin to the means of transportation; and sending a second notification message to a preset communication terminal. and transmitting. For example, it is possible to record "At 19:00:35 on March 22, 2020, a user whose ID is XXX took out a pet." If first control information for adjusting an environmental parameter in the cabin is previously transmitted to the means of transportation, second control information for adjusting the environmental parameter in the cabin may be transmitted to the means of transportation. For example, windows and/or air conditioners of the vehicle may be closed by transmitting window closing control information and/or air conditioner closing control information to said vehicle. In this way, the energy consumption of the means of transportation can be reduced, the manual operation of personnel can be reduced, and the complexity of operation can be reduced. A second notification message can be generated and sent to the communication terminal based on the recorded information, and the second notification message includes the name, type, time of removal, and personnel of the removed object. may include at least one of the following identification information. In this way, notification can be sent quickly when a retained object is retrieved, reducing the possibility of erroneous retrieval.

Optionally, a third image of the inside of the cabin is acquired within a predetermined time period after the passenger leaves the means of transportation, and based on the third image and the reference image, the object remains in the cabin. It is possible to determine whether or not it was taken out from within. Specifically, based on the difference between the third image and the reference image, it can be determined whether the remaining object has been taken out from inside the cabin. If there is at least one target object that is included in the third image but not included in the reference image, it is determined that there is a remaining object that has not been taken out. If at least one target object included in the third image but not included in the reference image does not exist, it is determined that all remaining objects have been extracted. It detects whether a retained object has been removed by acquiring an image, and is easy to implement and has low detection cost.

Optionally, a third image of the interior of the cabin is acquired within a predetermined time period after the passenger leaves the vehicle, and the remaining object is located in the cabin based on the third image and the first image. It is possible to determine whether or not it was taken out from within. Specifically, based on the difference between the third image and the first image, it can be determined whether the remaining object has been removed from inside the cabin. If there is at least one target object included in the third image but not included in the first image, it is determined that the remaining object has been retrieved. If at least one target object included in the third image but not included in the first image does not exist, it is determined that the remaining object has not been taken out.

In some embodiments, if a lingering object is detected, information about the lingering object may be stored in a database. The information on the retained object may include at least one of information such as an image of the retained object, a position, a type, a residence time, a person to whom the retained object belongs, and a person who took out the retained object. By constructing a database, it is convenient to inquire about information about the remains.

In the above embodiments, an image collection device (e.g., a camera) on the vehicle can be activated to collect images when images need to be collected, and further when images do not need to be collected. The image acquisition device can be turned off. By doing so, the image acquisition device does not have to be in a working state all the time, and energy consumption can be reduced. For example, activating an image collection device on the vehicle when the vehicle is started. or activating an image acquisition device on the vehicle when it is determined that a person has entered the vehicle; Alternatively, if it is determined that a person is leaving the vehicle, an image acquisition device on the vehicle is activated. Further, for example, when it is necessary to detect whether or not the remaining object has been taken out from inside the cabin, if it is determined that the remaining object has been taken out from inside the cabin, the image collecting device may be turned off. do. It is only necessary to detect lingering objects, turning off the image acquisition device when personnel leave the vehicle.

The remaining object detection method according to the embodiment of the present invention can detect not only living bodies but also static objects, and has a high detection accuracy rate. Embodiments of the present invention have wide applicability and can be used in different application scenarios such as private cars, network reserved vehicles, or school buses. Here, the person getting on and off the car can be determined in different ways depending on the actual scene. For example, in a private car scenario, based on the signal strength of the communication connection between the driver's communication terminal and the vehicle, it is determined whether the driver has entered the car or not and whether the driver has exited the car. be able to. In a network reserved vehicle scenario, the driver determines whether or not a passenger is in the vehicle based on actions in the network reserved vehicle application program (e.g., confirming that a passenger has been received or confirming that the destination has been reached). It is possible to determine whether or not the user got into the vehicle and whether or not he or she exited the vehicle.

Those skilled in the art will understand that in the above method of the specific embodiment, the described order of each step does not mean a strict execution order and constitutes a restriction on the implementation process, and the specific execution order of each step is It can be understood that it should be determined by the internal logic of its functionality and possibilities.

As shown in FIG. 5, the present invention further provides a residual object detection device, which includes a first acquisition module 501, a first collection module 502, and a detection module 503.

The first acquisition module 501 acquires a reference image inside the cabin of the means of transportation when there is no object left behind in the cabin.

A first collection module 502 collects a first image within the cabin when a person leaves the vehicle.

A detection module 503 detects objects left behind in the cabin when the person leaves the vehicle based on the first image and the reference image.

In some embodiments, the first acquisition module 501 collects a target image within the cabin before the person enters the vehicle, where the target image is the reference image; A detection module detects objects left behind in the cabin when the person leaves the vehicle based on a difference between the first image and a reference image.

In some embodiments, the apparatus includes a second collection module for collecting a second image of the interior of the cabin when the person enters the vehicle; and between the second image and the reference image. The apparatus further includes a first determination module for determining a remaining target waiting to be detected based on a difference between the following.

In some embodiments, the first determination module includes a first acquisition unit for acquiring target object information of the second image and the reference image, respectively; and a first determining unit for determining at least one target object not included in the image as the remaining object waiting to be detected.

In some embodiments, the detection module 503 includes a first collection unit for collecting a second image inside the cabin when the person enters the vehicle; a second determination unit for determining the position of the residual object awaiting detection in the first image based on the position of the residual object awaiting detection in the first image; and a detection unit for detecting the remaining object from among waiting remaining objects.

In some embodiments, the second determining unit includes a tracking subunit for performing target tracking on a remaining object awaiting detection based on the reference image and a plurality of frames of the second image; a determining subunit for determining the position of the remaining object waiting to be detected in the first image based on the tracking result.

In some embodiments, the device receives a left object confirmation instruction before detecting a left object in the cabin when the person leaves the vehicle based on the first image and a reference image. and an exclusion module for excluding non-remaining objects from the first image according to the remaining object confirmation command.

In some embodiments, the apparatus further comprises a second determination module for determining the location of the remaining object in the cabin and/or the type of the remaining object when the remaining object is detected.

In some embodiments, the device further comprises a first sending module for sending a first notification message to the means of transportation and/or a preconfigured communication terminal when the remaining object is detected. .

In some embodiments, the apparatus provides first control information to the vehicle for adjusting environmental parameters within the cabin based on the type of the remaining object when the remaining object is detected. The apparatus further includes a second transmitting module for transmitting.

In some embodiments, the second transmitting module transmits first control information for adjusting environmental parameters in the cabin to the means of transportation when the type of the remaining object is a biological type.

In some embodiments, the second transmitting module transmits window opening control information and/or air conditioner operation control information to the means of transportation.

In some embodiments, the apparatus includes a third determination module for determining whether the retained object has been removed from within the cabin when the retained object is detected; and an execution module for executing a specific operation when it is determined that the remaining object has been removed from the cabin, and the specific operation is a time when the remaining object is removed from the cabin, and/or , recording the identity information of the person who took out the remaining object; transmitting second control information for adjusting environmental parameters in the cabin to the means of transportation; and transmitting second control information to a preset communication terminal. 2. Sending a notification message.

In some embodiments, the third determination module includes a second acquisition unit for acquiring a third image of the interior of the cabin within a predetermined time period after the passenger leaves the vehicle; and a third determining unit for determining whether or not the remaining object has been removed from the cabin based on the third image and the reference image.

In some embodiments, the apparatus further comprises an activation module for activating an image collection device on the vehicle when the vehicle is activated.

In some embodiments, the apparatus further comprises an off module for turning off the image acquisition device when the retained object is determined to have been removed from within the cabin.

The functions included in or the modules included in the apparatus provided by the embodiments of the present invention can perform the methods described in the above method embodiments, and the specific implementation thereof is the above method embodiments. For the sake of brevity, it will not be repeated here.

Embodiments of the invention further include a computing device that includes a memory and a processor. A computer program is stored in the memory, and when executed by the processor, the method described in any embodiment is implemented.

FIG. 6 is a schematic diagram of the hardware configuration of a more specific computer device provided by an embodiment of the present specification, and the device includes a processor 601, a memory 602, an input/output interface 603, and a communication interface. 604 and a bus 605. Here, a processor 601, a memory 602, an input/output interface 603, and a communication interface 604 realize mutual communication connection within the device via a bus 605.

The processor 601 may be implemented by a general CPU (Central Processing Unit), a microprocessor, an Application Specific Integrated Circuit (ASIC), one or more integrated circuits, or the like. The processor 601 implements the technical solution provided by the embodiments herein by executing the associated programs.

Memory 602 may be implemented by ROM (Read Only Memory), RAM (Random Access Memory), static storage devices, dynamic storage devices, and the like. Memory 602 can store an operating system and other application programs. When implementing the technical solution provided by the embodiments herein using software or firmware, the relevant program code is stored in the memory 602 and called up and executed by the processor 601.

The input/output interface 603 realizes input and output of information by being connected to an input/output module. The input/output module may be disposed within the device as a component (not shown) or may be connected to the device to provide the appropriate functionality. However, input devices may include keyboards, mice, touch screens, microphones, various sensors, etc., and output devices may include displays, speakers, vibrators, indicator lights, etc.

Communication interface 604 is connected to a communication module (not shown) to enable communication interaction between the present device and other devices. However, the communication module may realize communication by wired means (e.g. USB, network cable, etc.) or by wireless means (e.g. mobile network, WIFI, Bluetooth (registered trademark), etc.) .

Bus 605 includes a path and transmits information between each component of the device (eg, processor 601, memory 602, input/output interface 603, and communication interface 604).

Although only a processor 601, a memory 602, an input/output interface 603, a communication interface 604, and a bus 605 are shown for the above device, in the specific implementation process, the device is used to realize normal operation. It is necessary to explain that it may further include other components necessary for. It should be noted that those skilled in the art will understand that the above-described device may include only the components necessary to realize the solutions of the embodiments herein, and does not need to include all the components shown in the drawings. Should.

As shown in FIG. 7, an embodiment of the present invention further provides a means of transportation, wherein within the cabin of the means of transportation there is an image acquisition device and an image capturing device of the present invention communicatively connected to the image acquisition device. A residual object detection device according to any embodiment or a computer device according to any embodiment of the present invention is provided.

The image acquisition device is used to acquire the first image. The image acquisition device may take images of the interior of the cabin to be processed starting from when the person enters the cabin and until the person leaves the cabin, or may capture images of the interior of the cabin to be processed after the person enters the cabin. It is also possible to start capturing images of the interior of the cabin that are waiting to be processed after the period has elapsed.

In some embodiments, the image acquisition device may be located in an upper portion within the cabin. The number of image acquisition devices within the cabin may be one or more. When the number of image collecting devices is one, the image collecting device is used to collect images of the entire cabin to be processed. If the number of image acquisition devices is greater than one, each image acquisition device is utilized to acquire a respective pending image of one subarea within the cabin. The number, location and distribution within the cabin of the image acquisition devices can be determined according to the shape and size within the cabin and the field of view of the image acquisition devices. For example, for a narrow and long area such as a vehicle, one image acquisition device can be placed in the center of the top (inside top) of the vehicle, as shown in Figure 8A, and Alternatively, one image acquisition device may be placed above each row of seats. By arranging multiple image acquisition devices, the photographed area is made more complete.

In some embodiments, the first image collected by the image collection device may be detected frame by frame, and in some embodiments, the frame rate of the image collected by the image collection device may be Since images are often relatively large, e.g. tens of frames per second are collected, frame-skip detection may be performed on the first image, e.g. the first, third, fifth image taken. Only the first image of the frame may be detected. The frame skipping step (ie, the frame number interval between detected adjacent image frames) can be determined according to the actual scene. For example, if the light is weak, there are more detection objects, and the resolution of the first image taken is lower, the frame skip step can be set smaller, the light is better, there are fewer detection objects, If the first image taken has a higher resolution, the frame skip step can be set larger.

The field of view of the image acquisition device may be larger and includes areas where lingering objects are likely to appear and areas where lingering objects typically do not appear. Therefore, when detecting a potential remaining object in the cabin in the first image, an area of interest in the first image can be determined first, and then a remaining object can be detected within the area of interest. For example, seats are an area of interest because leftover objects are likely to appear on seats in a car cabin, while leftover objects generally do not appear on a center console.

Embodiments herein further provide a computer readable storage medium having a computer program stored thereon, which when executed by a processor implements the method described in any of the preceding embodiments.

Computer-readable media includes both persistent and non-persistent, removable and non-removable media, and may accomplish storage of information by any method or technique. The information may be computer readable instructions, data structures, program modules, or other data. Examples of computer storage media include phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only optical disk (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassette A device including, but not limited to, tape, magnetic tape storage or other magnetic storage or other non-transmission media may store information that is accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media such as modulated data signals or carrier waves.

As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that the embodiments herein can be implemented by software and the necessary common hardware platforms. Based on this understanding, the technical solutions of the embodiments herein can essentially be embodied in the form of a software product, or the parts contributing to the prior art can be embodied in the form of a software product. It can be made concrete. The computer software product can be stored in a recording medium, such as ROM/RAM, magnetic disk, optical disk, etc., and includes a number of instructions to run a computer device (such as a personal computer, a server, or a network device). (possibly) to carry out the method described in each example or portion of an example herein.

The systems, devices, modules, or units illustrated in the examples above may be specifically implemented by computer chips or entities, or may be implemented by products with specific functionality. Typical implementation devices are computers, and specific forms of computers include personal computers, laptop computers, mobile phones, camera phones, smart phones, personal digital assistants, multimedia players, navigation devices, and email sending and receiving devices. , and game controls, tablet computers, wearable devices, or any combination of these devices.

The various embodiments herein are described progressively, and the same or similar parts between the various embodiments may be referred to with each other. Each example focuses on differences from other examples. In particular, since the apparatus embodiment is basically the same as the method embodiment, the explanation is relatively simple, and for related parts, please refer to part of the description of the method embodiment. The device embodiments described above are merely schematic. The modules described above as separate parts may or may not be physically separated, and each module's functionality may be implemented in one or more ways when implementing the example solutions herein. may be implemented in software and/or hardware. In addition, some or all of the modules may be selected according to actual needs to realize the purpose of the solution of this embodiment. Those skilled in the art can understand and implement without creative effort.

Claims (15)

A method for detecting a remaining object,
Obtaining a reference image of the inside of the cabin of the means of transportation when there is no object left behind in the cabin of the means of transportation;
collecting a second image of the interior of the cabin when a person is in the vehicle;
collecting a first image within the cabin when the person leaves the vehicle;
determining the position of the remaining object waiting to be detected in the first image based on the reference image and the second image;
Based on the position of the remaining object waiting to be detected in the first image, a remaining object in the cabin when the person leaves the means of transportation is detected from among the remaining objects waiting to be detected in the first image. including,
Determining the position of the remaining object waiting to be detected in the first image based on the reference image and the second image,
Performing target tracking on a remaining target waiting for detection based on the reference image and the second image of a plurality of frames;
determining the position of the remaining object awaiting detection in the first image based on the result of target tracking;
A method for detecting a remaining object characterized by the following. Obtaining a reference image inside the cabin of the means of transportation when there is no object left behind in the cabin is as follows:
The method of detecting a remaining object according to claim 1, further comprising: collecting a target image in the cabin before the person enters the means of transportation, and determining the target image as the reference image. The remaining object detection method according to claim 1 or 2, further comprising determining a remaining object waiting to be detected based on a difference between the second image and the reference image. Determining the remaining object awaiting detection based on the difference between the second image and the reference image,
obtaining target object information in the second image and the reference image, respectively;
and determining at least one target object included in the second image but not included in the reference image as the remaining object waiting to be detected. Described method for detecting remaining objects. receiving a remaining object confirmation command and excluding non-remaining objects from the first image in accordance with the remaining object confirmation instruction;
determining the location of the remaining object in the cabin and/or the type of the remaining object when the remaining object is detected;
Any one of claims 1 to 4 , further comprising transmitting a first notification message to the means of transportation and/or a preset communication terminal when the remaining object is detected. Detection method for remaining objects as described in section. The method further comprises transmitting first control information for adjusting environmental parameters in the cabin to the means of transportation based on the type of the remaining object when the remaining object is detected. The remaining object detection method according to any one of claims 1 to 5 . Sending first control information for adjusting an environmental parameter in the cabin to the means of transportation based on the type of the remaining object;
7. The method according to claim 6, further comprising: transmitting first control information for adjusting environmental parameters in the cabin to the means of transportation when it is detected that the type of the remaining object is a living body. Described method for detecting remaining objects. Sending first control information for adjusting an environmental parameter within the cabin to the vehicle;
The remaining object detection method according to claim 6 or 7 , further comprising transmitting window opening control information and/or air conditioner operation control information to the means of transportation. determining whether the remaining object has been removed from the cabin when the remaining object is detected;
further comprising: performing a specific operation if the remaining object is determined to have been removed from the cabin;
The specific operation is
recording the time at which the retained object was removed from the cabin;
Recording the identity information of the person who took out the retained object;
transmitting second control information to the vehicle for adjusting environmental parameters within the cabin;
The remaining target detection method according to any one of claims 1 to 8 , comprising at least one of: transmitting a second notification message to a preset communication terminal. Determining whether the retained object has been removed from the cabin includes:
acquiring a third image within the cabin within a predetermined time period after the person leaves the vehicle;
determining whether the remaining object has been removed from the cabin based on the third image and the reference image;
and/or
The remaining object detection method further includes activating an image collection device on the means of transportation when the means of transportation is started;
and/or
The remaining object detection method according to claim 9 , wherein the remaining object detection method further includes turning off the image collecting device when it is determined that the remaining object has been taken out from inside the cabin. Method. A remaining object detection device,
a first acquisition module for acquiring a reference image inside the cabin of the means of transportation when there is no remaining object in the cabin;
a second acquisition module for collecting a second image within the cabin when personnel are aboard the vehicle;
a first collection module for collecting a first image within the cabin when personnel leave the vehicle;
A detection module,
a second determining unit for determining the position of the remaining object waiting to be detected in the first image based on the reference image and the second image;
Based on the position of the remaining object waiting to be detected in the first image, a remaining object in the cabin when the person leaves the means of transportation is detected from among the remaining objects waiting to be detected in the first image. a detection unit for; a detection module including ;
The second confirmation unit is
a tracking subunit for performing target tracking on a remaining target awaiting detection based on the reference image and the second image of a plurality of frames;
a determining subunit for determining the position of the remaining object waiting to be detected in the first image based on the result of target tracking.
A remaining object detection device characterized by the following. Equipped with memory and processor,
A computer program is stored in the memory, the computer program is operable on the processor, and the computer device comprises:
A computer device, characterized in that the method according to any one of claims 1 to 10 is implemented when the processor executes the program. A means of transportation,
The cabin of the means of transportation is provided with an image collecting device and the remaining object detection device according to claim 11 or the computer device according to claim 12 , which is communicably connected to the image collecting device. A means of transportation characterized by A computer readable recording medium in which a computer program is stored,
A computer-readable recording medium, characterized in that when the program is executed by a processor, the method according to any one of claims 1 to 10 is realized. A computer program comprising computer readable code, the computer program comprising:
A computer program product, characterized in that when the computer readable code is run on an electronic device, a processor in the electronic device executes the method according to any one of claims 1 to 10 .