KR20210121015A - Detection of leftover objects - Google Patents

Abstract

A method for detecting a leftover object is provided, the method comprising: acquiring a reference image in the cabin when there is no object left in the cabin of a transportation means, and collecting a first image in the cabin when a person leaves the transportation means, , and also detect the object left in the cabin when the person leaves the transportation means based on the first image and the reference image.

Description

Detection of leftover objects

[Citation of cross-references to related applications]

The present invention claims priority to the Chinese patent application with an filing date of March 25, 2020 and an application number of 202010217625.9, which is incorporated into this application in a manner of citing all contents of the Chinese patent application.

[Technology]

The present invention relates to the field of computer vision technology, and more particularly, to a method, apparatus and means of transportation for detecting a leftover object.

When people use transportation, they often leave objects (eg wallets, keys, etc.) and even living things (eg pets, children, etc.) There are also risks. Therefore, it is necessary to reduce the loss or risk by taking a countermeasure according to the detection result by detecting the object left in the transportation means.

The present invention provides a method, apparatus and means of transportation for detecting a leftover object.

According to a first aspect according to an embodiment of the present invention, there is provided a method for detecting a leftover object, the method comprising: acquiring a reference image in a cabin of a transportation means when there is no object left in the cabin; collecting a first image within the cabin when a person leaves the vehicle; and detecting a leftover object in the cabin when the person leaves the transportation means based on the first image and the reference image.

According to a second aspect according to an embodiment of the present invention, there is provided a device for detecting a leftover object, wherein the device includes a first acquiring module for acquiring a reference image in the cabin when there is no object left in the cabin of the transportation means ; a first collection module for collecting a first image within the cabin when a person leaves the transportation means; and a detection module for detecting an object left in the cabin when the person leaves the transportation means based on the first image and the reference image.

According to a third aspect according to an embodiment of the present invention, there is provided a computer device, the computer device comprising a memory and a processor. A computer program is stored in the memory, and the computer program is executed by the processor to realize the method described in any embodiment.

According to a fourth aspect according to an embodiment of the present invention, there is provided a means of transportation, comprising: an image collecting device in a cabin of the means of transportation; and the apparatus for detecting a leftover object described in any embodiment of the present invention or the computer device described in any embodiment of the present invention, communicatively connected with the image collection apparatus, are provided.

According to a fifth aspect according to an embodiment of the present invention, there is provided a computer-readable recording medium having a computer program stored thereon. When the program is executed by a processor, the method described in any of the embodiments is realized.

According to a sixth aspect according to an embodiment of the present invention, there is provided a computer program product, wherein when the computer program product is read and executed by a computer, the method described in any of the embodiments is realized.

According to a seventh aspect according to an embodiment of the present invention, there is provided a computer program comprising computer readable code, wherein when the computer readable code is run on an electronic device, the present invention is executed by a processor in the electronic device. The methods described in any of the examples are practiced.

According to an embodiment of the present invention, a reference image in the cabin when there are no objects left in the cabin of a transportation means is acquired, and a first image in the cabin when a person leaves the transportation means is collected, the first image and detecting the object left in the cabin when the person leaves the transportation means based on the reference image. The above method is capable of detecting not only living organisms but also non-living objects, has a simple method, has a wide application range, and has a high detection accuracy.

It should be understood that the above general description and the following detailed description are merely exemplary and interpretative, and not limiting of the present invention.

The drawings herein are incorporated in and constitute a part of this specification. These drawings show embodiments consistent with the present invention, and together with the specification, they may be used to describe embodiments of the present invention.
1 is a flowchart of a method for detecting a leftover object according to an embodiment of the present invention.
2A is a schematic diagram of a first image of an embodiment of the present invention;
2B is a schematic diagram of a reference image of an embodiment of the present invention.
Fig. 3 is a schematic diagram of the relationship between the machine learning model and the image acquisition device in the embodiment of the present invention.
4 is a schematic diagram of a message notification interface of a communication terminal according to an embodiment of the present invention.
5 is a block diagram of an apparatus for detecting a leftover object according to an embodiment of the present invention.
6 is a schematic diagram of a computer device according to an embodiment of the present invention.
7 is a schematic diagram of a transportation means according to an embodiment of the present invention.
8A is a distribution schematic diagram of an image acquisition device according to an embodiment of the present invention.
8B is another work distribution schematic diagram of the image collecting apparatus according to the embodiment of the present invention.

Exemplary embodiments are described in detail herein and examples thereof are shown in the drawings. When reference is made to drawings in the following description, the same numbers in different drawings refer to the same or similar elements, unless specifically stated otherwise. The embodiments described in the following exemplary embodiments are not representative of all embodiments consistent with the present invention. To the contrary, these are merely examples of apparatus and methods consistent with some aspects of the invention as set forth in the appended claims.

The terminology used in the present invention is for the purpose of describing specific embodiments only, and is not intended to limit the present invention. As used in the present invention and the appended claims, the singular forms of "a kind," "the," "the," and the like are intended to include the plural unless the context clearly dictates otherwise. It is to be understood that the term "and/or" as used herein indicates that one or more includes any one or all possible combinations of related listed items. In addition, the term "at least one" in the present specification means any one of a plurality of types or an arbitrary combination of at least two of the plurality of types.

In the present invention, various information is described using terms such as first, second, third, etc., but it should be understood that such information is not limited by these terms. These terms are only used to distinguish the same kind of information from each other. For example, the first information may be referred to as second information, and likewise second information may be referred to as first information, without departing from the scope of the present invention. Depending on the context, the word “if” as used herein may be interpreted as “if…”, “if…”, or “in response to…”.

Hereinafter, in order for those skilled in the art to better understand the technical solutions of the embodiments of the present invention, and to make the above objects, features and advantages of the embodiments of the present invention clearer and easier to understand, the technical solutions of the embodiments of the present invention with reference to the drawings The method will be described in more detail.

An embodiment of the present invention provides a method for detecting an object left in a transportation means, and as shown in FIG. 1 , the method may include the following steps 101 to 103 .

In step 101, a reference image in the cabin in the case where there is no object left in the cabin of the transportation means is acquired.

In step 102, collecting a first image in the cabin when a person leaves the transportation means.

In step 103, an object left in the cabin when the person leaves the transportation means is detected based on the first image and the reference image.

In an embodiment of the present invention, the means of transportation may be a vehicle, such as a private car, a bus, a school bus, a truck, or a train, or a means used to transport people or cargo, such as a ship or an airplane. Accordingly, the cabin of the transportation means may be a vehicle cabin, a ship cabin, or an airplane cabin. Hereinafter, a solution of an embodiment of the present invention will be described by taking an example in which the means of transportation is a vehicle and the cabin is a vehicle cabin. The realization of the remaining object detection technology of other types of transportation is also the same, so it will not be repeated.

In step 101, a reference image in the vehicle cabin when there is no object left in the vehicle cabin may be acquired. Here, in the reference image in the vehicle cabin, only objects unique to the vehicle cabin (eg, a seat, a steering wheel, a vehicle trim, etc.) are included, and the remaining objects are not included. The reference image may be collected and stored when there is no object left in the vehicle cabin, and may be used once or a plurality of times. For example, it may be repeatedly used a plurality of times when it is necessary to perform detection of a left object. For example, before the vehicle is shipped, it may be collected using an image collecting device on the vehicle and stored in a storage unit of the vehicle. to be collected, or by using a user terminal (eg, a mobile phone, a tablet computer, a camera, etc.) and stored in the vehicle's storage unit. In addition, by performing image compression processing on the collected original images in the cabin and storing the compressed image as a background image, it is possible to reduce storage space and improve image processing efficiency.

A reference image is collected once, and the collected reference image can be used as a reference image when the detection of the leftover object is executed every time. Alternatively, the reference image may be updated according to a specific method. Updating the reference image according to the specific method may include updating the reference image at regular intervals (eg, 1 day or 1 week, etc.), and updating the reference image in response to a trigger of a specific event. it might be The specific event may be that an object left behind has been detected and a reference image update command has been received, for example, the background in the vehicle cabin has changed, or a person is attempting to enter the cabin of the transportation means (eg, vehicle locking has occurred). released) is detected, triggering an update of the reference image.

The background image may be a background image of the entire vehicle cabin, or a specific area within the vehicle cabin (for example, at least one of a driving area, a co-pilot area, a rear seat, a child seat, a love seat area, a trunk area, a luggage area, etc.) It may be a background image of After the image of the entire vehicle cabin is collected, the directly collected image may be used as a background image, or the image after cutting by cutting the image as needed may be used as a background image. The number of background images may be one or plural. For example, when one image collecting device is included in the vehicle cabin, the background image may be one image collected by the image collecting device. When a plurality of image collecting devices are included in the vehicle cabin, the number of the background images is greater than one, and each background image is collected by one image collecting device among them.

In actual application, if there are no objects left in the vehicle cabin after the personnel get out of the vehicle, by shooting and memorizing an image in the vehicle cabin, a reference for executing the detection of the remaining objects when the next person leaves the means of transportation Can be used as an image. Alternatively, a target image in the cabin may be collected before a person enters the means of transportation, and used as a reference image for performing object detection left when the person leaves the means of transportation. The left object may include things such as a wallet, a key, a mobile phone, an umbrella, a brief case, and a suit case, and may also include a living body such as a person or a pet. A person may be a person who is likely to be left in various vehicles, such as a child, an elderly person, a person sleeping, a person with a physical disability, and the like.

In step 102, the personnel may include any personnel in the means of transportation, such as, for example, drivers, crew members, passengers, and the like. Based on at least one of an opening situation of a cabin door of a transportation means, a driving situation of the transportation means, a movement trajectory of a person, and a specific command, it may be determined whether the person leaves the transportation means. For example, when it is detected that the cabin door is opened or that the seat belt of the person is loosened, it can be determined that the person is getting out of the vehicle. In addition, for example, when it is detected that the power supply of the vehicle is turned off and the cabin door is opened, it can be determined that the person is getting out of the vehicle. Further, for example, when it is detected that the movement trajectory of the personnel is from the inside of the cabin to the outside of the cabin, it can be determined that the personnel are getting off the lane. Also, for example, when an order to confirm the end of the journey is detected, it may be determined that the person is getting out of the vehicle. In addition, other methods may determine whether the personnel have left the means of transportation, which will not be repeated here.

The first image in the cabin may include one image or a plurality of images. Each first image may be collected by one image acquisition device in the vehicle cabin, for example, one image or a plurality of images are acquired as the first image from a video stream collected by using the image acquisition device. can do.

In step 103, an object left in the cabin when the person leaves the transportation means may be detected based on the first image and the reference image. Here, the object left behind may be a living body and/or object that has been moved into the cabin by the person but is forgotten in the cabin when the person leaves the transportation means. If the quantity of both the first image and the reference image is greater than 1, based on each of the first image and the corresponding one reference image, respectively, it is possible to detect the object left in the cabin when the person leaves the transportation means. have. For example, if both the i-th first image and the i-th reference image are images corresponding to the i-th sub-region in the cabin, based on the i-th first image and the i-th reference image, when the person leaves the transportation means The object remaining in the i-th sub-region in the cabin may be detected. Here, whenever the remaining object is detected, the remaining object detection may be executed for all sub-regions, or the remaining object detection may be executed only for some of the sub-regions.

The operation for detecting the leftover object may be always executed, or may be triggered and executed in a specific case. It can be triggered by means of transportation, for example. It may also be triggered manually, for example, by a user terminal that has established a communication connection in advance with the means of transport. A detection trigger command is sent by the user terminal, and after the detection trigger command is received, an operation of executing detection of the remaining object can be started. The detection trigger command may further include a target detection type, and determine whether a specific type of remaining object is included in the image to be processed. For example, a user may notice that a key has been lost after getting out of a car. At this time, it is possible to send a detection trigger command including the type of "key" through the mobile phone to trigger detection of the remaining object of the type "key".

In some embodiments, based on the difference between the first image and the reference image, it is possible to detect the object left in the cabin when the person leaves the transportation means. For example, at least one target object included in the first image but not included in the reference image may be determined as the remaining object. As shown in Figure 2a, the first image includes a mobile phone, a child, a seat and a pillow, but as shown in Figure 2b, the reference image includes a seat and a pillow, and the first image includes Cell phones and children, but not included in the reference image, may be identified as objects left in the cabin when the personnel leave the means of transport. In this way, the remaining object can be detected more intuitively and the implementation is simple, so the realization cost is low.

Here, the target object in the reference image may be obtained through a method of labeling the reference image or may be obtained through a method of detecting the reference image. Whenever a difference between the first image and the reference image is determined, the target object may be detected from the reference image once, or a target object detected in the past may be directly employed as the target object.

In some embodiments, a second image in the cabin when the person enters the transportation means may be collected, and a remaining object to be detected may be determined based on a difference between the second image and the reference image. . Instances of the personnel entering the means of transportation may include cases when the personnel are entering the means of transportation, and may include cases of personnel entering the means of transportation and before leaving the means of transportation. For example, the second image may be collected when the means of transportation is about to arrive at a destination. In an actual application, it is possible to determine whether the transportation means is about to arrive at a destination through an application program (eg, a map application program or a network vehicle reservation application program, etc.) running on a smart terminal on the transportation means. By determining the left object to be detected related to the specific person through this method, the left object detection can be executed only for the left object to be detected related to the specific person, so that the detection precision can be improved and the detection resource consumption can be reduced.

In some embodiments, at least one target object included in the second image but not included in the reference image may be determined as the remaining object to be detected. In this way, a relationship between the left object and the personnel may be established, and only the remaining objects related to a specific person may be detected. The association can be established by allocating identity information to a person entering the means of transportation, and binding the left object to be detected and the identity information of the person determined when the person enters the means of transportation. Based on this, when the person leaves the transportation means, the probability of determining the remaining object from only the remaining objects to be detected related to the person and determining the object left by other personnel in the transportation means as the remaining object of the person can be reduced

For example, the second image taken when Person A gets into a car includes a mobile phone, and since there are a series of keys in the vehicle before Person A gets into the car, only the mobile phone corresponding to Person A is left to be detected. check as an object. If a mobile phone is included in the first image photographed when the person gets off the car A, it is determined that there is an object left in the car when the person gets off the car A. If the mobile phone is not included in the first image photographed when Person A gets out of the vehicle, it is determined that there are no objects left in the vehicle when Person A gets out of the vehicle.

When there are a plurality of persons entering the cabin, a second image in the cabin when each person enters the transportation means is respectively collected, and based on the difference between the second image corresponding to each person and the reference image, It is possible to determine the remaining objects to be detected for each of the persons.

In some other embodiments, collecting a second image within the cabin when the person enters the means of transportation, and a location within the first image of a leftover object to be detected based on the reference image and the second image , and based on the position of the remaining object to be detected in the first image, the remaining object may be detected from among the remaining objects to be detected in the first image. By doing so, the detection efficiency is improved by first roughly determining the position of the remaining object to be detected, and then executing the remaining object detection based on the position.

Here, the second image and the reference image are input to the first machine learning model trained in advance, and the left object to be detected (called the object suspected of being left) and the left one based on the result of the first machine learning model After determining the location of the suspected object in the second image, the location of the object suspected to be left in the first image may be determined based on the location of the object suspected to be left in the second image. The machine learning model may employ a neural network, and the machine learning model is a model in which a neural network and a conventional visual algorithm (for example, an optical flow method, an image sharpening method, an image difference algorithm, or a Carter tracking algorithm) are combined. can be hired A neural network according to an embodiment of the present invention may include an input layer, at least one intermediate layer, and an output layer, wherein the input layer, the at least one intermediate layer, and the output layer all include one or a plurality of neurons. Here, the intermediate layer generally refers to a layer positioned between an input layer and an output layer, such as a hidden layer. In one optional example, the intermediate layer of the neural network may include, but is not limited to, at least one of a convolutional layer, a Rectified Linear Units (ReLU) layer, and the like. The greater the number of intermediate layers included in the neural network, the deeper the network. The neural network may specifically be a deep neural network or a convolutional neural network.

For example, target tracking is executed on the remaining object to be detected based on the reference image and the second image of a plurality of frames, and the position of the remaining object to be detected is determined based on the result of the target tracking in the first image. can In this way, we first determine the object likely to be forgotten within the cabin (i.e. the object suspected to have been left behind), determine the location of the object suspected to have been left behind more precisely, and then continue based on the location of the object suspected to have been left behind. The detection efficiency and precision were improved by detecting the object left by the method.

As shown in FIG. 3 , when the second image is collected using a plurality of image acquisition devices in the cabin, each image acquisition device corresponds to one machine learning model, and each machine learning model corresponds to one image acquisition device. used to detect the second image collected by For example, when N image acquisition devices are included in the cabin, the second image and background image collected by the image acquisition device 1 are input to the machine learning model 1, and the second image collected by the image acquisition device 1 Detects an object suspected of being left from the image acquisition device 2, inputting the second image and background image collected by the image acquisition device 2 to the machine learning model 2, and the object suspected of being left from the second image collected by the image acquisition device 2 to detect, … , it is inferred that The machine learning model may be shared to detect images collected by a plurality of other image acquisition devices, but the present invention is not limited thereto.

In some embodiments, before detecting the object left in the cabin when the person leaves the vehicle based on the first image and the reference image, first exclude objects that are not left in the first image. can The object that is not left is an object that has been moved into the cabin by personnel, but is expected to remain in the cabin, and may include, for example, a pillow or car accessory. For example, it is possible to receive a left object confirmation command, and exclude an object that is not left from the first image according to the left object confirmation command. As one implementation manner, before the person leaves the means of transportation, a third image is taken in the cabin of the means of transportation, and the third image is displayed on a display device (for example, a central control screen of a vehicle or a display interface of a user terminal) By sending and displaying to , it is possible to transmit the object confirmation command left by the personnel using the user terminal or the central control screen. Also, for example, it is possible to obtain past processing results of the remaining objects. When a specific object is confirmed as a leftover object through a past detection process, but has been processed for a long period of time or only a few times (for example, it has not been taken out of the cabin), the object is determined as a non-remained object. Through this method, it is possible to reduce the probability of erroneously judging an object that is not left, such as an object that has been moved into the cabin by the person, but is expected to stay in the cabin, as a remaining object, and can reduce erroneous detection.

When the left object is detected, the location of the left object in the cabin and/or the type of the left object may be determined. The location may be an approximate location such as a co-pilot's seat or a back seat, or may be more accurate location information, for example, coordinates in the cabin of the object left behind. The type of the remaining object may be simply classified into a biological type and an object type, and each type may be classified into a more detailed sub-type. For example, classify biometric types into pet types and children types, and classify object types into key types, wallet types and. It can be classified by type of mobile phone, etc. By determining the location and/or type of the left object, subsequent operations can be easily performed based on the location and/or type of the left object, for example, by sending a notification message to control the environmental parameters in the cabin By doing so, it is possible to reduce the probability of a security problem of the left object.

The first image may be input to a second machine learning model trained in advance, and the position and/or type of the remaining object may be obtained based on an output result of the second machine learning model. The second machine learning model and the first machine learning model may be the same model or different models. In addition, the second machine learning model may include a first sub-model and a second sub-model, wherein the first sub-model detects a biological object, and the second sub-model detects an object object. Here, the first sub-model may be pre-trained through a sample image of a living body object, and the second sub-model may be pre-trained through a sample image of an object object. Sample images may include images taken at different light intensities and different scenes, improving the accuracy of the trained object recognition model.

In some embodiments, when the left object is detected, a first notification message may be transmitted to the transportation means and/or a preset communication terminal. By sending the first notification message, it is helpful for personnel to find the leftover object and quickly retrieve the leftover object. The first notification message may include prompt information for indicating that the currently left object exists. In addition, the first notification message further includes a time of leaving the left object, a type and/or location of the left object. After sending the first notification message to the transportation means, the transportation means may output prompt information, and the prompt information may include car audio or sound prompt information output through a horn and/or a light output through a vehicle light Contains prompt information. In addition, by outputting other voice prompt information or outputting light prompt information through a light emitting device located at another location, the position of the remaining object may be indicated.

The communication terminal can establish a communication connection with a transportation means through any connection method such as mobile data connection, Bluetooth connection, WiFi connection, etc., and the communication terminal is a smart terminal such as a mobile phone, a tablet computer, a smart watch and a note computer can be After sending the first notification message to the communication terminal, the communication terminal may output prompt information, and as shown in FIG. 4 , the prompt information includes at least one of text prompt information and image prompt information. The text prompt information may be textual content in the form of "There is an object left in the backseat of the car" or "There is a child in the car". The time at which the object left behind was detected may be further included, for example, text content such as "Time: February 13, 2020 18:35; Location: the back seat of the car; Type of object left: Wallet". The image prompt information may include the first image, or may include only the image of the remaining object cut from the first image. The amount of data transmission can be reduced by sending an image of the cut object left behind.

When the left object is detected, by transmitting first control information for adjusting the environmental parameter in the cabin based on the type of the left object to the transportation means, it is possible to provide a more comfortable environment in the cabin for the left object have. For example, when the type of the left object is a biological type, first control information for adjusting an environmental parameter in the cabin may be transmitted to the transportation means, for example, window opening control information and/or air conditioning operation By transmitting the control information, it is possible to reduce the probability of creating a hazard to the leftover object of the living type due to the relatively bad environment in the cabin (eg, the temperature is too high or the oxygen content is too low). Specifically, when the type of the left object is a biological type, window opening control information is transmitted to the transportation means to open the window of the transportation means.

The window opening control information includes, but is not limited to, quantity information, location information, and/or opening degree information of windows to be opened. Optionally, the window opening control information may be generated based on at least one of the location of the remaining object, an environmental parameter in the cabin, and an environmental parameter outside the cabin. For example, when the left object is located in the rear seat of the vehicle, one window at the rear of the vehicle is controlled to be opened. Also, for example, when the oxygen content in the cabin is lower than a predetermined limit value, the control is made to open the two windows on the rear side of the vehicle. The degree of opening can be set in advance, for example, the moving distance of the window opening can be fixedly set to 5 cm. By doing so, the oxygen content in the cabin is maintained within the required range, and at the same time, the object left by personnel outside the cabin The safety of the left object can be secured by harming the user or preventing the left object from coming out of the window. The degree of opening can be dynamically set according to environmental parameters other than the cabin. For example, when the environmental temperature outside the cabin is out of a predetermined range, it is set small according to the degree of opening (for example, 5 cm), and vice versa. In this case, the opening degree can be set to be large (eg, 8 cm). By doing so, it is possible to reduce the effect on the remaining objects of the environment other than the cabin.

When the type of the left object is a living body type, air conditioner operation control information may be transmitted to the transportation means, and the temperature and/or operation mode (eg, cooling or heating) of the air conditioner may be controlled. By controlling the windows and/or air conditioning, it is possible to reduce the probability that the temperature in the cabin is too high, or the oxygen content is insufficient, and the probability that a hazard is created for the leftover objects of the living species.

In one example, when the left object is detected, the first control information can be directly transmitted. For example, by transmitting the air conditioner operation control information to the transportation means, the air conditioner controls the temperature/humidity suitable for the living body. mode, or by sending window opening control information to the means of transportation, the degree of opening of the window can be controlled. For example, a window is controlled to open a gap rather than fully open, so that the air environment in the vehicle is improved, while at the same time, living organisms in the vehicle cabin do not escape from the inside of the vehicle cabin or are threatened outside the vehicle cabin.

In another example, first, an environmental parameter in the cabin may be detected, and the first control information may be transmitted only when the environmental parameter exceeds a predetermined range. For example, when it is detected that the temperature in the cabin is too high or too low, the air conditioner operation control information may be transmitted to the transportation means. After the cabin temperature has become adequate, the air conditioner can be controlled to turn it off again. Also, for example, when it is detected that the oxygen content in the cabin is too low, window opening control information can be transmitted to the vehicle to control the window to open the gap.

In some embodiments, when the leftover object is detected, it may be determined whether the leftover object has been taken out of the cabin. When it is determined that the left object has been taken out of the cabin, a specific operation may be executed, wherein the specific operation includes a time at which the left object was taken out of the cabin and/or the identity of the person who took out the left object. recording information; sending second control information for adjusting an environmental parameter in the cabin to the transportation means; and transmitting a second notification message to a preset communication terminal; at least one of For example, you can record "A user with ID XXX took out a pet on March 22, 2020 at 19:00:35". When the first control information for adjusting the environmental parameter in the cabin is transmitted to the transportation means before that, the second control information for adjusting the environmental parameter in the cabin can be transmitted to the transportation means, for example, By sending the window closing control information and/or air conditioner closing control information to the transportation means, it is possible to close the window and/or air conditioner of the transportation means. In this way, it is possible to reduce the energy consumption of the transportation means, reduce the manual operation of personnel, and reduce the complexity of the operation. A second notification message may be generated based on the recorded information and transmitted to the communication terminal, wherein the second notification message includes at least one of the name, type, time of extraction, and identification of the removed person. may include In this way, it is possible to quickly send a notification when the leftover object is ejected, thereby reducing the possibility of erroneous ejection.

Optionally, acquire a third image in the cabin within a predetermined time period after the passenger leaves the transportation means, and determine whether the leftover object has been taken out of the cabin based on the third image and the reference image can do. Specifically, based on the difference between the third image and the reference image, it may be determined whether the remaining object is taken out of the cabin. When there is at least one target object included in the third image but not included in the reference image, it is determined that the remaining object not taken out exists. If at least one target object included in the third image but not included in the reference image does not exist, it is determined that all remaining objects are taken out. It is possible to detect whether or not the left object has been taken out by acquiring the image, and since the implementation is simple, the detection cost is low.

Optionally, acquire a third image in the cabin within a predetermined time period after the passenger leaves the transportation means, and determine whether the leftover object has been taken out of the cabin based on the third image and the first image can be confirmed Specifically, based on a difference between the third image and the first image, it may be determined whether the remaining object has been taken out of the cabin. If there is at least one target object included in the third image but not included in the first image, it is determined that the remaining object is taken out. If at least one target object included in the third image but not included in the first image does not exist, it is determined that the remaining object is not taken out.

In some embodiments, when the left object is detected, information on the left object may be recorded in the database. The information on the left object may include at least one of information such as an image of the left object, a location, a type, a left time, a person to which the left object belongs, and a person who took out the left object. By building a database, it is convenient to inquire information about the object left behind.

In the above embodiment, when it is necessary to collect an image, an image collecting device (for example, a camera) on the transportation means may be activated to collect the image, and when it is not necessary to collect the image, the image may be collected. The collection device can be turned off. By doing so, energy consumption can be reduced since the image collection device does not always have to be in the working state. For example, when the vehicle is started, the image collection device on the vehicle is activated. or when it is determined that the person has entered the means of transportation, the image collecting device on the means of transportation is activated. or when it is determined that the person is about to leave the means of transportation, the image collection device on the means of transportation is activated. Also, for example, when it is necessary to detect whether the leftover object has been taken out of the cabin, when it is determined that the leftover object has been taken out of the cabin, the image collecting device is turned off. If it is only necessary to detect the object left behind, turn off the image collection device when a person leaves the vehicle.

The remaining object detection method of the embodiment of the present invention can detect not only a living body but also a static object, and the detection accuracy is high. Since the embodiment of the present invention can be used in other application scenarios such as private cars, network reservation vehicles, or school buses, the scope of application is wide. Here, getting on and off the car of the person may be determined in different ways depending on the actual scene. For example, in a private vehicle scenario, it is possible to determine whether or not the driver gets in and out of the vehicle based on the strength or weakness of the communication connection signal between the driver's communication terminal and the vehicle. In the network reservation vehicle scenario, based on the operation in the network reservation vehicle application program (for example, operation to confirm that the passenger has been received or operation to confirm that the driver has arrived at the destination), whether or not the passenger is in the vehicle and getting out of the vehicle It can be determined whether or not

Those skilled in the art will know that the description order of each step in the above method of specific embodiments means a strict execution order and does not constitute a limitation on the implementation process, the specific execution order of each step is determined by its function and possible internal logic You have to understand that it should be

As shown in Fig. 5, the present invention further provides a device for detecting a leftover object, the device comprising: a first acquiring module 501; a first collecting module 502; and a detection module 503 .

The first acquisition module 501 acquires a reference image in the cabin of the transportation means when there is no object left in the cabin.

The first collection module 502 collects a first image in the cabin when a person leaves the transportation means.

The detection module 503 detects an object left in the cabin when the person leaves the transportation means based on the first image and the reference image.

In some embodiments, the first acquisition module 501 collects a target image in the cabin before the person enters the means of transportation, wherein the target image is the reference image, and the detection module is configured to: Based on the difference between the image and the reference image, detect the object left in the cabin when the person leaves the transportation means.

In some embodiments, the apparatus further comprises: a second collection module for collecting a second image within the cabin when the person enters the means of transportation; and a first determining module configured to determine the remaining object to be detected based on the difference between the second image and the reference image.

In some embodiments, the first determining module comprises: a first acquiring unit for acquiring target object information of the second image and the reference image, respectively; and a first determining unit configured to determine at least one target object included in the second image but not included in the reference image as the remaining object to be detected.

In some embodiments, the detection module 503 comprises: a first collection unit for collecting a second image in the cabin when the person enters the means of transportation; a second determining unit for determining a position in the first image of a left object to be detected based on the reference image and the second image; and a detecting unit for detecting the leftover object from among the leftover objects to be detected in the first image based on the position of the leftover object to be detected in the first image.

In some embodiments, the second determining unit comprises: a tracking sub-unit configured to execute target tracking on the remaining object to be detected based on the reference image and the second image of a plurality of frames; and a determining sub-unit, configured to determine a position in the first image of the left object to be detected based on a result of target tracking.

In some embodiments, the device is a receiving module for receiving a left object confirmation command before detecting the left object in the cabin when the person leaves the transportation means based on the first image and the reference image ; and an exclusion module for excluding objects that are not left from the first image according to the left object confirmation command.

In some embodiments, the apparatus further comprises a second determining module for determining a location of the left object in the cabin and/or a type of the left object when the left object is detected.

In some embodiments, the device further comprises a first sending module for sending a first notification message to the transportation means and/or a preset communication terminal when the leftover object is detected.

In some embodiments, the device is configured to send, when the leftover object is detected, first control information for adjusting an environmental parameter in the cabin based on the kind of the leftover object to the transportation means A module is further provided.

In some embodiments, the second transmitting module transmits first control information for adjusting an environmental parameter in the cabin to the transportation means when the type of the remaining object is a biological type.

In some embodiments, the second transmitting module transmits window opening control information and/or air conditioning operation control information to the transportation means.

In some embodiments, the apparatus further comprises: a third determining module, configured to, when the leftover object is detected, determine whether the leftover object has been taken out of the cabin; and an execution module for executing a specific operation when it is determined that the left object has been taken out of the cabin, wherein the specific operation is performed at a time when the left object is taken out of the cabin and/or the left object recording identification information of the person who took out the vehicle, sending second control information for adjusting an environmental parameter in the cabin to the transportation means; and transmitting a second notification message to a preset communication terminal; at least one of

In some embodiments, the third determining module comprises: a second acquiring unit for acquiring a third image in the cabin within a predetermined time period after the passenger leaves the transportation means; and a third determining unit for determining whether the leftover object has been taken out of the cabin based on the third image and the reference image.

In some embodiments, the device further comprises a triggering module for activating the image collection device on the means of transportation when the means of transportation is started.

In some embodiments, the device further comprises an off module for turning off the image collection device when it is determined that the leftover object has been taken out of the cabin.

A function or a module included in the apparatus provided by the embodiment of the present invention may execute the method described in the embodiment of the method, the specific implementation of which may refer to the description of the embodiment of the method above. However, for the sake of brevity, the description is not repeated here.

An embodiment of the present invention further includes a computer device, the computer device having a memory and a processor. A computer program is stored in the memory, and when the computer program is executed by the processor, the method described in any of the embodiments is realized.

6 is a schematic diagram of a hardware configuration of a more specific computer device provided by an embodiment of the present specification, the device comprising: a processor 601; memory 602; input/output interface 603; communication interface 604; and a bus 605 . Here, the processor 601 , the memory 602 , the input/output interface 603 , and the communication interface 604 realize a mutual communication connection in the device via the bus 605 .

The processor 601 may be implemented by a general CPU (Central Processing Unit, Central Processing Unit), a microprocessor, an Application Specific Integrated Circuit (ASIC), or one or a plurality of integrated circuits, and the processor 601 . ) implements the technical solution provided by the embodiment of the present specification by executing the related program.

The memory 602 may be implemented by a read only memory (ROM), a random access memory (RAM), a static storage device, a dynamic storage device, or the like. Memory 602 may store operating system and other application programs. When implementing the technical solution provided by the embodiment of the present specification using software or firmware, the related program code is recorded in the memory 602 , and is called and executed by the processor 601 .

The input/output interface 603 is connected to the input/output module to realize input and output of information. The input/output module may be disposed in the device as a component (not shown), or may be connected to the device to provide a corresponding function. Here, the input device may include a keyboard, a mouse, a touch screen, a microphone, various sensors, and the like, and the output device may include a display, a speaker, a vibrator, a display, and the like.

The communication interface 604 is connected to a communication module (not shown), thereby realizing communication interaction between the present device and other devices. Here, the communication module may realize communication by wired means (eg, USB, network cable, etc.) or may realize communication by wireless means (eg, mobile network, WIFI, Bluetooth, etc.).

The bus 605 includes one path and transfers information between each component of the device (eg, the processor 601 , the memory 602 , the input/output interface 603 , and the communication interface 604 ).

For the above device, only the processor 601, the memory 602, the input/output interface 603, the communication interface 604 and the bus 605 are shown, but in a specific implementation process, the device is used to realize normal operation. It is necessary to explain that it may further include other necessary components. Meanwhile, those skilled in the art should understand that the above device may include only the components necessary to realize the solutions of the embodiments of the present specification, and need not include all the components shown in the drawings.

As shown in FIG. 7 , the embodiment of the present invention further provides a means of transportation, wherein in the cabin of the means of transportation, an image collection device and the present invention communicatively connected to the image collection device are any embodiment The apparatus for detecting a leftover object described in , or the computer device described in any embodiment of the present invention is installed.

The image acquisition device is used to acquire the first image. The image collecting device may take images to be processed in the cabin from after the personnel enter the cabin until the personnel leave the cabin, and the images to be processed in the cabin after a certain period of time has elapsed after the personnel enter the cabin You can also start filming.

In some embodiments, the image collection device may be disposed at an upper portion of the cabin. The number of image collection devices in the cabin may be one or plural. When the number of the image collecting device is one, images to be processed in the entire cabin are collected by using the image collecting device. When the number of image acquisition devices is greater than 1, images to be processed of one sub-area in the cabin are respectively collected by using each image collection device. The quantity, location, and distribution within the cabin of the image collection device may be determined according to the shape, size, and field of view of the image collection device within the cabin. For example, as shown in FIG. 8A , in the case of a narrow and long area such as a vehicle, one image collection device may be placed in the center of the upper part of the vehicle (the upper part of the inside), and as shown in FIG. 8B , each row It is also possible to place one image collection device each above the seat. By arranging a plurality of image acquisition devices, the imaged area becomes relatively full-scale.

In some embodiments, detection may be performed on a frame-by-frame basis for the first image collected by the image collection device, and in some embodiments, the frame rate of the image collected by the image collection device is relatively For example, since images of tens of frames are collected every second, frame skip detection can be performed on the first image. For example, only the first image of the captured first, third, and fifth frames may be detected. The frame skip stride (ie, the interval of frame numbers between adjacent image frames to be detected) may be determined according to an actual scene. For example, when the light beam is weak, the number of detection objects is relatively large, and the resolution of the photographed first image is relatively low, the frame skip stride can be set small, the beam is good, the number of detection objects is relatively small, and the When the resolution of the first image is relatively high, the frame skip stride may be set to be relatively large.

The field of view of the image collection device is likely to be relatively large, and includes an area in which the remaining object is likely to appear and an area in which the remaining object is not generally displayed. Accordingly, when detecting a potential remaining object in the cabin with the first image, the ROI in the first image may be first determined, and then the object remaining within the ROI may be detected. The seat is an area of interest because, for example, leftover objects are likely to appear on car cabin seats, and the center console usually does not.

Embodiments of the present specification further provide a computer-readable recording medium in which a computer program is stored, and when the program is executed by a processor, the method described in any of the above-described embodiments is realized.

The computer-readable medium includes persistent and non-persistent media and removable and non-removable media, and may realize storage of information through any method or technology. The information may be computer readable instructions, data structures, program modules, or other data. Examples of computer recording media include phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read only memory (ROM), electrically erasable Programmable read-only memory (EEPROM), flash memory or other memory technology, read-only optical disk, read-only memory CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassette tape, magnetic tape storage or other magnetic storage may store information accessed by a computing device including, but not limited to, apparatus or other non-transmission medium. As defined herein, computer readable media does not include transitory computer readable media such as modulated data signals or carrier waves.

As can be seen from the description of the above embodiments, those skilled in the art can clearly understand that the embodiments of the present specification can be implemented by software and a necessary general hardware platform. Based on this understanding, the technical solutions of the embodiments of the present specification may be embodied in the form of essentially a software product, or a part contributing to the prior art may be embodied in the form of a software product. The computer software product may be stored in a recording medium such as ROM/RAM, magnetic disk, optical disk, and the like, and by including some instructions, a computer device (which may be a computer, server, or network device, etc.) A method described in an example or part of an example is to be practiced.

The system, apparatus, module, or unit shown in the above embodiment may be implemented specifically by a computer tip or entity, or may be implemented by a product having a specific function. Typical implementation devices are computers, and specific types of computers are computers, laptop computers, mobile phones, camera phones, smartphones, personal digital processing devices, multimedia players, navigation devices, devices for sending and receiving electronic mail and game consoles, tablet computers, wearables. device, or any combination of such devices.

Various embodiments herein have been described progressively, and the same or similar parts among the various embodiments may be referred to with each other. Each embodiment focuses on differences from other embodiments. In particular, since the embodiment of the apparatus is basically the same as the embodiment of the method, the description is relatively simple. For related parts, refer to the description of the embodiment of the method. The embodiment of the above device is only schematic. The modules described as separate parts may or may not be physically separated, and when implementing the solutions of the embodiments of the present specification, the functions of each module may be implemented with one or a plurality of software and/or hardware. have. In addition, according to actual needs, some or all of the modules may be selected to realize the purpose of the solution of the present embodiment. Those skilled in the art can understand and practice without creative work.

Claims (20)

In the detection method of the left object,
acquiring a reference image in the cabin of the transportation means when there are no objects left in the cabin;
collecting a first image within the cabin when a person leaves the vehicle; and
detecting, based on the first image and the reference image, an object left in the cabin when the person leaves the means of transportation;
A method of detecting a leftover object, characterized in that. According to claim 1,
Acquiring a reference image in the cabin when there is no object left in the cabin of the transportation means,
collecting a target image in the cabin before the person enters the means of transportation, and determining the target image as the reference image;
detecting, based on the first image and the reference image, the object left in the cabin when the person leaves the transportation means,
detecting, based on a difference between the first image and the reference image, an object left in the cabin when the person leaves the transportation means
A method of detecting a leftover object, characterized in that. 3. The method of claim 1 or 2,
before detecting the object left in the cabin when the person leaves the transportation means based on the first image and the reference image;
collecting a second image within the cabin when the person enters the vehicle; and
Further comprising determining a remaining object to be detected based on a difference between the second image and the reference image
A method of detecting a leftover object, characterized in that. 4. The method of claim 3,
determining the remaining object to be detected based on the difference between the second image and the reference image,
acquiring target object information in the second image and the reference image, respectively; and
Determining at least one target object included in the second image but not included in the reference image as the remaining object to be detected
A method of detecting a leftover object, characterized in that. According to claim 1,
detecting, based on the first image and the reference image, the object left in the cabin when the person leaves the transportation means,
collecting a second image within the cabin when the person enters the vehicle;
determining a position in the first image of a remaining object to be detected based on the reference image and the second image; and
detecting the leftover object from among the leftover objects to be detected in the first image based on the position of the leftover object to be detected in the first image
A method of detecting a leftover object, characterized in that. 6. The method of claim 5,
Determining a position in the first image of the remaining object to be detected based on the reference image and the second image comprises:
executing target tracking on the remaining object to be detected based on the reference image and the second image of a plurality of frames;
determining a position in the first image of the left object to be detected based on a result of target tracking
A method of detecting a leftover object, characterized in that. 7. The method according to any one of claims 1 to 6,
before detecting the object left in the cabin when the person leaves the transportation means based on the first image and the reference image;
receiving a command to confirm the object left behind; and
Further comprising excluding an object that is not left from the first image according to the left object confirmation command
A method of detecting a leftover object, characterized in that. 8. The method according to any one of claims 1 to 7,
When the left object is detected, further comprising determining a location of the left object in the cabin and/or the type of the left object
A method of detecting a leftover object, characterized in that. 9. The method according to any one of claims 1 to 8,
When the left object is detected, further comprising transmitting a first notification message to the transportation means and / or preset communication terminal
A method of detecting a leftover object, characterized in that. 10. The method according to any one of claims 1 to 9,
When the left object is detected, the method further comprising transmitting first control information for adjusting an environmental parameter in the cabin based on the type of the left object to the transportation means
A method of detecting a leftover object, characterized in that. 11. The method of claim 10,
Transmitting the first control information for adjusting the environmental parameter in the cabin based on the type of the object left to the transportation means,
When it is detected that the type of the object left behind is a living body, transmitting first control information for adjusting an environmental parameter in the cabin to the transportation means
A method of detecting a leftover object, characterized in that. 12. The method of claim 10 or 11,
Transmitting first control information for adjusting an environmental parameter in the cabin to the means of transportation comprises:
Transmitting window opening control information and/or air conditioning operation control information to the means of transportation
A method of detecting a leftover object, characterized in that. 13. The method according to any one of claims 1 to 12,
when the leftover object is detected, determining whether the leftover object has been taken out of the cabin;
When it is determined that the leftover object has been taken out of the cabin, further comprising: executing a specific operation;
The specific operation is
recording the time the leftover object was taken out of the cabin;
recording the identity information of the person who took out the leftover object;
sending second control information for adjusting an environmental parameter in the cabin to the transportation means; and
sending a second notification message to a preset communication terminal; comprising at least one of
A method of detecting a leftover object, characterized in that. 14. The method of claim 13,
Determining whether the leftover object has been taken out of the cabin comprises:
acquiring a third image within the cabin within a predetermined time period after the passenger leaves the vehicle; and
determining whether the left object has been taken out of the cabin based on the third image and the reference image;
and/or;
The method of detecting the leftover object further comprises activating an image collection device on the means of transportation when the means of transportation is started,
and/or;
The method for detecting the leftover object further includes turning off the image collection device when it is determined that the leftover object has been taken out of the cabin.
A method of detecting a leftover object, characterized in that. A device for detecting a leftover object, comprising:
a first acquisition module for acquiring a reference image in the cabin when there are no objects left in the cabin of the transportation means;
a first collection module for collecting a first image within the cabin when a person leaves the transportation means; and
a detection module for detecting an object left in the cabin when the person leaves the transportation means based on the first image and the reference image
Remaining object detection device, characterized in that. A computer device comprising a memory and a processor, wherein the memory stores a computer program operable on the processor,
When the processor executes the program, the method according to any one of claims 1 to 14 is realized
Computer device, characterized in that. In the means of transportation,
an image collecting device in the cabin of the transportation means; and the apparatus for detecting the object left according to claim 15 or the computer device according to claim 16, which is communicatively connected with the image collection apparatus, is installed.
means of transportation, characterized in that In a computer-readable recording medium storing a computer program,
When the program is executed by the processor, the method according to any one of claims 1 to 14 is realized
A computer-readable recording medium, characterized in that. A computer program product comprising:
15. The method according to any one of claims 1 to 14 is realized when the computer program product is read and executed by a computer.
A computer program product, characterized in that A computer program comprising computer readable code, comprising:
15. When the computer readable code is run to an electronic device, a processor in the electronic device executes the method of any one of claims 1 to 14.
A computer program characterized in that.

Images