JP2015007874A - Vehicle cabin monitoring device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To alert a vehicle passenger to an object in a vehicle cabin with the intensity according to the possibility that an object such as luggage or passenger is left in the cabin.SOLUTION: Provided is a vehicle cabin monitoring device 100 which comprises a control device 10. The control device 10 executes: an image acquisition function which acquires a captured image of a vehicle cabin; a vehicle information acquisition function which acquires vehicle information; an object detection function which detects an object existing in the vehicle cabin; a determination function which refers to registration information that associates in advance position information with information about a possibility that an object is left in the vehicle cabin, and determines a possibility that an object is left in the vehicle cabin at the current position; and an alert function which alerts the vehicle passenger by outputting alert information about the object detected in the vehicle cabin, on the basis of the acquired vehicle information and the determination result concerning the possibility that an object is left in the vehicle cabin.

Description

  The present invention relates to a vehicle interior monitoring device that monitors a vehicle interior based on a captured image of a camera provided in the vehicle interior.

  With regard to this type of device, there is known a device that detects forgotten things in the passenger compartment when the shift position is "P" and notifies the passenger that there is leftover when the door is open (Patent Document 1). .

JP 2009-75756 A

  However, in the conventional device, the notification is made by the same method under the condition of the door opening operation regardless of the scene where the occupant is present, so there is a problem that the occupant feels annoyed and alerting the occupant becomes insufficient. is there.

  The problem to be solved by the present invention is to make the passenger's attention sufficiently attracted while preventing the passenger from feeling bothersome.

  The present invention refers to registration information in which position information is associated in advance with information related to the possibility of misplacement of objects such as articles and passengers in the passenger compartment. The above problem is solved by outputting the alert information to the vehicle occupant based on the acquired vehicle information and the determination result relating to the possibility of misplacement.

  According to the present invention, the possibility of misplacement occurring under the situation at the current position of the host vehicle is determined, and the higher the possibility of misplacement is, the higher the alert level when alerting is performed. The driver can be alerted sufficiently by changing the alerting level to a high level at a point where misplacement is likely to occur, while preventing the driver from feeling annoyed at a point where it is difficult for the driver to occur. As a result, it is possible to prevent an occupant or article from being left behind in the passenger compartment without causing the driver to feel bothered.

It is a block block diagram of the compartment monitoring system 1 which concerns on this embodiment. It is a figure which shows the example of installation of the camera of this embodiment. It is a figure which shows an example of the image | video of a backseat. It is a figure which shows an example of registration information It is a figure which shows an example of an alerting level. It is a figure which shows an example of a 1st alerting level. It is a figure which shows an example of the 2nd alerting level. It is a flowchart for demonstrating the control procedure of the compartment monitoring system 1 which concerns on this embodiment. It is a figure for demonstrating the judgment method of the presence or absence of a passenger | crew etc. in a vehicle interior. It is a figure for demonstrating the judgment method of misplacement of the passenger | crew etc. in a vehicle interior.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. In the present embodiment, a case where the vehicle interior monitoring device according to the present invention is applied to the vehicle interior monitoring system 1 will be described as an example.

  FIG. 1 is a diagram illustrating a block configuration of a passenger compartment monitoring system 1 including a passenger compartment monitoring device 100 according to the present embodiment. The vehicle interior monitoring device 100 and the vehicle interior monitoring system 1 including the vehicle interior monitoring device 100 are mounted on a vehicle. The vehicle compartment monitoring system 1 monitors the state of the vehicle interior of the host vehicle. The vehicle interior monitoring device 100 alerts the occupant so that passengers and luggage in the vehicle interior cannot be left behind.

  As shown in FIG. 1, the vehicle interior monitoring system 1 includes a vehicle interior monitoring device 100, an in-vehicle device 200, and a server 300. The in-vehicle device 200 includes a vehicle controller 50, various devices 60, and a navigation device 70. The vehicle interior monitoring device 100 and each vehicle-mounted device 200 are connected by a CAN (Controller Area Network) or other vehicle-mounted LAN in order to exchange information with each other. The server 300 includes a communication device 320, and exchanges information with the communication device 120 of the vehicle interior monitoring device 100.

  The various devices 60 of the present embodiment detect the position of the shift lever, output the shift lever 61, detect the opening / closing of the door of the vehicle, output the detection result, and detect the speed of the vehicle. A vehicle speed sensor 63 for output, a lock device 64 for detecting a lock state of the vehicle and outputting a door lock signal and a door lock release signal, a clock 65 for measuring the time in the in-vehicle device 200, and a seat belt of the vehicle A seat belt device 66 for detecting and outputting a use situation, an ignition key 67 for detecting on / off of an ignition in accordance with an operation of the driver, and an occupant specifying device 68 for specifying an occupant who has boarded the vehicle in the current travel; Have These various device groups 60 output the detected information to the vehicle interior monitoring apparatus 100 via the vehicle controller 50 or directly.

  The shift position information detected by the shift lever 61 is used for travel start determination processing or travel end determination processing.

  The door device 62 determines whether the driver has boarded, and determines whether the driver has boarded based on the open / close flag of each door. The determination result is used for a travel start / end determination process.

  The locking device 64 detects a door lock / door lock release signal based on an input to the locking device 64, and sends the detection result to the control device 10. Each lock information (unlock information) is used in the door opening / closing detection process and the second alert execution process. The locking device 64 detects the opening / closing operation of the door, and after detecting the occupant's key operation signal obtained from the key operation unit, the locking device 64 opens / closes each door based on the door opening / closing signal obtained from the door device 62. Determine whether it was done. The determination results are used for a driver's boarding determination process, a driver's getting-off preparation determination process (first unloading state), an unloading completion determination process (second unloading state), and a lock completion determination process (third unloading state).

  The occupant specifying device 68 specifies the occupant who has boarded the vehicle in the current travel based on the key identifier input to the locking device 64, the position information input to the seat position setting device of the vehicle, and the like. The specific result is sent to the control device 10 and used to create a warning execution history. By referring to the execution history including the occupant specific information, the execution history of each occupant can be acquired.

  The vehicle controller 50 detects a signal indicating the engine operating state, and determines whether or not the engine is operating based on a signal obtained from the engine ECU. The determination result is used for determination processing for the start and end of travel.

  The navigation device 70 of the present embodiment includes a position detection device 71 and map information 72, and guides the driver by showing a route from the current position of the host vehicle to the destination. The navigation device 70 accesses the road attribute and the point attribute stored in the map information 72, obtains the road attribute of the running road and the POI attribute of the current position, and presents them to the driver. In the map information 72 of the present embodiment, the POI information of the position is associated with the position information such as latitude and longitude. For this reason, if position information is specified, the facility (park, entertainment facility, supermarket, etc.) to which the position belongs can be specified. In addition, the navigation device 70 of the present embodiment is capable of communicating with the outside, obtains the road attribute of the road that is running and the POI attribute of the current position, and provides the information to the passenger 73 is provided. Thus, the information on the current position can be associated with the situation of the vehicle and the occupant existing at the point. For example, when a point associated with an amusement facility and the current position of the vehicle substantially match, it can be determined that the vehicle occupant is attempting to use the amusement facility.

  The vehicle interior monitoring device 100 according to the present embodiment includes a control device 10, a camera 20, an output device 30, and a communication device 120. The communication device 120 exchanges information with the server 300 and enables the registration information stored in the server 300 to be read.

  The camera 20 of the present embodiment is a camera that uses an image sensor such as a CCD, and images a rear seat in the vehicle interior. FIG. 2 is a diagram illustrating an installation example of the camera 20. 2A shows the installation position of the camera 20 when the vehicle V is viewed from above, and FIG. 2B shows the installation position of the camera 20 when the vehicle V is viewed from the side. The number and arrangement location of the cameras 20 are not particularly limited, and in the present embodiment, one wide-angle camera 20 is installed near the vehicle interior ceiling of the vehicle V as shown in FIG.

  FIG. 3 shows an example of the rear seat image captured by the camera 20. As shown in FIG. 3, the camera 20 includes images of the front driver seat STFR and the passenger seat STFL, and the rear right seat STRR and the left seat STRL in the imaging region P. Take an image. This image includes an image of the occupant (child) PS1 sitting in the left seat STRL of the rear seat.

  The output device 30 includes a display 31, a speaker 32, a vibration device 33, a hazard lamp 34, an illumination lamp 35, and an external speaker 36 that are equipped on the vehicle and can output alert information to the occupant. The display 31 is installed around a dashboard that can be visually recognized by the driver of the vehicle, and presents a captured image of a rear seat in the vehicle interior, guidance information of the navigation device 70, and the like. The speaker 32 in the passenger compartment and the speaker 36 outside the passenger compartment generate sound that stimulates the hearing of the occupant when the alert information is output. Further, the hazard lamp 34 and the headlight 35 such as a headlight are for presenting light that stimulates the occupant's vision by blinking the lamp when alerting information is output. The vibration device 33 is a mobile phone or the like having a vibration function, and generates vibration that stimulates the occupant's tactile sense when outputting alert information.

  Then, the control apparatus 10 of this embodiment is demonstrated. The control device 10 of the vehicle interior monitoring device 100 according to the present embodiment detects an object on the rear seat based on an image captured by the camera 20, monitors the object, and may cause the occupant to leave the object in the vehicle interior. A ROM (Read Only Memory) 12 in which a program for alerting the passenger is stored in response, and an operation that functions as the vehicle interior monitoring device 100 and the vehicle interior monitoring system 1 by executing the program stored in the ROM. The computer includes a CPU (Central Processing Unit) 11 as a circuit and a RAM (Random Access Memory) 13 that functions as an accessible storage device.

  The control device 10 of the vehicle interior monitoring apparatus 100 according to the present embodiment has an image acquisition function, a vehicle information acquisition function, an object detection function, a determination function, and a warning function. Furthermore, the control device 10 according to the present embodiment further includes a getting-off state determination function, an update function, and an occupant specifying function according to the embodiment. The control device 10 of the present embodiment is a computer that executes each function by cooperation of software for realizing the above function and the above-described hardware.

  Hereinafter, each function of the vehicle interior monitoring apparatus 100 according to the present embodiment will be described.

  First, the image acquisition function of the control apparatus 10 of this embodiment is demonstrated. The control device 10 acquires a captured image in the passenger compartment of the host vehicle captured by the camera 20. The acquired captured image is used for the detection process of the occupant in the rear seat and is also used as an image presented on the display 31 as a display device as an image of the rear seat. In the present embodiment, the rear seat image is included in the alert information that notifies the passenger of the possibility of misplacement. That is, when the occupant tries to leave the vehicle while leaving the luggage or other occupants in the passenger compartment, the rear seat image can be displayed on the display 31.

  The vehicle information acquisition function of the control apparatus 10 of this embodiment is demonstrated. The control device 10 acquires vehicle information detected by the vehicle controller 50, the various devices 60, and the navigation device 70 in the vehicle. The vehicle information may be acquired directly from the vehicle controller 50, the various devices 60, and the navigation device 70, or may be acquired from the various devices via the vehicle controller 50. The vehicle information obtained from the various devices 60 includes shift lever information obtained from the shift lever 61, door opening / closing information obtained from the door device 62, vehicle speed of the host vehicle obtained from the vehicle speed sensor 63, and locking information obtained from the locking device 64. , Time information acquired from the clock 65, seat belt attachment / detachment information acquired from the seat belt device 66, ignition ON / OFF state acquired from the ignition key 67, occupant specifying information acquired from the occupant specifying device 68, navigation The position information of the current position acquired from the position detection device 71 of the device 70 can be exemplified.

  The object detection function of the control apparatus 10 of this embodiment is demonstrated. The control device 10 performs a difference comparison on the captured image captured by the camera 20 and detects an object present in the vehicle interior from the difference value of the image. When a predetermined feature is extracted from a difference between captured images acquired at different times, the control device 10 determines that an object is present at the location where the feature is extracted.

  The method for determining the presence / absence of an occupant other than the driver is not particularly limited, but the control device 10 according to the present embodiment acquires an image captured by the camera 20 at the determined timing when the vehicle is traveling as a comparative image, and performs this comparison. The difference between the images is compared by the gradation difference processing for each pixel between the image and the reference image acquired when the vehicle is unlocked. Since it can be predicted that there is an occupant if there is a difference in the result compared with the comparative image with respect to the reference image obtained when there is no occupant obtained at the time of unlocking the vehicle, it is determined that there is an occupant other than the driver.

  Furthermore, the method for determining whether or not a passenger is left in the passenger compartment is not particularly limited, but when it is determined that there is a passenger other than the driver by the above-described method, the timing when the passenger gets ready to get off, It is determined whether or not an occupant other than the driver remains in the passenger compartment at the timing when the vehicle is completed or when the vehicle is locked. Although not particularly limited, in the present embodiment, it is determined whether or not an occupant is left behind in the passenger compartment at the timing when the door of the driver's seat is locked. In the present embodiment, an image captured by the vehicle interior camera 20 at each timing is acquired as a comparative image, and the difference between the images is compared by a pixel-by-pixel density difference process between the comparative image and the reference image acquired during preparation for getting off the vehicle. To do. And if there is no difference in the result compared with the comparison image with respect to the reference image when there is an occupant acquired during the preparation for getting off, the occupant remains and is determined to be left behind.

  Next, the determination function of the control device 10 of this embodiment will be described. The control device 10 refers to registration information in which position information is associated in advance with information on the possibility of misplacement of articles and passengers in the passenger compartment, and misplacement of articles and passengers in the passenger compartment at the current position of the vehicle occurs. Determine the possibility.

  FIG. 4 shows an example of registration information. As shown in FIG. 4, the position information of the present embodiment includes information in which position information and facility information are associated, information in which position information and driving behavior are associated, and position information and misplaced history. And associated information. The registration information of the present embodiment is associated with facilities, driving behaviors, and histories that are likely to be misplaced. Of course, facilities, driving behaviors, and histories that have a medium or low possibility of misplacement may be stored in the registration information in association with each other. In the present embodiment, the possibility of misplacement is determined according to the degree of possibility of misplacement of articles or passengers in the passenger compartment.

  In the present embodiment, the possibility that an article or an occupant is left behind is defined for each piece of positional information that specifies the facility, from the viewpoint that the possibility that an article or an occupant is left behind in the vehicle compartment varies from facility to facility. As shown in FIG. 4, in entertainment facilities, shopping centers, convenience stores, highways SA, PA, etc., it is defined that there is a high possibility of misplacement of goods and passengers.

  In the present embodiment, the possibility that misplacement of an article or an occupant will occur for each driving action is defined from the viewpoint that the possibility of misplacement of an article or occupant in the passenger compartment varies for each driving action. As shown in FIG. 4, it is defined that there is a high possibility that an article or an occupant is misplaced at a destination when the occupant reaches the destination by route guidance.

  In the present embodiment, from the viewpoint that the possibility of misplacement of articles or passengers in the passenger compartment varies depending on the misplacement history of the occupant himself or another person, misplacement of articles or occupants may occur for each occurrence of misplacement s history. Define gender. As shown in FIG. 4, an article or occupant at a point where the occupant has misplaced himself / herself, at a point where the number of misplacements by others has exceeded a predetermined value, or at a point where the frequency of misplacement by others has exceeded a predetermined value. It is defined that there is a high possibility that misplacement will occur.

  The control device 10 refers to such registration information, and determines the possibility that an article or an occupant is left behind in the current position of the host vehicle. By referring to the registered information, it is possible to determine the possibility of the passenger being left behind under the circumstances at a certain point. From the viewpoint that the behavior and judgment of the occupant in a certain situation is addictive, it is considered that an occupant who has previously left the baggage or occupant at a certain point may misplace the baggage or occupant again at the same location . In particular, when you use an amusement facility that an occupant is interested in, or when you are working on an affair that the occupant thinks it will take only a short time, you temporarily forget that you are carrying luggage or the occupant on the vehicle. Or you may leave it consciously. In other words, depending on the occupant, there is a tendency to repeatedly forget about the presence of luggage and occupants in the vehicle and to estimate the time for performing the task in a short time at an amusement facility of interest. By referring to the registration information that associates such points with the possibility of misplacement, it is possible to determine the possibility that the occupant misplaces the article or the occupant under the situation at each point.

  Furthermore, by performing the process of specifying the occupant described later, the control device 10 refers to the registered information of the specified occupant, and misplacement of articles or occupants in the vehicle interior at the current position corresponding to the specified occupant occurs. The possibility of doing it can be judged. As a result, it is possible to accurately determine the misplacement of articles and passengers in the passenger compartment that occur according to the customs and preferences of individuals at a certain point.

  Next, an update function related to registration information will be described. When the alert is executed by the alerting means, the control device 10 of this embodiment stores the fact that the alert has been executed and the location where the alert has been executed in association with each other in the registration information. Then, the control device 10 also updates the registration information by recalculating information regarding the possibility of misplacement of articles and passengers in the vehicle interior in consideration of the added registration information. Thereby, the registration information can be kept up-to-date. By reflecting the fact that the article or the occupant is left behind in the registration information, it is possible to accurately obtain the possibility that the article or the occupant is left behind using the registration information.

  The occupant specifying function of this embodiment will be described. The control device 10 specifies an occupant based on the vehicle information. Although not particularly limited, the control device 10 can identify the occupant carrying the unlocking key based on the identifier of the unlocking key acquired by the ignition key 67 of the vehicle. Further, the control device 10 can acquire the vehicle seat position information as vehicle information, and can specify an occupant based on the seat position information. The identification information for identifying the occupant is stored with the fact that the article or the occupant has been left behind. That is, the registration information stores position information and information on the possibility of misplacement of articles and passengers in the passenger compartment for each passenger attribute. In this way, by identifying the occupant, you can identify yourself and others, registration information of the history that you have misplaced the article or occupant, and registration information of the history that the third party has misplaced the article or occupant Can be created.

  Next, the getting-off state determination function of this embodiment will be described. The control device 10 identifies, based on the vehicle information, a first getting-off state in which the occupant is preparing to get off, a second getting-off state in which the occupant is getting off, and a third getting-off state in which the occupant has finished getting off. The vehicle getting-off state is determined. The getting-off state determination function does not have to determine all of the first to third getting-off states, and can determine one or more getting-off states among them. The control device 10 determines the result of the object detection function, the number of passengers determination function, the determination result of the traveling state determination function, the engine operation information acquired from the vehicle controller 50, the shift position information acquired from the shift lever 61, and the door device 62. Acquired door opening / closing information, speed information acquired from speed sensor 63, locking information acquired from locking device 64, seat belt attachment / detachment information acquired from seat belt device 66, ignition on / off information acquired from ignition key 67, occupant specifying device Based on the end information of the occupant identification process acquired from 68, whether the occupant has entered the vehicle and started traveling, whether the occupant has finished traveling, has started preparations for getting off, has been prepared for getting off, has been prepared for getting off, It is determined whether or not is completed and output.

  Although not particularly limited, in the present embodiment, the time zone during which passengers are preparing to get off is in the first disembarkation state, the time zone during which passengers are disembarking is in the second disembarkation state, and the time zone after passengers are disembarking is the third disembarkation Define as state. The first dismounting state is a state in which the vehicle's ignition switch changes from the on state to the off state, and the vehicle is being prepared for dismounting until the driver's door of the vehicle is opened. This is a state in which the driver's door is being unloaded from when the door of the driver's seat is opened until the door is closed, and the third unloading state is a state at the time of locking and after the locking of the vehicle door. The first getting-off state is determined based on the ignition information and the door opening information, the second getting-off information is determined based on the door opening information, and the third getting-off information is determined based on the door locking information. . The definition of each state is not limited, and it is possible to prepare for getting off until the driver's seat door is closed, to complete the getting off from the closing of the driver's seat door to the time of locking, and after that, after getting off.

  Finally, the alerting function of this embodiment will be described. The control device 10 outputs warning information for the detected object in the vehicle interior to the vehicle occupant based on the acquired vehicle information and a determination result relating to the possibility of misplacement of articles or occupants in the vehicle interior To call attention. Specifically, the control device 10 of the present embodiment changes the alerting level at the time of alerting so as to increase the possibility that an article in the passenger compartment or an occupant is left behind at the current position. As a result, alerts can be executed in consideration of the possibility of misplacement of articles and passengers in the passenger compartment, so that the driver will not be bothered at places where misplacement is unlikely to occur. However, at points where misplacement is likely to occur, the alert level can be changed to a high level to fully alert the passenger. That is, without making the driver feel bothersome, it is possible to prevent an occupant or an article from being left behind in the vehicle interior, and to perform appropriate alerting according to the possibility of an article or occupant being left behind in the vehicle interior.

  In the present embodiment, the control device 10 determines whether the first getting-off state in which the occupant is ready to get off (no getting-off is completed) when an article or an occupant is present in the passenger compartment. Perform alerts. On the other hand, when an article or an occupant is present in the passenger compartment when determining the third dismounting state in which the occupant has completed the disembarkation, a second alert is performed that is different from the first alert. In other words, the first warning is issued at the timing when the preparation for getting off is completed (the first getting-off state) before the passenger closes the door, and the passenger gets out of the car and locks after the door is closed to complete the getting-off. The second alert is issued at the timing when the unloading is completed (the third unloading state). The control device 10 of the present embodiment performs alerting twice or more. Note that the second alerting can also be performed in the second getting-off state triggered by the door being closed.

  When the control device 10 of the present embodiment alerts the occupant, it is determined by the determination function that the possibility of misplacement of an article or an occupant in the vehicle interior at the current position is higher than a predetermined first threshold. In such a case, the alerting level for performing the second alerting is changed to be higher than the preset normal alerting level.

  When it is determined that the possibility of misplacement of an article or an occupant in the passenger compartment is higher than the first threshold, the control device 10 according to the present embodiment performs the second alert when performing this alert. Change the level higher than the normal alert level. By raising the second alert level, the passenger's attention can be strongly alerted. The occupant's attention is evoked more strongly than usual by stimulating the occupant's five senses with a method different from the normal time, or by stimulating the occupant's five senses with an output intensity different from the normal time. As described above, it is considered that there is a high possibility that misplacement of an article or an occupant will occur again in a place or situation where the misplacement of the article or an occupant is likely to occur. Thus, by changing the level of the current alert based on the past alert execution history, the alert can be executed at a level corresponding to the tendency of the occupant's behavior.

  Further, in the present embodiment, when alerting is performed based on the “possibility of misplacement of articles and passengers in the passenger compartment” defined in the registration information, there is a predetermined possibility that misplacement of articles and passengers will occur. If it is higher than the first threshold value, the second alert level is changed to a higher value. There is a high possibility that misplacement of articles and passengers in the passenger compartment will occur. This means that there is a high possibility that the passengers to be alerted will either leave things in the passenger compartment or leave the vehicle while leaving other passengers. I can say that. For this reason, in the present embodiment, attention is urged at different alerting levels depending on the points, situations, and histories that are likely to get off while leaving passengers or objects in the passenger compartment. Expecting the effect of raising the caution of the occupant more strongly than the usual caution by changing the second caution level higher when the occupant finishes getting off (third or second getting-off state) Can do. That is, it is possible to effectively prevent the passenger from being alerted by preventing a reduction in the degree of recognition of the passenger due to the familiarity with the alert.

  Although not particularly limited, FIG. 5 shows an example of the alert level. As shown in FIG. 5, the first alert is usually made by displaying the vehicle interior image on the display 31 in the vehicle interior, but there is a high possibility that misplacement of indoor items or passengers will occur. When changing the alert level to a high level, in addition to displaying the interior image on the display 31 in the interior of the vehicle, a voice message for alerting is output from the speaker 32. In addition, as shown in the figure, normally, an intermittent notification sound is output from the speaker 32 and the hazard lamp 34 is turned on three times to perform the second alert, while the indoor article When the warning level is changed to a high level because there is a high possibility that an occupant will be left behind, the hazard lamp 34 is turned on five times in addition to outputting a continuous notification sound from the speaker 32. Thereby, when there is a high possibility of misplacement of indoor items or passengers, it is possible to effectively prevent the passengers from being misplaced.

  Although not particularly limited, FIG. 6 shows an example of output processing defined according to the alert level when performing the first alert activation work. Information to be output and output means are defined for each alert level. The first alert activation work is executed at the timing when the occupant completes preparation for getting off the vehicle, that is, in a state where the occupant is in the vehicle interior, and therefore the alert information is mainly output using an output device in the vehicle interior.

  Although not particularly limited, FIG. 7 shows a definition example of the strength of the first alert activation work that alerts the occupant in the present embodiment. The alerts in this example include display of a monitor image using the display 31, text display using the display 31, output of a beep sound using the speaker 32, vibration operation using the vibration device 33, and sound using the speaker 32. This is done by output. Since the first alert activation work is performed when the occupant is ready to get off, the occupant is alerted mainly using the equipment in the passenger compartment. Means for alerting passengers are not particularly limited. Means for stimulating the five senses of passengers, such as lighting and blinking of lamps in the passenger compartment, ejection of fragrance into the passenger compartment, and blowing of cool air or warm air into the passenger compartment. Can be adopted.

As shown in FIG. 6, in this embodiment, execution / non-execution is defined for each means of monitor video, text display, beep sound, vibration, and audio output for each level. In this embodiment, the degree of output intensity of each means is changed for each level. For example, for text display, the character is enlarged and displayed, the font of the character is changed to an emphasized font, and the color of the character is changed to an emphasized color. The degree of the output intensity is changed by a method such as blinking the character display. About the beep sound, the degree of the output intensity is changed by a technique such as increasing the volume or changing the frequency of the sound. For audio output, increase volume, change audio frequency, change audio tone, change audio type, eg speaker (male / female, adult / children, etc.), message content (caution content) The degree of output intensity is changed by a method such as changing the warning content). By changing the means of alerting and the strength of each means, it is possible to adjust the strength of alerting the occupant comprehensively.
In addition, the combination of each alerting level, its output information, and output mode is not limited to what is shown in FIG. 6, and can be edited suitably.

  Although it does not specifically limit, the example of a definition of the intensity | strength of the 2nd alerting start work which alerts a passenger | crew in this embodiment is shown in FIG. The alert in this example includes display of a monitor image using the display 31, text display using the display 31, lighting display using a lamp 35 such as a hazard lamp 34, a blinker lamp, or a headlight, and the outside of the vehicle including the horn. This is performed by outputting an alarm using the speaker 36. Means for alerting the occupant is not particularly limited, and means such as lighting / flashing of a lamp in the vehicle interior and lighting of the display 31 that can be confirmed from outside the vehicle can be employed. The second alert activation work is performed when the occupant completes getting off, such as when the door is closed or when the door is locked, that is, when the occupant is outside the passenger compartment, so check mainly from the equipment outside the passenger compartment or outside the passenger compartment. Raise occupant's attention using possible means.

  As shown in FIG. 7, in this embodiment, execution / non-execution is performed for each means of the lighting 35 such as notification sound, monitor display, hazard lamp lighting, front lamp and hazard lamp lighting, blinker lamp lighting, etc. for each level. Define In this embodiment, the degree of output intensity of each means is changed for each level. For example, for the notification sound, the hazard lighting cycle, which is intermittent notification sound or continuous notification sound, is changed. By changing the means of alerting and the strength of each means, it is possible to adjust the strength of alerting the occupant comprehensively. In addition, the combination of each alerting level, its output information, and output mode is not limited to what is shown in FIG. 7, and can be edited suitably.

  In addition, the control device 10 according to the present embodiment is less likely to cause misplacement of articles or passengers in the vehicle interior at the current position, or when the current position is not included in the registration information. The first alert is not executed. The second threshold may be the same value as the first threshold described above, or may be a different value. In this way, in situations where the possibility of misplacement of articles or passengers in the vehicle interior at the current position is relatively low, the first alert is not executed and the passengers are not likely to leave misplaced. It can be made not to feel annoyance. The non-execution of the first alert may be activated by a passenger input.

  As described above, the vehicle interior monitoring apparatus 100 according to the present embodiment changes the alerting method according to the possibility of misplacement of articles or passengers in the vehicle interior, and may cause misplacement of articles or passengers in the vehicle interior. When the value is low, the alerting is not executed, so that the alerting can be controlled according to the situation. Thereby, it is possible to achieve both the execution of alerting in a necessary scene and the reduction of troublesomeness in an unnecessary scene.

  Next, the monitoring control procedure of the passenger compartment monitoring system 1 of the present embodiment will be described based on the flowchart of FIG. FIG. 8 shows a process of performing alerting when an occupant in the vehicle compartment is left behind. The processing content shown in FIG. 8 is continuously performed at a predetermined cycle.

  As shown in FIG. 4, in step S1, initialization processing for initial setting of timers, counters, and flags used for subsequent processing is executed.

  In step S2, vehicle information such as a lock signal, an unlock signal, a shift position signal, and a vehicle speed signal, which are vehicle information, is acquired, and the current vehicle state is determined based on the acquired vehicle information. The determination of the vehicle state includes determination of the traveling state of the vehicle, the determination of the occupant's dismounting state, and the determination of any of the first to third dismounting states. In this determination, it is determined that the vehicle is running when the shift position is other than the parking position and the vehicle speed signal is not 0 km / h after unlocking based on the unlocking signal. Similarly, when the vehicle speed signal is 0km / h and the shift position is in the parking position, preparations for getting off are made. After that, when the seat belt release signal and the door opening signal are obtained, the passenger gets ready to get off. Judged as a state. Then, it determines with the state in which the door was closed being a 2nd getting-off state. Furthermore, if the vehicle is locked and a locking signal is acquired, it is determined that the vehicle is in the third getting-off state where the passenger has got off.

  If it is determined in step S3 that the vehicle is not traveling based on the vehicle state including the dismounting state determined in step S2, the process proceeds to step S6. On the other hand, if it is determined that the vehicle is traveling, the process proceeds to step S4, the image captured by the camera 20 is captured as a reference image, and stored in the memory as a reference image used for subsequent processing in step S5. An example of the reference image stored in step S5 is shown in FIG. Since the reference image is captured when the vehicle is unlocked, the reference image does not include an occupant.

  In step S6, a branching process is performed based on the vehicle state determined in step S2. Here, it is determined in step S3 that the vehicle is not unlocked, and if it is determined in step S6 that the vehicle state is the start of travel, the process proceeds to the next step S7, and if it is not the start of travel, step S13 is performed. Proceed to

  In step S7, the captured image of the camera 20 is taken into the processing area as a comparison image, and the process proceeds to the next step. An example of the comparative image acquired here is shown in FIG. FIG. 9B shows a case where there is no passenger, and FIG. 9C shows a case where there is a passenger. In step S8, the comparison image captured in step S7 is compared with the reference image stored in step S5. The method of comparison is not particularly limited. For example, the density difference for each pixel of the comparison image and the reference image is calculated in the processing region, and the number of pixels having a density difference equal to or greater than a predetermined value is counted.

  Next, in step S9, branch processing is performed based on the processing result of step S8. If the count value (that is, the area in the image) that is the difference between the comparison image calculated in step S8 and the reference image is equal to or greater than a predetermined value TH_E (for example, 1/8 of the entire image), there is a difference from the reference image. That is, it is determined that there is an occupant, and the process proceeds to step S10. If it is smaller than the predetermined value TH_E, it is determined that there is no difference from the reference image, that is, there is no occupant, and the process proceeds to step S12. In step S12, since it is determined that there is no occupant other than the driver, the occupant flag is set to 0 for the subsequent processing, and the process returns to step S2 to determine the vehicle state. This corresponds to detection processing of an object present in the passenger compartment.

  In step S10, since it is determined that there is an occupant other than the driver, 1 is set in the occupant flag for the subsequent processing, and the process proceeds to the next step. In step S11, the registration information (see FIG. 4) stored in advance in the server 300 in which the positional information is associated with information related to the possibility of misplacement of articles and passengers in the passenger compartment is read, and the vehicle state is returned to step S2. Perform the determination.

  In step S13, branch processing is performed based on the vehicle state determined in step S2. Here, it is determined in step S6 that the vehicle state is not running, and if it is determined in step S2 that the vehicle state is preparing for getting off, the process proceeds to the next step S14. Proceed to step S23.

  In step 14, branch processing is performed based on the occupant flag set in step S10 or step S12. If the occupant flag is 1, that is, if there is an occupant other than the driver, the process proceeds to the next step S15. On the other hand, when the occupant flag is 0, that is, when there is no occupant other than the driver, it is determined that there is no need to call attention to the misplacement of the article or occupant, and the entire process is terminated.

  In step S15, the image captured by the camera 20 is taken into the processing area as a reference image, and the process proceeds to the next step. At this time, depending on the installation position of the camera 20 in the vehicle interior, it is possible to take the passenger seat and the rear seat as one image and determine whether a passenger other than the driver for each seat is left behind, Processing in the case where determination is made only for the rear seats that are likely to be left behind will be described. In this process, as shown in FIG. 10, among the images of the rear seat taken by the camera 20 installed on the ceiling of the passenger compartment as shown in FIG. 2, the seat on the left or right side of the rear seat is shown in FIG. Although an example of processing will be described, the same processing can be performed when processing is performed for both the left and right seats of the rear seat.

  In step S16, the image captured in the processing area in step 15 is stored in the memory as a reference image for subsequent processing, and the process proceeds to the next step S16. An example of the reference image stored here is shown in FIG. Since it is determined that there is an occupant other than the driver in the processing so far, the reference image is in a state where there is an occupant.

  In step S17, the registration information read in step S11 is referred to to determine the possibility of the object being left behind. The registration information in this example is registration information in which position information is associated in advance with information on a facility or the like that is likely to be misplaced by an article in the passenger compartment or an occupant. Next, in step S18, branch processing is performed based on the registration information referred to in step S17. If the referenced registration information includes position information corresponding to the current position of the vehicle, the current position has a high frequency of occurrence of misplacement of objects in the passenger compartment, and this time there is a high possibility that misplacement of objects will occur. If not, the process proceeds to the next step. Otherwise, the process proceeds to step S20.

  In step S19, it is determined that there is an occupant other than the driver in the processing so far, and it is determined in step S18 that the current location is a location where the frequency of misplacement is high. As shown in FIG. 5, an image of the vehicle interior camera 20 is displayed on the display 31 and a flag for outputting a voice message to the speaker 32 in the vehicle interior is set. Proceed to step S22.

  In step S20, a branch process is performed depending on whether or not the execution of the first alert function is canceled by an input operation of the driver. Here, in order to reduce the troublesomeness of alerting the driver, if the frequency of misplacement is not high, canceling the first alert function is given priority to the occupant's intention. If the first alert function is not cancelled, the process proceeds to the next step S21, and if cancelled, it is not necessary to execute the first alert, so the process returns to step S2 to determine the vehicle state.

  Next, in step S21, since it is determined that there is an occupant other than the driver in the process so far, in order to alert the driver that there is a passenger, as shown in FIG. A flag for displaying an image on the display 31 is set, and the process proceeds to step S22.

  In step S22, based on the alert mode set in step S19 or step S21, a first alert to the effect that there is a passenger is executed to the driver, and the process returns to step S2 to determine the vehicle state. To do.

  In step S23, a branching process is performed based on the vehicle state determined in step S2. Here, if it is determined in step S6 that the vehicle state is not running, and it is determined in step S13 that the vehicle is not getting off, and it is determined in step S2 that the vehicle is locked, the next step Proceed to S24. If the vehicle is not locked, the process returns to step S2 to determine the vehicle state. The locked state includes a second unloading state in which the door is closed before locking and a third unloading state in which the door is locked after closing.

  Next, in step S24, similarly to step S14, branch processing is performed based on the occupant flag set in step S10 or step S12. If the occupant flag is 1, that is, if there is an occupant other than the driver, the process proceeds to the next step S25, and if the occupant flag is 0, that is, if there is no occupant other than the driver, it is determined that the second alert is not necessary. The entire process ends.

  In step S25, the image captured by the vehicle interior camera 20 is captured as a comparative image in the processing area, and the process proceeds to the next step. An example of the comparative image acquired here is shown in FIG. FIG. 10B shows a case where there is no passenger, and FIG. 5C shows a case where there is a passenger.

  In step S26, the comparison image captured in step S25 is compared with the reference image stored in step S16. The comparison method is not particularly limited. For example, the density difference for each pixel of the comparison image and the reference image is calculated in the processing region, and the number of pixels having a density difference equal to or greater than a predetermined value is counted.

  In step S27, branch processing is performed based on the processing result of step S26. If the count value (that is, the area in the image) that is the difference between the comparison image calculated in step S26 and the reference image is equal to or less than a predetermined value TH_R (for example, 1/8 of the entire image), there is no difference from the reference image. That is, it is determined that there is a passenger and the process proceeds to the next step S28. On the other hand, if the count value (that is, the area in the image) that is the difference between the comparison image calculated in step S26 and the reference image is larger than the predetermined value TH_R, it is determined that there is a difference from the reference image, that is, there is no occupant. . If no passenger is present, it is determined that there is no need to call attention to misplacement, and the entire process is terminated.

  In step S28, since it is determined in step S27 that passengers other than the driver remain in the passenger compartment, the registration information of the server 300 is updated using the current position information, and the updated registration information is read. Proceed to the next Step S29.

  Finally, in step S29, since it is determined in step S27 that passengers other than the driver remain in the passenger compartment, the speaker 36 outside the passenger compartment including the horn and the like, the hazard lamp 34, or the lighting as a second alert 35 is used to notify the driver outside the passenger compartment that the luggage has been misplaced, the driver's attention is drawn, and the entire process is terminated. In this embodiment, the alert is issued and the process is terminated immediately, but an input signal from the driver that the misplacement of the luggage in the passenger compartment is resolved or that the driver is informing the luggage with the intention. You may continue performing alerting until it can confirm by (for example, again locking operation).

  In the present embodiment, when notifying the passenger of misplacement from the captured image of the camera 20, the alerting method is changed according to the possibility of misplacement, and it is troublesome if the possibility of misplacement is low. By not performing the alerting, the alerting can be performed in a manner that attracts the occupant's attention according to the situation while reducing the troublesomeness caused by unnecessary alerting.

  According to the vehicle interior monitoring apparatus 100 of the present embodiment, the following effects are obtained.

  [1] According to the vehicle interior monitoring apparatus 100 of the present embodiment, based on the acquired vehicle information and the determination result relating to the possibility of misplacement of articles or passengers in the vehicle interior, The higher the possibility of misplacement of articles and passengers, the higher the level of alerting at the time of alerting, so alerting in a manner that takes into account the possibility of misplacement of articles and passengers in the passenger compartment Can be executed. In other words, while avoiding annoyance to the driver at points where misplacement is unlikely to occur, change the alert level to a high level at points where misplacement is likely to occur, so that the passenger's attention is sufficient Can be aroused. As a result, without causing the driver to feel annoyance, it is possible to prevent the passengers, articles, and passengers from being misplaced in the passenger compartment, and to perform appropriate alerts according to the possibility of misplacement of articles or passengers in the passenger compartment. be able to.

  [2] According to the vehicle interior monitoring apparatus 100 of the present embodiment, when the “possibility of misplacement of articles or passengers in the vehicle interior” defined in the registration information is higher than the first threshold, the second Change the alert level for alerts higher than normal. There is a high possibility that misplacement of articles and passengers in the passenger compartment will occur. This means that there is a high possibility that the passengers to be alerted will either leave things in the passenger compartment or leave the vehicle while leaving other passengers. I can say that. For this reason, in the present embodiment, attention is urged at different alerting levels depending on the points, situations, and histories that are likely to get off while leaving passengers or objects in the passenger compartment. By changing the alerting level, it is possible to expect an effect that can alert the passenger more strongly than the usual alerting. That is, it is possible to effectively prevent the passenger from being alerted by preventing a reduction in the degree of recognition of the passenger due to the familiarity with the alert.

  [3] According to the vehicle interior monitoring apparatus 100 of the present embodiment, the possibility that an article or an occupant is left behind at the current position is lower than a predetermined second threshold or the current position is included in the registration information. If not, the first alert is not executed. In situations where the possibility of misplacement of articles or passengers in the vehicle interior at the current position is relatively low, the first alert is not executed, and the passengers feel annoyed in situations where there is a high possibility that they will not be misplaced. You can prevent alerts from being executed.

  [4] According to the vehicle interior monitoring apparatus 100 of the present embodiment, when alerting is executed, the fact that the alerting is executed and the location where the alerting is executed are associated with the registration information. Since it is stored and updated, the registration information can be kept up-to-date. By reflecting the fact that the article or the occupant is left behind in the registration information, it is possible to accurately obtain the possibility that the article or the occupant is left behind using the registration information.

  [5] According to the vehicle interior monitoring apparatus 100 of the present embodiment, by performing the process of specifying the occupant, the registration information of the specified occupant is referred to, and the article in the vehicle interior or the occupant is left behind at the current position. The possibility of occurrence can be determined. As a result, it is possible to accurately determine the misplacement of articles or passengers in the passenger compartment that occur according to personal habits and preferences.

  The embodiment described above is described for facilitating the understanding of the present invention, and is not described for limiting the present invention. Therefore, each element disclosed in the above embodiment is intended to include all design changes and equivalents belonging to the technical scope of the present invention.

  That is, in this specification, the vehicle interior monitoring system 1 including the vehicle interior monitoring device 100, the in-vehicle device 200, and the server 300 according to the present invention will be described as an example, but the present invention is not limited to this. .

  In this specification, as one aspect of the vehicle interior monitoring device according to the present invention, the control device 10, the camera 20, the display 31, the speaker 32, the vibration device 33, the hazard lamp 34, the lighting 35, and the outside of the vehicle. Although the vehicle interior monitoring apparatus 100 provided with the speaker 36 is demonstrated as an example, it is not limited to this.

  In the present specification, as an example of a vehicle interior monitoring device according to the present invention that includes an image acquisition unit, a vehicle information acquisition unit, an object detection unit, a determination unit, and a warning unit, an image acquisition function, a vehicle The vehicle interior monitoring device 100 including the control device 10 that executes the information acquisition function, the object detection function, the determination function, and the alerting function will be described as an example, but is not limited thereto.

  In this specification, in addition to the above-described means, as an example of a vehicle interior monitoring device according to the present invention further comprising a getting-off state judging means, an updating means, and an occupant specifying means, a getting-off state judging function and an updating function The vehicle interior monitoring device 100 having the control device 10 that executes the occupant specifying function will be described as an example, but is not limited thereto.

DESCRIPTION OF SYMBOLS 1 ... Vehicle interior monitoring system 100 ... Vehicle interior monitoring apparatus 10 ... Control apparatus 11 ... CPU
12 ... ROM
13 ... RAM
DESCRIPTION OF SYMBOLS 20 ... Camera 31 ... Display 32 ... Speaker 33 ... Vibration apparatus 34 ... Hazard lamp 35 ... Illumination 36 ... Speaker 200 ... In-vehicle device 50 ... Vehicle controller 60 ... Various equipment 61 ... Shift lever 62 ... Door device 63 ... Vehicle speed sensor 64 ... Locking device 65 ... Clock 66 ... Seat belt device 67 ... Ignition key 68 ... Occupant identification device 70 ... Navigation device 300 ... Server 310 ... Registration information 311 ... Map information 312 ... Notification execution history

Claims (5)

Image acquisition means for acquiring a captured image from a camera that images the interior of the vehicle;
Vehicle information acquisition means for acquiring vehicle information including a current position detected in the vehicle;
Object detection means for detecting an object present in the passenger compartment based on the captured image;
Judgment means for referring to registration information in which position information and information related to the possibility of misplacement of an object in the vehicle interior are associated in advance, and determining the possibility of misplacement of the object in the vehicle interior at the current position; ,
Based on the acquired vehicle information and a determination result relating to the possibility of misplacement of an object in the vehicle interior, warning information for the detected object in the vehicle interior is output to a passenger of the vehicle. An alerting means for performing alerting,
The vehicle is characterized in that the alerting means changes the alerting level at the time of alerting higher as the possibility that the object in the vehicle interior is left behind at the current position is higher by the determining means. Indoor monitoring device. Based on the vehicle information, a first getting-off state in which the occupant is ready to get off and a second getting-off state in which the occupant has finished getting off are identified, and a getting-off state judging means for judging the getting-off state of the vehicle, respectively. Further comprising
When the first alighting state is identified, the alerting means executes a first alerting, and when the second alighting state is identified, a second alerting that is different from the first alerting is performed. Run,
When the determination means determines that the possibility that an object in the vehicle interior is left behind at the current position is higher than a predetermined first threshold when calling the occupant's attention, the second 2. The vehicle interior monitoring device according to claim 1, wherein a warning level at the time of warning is changed to be higher than a preset normal warning level.   The warning means is when the possibility that the object in the vehicle interior is left behind at the current position is lower than a predetermined second threshold or the current position is not included in the registration information. The vehicle interior monitoring device according to claim 1 or 2, wherein the first alerting is not executed.   When alerting is executed by the alerting means, the fact that the alert has been executed and the place where the alert has been executed are stored in the registration information in association with each other, and the object in the passenger compartment The vehicle interior monitoring device according to any one of claims 1 to 3, further comprising an update unit that recalculates information on a possibility of misplacement and updates the registration information. Further comprising occupant specifying means for specifying the occupant based on the vehicle information;
The registration information stores, for each occupant, position information and information related to the possibility of misplacement of objects in the passenger compartment,
The determination means refers to registration information corresponding to the specified occupant, and determines a possibility that an object is left behind in the vehicle interior at the current position according to the occupant. The vehicle interior monitoring device according to any one of 1 to 4.

Images