JP2019077330A - Monitoring device, monitoring system, monitoring method and monitoring program - Google Patents

Abstract

To provide a monitoring device which can perform notification according to an external situation of an automobile when an operator exists at a place separated from the automobile, a monitoring system, a monitoring method and a monitoring program.SOLUTION: A monitoring device 100, which monitors the exterior of an automobile, has: image data acquisition means which acquires image data of an object; object recognition means which recognizes a monitoring object among objects included in the image data; notification condition determination means which determines whether a notification condition for notification to the exterior is satisfied or not when the object recognition means recognizes the monitoring object; and notification means for notification to the exterior when notification condition is satisfied.SELECTED DRAWING: Figure 3

Description

  The present invention relates to a monitoring device, a monitoring system, a monitoring method, and a monitoring program.

  2. Description of the Related Art Conventionally, a technology has been proposed in which a camera is mounted on a car to monitor the outside (for example, Patent Document 1).

Patent No. 4161225

  By the way, the above-described technology is effective when the driver is in a car. However, while the car is parked, it is not possible to prevent the car from being harmed when the driver is away from the car.

  The present invention is an attempt to solve such a problem, and when the driver is away from the vehicle, a monitoring device, a monitoring system, a monitoring method, and a notification that can be notified according to the external situation of the vehicle. It aims to provide a monitoring program.

  A first invention is a monitoring device for monitoring the outside of a car, comprising: image data acquisition means for acquiring image data of an object; and object recognition means for recognizing a monitoring target among the objects included in the image data And, when the object recognition unit recognizes the monitoring target, a notification condition determination unit that determines whether a notification condition for notifying an outside condition is satisfied, and that the notification condition is satisfied by a predetermined notification mode And a notification means for notifying to.

  According to the configuration of the first aspect of the present invention, the monitoring device can acquire image data by the image data acquisition unit, and can recognize the monitoring target among the objects included in the image data by the object recognition unit. Then, the notification condition judging means can judge whether or not the notification condition is satisfied, and the notification means can notify the outside. Thus, when the driver is away from the vehicle, it is possible to notify in accordance with the external situation of the vehicle.

  A second invention is an unmanned surveillance system according to the first invention, wherein the object recognition means refers to data generated by deep learning.

  A third invention according to the first invention or the second invention further comprises distance estimation means for estimating the distance between the monitoring target and the vehicle, and the notification condition includes the monitoring It is a monitoring apparatus whose object is located within predetermined distance from the said motor vehicle.

  A fourth invention is a monitoring device according to the configuration of the third invention, comprising notification mode determination means for determining a notification mode for notifying outside based on a distance between the monitoring target and the vehicle.

  According to a fifth invention, in the configuration of the first invention or the second invention, the fifth aspect of the invention has an approach judging means for judging whether the monitoring target is approaching the car, and the notification condition is the monitoring target Is a monitoring device that is approaching the car.

  A sixth invention according to the first invention or the second invention has speed change recognition means for recognizing a change in speed at which the monitoring target approaches the vehicle, and the notification condition is that the speed is It is a monitoring device that is faster.

  A seventh invention is the configuration according to the first invention or the second invention, wherein position information indicating the position of the vehicle is provided when the notification means notifies outside by a predetermined notification mode that the notification condition is satisfied. , And tracking means for tracking the vehicle.

  An eighth aspect of the present invention is a monitoring system for monitoring the outside of a car, comprising: image data acquisition means for acquiring image data of an object photographed by the photographing means; and a monitoring target among the objects included in the image data. The object recognition means to be recognized, the notification condition judgment means for judging whether the notification condition for notifying to the outside is satisfied when the object recognition means recognizes the monitoring target, and the condition that the notification condition is satisfied And a notification unit for notifying the outside according to the notification mode of

  A ninth invention is an image data acquisition step of acquiring image data of an object, and an object recognition step of recognizing an object to be monitored among the objects included in the image data. When the monitoring target is recognized in the object recognition step, a notification condition determination step of determining whether a notification condition for notification to the outside is satisfied and notification that the notification condition is satisfied are notified to the outside according to a predetermined notification mode And a notifying step.

  A tenth aspect of the invention relates to a control device that manages a monitoring device that monitors the outside of a vehicle, an image data acquisition unit that acquires image data of an object, and an object that recognizes a monitoring target among the objects included in the image data A recognition unit, a notification condition judging unit that judges whether a notification condition for notifying outside is satisfied when the object recognition unit recognizes the monitoring target, and a predetermined notification aspect that the notification condition is satisfied Is a monitoring program for functioning as notification means for notifying the outside by means of

  As described above, according to the present invention, when the driver is away from the car, it is possible to notify in accordance with the external situation of the car.

FIG. 1 shows an overview of an embodiment of the present invention. It is the schematic which shows the functional block of an imaging device. It is the schematic which shows the functional block of a monitoring apparatus. It is a conceptual diagram for demonstrating the feature data for image recognition. It is a figure for demonstrating notification conditions. It is a figure for demonstrating notification conditions. It is a conceptual diagram which shows notification mode data. It is a figure for demonstrating a tracking program. It is a flowchart which shows operation | movement of a monitoring apparatus etc.

  Embodiments of the present invention will be described with reference to the drawings. The description of the configurations that can be appropriately implemented by those skilled in the art will be omitted, and only the basic configuration of the present invention will be described.

  As shown in FIG. 1, the monitoring device 100 is stored in the coat pocket of the driver 350 who has left the automobile 300. The monitoring device 100 is, for example, a mobile phone (including a smart phone). The driver 350 is about to park the car 300 in the parking space on the shoulder of the road R1 and go out for a meal with the woman 352. Near the inner upper portion of the windshield of the automobile 300, a photographing device 10 is disposed. The camera (not shown) which comprises the imaging device 10 is an example of an imaging means. The imaging device 10 is, for example, a drive recorder.

  The monitoring device 100 is configured to obtain (receive) image data of an image captured by the imaging device 10 from the imaging device 10 when the driver 350 leaves the vehicle 300.

  When the monitoring target is included in the image data received from the imaging device 10, the monitoring device 100 recognizes the monitoring target by image recognition. The monitoring target is, for example, a person in a predetermined clothes, and corresponds to humans 360 and 362. Humans 360 and 362 wear red jackets 360a and 362a, respectively. The monitoring apparatus 100 recognizes the monitoring target in the image data, and when the notification condition is satisfied, notifies the driver 350 of the satisfaction of the notification condition by sound or vibration. Furthermore, when the predetermined condition is satisfied, the monitoring device 100 requests the imaging device 10 for position information indicating the position of the imaging device 10. The position of the photographing device 10 is also the position of the car 300. Thus, when the vehicle 300 is moved by a person other than the driver 350, the monitoring device 100 can track the vehicle 300 by receiving position information from the imaging device 10.

  As shown in FIG. 2, the imaging device 10 includes a central processing unit (CPU) 50, a storage unit 52, a wireless communication unit 54, a GPS (global positioning system) unit 56, an image processing unit 60, and a power supply unit 70. .

  The imaging device 10 can operate the camera (not shown) that is the central configuration of the imaging device 10 by the image processing unit 60 to acquire an external image.

  The imaging device 10 can transmit image data to the monitoring device 100 by the wireless communication unit 54.

  The imaging device 10 can measure the position of the imaging device 10 itself by the GPS unit 56. The GPS unit 56 basically receives radio waves from four or more GPS satellites (navigation satellites) to measure the position of the imaging device 10.

  The power supply unit 70 is connected to the power supply of the automobile 300 and receives power supply from the automobile 300.

  The storage unit 52 stores a stop shooting program, an image data transmission program, and a position information transmission program, in addition to a general program as a drive recorder. The CPU 50 and the at-stop imaging program are an example of the at-stop imaging means. The CPU 50 and the image data transmission program are an example of an image transmission unit. The CPU 50 and the position information transmission program are an example of position information transmission means.

  The photographing device 10 photographs the outside while the car 300 is stopped by the photographing program at the time of stop. The photographing device 10 continuously photographs the outside while the car 300 is traveling, but even when the car 300 is stopped, the switch of the camera of the photographing device 10 is operated by the driver 350 who is the user. The external shooting is continued unless the shooting interruption process is performed, for example, the camera is turned off.

  The imaging device 10 is configured to continuously transmit image data of an image captured while the vehicle is stopped to the monitoring device 100 by an image data transmission program.

  The imaging device 10 is configured to continuously transmit the position of the imaging device 10 to the monitoring device 100 when the predetermined condition is satisfied by the position information transmission program. The predetermined condition is that a transmission request for position information has been received from the monitoring device 100.

  As shown in FIG. 3, the monitoring device 100 includes a CPU 150, a storage unit 152, a wireless communication unit 154, a GPS unit 156, and a power supply unit 170.

  The monitoring device 100 can receive image data from the imaging device 10 by the wireless communication unit 154. Further, it is possible to transmit a transmission request for position information to the imaging device 10 and receive the position information.

  The monitoring device 100 can measure the position of the monitoring device 100 itself by the GPS unit 156.

  The power supply unit 170 is a chargeable rechargeable battery, and supplies power to each unit of the monitoring device 100.

  The storage unit 152 includes a general program as a mobile phone, an image data acquisition program, an object recognition program, a distance estimation program, an approach determination program, a speed change recognition program, a notification condition determination program, a notification mode determination program, notification A program and a tracking program are stored. The CPU 150 and the image data acquisition program are an example of an image data acquisition unit. The CPU 150 and the object recognition program are examples of object recognition means. The CPU 150 and the distance estimation program are examples of distance estimation means. The CPU 150 and the approach determination program are an example of the approach determination means. The CPU 150 and the speed change recognition program are an example of the speed change recognition means. The CPU 150 and the notification condition determination program are an example of notification condition determination means. The CPU 150 and the notification mode determination program are an example of the notification mode determination means. The CPU 150 and the notification program are an example of notification means. The CPU 150 and the tracking program are an example of the tracking means.

  The monitoring device 100 acquires image data from the imaging device 10 according to an image data acquisition program.

  The monitoring apparatus 100 recognizes an object included in the image data by an object recognition program, and recognizes a monitoring target. In the present embodiment, it is assumed that the monitoring target is a man wearing a red jacket.

  The monitoring apparatus 100 refers to the feature data generated by deep learning by the object recognition program. Here, deep learning is machine learning of a multi-layered neural network, and the field of image recognition is a powerful application field.

  The monitoring device 100 identifies a feature of an object included in image data acquired from the imaging device 10 based on a large number of image recognition feature data (hereinafter also referred to as “feature information”) for each category, It is possible to recognize the category of The feature information may be stored in the storage unit 152, or may be stored in an external storage device so that the monitoring apparatus 100 can access the external storage device. This category of recognition is called general object recognition. As shown in the conceptual diagram of the image recognition feature data in FIG. 4, the categories are, for example, “human”, “dog”, “cat” and the like. Each category is subdivided, and there is a subcategory 1 such as "male" or "female" in the "human" category, and a "jacket" or "jacket" in the subcategory 1 "male". There is a subcategory 2 such as "pants" and a subcategory 3 such as "red" and "blue" in the "jacket" of the subcategory 2. A large number of feature data are stored in the storage unit 152 for each of the categories described above. The monitoring apparatus 100 can refer to feature information of “human” acquired based on, for example, a number of image examples of “human”.

  In general object recognition, for image data, for example, a large number of features such as contours and directions of individual configurations are extracted, and the correlation (degree of correlation) is determined in comparison with the features of each category acquired by deep learning. The higher the degree of correlation, the more likely an object in the acquired image belongs to a particular category. For example, when the degree of correlation is 0, the possibility of belonging to a specific category (hereinafter referred to as "category common probability") is 0, and when the degree of correlation indicates the maximum value, the category common probability is 100%. It is defined as The monitoring apparatus 100 determines that the category of the object in the acquired image belongs to a specific category when the category common probability is a predetermined reference value, for example, 95% or more.

  When the acquired image includes a man wearing a red jacket, the monitoring apparatus 100 refers to the feature data by the general object recognition described above, and is “human” and “male”, The "jacket" is worn, and the jacket is recognized as "red", and the object in the image is recognized as a monitoring target.

  The monitoring apparatus 100 estimates the distance between the monitoring target and the vehicle 300 by means of a distance estimation program. The number of pixels of the imaging device 10 is, for example, 3 million. The number of pixels of the photographing device 10 and the number of pixels of the monitoring device 100 do not necessarily have to be the same. However, in the present embodiment, it is assumed that the total number of images of the monitoring device 100 is the same as the total number of images of the photographing device 10.

  As shown in FIG. 5A, the monitoring target occupies a predetermined number of pixels in the image data according to the size thereof. As the number of pixels occupied by the monitoring target is larger, the distance to the car 300 is shorter. For example, the monitoring apparatus 100 may be configured such that the number of pixels in the height direction when the 1 m (bar) height bar is 500 m away, the number of pixels when 200 m away, and 100 m away. It has basic reference values such as the number of pixels at one time, the number of pixels at a position separated by 50 mails, and the number of pixels at a position separated by 10 meters. Also, for each category, for example, in the case of a "man" "human", it has a category reference value that the average height is 170 cm (centimeter). The monitoring apparatus 100 is configured to estimate the distance between the monitoring target and the vehicle 300 based on the number of pixels occupied by the actual monitoring target with reference to the basic reference value and the category reference value. For example, in the case of a "man" "human", the average height is 170 cm, so the number of pixels in the height direction at a predetermined distance is 1.7 times the basic reference value A based on a height of 1 m. Calculate as a number (A × 1.7). In the present embodiment, since the estimated distance does not need to be accurate and a simple configuration is desirable, the distance is estimated based on the number of pixels in the image.

  The monitoring apparatus 100 determines whether the monitoring target approaches the vehicle 300 according to the approach determination program. For example, as shown in FIG. 6, the number of pixels occupied by the monitoring target at time t2 when time Δt has passed from time t1 than the number of pixels occupied by the monitoring target at time t1 (see FIG. b) When there are many cases), the monitoring apparatus 100 determines that the monitoring target is approaching the car 300.

  The monitoring apparatus 100 recognizes a change in speed at which the monitoring target approaches the vehicle 300 by the speed change recognition program. For example, the monitoring apparatus 100 determines that the increase δB1 in the number of pixels occupied by the monitoring target between time t1 and time t2 when time Δt has elapsed from time t1, which is the time zone thereafter, If the increase in the number of pixels occupied by the monitoring target during the period is smaller than .delta.B2 (.delta.B1 <.delta.B2), it is determined that the monitoring target is accelerating.

  The monitoring device 100 determines whether to notify the driver 350 who is the user of the monitoring device 100 of the establishment of the notification condition by the notification condition determination program. For example, as shown in FIG. 5A, when one monitoring target is recognized in the image data, as the notification condition, as shown in FIG. 5B, two monitoring targets are displayed in the image data. When it is recognized, or as shown in FIGS. 6 (a) and 6 (b), the monitored object approaches the vehicle 300, or the monitored object approaches the vehicle 300 at a high speed. is there. The notification conditions can be selected and set in advance by the driver 350 who is the user of the monitoring apparatus 100.

  The monitoring apparatus 100 determines a notification mode for notifying the outside (driver 350) based on the distance between the monitoring target and the vehicle 300 by the notification mode determination program. As shown in FIG. 7, the monitoring apparatus 100 refers to the notification mode data, and determines the notification mode according to the distance between the monitoring target and the vehicle 300. The notification mode data is stored in the storage unit 152. The notification mode A in the case of a distance of 200 m to 500 m is, for example, a relatively small sound (or low frequency vibration or low sound), and the notification mode B of a distance 100 m to 200 m is a sound (or It is specified that the louder the sound (or the vibration with a high frequency, the sound with a high frequency) the greater the distance between the automobile 300 and the monitoring target, such as vibration with a large frequency or sound with a high frequency). ing.

  The monitoring apparatus 100 notifies the outside (driver 350) of the notification mode determined by the notification mode determination program by the notification program.

  When the monitoring device 100 notifies the outside (driver 350) of the establishment of the notification condition, the tracking program acquires position information from the imaging device 10 and tracks the position of the automobile 300 by the tracking program. That is, when the monitoring device 100 notifies the outside of establishment of the notification condition, the monitoring device 100 transmits a transmission request for position information to the imaging device 10, and continuously receives the position information from the imaging device 10. Note that the monitoring apparatus 100 may not always perform tracking after notification to the outside, and may be performed when a predetermined condition is satisfied. The predetermined condition is, for example, that the driver 350 has input an instruction to start tracking to the monitoring device 100. When receiving the position information from the photographing device 10, the monitoring device 100 displays the position of the photographing device 10 (the position of the car 300) on the display screen 100a, as shown in FIG. In the example of FIG. 8, it is shown that the display screen 100 a has passed the route R1 from the first position P1 of the imaging device 10 and has reached the current position P2. That is, since the monitoring device 100 can track the position of the photographing device 10 (the position of the car 300) when the notification condition is satisfied, it can track even if the automated worker 300 is stolen.

  Hereinafter, an operation example of the monitoring device 100 and the photographing device 10 will be described with reference to the flowchart of FIG. The imaging device 10 acquires an image (step ST1), and transmits image data (step ST2), the monitoring device 100 receives the image data (step ST3). The monitoring apparatus 100 performs image recognition (step ST4), determines whether or not the notification condition is satisfied (step ST5), determines the notification mode when the notification condition is satisfied (step ST6), and notifies (step ST6) Step ST7). Subsequently, it is determined whether or not a predetermined condition (tracking condition) is satisfied (step ST8), and when the tracking condition is satisfied, the position information is requested to the imaging device 10 (step ST9). Are received (step ST10), and the imaging device 10 (car 300) is tracked (step ST11).

  In the above embodiment, the imaging device 10 acquires an image, and the monitoring device 100 is described as receiving image data from the imaging device 10 and performing processing after image recognition, but the driver 350 is notified. Processing other than the above may be performed by the imaging device 10. In this case, the imaging device 10 transmits information indicating that the notification condition is satisfied to the monitoring device 100, and the monitoring device 100 notifies the outside, and the imaging device 10 and the monitoring device 100 are integrated as a monitoring device. Or, configure a monitoring system. Alternatively, part of the processing other than the notification to the driver 350 may be performed by the imaging device 10. Also in this case, the imaging device 10 and the monitoring device 100 integrally constitute a monitoring device or a monitoring system. Moreover, the imaging device 10 is not limited to the drive recorder, and may be a mobile phone or a smartphone as long as the imaging device 10 can capture an image.

  The monitoring apparatus, the monitoring system, the monitoring method, and the monitoring program of the present invention are not limited to the above embodiments, and various changes can be made without departing from the scope of the present invention.

10 Shooting device 100 Monitoring device 300 Automobile 352 Driver 360, 362 Monitored object

Claims (10)

A monitoring device for monitoring the outside of a car,
Image data acquisition means for acquiring image data of an object;
An object recognition unit that recognizes a monitoring target among the objects included in the image data;
A notification condition determination unit that determines whether the notification condition for notifying the outside is satisfied when the object recognition unit recognizes the monitoring target;
A notification unit that notifies the outside of the notification condition according to a predetermined notification mode;
Monitoring device having.   The monitoring device according to claim 1, wherein the object recognition means refers to data generated by deep learning. A distance estimation unit configured to estimate a distance between the monitoring target and the vehicle;
The notification condition is that the monitoring target is located within a predetermined distance from the vehicle.
The monitoring device according to claim 1 or 2. It has notification mode determination means for determining a notification mode to be notified to the outside based on the distance between the monitoring target and the car.
The monitoring device according to claim 3. It has an approach judgment means which judges whether the said monitoring object is approaching the said motor vehicle,
The notification condition is that the monitored object is approaching the car.
The monitoring device according to claim 1. It has speed change recognition means for recognizing a change in speed at which the monitoring target approaches the vehicle,
The notification condition is that the speed has become faster.
The monitoring device according to claim 1 or 2. It has tracking means for acquiring position information indicating the position of the vehicle and tracking the vehicle when the notification means notifies the outside that the notification condition is satisfied by a predetermined notification mode.
The monitoring device according to claim 1 or 2. A monitoring system for monitoring the outside of a car,
Image data acquisition means for acquiring image data of an object photographed by the photographing means;
An object recognition unit that recognizes a monitoring target among the objects included in the image data;
A notification condition determination unit that determines whether the notification condition for notifying the outside is satisfied when the object recognition unit recognizes the monitoring target;
A notification unit that notifies the outside of the notification condition according to a predetermined notification mode;
Monitoring system with. A monitoring device that monitors the outside of the car
An image data acquisition step of acquiring image data of an object;
An object recognition step of recognizing a monitoring target among the objects included in the image data;
A notification condition determination step of determining whether or not a notification condition for notification to the outside is satisfied when the monitoring target is recognized in the object recognition step;
A notification step of notifying the outside of the notification condition according to a predetermined notification mode;
Monitoring method. A control device that manages a monitoring device that monitors the outside of the vehicle,
Image data acquisition means for acquiring image data of an object,
Object recognition means for recognizing a monitoring target among the objects included in the image data,
A notification condition determination unit that determines whether a notification condition for notification to the outside is satisfied when the object recognition unit recognizes the monitoring target;
Notification means for notifying the outside of the notification condition by a predetermined notification mode,
Monitoring program to function as.