JP2023061984A - System, program, and the like - Google Patents

Abstract

PROBLEM TO BE SOLVED: To use an image obtained by a camera attached to a vehicle to identify the cause of an accident more effectively than before, and to utilize the image more effectively without being limited to the identification of the cause of the accident.

SOLUTION: Processing means of a system has a function to record an image captured by a camera attached to a vehicle, perform image analysis of the images captured by the camera, and a function that executes processing corresponding to the analysis results on the basis of the analysis results.

SELECTED DRAWING: Figure 2

COPYRIGHT: (C)2023,JPO&INPIT

Description

The present invention relates to, for example, a system and program for processing an image captured by a camera or the like attached to a vehicle or the like.

Patent Document 1 below discloses a system applied to a drive recorder or the like. This system repeats the cycle of recording video data in the recording area. When a predetermined cycle termination condition is satisfied, the current cycle is terminated and a new cycle is started in which the video data recorded in the cycle that has been completed is overwritten with new video data in order of oldest. based on at least one of the amount of video data recorded in the current cycle and the amount of video data recorded in the immediately preceding cycle that has not been overwritten;
Causes the output means to output information about the overwritten position.

Japanese Unexamined Patent Application Publication No. 2018-6969

Images recorded by the drive recorder are effectively used to identify the cause of an accident or the like. It is desired that the images acquired by the cameras attached to the vehicle or the like are used more effectively than before to identify the cause of the accident and are not limited to the identification of the cause of the accident and are used more effectively.

The purpose of the invention of the present application is not limited to this, and we intend to acquire rights for the structure aiming to obtain the effect produced by the structure disclosed in the specification and drawings, etc. by divisional application, amendment, etc. . For example, the present specification discloses a problem in which the phrase "can be done" is read as "is the problem." Each problem is described as being independent, and the structure for solving this problem is also separately filed as a divisional application.
I have the intention to acquire the right by amendment, etc. Even if the problem is implicitly grasped from the description of the specification, the applicant intends to claim part of the structure described in this specification in an amended or divisional application. Also disclosed is a combination of these independent tasks.

(1) A function to record images captured by a camera mounted on a vehicle, and a function to perform image analysis on the images captured by the camera and execute processing corresponding to the analysis results based on the analysis results. It is preferable that the system has a processing means having

It is possible not only to record the image captured by the camera attached to the vehicle, but also to use the image more effectively.

Examples of vehicles include passenger cars, freight cars, and special cars. The camera
For example, at least one of the interior and exterior of the vehicle may be imaged. Next, "inside the car" and "
"Outside the vehicle" may be interpreted, for example, as follows.

When the interior space and the exterior space are clearly separated by the body of the vehicle, the interior of the body may be referred to as the "vehicle interior" and the exterior of the body may be referred to as the "exterior." For example, in a van with a contiguous passenger space and cargo space, the contiguous passenger space and cargo space may be referred to as the interior. For sedans with separate passenger and cargo spaces, both the passenger and cargo spaces may be referred to as the interior. In a covered truck having a cargo bed with a closed structure, the passenger space and the inside of the closed cargo bed may be referred to as the vehicle interior. In open trucks (flat-body trucks) with an open cargo bed, it is difficult to clearly separate the space for storing the cargo on the cargo bed from the space around it. The space in which it is housed can also be called the inside of the vehicle. It should be noted that the occupant space of an open truck is also referred to as the inside of the vehicle.
In some specialized vehicles, such as forklifts, both the passenger space and the cargo space are open, and it may be difficult to clearly separate the passenger space and the cargo space from the surrounding space. Even in such cases, both the passenger space and the cargo space may be referred to as the interior. For example, the space above the seat where the occupant sits and the space above the forks that hold and lift the cargo may be referred to as the vehicle interior. In this way, the space that can be considered to move with the vehicle as it moves should be called the inside of the vehicle, and the surrounding space should be called the outside of the vehicle.

As the "vehicle-mounted camera", it is preferable to use a camera whose relative position and orientation with respect to the vehicle are fixed in normal use. Also, as the "camera attached to the vehicle", it is preferable to use a semi-spherical or omnidirectional 360-degree camera. Using a hemispherical or omnidirectional 360-degree camera can acquire a wider range of images than when using a normal camera. For example, a single camera can capture images both inside and outside the vehicle. A plurality of ordinary cameras may be attached to the vehicle instead of the 360-degree camera.

An image captured by a camera may be a moving image. If the image captured by the camera is a moving image, the processing means can easily determine the movement of the subject, the presence or absence of the subject before and after a certain time, etc. by analyzing the image.

The function of recording an image captured by a camera may be implemented by a server that communicates with an in-vehicle device of a vehicle in which the camera is mounted, but it is even better if the in-vehicle device performs the function. A removable medium such as an SD card may be used as the recording medium. If a removable medium is used as the recording medium, the removable medium on which the image is recorded can be removed from the in-vehicle device and the image can be displayed on another terminal such as a personal computer.

The function of analyzing an image captured by a camera may be realized by a server that communicates with an in-vehicle device equipped with a camera, but it is even better if the in-vehicle device performs the function. When the in-vehicle device analyzes the image, it is possible to quickly execute processing corresponding to the analysis result in the vehicle.

The processing means may be composed of one device, or may be composed of a plurality of devices. For example, a drive recorder has a function of recording images captured by a camera. The functions of the processing means of this system may be implemented by a microcomputer incorporated in this drive recorder and an application program stored in a memory. Further, for example, the processing means may be composed of two devices, an in-vehicle device and a server, so that the in-vehicle device records images and the server analyzes the images.

In addition, the function to record the image captured by the camera attached to the object other than the vehicle and the image analysis of the image captured by the camera are performed, and based on the analysis result, the processing corresponding to the analysis result is executed. The system may have processing means with functions. However, in particular, (1) is preferable.

(2) The processing means has a function of determining whether or not the movement of a person in an image corresponds to a specific movement registered in advance as a function of the image analysis. If applicable, as processing corresponding to the analysis results, record an image of a specific movement of a person, send the image of the specific movement to a server, and report that a person has made a specific movement. , and issuing an alarm.

For example, a recorded image or an image sent to a server can be used as evidence that a person was making a particular move. For example, by notifying that a person has made a specific movement, it is possible to inform an external person that there is a person making a specific movement. For example, an issued alert may motivate a person performing a particular motion to stop the particular motion by noticing that the alert has been issued to him/herself.

A person who makes a specific movement may be, for example, a driver of a vehicle. As a specific movement
For example, it may be an action other than driving by the driver. For example, the action of looking at a mobile terminal such as a smartphone, the action of eating food, the action of looking away or looking aside, and dozing off may be used. for example,
Whether or not there is an inattentive action may be determined based on the detection result of the position of the driver's eyes. The recorded images can be used to improve driver awareness, analyze the causes of accidents, and the like. The processing means
When it is detected that the driver is operating a mobile terminal such as a smartphone, an alarm should be issued to remind the driver not to hold the mobile terminal while driving. If the driver notices the warning and stops the specific movement, the danger can be avoided in advance. The server to which the data is sent may be the server of the vehicle administrator or the server of the police. A vehicle manager can evaluate the driver's driving behavior based on the transmitted data. The police can evaluate the driving attitude of the driver, analyze the cause of the accident, etc., based on the transmitted image.

For example, a person who makes a specific movement is a suspicious person outside the vehicle, and the specific movement is, for example, an attempt to illegally enter the vehicle, such as breaking a glass window with a hammer, looking into the vehicle, etc. should be By issuing an alarm, it is possible to dissuade a suspicious person who intends to illegally enter the vehicle from entering the vehicle. The recorded images or images sent to the server can be used as useful information for thief arrests after a vehicle-based crime has occurred. By notifying the owner or driver of the vehicle that a person has made a specific movement, the owner or driver can immediately return to the vehicle and prevent damage aimed at the vehicle. In addition, the driver or owner who receives the notification can also report to the police. Thus, this function can be used for crime prevention.

For example, a person making a specific movement is assumed to be a suspicious person inside a vehicle, and the specific movement is defined by various methods aimed at the inside of the vehicle, such as illegal entry into a parked vehicle, movement of a suspicious person inside a parked vehicle, and movement of a suspicious person inside the vehicle. It is preferable to perform the operation of removing the device. When the thief flees in a hurry by issuing the alarm, the effect of reducing the damage aimed at on the vehicle can be obtained. The recorded images or images sent to the server can be used as useful information for thief arrests after a vehicle-based crime has occurred. By notifying the owner or driver of the vehicle that a person has made a specific movement, the owner or driver can immediately return to the vehicle or report to the police, thereby reducing the damage targeted at the vehicle. It becomes possible to let An example of a report to the outside includes transmission of an e-mail attached with an image of a suspicious person. The e-mail may be sent to a vehicle manager, driver, owner, police, or the like.

A person who makes a specific movement is, for example, a passenger in a commercial vehicle in which an unspecified person rides, and as a specific movement, various robbery tricks, such as flickering a knife, holding a handgun, committing violence, etc. should be By executing the process of issuing an alarm by the processing means, damage can be prevented in advance when the robber stops committing a crime. The recorded images or the images sent to the server can be used as useful information for arresting robbers. By notifying the manager of the vehicle that the person has made a specific movement, the manager can notice the occurrence of the robbery and report it to the police. Moreover, it is preferable that the processing means has a function of reporting to the police using a communication function.

For example, a person who makes a specific movement may be an infant in a car, and the specific movement may be, for example, a movement when waking up from sleep. When the image analysis detects that the infant is awake, the processing means preferably notifies the guardian by e-mail or the like. There are cases in which an infant falls asleep in the vehicle while the vehicle is running, and the user wants to go shopping or do some errands with the engine running. At this time, there is a demand to keep the infant asleep, because if the infant is forced to wake up, he or she becomes in a bad mood. However, when the guardian is away from the vehicle,
Worried that infants are awake? Using this system, when an infant wakes up, an e-mail is sent to the parent's mobile terminal, so the parent can immediately rush to the vehicle and deal with the awakening of the infant.

For example, the person performing the specific movement may be a person outside the vehicle requiring care or a person requiring rescue, and the specific movement may be, for example, a movement peculiar to the person requiring care or a movement corresponding to a situation in which rescue is considered necessary. For example, as an example of a motion peculiar to a person requiring assistance, there is a wobbling motion with a cane bent and the waist bent. An example of a motion that corresponds to a situation in which rescue is considered to be necessary includes a motion that corresponds to a situation in which a person has collapsed and is blowing foam from his or her mouth.

The function to detect people making specific movements includes the function to select at least one image from the image in front of the vehicle, the image behind the vehicle, and the image inside the vehicle, and detect the relevant person from the selected images. It is better to The selection of at least one image may be designated by the user. This eliminates the need for the processing means to have excessive processing power, since the processing means does not analyze unnecessary or less important images.

It is preferable to include a function of detecting drowsiness of the driver in the function of detecting a person making a specific movement. For example, the pulse is measured from the variation in the redness of the driver's skin, and based on the pulse variation,
It is preferable to determine whether or not the driver is drowsy. It is recommended to increase the detection accuracy of drowsiness by coordinating body temperature measurement with a thermo camera, pulse measurement with microwaves, and the like. When the processing means of this system detects drowsiness, it is preferable to lower the set temperature of the air conditioner or issue a warning by voice or sound.

(3) The processing means has a function of determining whether or not a person in the image has a specific feature registered in advance as a function of the image analysis, and as a process corresponding to the analysis result, It is preferable that the system executes processing according to the specific characteristics of a person.

Appropriate processing can be performed according to the features of the person captured in the image.

For example, a face recognition function may be included in the function of determining whether a person in an image has specific characteristics. For example, when the face of a person sitting in the driver's seat is authenticated and the person sitting in the driver's seat does not correspond to a pre-registered person, the processing means preferably accesses the vehicle so that the engine cannot be started. For example, from the processing means to an engine control unit (ECU
) to prohibit the engine from starting. Accordingly, it is possible to register persons who can drive the vehicle in advance and to prohibit unregistered persons from driving the vehicle. As a result, the security function of the vehicle can be enhanced. It is also effective as a countermeasure against children's pranks. In addition to prohibiting the engine from starting,
It is also possible to perform processing such as preventing the side brake (parking brake) from being released, preventing the shift lever from entering the drive position, sounding an alarm, and sending an e-mail together with the image to a registered e-mail address.

In addition, for example, face recognition of a person outside the vehicle may be performed, and if the person outside the vehicle corresponds to a pre-registered person, the processing means may control the vehicle to unlock the doors. for example,
If the vehicle is provided for a car sharing service, it is preferable to register in advance a registered member of the car sharing service. The registered member can use the vehicle without having a card key or the like by approaching the vehicle and moving his or her face within the angle of view of the camera of the vehicle. This will improve the usability of vehicles for car sharing services.

The vehicle is a commercial vehicle in which an unspecified person rides, and the processing means preferably uses a face authentication function to determine whether or not the passenger is a wanted criminal. The photograph of the face of the criminal being sought is preferably transmitted from the police database to the processing means of this system through communication via LTE, wireless LAN, etc., and registered in the processing means in advance. If the processing means determines that the passenger is a wanted criminal, it is preferable to notify the vehicle manager, the police, etc. via the communication device and the communication network. Further, the processing means may notify the driver in a manner that is transparent to the passengers. The vehicle manager, police, driver, etc. can recognize that the criminal being wanted has boarded the vehicle, and can quickly take appropriate measures.

For example, taxis are in a closed room where only the crew and passengers get on board, so it seems that there is a lot of demand for this system in terms of ensuring the safety of the crew. When the processing means detects that a criminal has boarded a taxi, it is preferable to notify the taxi company or the police of the criminal's boarding. Also, according to this system, it is possible to detect the criminal's boarding when the criminal gets into the vehicle, before the criminal actually commits a criminal act in the vehicle. Therefore, it becomes possible to prevent crimes from occurring.

A function that determines whether or not a person in an image has a specific feature, for example, a function that compares the appearance of the person in the image with a specific pre-registered appearance and determines whether the two match. (hereinafter referred to as the appearance authentication function) should be included. Appearances peculiar to loitering persons, appearances peculiar to criminals, etc. may be registered in advance as specific appearances. As an example of the appearance peculiar to criminals,
For example, carrying a knife on the road, carrying a dangerous object, and the like. In addition, it is preferable to combine images from an infrared camera, an ultraviolet camera, etc. with images from a normal camera to detect a person who is in possession of an illegal drug. When the processing means detects a person with a particular appearance in the image, the processing means may notify the police or the vehicle manager via the communication device and the communication network.

If it is desired to find a specific missing person or wandering person, for example, the appearance at the time of the disappearance may be individually registered in the processing means as search information. For example, individual search information such as gender, height, type of clothing, color of clothing, whether or not a person is wearing a hat, whether or not he/she has a cane, and whether or not the waist is bent may be registered. Furthermore, it is preferable to register an area (search target area) where it is thought to be wandering. The processing means preferably executes a process of detecting a person who matches the individual search information when the vehicle is located in the search target area. Such individual search information may be sent to the processing means of this system from a database (server) of the police, local support center, or the like. The in-vehicle device performs a function to determine whether the person in the image has a specific feature,
As the vehicle enters the search area, search information may be downloaded from the server to the vehicle's processing means.

A function that determines whether or not a person in an image has specific characteristics, for example, a function that determines whether or not a person sitting in the front passenger seat or the back seat is wearing a seatbelt (hereafter referred to as the appearance authentication function) ) should be included. When the processing means of the system detects a person who is not wearing a seatbelt, the processing means may issue a message prompting the person to fasten the seatbelt.

In particular, when a person in the back seat approaches the entrance of the highway without wearing a seat belt,
A message prompting the user to fasten the seat belt should be issued. For example, when the distance from the current position of the vehicle to the entrance of the expressway becomes equal to or less than a reference distance, it is determined that the entrance of the expressway is approached. In addition, it is preferable to determine that the vehicle has approached the entrance of the expressway when receiving a signal for checking that the ETC card has been forgotten to be inserted. Further, when an image in front of the vehicle is analyzed and an entrance to the expressway is detected in the image, it may be determined that the entrance to the expressway is approached.

(4) The processing means has a function of counting the number of people in the image as a function of the image analysis, and the system executes a process of recording or transmitting the number of people in the image as a process corresponding to the analysis result. good.

The number of people can be counted by this system without using a special device for counting the number of people.

For example, it is preferable to attach a camera to a bus and use an image of the inside of the vehicle as the original image for counting the number of people. The processing means should display the counting result to notify the crew. The crew can easily grasp the number of passengers (passengers) on board the vehicle. This function can be used in place of the tour roll call. When the number of passengers does not reach zero when all the passengers should get off, the crew can know that there are people left behind in the car. For example, it is possible to detect lingering persons such as babies who cannot express their own intentions and persons who cannot get off voluntarily such as disabled persons.

For example, the vehicle to which the camera is attached is a vehicle with a predetermined number of passengers, such as a sightseeing bus, a long-distance express bus, etc., and the predetermined number of passengers is registered in the processing means of this system. Good. The processing means preferably compares the result of counting the number of people with the registered number of people, and if there is an excess or deficiency, warns the crew of the excess or deficiency. This prevents the train from departing with an insufficient number of passengers. In addition, it is possible to easily recognize that an unqualified person has boarded the vehicle, and to encourage the person to get off the vehicle.

For example, a camera (e.g., a semi-spherical camera on the ceiling) is installed inside a route bus, and the processing means performs image analysis to count the number of passengers, the number of standing passengers, and the number of sitting passengers. Good to have Furthermore, the processing means acquires the current position information of the bus, the open/close signal of the door of the bus, the buzzer of getting off the bus, the touch of the IC card when getting on and off the bus, and the signal of the fare box, and records each of them as a trigger. Good to have Accumulate this information in a data lake,
It will be possible to analyze big data about where people got on and where to get off, and how the seating situation was in the car during that time. Such data can be valuable information for consulting companies that carry out city planning and the like. Furthermore, it is preferable to acquire together the information transmitted through an in-vehicle network such as CAN and the driver's vital information. These pieces of information can be used to identify causal relationships such as whether the abnormality is due to the driver, the driver, or the vehicle when some abnormality occurs in the vehicle.

For example, an image outside the vehicle may be used as the original image for counting the number of people. The processing means of this system may record the counting result as the number of people data. Based on the recorded number of people data, it is possible to easily extract places where many people were present. It is preferable to record the image from which the counting is based, using the counting result as a trigger. For example, an image may be recorded when the number of people is equal to or greater than a predetermined threshold.

(5) The processing means has a function of detecting, as the image analysis function, the display content of the display device of the vehicle or the operation status of the operation device of the vehicle from the image inside the vehicle, and as the processing corresponding to the analysis result, the It is preferable that the system executes a process according to the display contents of the display device or the operation status of the operation device.

These data can be acquired by the present system without retrofitting a device for acquiring data that is the basis of the display content of the display device or a device for acquiring data according to the operation status of the operation device. For example, these data can be acquired without connecting an aftermarket device to the OBD connector. In particular, even if the vehicle manufacturer prohibits connecting third-party products to the OBD connector, the system allows these data to be obtained. The display device may include various gauges, warning lights (warning lamps) in the vehicle, a head-up display that projects an image onto the windshield or the like, and the like.

The display content of the display device to be detected may be, for example, the vehicle speed displayed on the speedometer. The processing means preferably acquires the vehicle speed data by performing pattern recognition on the position of the needle of the analog speedometer and character recognition on the numbers on the dial. Further, it is preferable to obtain vehicle speed data by character recognition of the numbers displayed on the digital speedometer. The acquired vehicle speed data can be used for various processes. An example of how the vehicle speed data is used will be described below.

The processing means preferably has a function of automatically detecting the speedometer by image recognition. This function allows the processing means to detect speedometers in a variety of vehicles, independent of vehicle type. In addition, the position and shape in the image of the speedometer of the vehicle in which the processing means is mounted may be registered in advance in the processing means. This allows the processing means to easily detect the speedometer.

A stand-alone car navigation device uses GPS position information to obtain the current position. In cases where GPS signals cannot be received, such as in tunnels, other methods must be used to determine the vehicle's current position. As an example, the current position of the vehicle within the map is moved for 5 minutes at a constant speed of about 80% of the speed immediately before the GPS signal is lost. A mode in which such processing is performed is called a tunnel mode. Tunnel mode is effective not only when entering a tunnel, but also when driving under an overpass. However, if you encounter a traffic jam while processing in tunnel mode, the method that assumes that you are traveling at a constant speed will result in an excessively large error in the current position, making it practically useless. By attaching an acceleration sensor to the vehicle and using the acceleration data from the acceleration sensor, the accuracy of the current position information in the tunnel mode can be improved. However, it is difficult for the user to attach the acceleration sensor, and it is expensive, so it is not realistic.

It is preferable to read the speed of the vehicle from the display contents of the speedometer and update the current position information of the vehicle based on the read speed using the map matching function implemented in the car navigation system. As a result, the accuracy of the current position information of the own vehicle can be improved.

At places where the road inside the tunnel runs parallel to the ground road, or where the elevated road runs parallel to the ground road under the elevated road, which road is matched based on whether the tunnel mode is activated or not. You can choose For example, when the tunnel mode is activated, the vehicle position is mapped on the road in the tunnel or on the ground road under the overpass, and when the tunnel mode is not activated, the vehicle position is mapped on the road in the tunnel. It is good to map on the ground road or elevated road you are running.

The display content of the display device to be detected may be, for example, the remaining amount of fuel displayed on the fuel gauge. The processing means preferably detects the remaining amount of fuel by recognizing the image of the fuel gauge. The processing means preferably sends a message prompting the driver to refuel when detecting that the remaining amount of fuel has fallen below a predetermined reference value. For example, in conjunction with the current location information of the own vehicle, when returning from an outing to a vehicle storage location near the home or company, the remaining amount of fuel is a reference value, for example, 1/4 or less when the tank is full. , the processing means may issue a warning and prompt refueling. In addition, the car navigation device and the system are interlocked, and the processing means compares the distance to the destination and the remaining amount of fuel, and refuels at the time of departure or during the route to the destination based on the comparison result. Encourage them. For example, a message prompting for refueling at a gas station near the current location may be output near a gas station on the route.

The function of extracting the speedometer and fuel gauge from the image may be realized by artificial intelligence (AI) installed in the processing means of this system. If it is difficult to extract the speedometer and fuel gauge from the image by artificial intelligence, the position and image information of the speedometer and fuel gauge in the image should be registered in the processing means in advance.

The processing means reads the fuel gauge of the vehicle provided to the car sharing service,
When it is detected that the remaining amount is less than a predetermined reference value, it is preferable to notify the driver that he/she can receive the benefits of the car sharing service by refueling. For example, a message such as "If you refuel now, you can receive a discount equivalent to 15 minutes."

The display content of the display device to be detected may be, for example, the presence or absence of lighting of a warning light in the instrument panel. The warning lights for which the presence or absence of lighting is determined may include a headlight lighting indicator, turn signal indicator, high beam indicator (high beam warning light), parking brake indicator, ajar door warning light, fuel warning light, and the like.
The processing means of this system can detect the state of the vehicle corresponding to the warning lights by determining whether or not these warning lights are lit.

The display content of the display device to be detected may be, for example, warning information displayed by a head-up display. Preferably, the processing means detects the content of the warning by detecting changes in the image displayed by the head-up display, and executes predetermined processing according to the content of the warning. For example, the warning may be issued by sound or voice. By changing the visual warning to auditory information and issuing the warning, the driver can easily notice the warning.

For example, the processing means of this system, in cooperation with the brightness information outside the vehicle and the current position information of the vehicle, detects that the lighting indicator of the headlight is not lit even though the headlight should be turned on. When this is detected, the driver may be notified to turn on the headlights. The situations in which the headlights should be turned on include, for example, situations where the outside of the vehicle is darker than a certain level of brightness, situations where the current position of the vehicle is inside a tunnel or inside a culvert, and the like. The processing means of the system may notify the driver to turn off the headlights when the headlight turn-on indicator remains on even though the headlight turn-on condition has been resolved. Since there are tunnels where the interior is bright due to the lighting, if it is determined whether or not the headlights should be turned on based only on the information on the brightness outside the vehicle, the situation where the headlights should be turned on despite being in a tunnel is determined. Otherwise, an erroneous determination may occur. Such an erroneous determination can be avoided by determining whether the vehicle is in a tunnel or not based on the current position information of the vehicle. In addition to the headlights, if it detects that the lighting indicator for the sidelights is not lit even though it is supposed to turn on the sidelights (small lights), it will turn on the sidelights. should be notified to the driver immediately.

The processing means of this system determines whether or not there is an oncoming vehicle from the image in front of the vehicle, and if it detects that the high beam indicator is lit even though there is an oncoming vehicle, it warns the driver to change to low beam. do it. Since the beam high/low switching operation differs depending on the vehicle type, when the driver is driving a vehicle of a vehicle type different from the vehicle type he or she normally uses, there is a tendency for the driver to turn on the low beam but remain on the high beam. In such a case, the function of issuing this warning is particularly effective.

The processing means of this system preferably detects blinking of the turn signal from an image outside the vehicle when it is dark outside the vehicle, and recognizes an indicator blinking inside the vehicle in synchronization with the blinking period of the turn signal as the turn signal indicator. . By doing so, the processing means can easily find out the blinker indicator in the instrument panel.

When the processing means of this system detects that the headlights and sidelights are on, even though it detects that no one is inside the vehicle from the image inside the vehicle, the driver outside the vehicle and the vehicle manager will be notified. should be notified that the user forgot to turn off the light. This notification may be made, for example, by sending an e-mail to a pre-registered e-mail address. In addition, the processing means
If the hazard lamps are forgotten to be turned off or the room light is left to be turned off, the notification may be sent by e-mail or the like. The lighting state of the hazard lamps may be detected from the blinking state of the blinker indicator. The lighting state of the room light may be detected from the brightness of the room light reflected in the image.

There are generally three types of room light settings: a constant lighting mode, a constant lighting mode, and a door open/close state interlocking mode (hereinafter simply referred to as an interlocking mode). In many vehicles the
These three types of modes are cyclically updated each time the room light on/off switch is pressed. Some vehicles have a function in which the room light turns on when the engine is stopped in a dark place (at night) and turns off automatically after a while (for example, after several seconds have passed). In some cases, it may be desirable to keep the interior lights on for a longer period of time to prepare for exiting the vehicle after the engine has been stopped. For example, after stopping the engine, it is desirable to turn on the room light during the period in which purchased items are sorted and taken out, portable terminals are taken out, and the like. In such a case, the driver switches the setting of the room light to the interlock mode or the constant lighting mode in order to prevent the room light from automatically turning off after a certain period of time has passed since the engine was stopped. However,
When the door is open, it is difficult to determine whether the room light is currently set to the constant lighting mode or the interlocking mode. Even though the room light is currently set to the constant lighting mode, if you misunderstand that it is in the interlocking mode and get out of the car, the room light will remain on. By using this system, the driver or manager can notice that the room light has been left on and turn off the room light, thereby preventing the vehicle's battery from running out due to the room light being on for a long period of time. A vehicle with a structure in which it is not possible to tell which mode the room light is currently set to just by looking at the operation switch. This feature of the system is particularly useful in.

When the room light setting is in the linked mode, the room light will remain on if any door is open or the door is ajar. A driver or manager who is notified that the room light is on can notice that the door may be ajar or the door on the side where it is difficult to see may be open.

Some vehicles have a function that keeps the room light on until a certain period of time (turn-off delay time) elapses after the door is closed when the room light is set to interlock mode. In such a vehicle, it is preferable that the processing means of the present system does not determine whether or not the lights have been forgotten until the lights-out delay time has elapsed after detecting that the door was closed. The turn-off delay time is preferably set by the system based on input from the user.

Further, the processing means preferably transmits a signal to turn off the room light to the controller of the vehicle when it detects that the room light remains on even after a certain period of time has passed since no one is inside the vehicle. .

When the processing means of this system detects that the door of the vehicle has been opened while the headlights, side lights, hazard lights, room lights, etc. are lit or blinking, one of the lights remains lit. It is preferable to have a function to issue an alarm to notify that As a result, the driver can notice that he has forgotten to turn off the light before he leaves the vehicle, and can prevent him from forgetting to turn off the light.

If the processing means of this system detects that the parking brake is applied based on the lighting of the parking brake indicator, it will issue an alarm even if it detects that the driver is dozing off from the image of the driver. should not be issued. In addition, when the parking brake is applied, an alarm is issued even if it detects that the driver is doing something that would be considered dangerous while driving, such as looking at a mobile device. It is better not to

When the system's processing means detects that the warning light has remained on for a certain period of time, signaling the need for early inspection or repair, the system will notify the vehicle's dealer or registered repairer. It is preferable to have a function of notifying the state of the vehicle. For example, a user who is not interested in automobiles may not notice the warning light when it is possible to drive even if the warning light is on.
It is often left unattended. For example, if a warning light for the engine itself lights up, the vehicle will normally be unable to run, but even if the warning light that indicates an abnormality in the engine oil is lit, the vehicle will still be able to run. Further, in the case of an O2 sensor abnormality, the fuel consumption becomes worse, but the vehicle itself can be driven.
If you continue to drive the vehicle without checking and repairing it, even if it is possible to drive, minor failures may become more severe. For dealers and repairers, the notification of the lighting of the warning light is useful information for carrying out sales activities for inspection and repair to the owner of the vehicle on which the warning light is lit. In addition, the vehicle owner can prevent the failure from becoming serious by performing inspection and maintenance at an early stage.

The display content of the display device to be detected may be, for example, the temperature displayed in the temperature display area in the instrument panel that displays the temperature measurement value measured by the temperature sensor that measures the temperature outside the vehicle. The processing means preferably has, as the image analysis function, a function of reading the temperature from the image displayed in the temperature display area of the instrument panel by character recognition. The processing means preferably executes a process of transmitting the temperature information to an external server via the communication device and the communication network as the process corresponding to the analysis result.

For example, in order to obtain data on pollen dispersal, sensors are installed in the homes of general users, and by collecting measurement data from the sensors, detailed predictions of pollen dispersal conditions and distribution of pollen dispersal information are provided. Provided by Weathernews, etc. In addition, weather forecasts incorporate outdoor photographs posted by general users and images collected from outdoor live cameras installed in ordinary homes. Also, a service for distributing these images (for example, Weather News SOLiVE24, etc.) is provided. A system that provides these weather information services can use many vehicles as temperature measurement probes by collecting outside air temperature information measured by many vehicles. When many vehicles transmitting temperature information are distributed over a wide area, the temperature information collected from many vehicles can be useful as spot information. For example, it is possible to issue a spot heatstroke warning that cannot be expressed by general weather information.

A company that provides such a weather information service can obtain the effect of being able to collect temperature information over a wide range without installing a temperature sensor in-house. The company that receives the temperature information should provide free or discounted paid weather information services provided by the company in return for the owner of the vehicle that provided the temperature information.

In addition to temperature information read from the temperature display area in the instrument panel, outside temperature information is obtained by character recognition of the numbers displayed on the roadside temperature display signs that are transferred to the image acquired by the camera. Good to get. Detection of the temperature display sign may be performed, for example, by recognizing characters such as "current temperature" in the sign. The acquired outside temperature information is
The processing means of this system may send the temperature information to an external server in the same manner as the temperature information obtained from the temperature display area in the instrument panel.

The function of detecting the operation status of the operation device by image analysis may include a function of detecting the position of the driver's foot relative to the accelerator pedal and the brake pedal. As the processing according to the operation status of the operation device, processing is performed to notify the driver so that the driver can easily recognize whether the driver's foot is at the position of the accelerator pedal or the position of the brake pedal. you should do it. For example, when the foot is on the accelerator pedal and when the foot is on the brake pedal, the driver may be notified by changing the color of light emitted by the light-emitting element. This system is installed in a vehicle that has a function to detect whether the foot is on the accelerator pedal or the brake pedal, and to display the detection results as visual information on the display. , image analysis of the content displayed on the display device may be performed to detect the position of the driver's foot with respect to the accelerator pedal and the brake pedal.

Furthermore, when the foot is at the position of the accelerator pedal even though an obstacle is detected in front by image analysis, it is preferable to issue an alarm by sound or the like. By the driver noticing the warning, it is possible to suppress the occurrence of erroneous depression of the accelerator pedal and the brake pedal.

It is preferable to include a function of detecting the rotation angle of the steering wheel (steering wheel) in the function of detecting the operation status of the operation device by image analysis. As the processing according to the operation state of the operating device, it is preferable to execute processing for determining the traveling direction and traveling distance of the vehicle based on the rotation angle of the steering wheel and the vehicle speed. The processing result may be provided to, for example, a stand-alone car navigation device. The stand-alone car navigation device can calculate the travel route of the vehicle based on the information about the direction of travel and travel distance given from this system even in situations where GPS signals cannot be received.

The image analysis function should include a function for detecting whether the vehicle is being driven automatically (including automatic driving levels 1 to 5) or manually (automatic driving level 0). This detection may be performed, for example, by recognizing from an image the display state of an indicator that indicates whether various functions of automatic driving displayed on the instrument panel are in operation. As a process corresponding to the analysis result, for example, when it is detected that the automatic driving function has been canceled, a process of notifying the driver that the automatic driving function has been canceled may be executed. In addition, when it detects that the driver is not putting his hand on the steering wheel even though the automatic driving function has been canceled, the processing means issues a warning to the driver to operate the steering wheel. do it.
Further, the processing means preferably records the image captured by the camera, triggered by the fact that the automatic driving function is canceled and the driver's hand is not on the steering wheel.
The recorded image can be used as evidence that the driver's operation was inappropriate.

In a vehicle equipped with a radar detector, the processing means preferably disables the function of the radar detector when the automatic driving function is enabled. This eliminates unnecessary alarms from radar detectors. When the automatic driving state is disabled, the processing means may enable the function of the radar detector.

Whether the vehicle is in automated driving or manual driving, which level (mode) of automated driving is, the movement of the indicator and steering wheel, the position of the driver's hands, the direction of the driver's head, etc. It is better to judge from the direction in which it is located.

(6) The processing means has a function of recognizing characters displayed on a sticker affixed to the vehicle from an image inside the vehicle as the function of image analysis. It is preferable that the system executes a process of calling attention according to the information indicated by .

By being alerted, the driver, manager, or owner of the vehicle can pay attention to the character information on the sticker attached to the vehicle and perform appropriate processing according to the character information.

For example, an inspection mark (vehicle inspection sticker) may be included in the stickers to be subjected to image analysis. The date on which the valid period of the vehicle inspection expires is displayed on the inside surface of the inspection mark. The processing means reads information on the expiration date of the validity period of the vehicle inspection by character recognition, and when the period from the current time to the expiration date of the validity period of the vehicle inspection (the expiration date of the vehicle inspection) becomes shorter than a predetermined period, the vehicle inspection is executed. It is good to call attention to receive it. For example, the processing means may alert by issuing a voice saying "The expiration date of the vehicle inspection valid period is approaching." If the period until the expiration date of the validity period of the vehicle inspection becomes shorter, for example, the processing means may issue a warning with a voice saying, "Please undergo a vehicle inspection."

For example, the stickers subject to image analysis may include a service check sticker. The due date for the next periodic inspection is displayed on the maintenance check sticker in terms of date and time. The processing means reads the date information indicating the due date of the periodic inspection by character recognition, and if the period from the current time to the due date of the periodic inspection becomes shorter than a predetermined period, it is preferable to call attention to receive the periodic inspection. For example, the processing means may alert the user by issuing a voice saying, "The periodical inspection deadline is approaching." If the period until the periodical inspection deadline becomes shorter, for example, the processing means may issue a warning with a voice saying, "Please undergo a periodical inspection."

The image analysis function may include a function of determining whether or not the inspection mark is included in the image captured by the camera. As a process corresponding to the analysis result, when the inspection mark is not captured in the image, the processing means executes a process of calling attention to adjust the mounting position and posture of the camera so that the inspection mark is captured. do it. For example, the processing means may issue a voice saying "The inspection mark is not captured. Please adjust the position and posture of the camera so that the inspection mark is captured." This makes it possible to easily adjust the position and orientation of the camera when it is installed. It is further preferable that the processing means has a function of reading the contents displayed on display devices in the vehicle, such as a speedometer and a fuel gauge, and a function of calling attention so that the display devices in the vehicle are reflected in the image.

The image analysis function should include a function to determine whether the windshield, driver's side window and front passenger's side window have stickers other than those permitted by law. As a process corresponding to the analysis result, when it is determined that a sticker other than the legally approved sticker is attached, a process of notifying the driver or the like to remove the sticker may be executed.

(7) The processing means includes, as the image analysis function, a function of analyzing the image of the area in which the front passenger seat is reflected, and as the processing corresponding to the analysis result, a system that executes processing according to the situation of the front passenger seat. should be

For example, appropriate processing can be executed depending on whether a person is in the front passenger seat or not, or whether an object is placed in the front passenger seat.

Preferably, the processing means is installed in the vehicle with the function of issuing a warning when the person sitting in the front passenger seat is not wearing a seat belt. The processing means preferably has, as an image analysis function, a function of determining whether or not a person is sitting in the passenger seat from the image of the area in which the passenger seat is reflected. As a process corresponding to the analysis result, the processing means preferably disables the function of issuing a warning that the passenger's seatbelt is not fastened when it detects that no one is sitting in the passenger's seat. The processing means preferably has a function of determining whether or not an object is placed on the front passenger seat as a function of image analysis. As a process corresponding to the analysis result, the processing means preferably executes a process of disabling the function of issuing a warning that the passenger's seat belt is not fastened when an object is placed on the passenger's seat.

A vehicle may erroneously determine that a person is in the vehicle when an object equivalent to the weight of a person is placed on the front passenger seat, and issue a warning that the seat belt is not fastened. When this system is installed in a vehicle, the vehicle will not issue an unfastened seatbelt warning if an object is placed in the front passenger seat. Therefore, the driver will not be misled by the warning. In particular, this function is effective in vehicles with specifications that continue to sound a warning buzzer when a heavy object is placed on the front passenger seat. To prevent the vehicle from issuing an unfastened seatbelt warning, for example, the processing means may send a signal to the vehicle's controller over the in-vehicle network (CAN) to disable the issuing of the warning.

As for the rear seat, in a vehicle having a similar function, it is preferable that the processing means cancels the seatbelt unfastened warning function when it detects that an object is placed on the rear seat.

(8) The processing means has a function of comparing an image of the interior of the vehicle at the present time with an image of the interior of the vehicle in the past as the function of the image analysis, and performs an image of the interior of the vehicle at the present time as a process corresponding to the analysis result, It is preferable that the system executes a process of notifying when the degree of disagreement with the image of the inside of the vehicle in the past exceeds a predetermined level.

By receiving the notification, the owner or manager of the vehicle can know that the situation inside the vehicle has changed beyond a certain level. As the image of the inside of the vehicle in the past, it is preferable to use an image of a state in which no objects other than those attached to the vehicle have been brought into the vehicle. By comparing this image with the image of the inside of the vehicle at the present time, it is possible to determine whether or not there is an object brought into the vehicle.

By comparing images taken at a plurality of times in the past with the current image, it is preferable to calculate the amount of time that an object brought into the vehicle remains in the vehicle. It is preferable that the processing means has a function of notifying the object when it is detected that the object has been left in the vehicle for a long time.

For example, when the vehicle is unloaded, preferably with only the driver on board, the initial setting button of the system is pressed to drive the vehicle. The processing means recognizes a portion in the image having an optical flow corresponding to the vehicle speed as a window. Other moving parts are recognized as the driver. A part other than the window and the part recognized as the driver is recognized as an area where an object may be placed, and is recorded as a no-luggage state image (no-luggage state video) in which no luggage is loaded on the vehicle. The processing up to this point completes the initial setting processing. Thereafter, when the engine is started or stopped, the difference between the no-luggage state image and the current image is taken to identify the presence of an object brought into the vehicle. A warning is issued when there is no change in the position of an object identified as being brought into the vehicle for a certain period of time. As a result, the driver or the like can be made to recognize an object that has been left in the vehicle for a long period of time. This function is especially useful for people whose car interiors are quickly filled with garbage.

(9) The processing means has, as the image analysis function, a function of determining whether or not a pre-registered object appears in the image inside the vehicle. is shown in the image inside the vehicle, the system may be configured to issue an alarm.

A driver or the like can be alerted to the presence of pre-registered items in the vehicle. For example, it is preferable to register an image of an item (specific bag, wallet, etc.) to be checked in the processing means. It is preferable that the processing means issues an alarm or notifies a forgotten item when it detects that the registered object is reflected in the image in a state where no one is inside the vehicle. As an example of issuing an alarm, for example, a buzzer or the like installed in the vehicle may be sounded. As an example of the notification of a forgotten item, for example, it is preferable to wirelessly notify the driver's smart phone, mobile phone, or the like by e-mail or the like.

The processing means may, for example, issue an alarm when detecting an item left behind in the vehicle and detecting a movement of opening the door. By the warning, the driver or the like can notice the forgotten item before leaving the vehicle. For example, when the processing means detects an item left behind in the vehicle and also detects a motion of closing the door, the processing means preferably notifies the driver or the like of the item left behind by e-mail or the like. A driver or the like can notice that there is an item left behind after leaving the vehicle. The detection of the movement to open and close the door,
For example, this may be done by detecting optical flow.

The processing means may be capable of detecting other movements within the vehicle by detecting optical flow. For example, the processing means may detect door opening/closing movements, window opening/closing movements, blinker operations, steering wheel operations, sun visor raising/lowering operations, and the like as image analysis functions. This system uses the in-vehicle network connector (
For example, it is possible to obtain these information without connecting to an OBD connector).

When the sun visor is down, especially when a 360-degree camera attached to the windshield is used to take an image of the driver's side, the sun visor becomes an obstacle and an image in that direction cannot be obtained at all. Therefore, when the processing means detects that the sun visor is down even though the brightness in front of the vehicle is such that the sun visor is not necessary, it is preferable to issue a warning. The processing means may determine whether the sun visor is down based on the proportion of the area of the sun visor in the image.

(10) The processing means has a function of capturing an image of an object brought close to the camera by the user and recognizing the image as the function of the image analysis, and performs processing corresponding to the analysis result to perform image recognition of the object brought close to the camera by the user. It is preferable that the system executes processing according to the information described in .

By bringing an object that the user is carrying close to the camera, the user can execute processing according to the information written on the object.

For example, the driver's driver's license may be brought closer to the camera. The processing means performs character recognition of the valid date written on the driver's license, and if the period from the current time to the valid date is shorter than a predetermined period, the user is notified that the driver's license is about to be renewed. should inform

For example, a receipt from a gas station should be close to the camera. The processing means preferably performs character recognition of the date, amount of fuel supplied, and amount of money displayed on the receipt, and records this information in a database.

For example, an identification card (for example, a driver's license, an employee ID card, a resident registry card, etc.) may be used as an item to be brought close to the camera. The processing means character-recognizes the name, etc., written on the identification card,
It is preferable to determine whether or not the person is pre-registered. If the person on the identification card is not registered, it is preferable to perform a process of prohibiting engine start-up. Thereby, the person who can drive the vehicle can be limited to the person registered in advance. Furthermore, the processing means
It is advisable to record the name of the person on the identification card. The recorded information can be used to identify the driver when managing vehicle operations.

As an image recognition function, the processing means preferably has a function of recognizing an object that the user has brought close to the camera outside the vehicle, and more preferably has a function of recognizing an object that the user has brought close to the camera inside the vehicle. For example, the processing means recognizes that there is an object brought closer to the camera when an object that is not in the normal image occupies a wider portion than a predetermined range within the angle of view of the camera compared to the normal image. do it. In particular, when the processing means recognizes that the user is holding an object that occupies a wider area than a predetermined range within the angle of view of the camera, it is preferable to determine that the user has brought the object closer to the camera.

(11) The processing means has, as the image analysis function, a function of performing an image analysis of an object reflected in the front image, and as a process corresponding to the analysis result, information to be noted regarding the running of the vehicle is generated. It is preferable that the system executes processing provided to users.

By calling the driver's attention according to the situation ahead, it is possible to suppress the occurrence of an accident that may occur in the future due to the vehicle traveling forward.

The image analysis function may include a function of calculating at least one of the inter-vehicle distance and the relative speed to the preceding vehicle. Based on the inter-vehicle distance or relative speed, it is preferable to detect a situation in which there is a high risk of collision, and notify the driver of the high risk of collision by issuing an alarm or the like. The inter-vehicle distance may be calculated from information such as the size of the forward vehicle in the image, for example, the size of the license plate, the size of the tread (tire gap), the width of the tail lamp, and the like. The relative speed may be calculated from the rate of change in inter-vehicle distance.

The image analysis function should include a function to detect the color of traffic lights ahead. The processing means preferably notifies the driver to stop the vehicle when the traffic light is in a red lighting state.

The image analysis function should include the function of recognizing road signs ahead. It is preferable that the processing means prompts the driver's attention according to the information indicated by the road sign. For example, if the road sign indicates a speed limit, the processing means may acquire the speed limit at the current location through character recognition and notify the driver of the speed limit. If the road sign indicates no entry, the processing means
If the vehicle attempts to enter a prohibited road, an alarm should be issued to notify the driver. For example, when it is detected that the vehicle is about to run in violation of traffic rules indicated by road signs, or that the vehicle is driving in violation of traffic rules, the driver may be notified of the violation of the rules. As a result, the occurrence of traffic rule violations can be prevented or suppressed. For example, by recognizing the character string of the road sign, the processing means warns the driver of excessive speed when detecting a sign indicating that the section in which the vehicle is currently traveling is a section where an automatic speed control machine is installed. You should notify

The image analysis function should include a function for detecting traffic congestion in lanes adjacent to the lane in which the vehicle is traveling. For example, whether or not there is traffic congestion may be determined based on the average traveling speed of vehicles in adjacent lanes and the inter-vehicle distance. For example, when the traveling speed of vehicles in adjacent lanes is less than or equal to a predetermined reference value, and the inter-vehicle distance is less than or equal to a predetermined reference value and a predetermined number of vehicles are lined up,
It is preferable to determine that traffic congestion is occurring. As a process corresponding to the analysis results, when it detects that the lane adjacent to the vehicle's driving lane is congested, it alerts the driver to be careful about jumping out from between the congested vehicles. should be executed. The lanes adjacent to the lane in which the vehicle is traveling may include oncoming lanes and adjacent lanes traveling in the same direction as the vehicle. especially,
In places where people gather, such as shopping malls, theme parks, and train stations, and in places where vehicles enter and leave, such as gas stations, there is a high possibility that the lanes in the same direction adjacent to the lane in which the vehicle is traveling are also congested. In such a case, it is effective to detect congestion in the adjacent lane traveling in the same direction as the own vehicle. As a result, it is possible to suppress the occurrence of accidents due to jumping out. For example, when the traveling speed of the own vehicle is less than a predetermined speed, for example, less than 30 km/h, it is preferable not to emit a warning sound to alert the driver to jump out. This makes it possible to avoid unnecessary warning sounds.
In addition, it is preferable to detect a situation in which there is a high risk of jumping out from the front image and alert the driver. For example, when there are vehicles parked on the side of the road in succession, when there are vehicles parked in a parking lot where a parking meter is installed, etc., it is preferable to call attention to jumping out.

For example, some car navigation devices are equipped with a function that calls the driver's attention while driving based on traffic congestion probe information, general road high-speed travel information, and information on surrounding facilities (residential areas, schools, etc.). By using this system, even in places where traffic congestion probe information cannot be obtained, it is possible to call attention due to traffic congestion.

Animals (dogs, cats, raccoon dogs, wild boars, etc.) in the vicinity of the vehicle can be detected by the image analysis function before starting.
It would be good to include a function to detect people and people. When the processing means detects an animal or person in the vicinity of the vehicle,
It is advisable to inform the driver that there are animals around the vehicle. As a result, it is possible to suppress the occurrence of an accident immediately after starting the vehicle.

The image analysis function should include a function to detect that the vehicle is moving slowly from the flow of the surrounding scenery. The processing means may issue an alarm when it detects that the vehicle is moving slowly forward. For example, if the vehicle starts to move little by little due to a creep phenomenon due to the loosening of the force applied to the brake pedal after the vehicle has stopped temporarily, the driver is notified that the vehicle is moving and that there is a risk of a rear-end collision. be able to. For example, it may be determined that the vehicle is slowly moving forward when the vehicle is moving at such a speed as to move forward due to a creep phenomenon. The speed of flow in the surrounding scenery may be determined by detecting optical flow.

The image analysis function may include a function of determining whether or not the preceding vehicle is a large vehicle such as a bus, or a box-type truck whose loading platform is made of aluminum. The processing means is configured such that the preceding vehicle is a large vehicle,
When it is detected that the vehicle is approaching a traffic signal, it is preferable to notify the driver of the presence of the traffic signal. for example,
It is preferable to issue a voice such as "Caution for signal". The presence or absence of a traffic light may be registered in the processing means in advance as POI information. For example, when a large vehicle is running ahead, it becomes difficult to see traffic signals in the direction of travel. It is preferable to anticipate the existence of traffic signals by looking at traffic signals and pedestrian traffic lights in the opposite lane, and to keep a sufficient inter-vehicle distance so that traffic lights in the direction of travel can be easily seen. However, if such attention is neglected, there is a danger of overlooking traffic signals in the direction of travel. Also, in rural areas, traffic signals for opposite lanes and pedestrian traffic signals may not be installed. By notifying the presence of traffic lights when the preceding vehicle is a large vehicle,
It is possible to reduce the risk of the driver overlooking the traffic signal. Also, instead of directly informing the driver of the presence of a traffic light, the processing means may prompt the driver to drive in such a way that the risk of overlooking the traffic light is reduced. For example, it is preferable to issue a voice such as "Keep a sufficient distance between vehicles." For example, when the distance to the intersection where the traffic light is installed is longer than the safe stopping distance of the own vehicle, it is preferable to alert the driver. The safe stopping distance of the own vehicle is the speed of the own vehicle,
It depends on the weather (rain or not). As the safe stopping distance, it is preferable to set an appropriate distance based on the vehicle speed and the weather (rain or not). For example, if the safe stopping distance in fine weather is 50 m, the processing means determines that the current weather is fine, the distance from the preceding large vehicle is less than 50 m, and the vehicle is outside an urban area in terms of a car navigation device. When there is an intersection with a traffic light within 100 m in the direction of travel, it is advisable to alert the driver.

The image analysis function should include a function to determine whether the road ahead is sloped and, if so, to detect the degree of inclination. It is preferable that the processing means prompts caution such as excessive speed, slow speed, setting to hill climb mode, etc., depending on the degree of inclination of the slope.

The vehicle is a vehicle that handles cargo, such as a forklift or truck, and the processing means recognizes signs such as character strings and barcodes displayed on the cargo from the imaged image, and determines whether the cargo should be handled by the own vehicle. It is preferable to have a function to determine whether or not It is preferable to register in the processing means in advance the information indicated by the signs attached to the packages to be handled by the own vehicle. The processing means preferably executes a process of issuing an alarm when detecting that the own vehicle is loading a load that should not be handled.

The processing means preferably has a function of detecting the window area in the image from the optical flow. Further, the processing means preferably has a function of determining whether the vehicle has entered a tunnel, exited a tunnel, or is traveling in a tunnel from the optical flow. In addition, the processing means preferably has a function of determining whether or not the vehicle is traveling under an overpass (under parallel roads) from the optical flow. Furthermore, the processing means, from the optical flow,
It would be good to have a function to determine whether there is a guardrail or not, and whether caution is required in the side ditches.

(12) The processing means has, as the image analysis function, a function of recognizing characters written on license plates of other vehicles reflected in the image outside the vehicle, and character recognition as a process corresponding to the analysis result. It is preferable that the system executes at least one of a process of recording license plate information specified by and a process of transmitting it via a network.

It is possible to identify the license plate information of other vehicles that are running or parked near the vehicle equipped with the vehicle-mounted device that has the function of the processing means of this system. The processing means preferably transmits the current position information of the vehicle equipped with the vehicle-mounted device and the date and time information when the license plate was imaged in association with the specified license plate information. License plate information acquired along with location information and date and time information is used together with license plate information acquired by the automatic number plate reader (N system) installed on the road to know the driving routes of many vehicles. Useful information. In this way, by associating the vehicle number with the GPS information, it is possible to construct a system that can be called a wide area movement N system.

It is preferable to build a center system that associates and collects location information, date and time information, and license plate information from this system installed in each of a plurality of vehicles via a network. This center system preferably has a function of extracting position information and date/time information of a vehicle having specific license plate information and identifying the travel route of the vehicle.

(13) The processing means has, as the image analysis function, a function of determining whether or not the edge of the windowpane exists in the window portion of the vehicle, and as a process corresponding to the analysis result, the window portion A system that performs a notification process when the edge of the glass is detected is preferable.

The presence of the edge of the windowpane in the window portion means that the windowpane has been left unclosed. A driver or the like can notice that he or she has forgotten to close the window glass by receiving the notification from this system. The edge of the window glass may be detected by detecting discontinuity in the image due to the difference in refractive index between the window glass and the atmosphere, for example. As the notification processing, for example, processing such as wireless notification to the driver's portable terminal or the like or notification by e-mail may be performed. Alternatively, the notification may be processed by sounding a buzzer attached to the vehicle. A smoke film is attached to the window of the vehicle in which the processing means is mounted, and the processing means determines whether the window is open based on the difference in brightness of the image between the area where the window is open and the smoke area of the window glass. It is good to detect that it is open.

(14) The processing means has, as the image analysis function, a function of detecting whether or not the wipers are operating or whether or not raindrops adhering to the windshield are increasing or decreasing, and determining whether or not it is raining, As a process corresponding to the analysis result, the system may execute a process of transmitting information notifying that it is raining from the communication device when it is determined that it is raining.

By receiving the information transmitted from this system, it is possible to know whether it is raining in the area where the vehicle is traveling. The processing means preferably notifies an external device such as a server that it is raining and notifies the current position information of the vehicle. The external device can create a map of areas where it is raining by collecting information from multiple vehicles. Furthermore, the processing means preferably notifies that it is raining and notifies the current time information. As a result, the external device can acquire information about temporal changes in areas where it is raining.

The external device may be, for example, a server of an organization that provides weather information. The processing means preferably uploads to the server an image at a timing when the wiper is not reflected, for example, at a timing when the wiper is lowered. From the images uploaded to the server, information about rainfall, such as the amount of rainfall, can be obtained. The server can utilize data such as rainfall collected from vehicles as big data in various application fields. For example, it can be used for prediction of localized torrential rain. The processing means preferably determines that the rain has stopped when there is no increase or decrease in the number (number) of the raindrops adhering to the windshield.
When the processing means determines that the rain has stopped, the processing means preferably notifies the server that the rain has stopped. The server can obtain more detailed information about rainfall.

(15) The processing means has, as the image analysis function, a function of judging the weather from the state of the sky reflected in the image, and as a process corresponding to the analysis result, a process of transmitting information about the weather from the communication device. should be a system that executes

Weather-related information collected from multiple vehicles can be used as weather-related big data. For example, the weather such as clear, sunny, or cloudy may be determined from the color of the sky reflected in the image, the presence or absence of clouds, and the like. Furthermore, it is preferable to detect the strength of the morning sun and the afternoon sun from the height of the sun and the brightness of the sky, and transmit the information that the morning sun and the afternoon sun are strong. It is preferable to transmit the current location information of the vehicle together with the information about the weather. The external device that has collected this information should provide the information that the morning sun and the afternoon sun are strong and the current position information of the vehicle to the drivers of other vehicles heading to that point from now on. Drivers who have been provided with this information can notice that they need to be careful in the morning sun or late afternoon sun.

(16) The processing means has a function of detecting sound information from a microphone mounted on the vehicle, associates the detection result of the sound information with the analysis result of the image analysis function, and performs processing according to the result of the association. A system that executes processing may be used.

Compared to executing processing based only on sound information detection results or image analysis results,
By associating the two, more appropriate processing can be performed.

The processing means has, as a function of image analysis, a function of detecting a change in the state of the display unit of a device having a display unit mounted in the vehicle, and as processing corresponding to the analysis result, detected sound information,
When the change in the state of the display unit is synchronized, it is preferable to execute a process of associating and storing the detected sound information and the device in which the state of the display unit has changed.

When detecting the sound information from the microphone, the processing means can specify the device that emitted the sound based on the stored information in which the sound information and the change in the state of the display unit are associated with each other.
For example, the processing means has a voice recognition function, determines whether or not the sound information from the microphone is emitted from a specific device, and if it is emitted from a specific device, recognizes the voice. It is better to cancel the function. This makes it possible to avoid executing unnecessary speech recognition processing. Further, when the processing means determines that the sound information from the microphone is emitted from a specific device, the processing means adds information capable of distinguishing the region of the device that generated the sound from the other regions in the image of the vehicle interior. It is better to have a function to record For example, it is preferable to superimpose and record an image of a rectangle or an arbitrary shape of a closed curve surrounding the area in which the device that generated the sound is captured. Alternatively, a translucent geometric pattern may be superimposed and recorded on the area where the device that generated the sound is shown.

As a function of image analysis, the processing means preferably has a function of determining whether or not an emergency vehicle such as an ambulance, a fire engine, or a police car appears in the image. When the processing means detects the sound of the siren of the emergency vehicle from the microphone and also detects the emergency vehicle in the image, it is preferable to determine that the emergency vehicle is in the vicinity of the own vehicle by associating both. When the processing means determines that there is an emergency vehicle, it is preferable to notify the driver of a message to alert the driver.

(17) there are multiple cameras installed in the vehicle;
The processing means has a function of detecting the positions of other cameras reflected in the image as a function of the image analysis, and executes a process of specifying the positional relationship of a plurality of cameras as a process corresponding to the analysis result. system.

It is possible to obtain an image of a wide area inside the vehicle, which is difficult to capture with a single camera. Once the positional relationship of the cameras is specified, the positional relationship of the images captured by each camera can be specified.

The processing means preferably has a function of identifying the model of another camera included in the image and identifying the angle of view from the model. The processing means preferably has a function of synthesizing images captured by a plurality of cameras. By synthesizing the images, it is possible to grasp the situation inside the vehicle more intuitively than when displaying the individual images of the cameras.

For example, at least part of the functions of the processing means may be realized by a viewer such as a personal computer.
It is preferable to display a marker such as a camera icon in the displayed image so that the position of the camera can be known. It is difficult to specify the position of the camera in the vehicle from the image captured by the camera (self). It is preferable to detect the position of the camera in the vehicle from images of other cameras in which the camera is captured. For example, when synthesizing and displaying images captured by three cameras, it is preferable to display the positions of the three cameras by icons or the like in the synthesized image. It is preferable to use different icons depending on the model or angle of view of the camera.

The processing means preferably has a function of displaying an image captured by each camera and displaying the position of the camera and the angle of view of the camera (the area captured by the camera) in the image. For example, depending on the function of a 360-degree spherical camera, a 360-degree semispherical camera, a wide-angle camera, or the like, the angle of view of the camera may be displayed in the image. A user who sees the image can intuitively understand whether each camera is capturing an image of the area.

(18) The camera has an image circle smaller than the length of the diagonal line of the imaging plane.
is a 60 degree camera,
The processing means may be a system having, as the image analysis function, a function of transforming a basic pattern for calculating the feature amount of the image according to the position within the image circle.

It is preferable to calculate the feature amount of the image using the basic pattern without correcting the distortion of the image within the image circle. This makes it possible to reduce the load on the processing means. A basic pattern is defined for an image obtained by imaging with a standard lens (a general wide-angle lens). Projection methods for obtaining an image circle include, for example, orthogonal projection, equidistant projection, stereographic projection, equisolid angle projection, etc. The basic pattern may be modified according to these projection methods. To obtain an image circle, a circular fisheye lens should be used. The imaging surface of the camera is rectangular, and the diameter of the image circle is preferably shorter than the long side, and more preferably equal to or shorter than the short side.

For example, a template pattern of an image to be detected may be prepared in advance as a basic pattern. In this case, the template pattern should be deformed according to its position within the image circle. In addition, for example, when recognizing an image using a Har-like feature, each of a plurality of rectangular patterns for obtaining a Har-like feature may be deformed according to its position within the image circle.

When performing image recognition, it is preferable to limit a region in which a specific object to be detected appears in the image circle, and perform image analysis within the limited region. For example, the functions of the vehicle-mounted device of this system may be realized by a drive recorder. In general, the recommended installation position of the drive recorder is determined in advance, so the area in the image circle where the driver, the traffic light, and the people in the back seat are reflected is almost fixed. . Therefore, depending on the person or object to be detected, it is preferable to limit the area in which the image analysis is performed to detect the person or object to the area in which the person or object is captured. For example, when analyzing an image in front of a vehicle, the area in which the windshield is reflected should be analyzed.

Alternatively, for example, a specific area of the image circle may be converted into a plane image (parallel projection image), and the plane image after conversion may be matched using a normal basic pattern.

Alternatively, the image within the image circle may be mapped onto a spherical surface using a predetermined conversion function, zoomed to a portion of the image mapped onto the spherical surface, and image analysis performed as a planar image ignoring distortion. When it is desired to detect a specific object, such as a road sign, it is preferable to perform image analysis by zooming in on the area in which the specific object is captured. When it is desired to detect a road sign, it is preferable to perform image analysis by zooming in on a region in which the road sign appears in the front image.

(19) The processing means has, as a function of the image analysis, a function of identifying an area of the image that should not be disclosed from the viewpoint of privacy protection as a privacy protection area,
As a process corresponding to the analysis result, the system may execute a process of generating an image in which a person captured in the privacy protection area cannot be identified by appearance from the image captured by the camera.

By making a person unidentifiable, the person's privacy can be protected. The area that should not be disclosed to the public may be, for example, an area in which the inside of the vehicle is reflected. In particular, when an image is acquired using an omnidirectional camera, the interior of the vehicle is imaged, but there may be cases where the image of the interior of the vehicle should not be retained. In such a case, for example, the area inside the vehicle may be painted black, or the area inside the vehicle may be mosaicked, and the image outside the vehicle may be recorded as it is. The processing means preferably performs image analysis, for example, to distinguish between an area showing the interior of the vehicle and an area showing the exterior of the vehicle. For example, an area in which optical flow occurs at a speed corresponding to the vehicle speed may be defined as outside the vehicle. The processing means preferably has a function that allows the user to preset the privacy protection area.

A normal mode that records an image that can identify a person in the privacy protection area,
It is preferable that the user can select a privacy mode for recording an image in which a person captured in the privacy protection area cannot be identified by appearance. The user can switch between the normal mode and the privacy mode according to the purpose of using the vehicle and the usage conditions.

The privacy area may be an area in which a person appears. For example, the processing means may detect a person by performing image analysis, and set the area surrounded by the outline of the detected person as the privacy area. Alternatively, only the detected person's face may be set as the privacy area. At this time, it is preferable to display the person as a silhouette. Alternatively, it is preferable to display only the outline of the person and display the image when there is no person inside the outline. At this time, the person appears transparent. As the outline of a person, it is preferable to display the edge of a moving object, for example.

The processing means preferably displays an avatar in place of the person in the area where the person is shown. The avatar to be displayed should be similar to the original person. For example, it is preferable to display an avatar that reflects the features according to the gender, body shape, facial shape, and the like. A person and an avatar may be associated in advance, and the processing means may identify the person appearing in the image and display the avatar associated with the identified person. An avatar should be created in advance for each person. It is good to move the avatar according to the movement of the person. For example, when a person raises his hand, the avatar should raise his hand, and when the person turns to the side, the avatar should also turn to the side.
It is preferable to change the facial expression of the avatar according to the facial expression of the person. For example, when the processing means detects a facial expression of a person laughing, sad, or angry, the facial expression of the avatar is changed to
It is preferable to change the facial expression into a smiling expression, a sad expression, or an angry expression. Preferably, the processing means reads the person's emotion and changes the color of the avatar according to the person's emotion. For example, when it is detected that a person is angry, it is preferable to change the color of the avatar to a color indicating that the person is angry, such as red.

Before recording the image, the processing means preferably generates an image in which a person appearing in the privacy protection area cannot be identified by appearance from the image captured by the camera, and records the generated image. Also, when reproducing an image captured by a camera, it is preferable to convert the person appearing in the privacy protection area into an image that cannot be identified by appearance and display the image.

(20) The processing means may be a system that executes a process of converting the result of image analysis into text data or encoding and recording it as a process corresponding to the analysis result.

Converting to text data or coding makes it possible to easily find a desired image from the recorded images. For example, "person detection" may be generated as text data when a person is detected in an image, and "red light" may be generated as text data when a red light is detected in an image. The generated text data should be recorded in association with the image from which the text data was generated. Instead of text data, the image analysis result may be coded (binary data) by associating the image analysis result with the code.

(21) The processing means has a function of detecting blocked black areas in the image as the function of the image analysis, and brightens the blocked black areas of the image as processing corresponding to the analysis result. It is preferable that the system executes processing for restoring and displaying on the display screen.

A user who sees the image can easily recognize an object or a person in the blocked-up area. In particular, when an image is acquired using an omnidirectional camera, the bright scenery outside the vehicle and the dark interior of the vehicle are simultaneously captured in the image. The process of detecting a blocked-up area and restoring and displaying it brightly is particularly effective when acquiring an image using an omnidirectional camera. For example, gamma correction, color correction, conversion to black-and-white image, contrast enhancement, etc. may be performed as processing for bright restoration.

(22) The processing means has a function of allowing the user to divide the imaging range captured by the camera into a plurality of sections, and a function of weighting each of the sections. It is preferable that the system has a function of determining imaging conditions based on the weights assigned to and causing the camera to perform imaging under the determined imaging conditions.

The user can obtain an image under imaging conditions suitable for the region desired to be viewed by increasing the weight of the section desired to be viewed in the image and lightening the weight of the section not required to be viewed. The user can divide the imaging range into an arbitrary shape according to the shape of the area that the user wants to see or the area that does not need to be seen. For this reason, for example, it is possible to perform more appropriate weighting for the user than in the case of uniformly weighting the square grid-shaped sections regardless of the content of the image.

In particular, when a 360-degree camera is used for imaging, the difference in the positions within the imaging range in which parts of the vehicle are captured increases for each vehicle type compared to when a general camera is used. For this reason, there is an increased possibility that the relationship between the general-purpose weighting table set in advance in the firmware of the processing means and the division of the imaging range that is actually expected does not match. The function that enables determination of weighting using PC browser software, etc., instead of using a general-purpose weighting table, is 360
A particularly noticeable effect is obtained when using a degree camera. For example, an image captured by a camera without weighting is analyzed using AI software on a PC, etc., and what part of the imaging range is estimated. A weight of "5" is assigned, a weight of "3" is assigned to the face portion, a weight of "1" is assigned to the instrument panel portion, and a weight of "0" is assigned to the area outside the circle image.

The processing means may determine, for example, gain, exposure time, etc. as imaging conditions. An omnidirectional camera may be constructed by arranging the rear surfaces of two semispherical cameras to face each other, and the imaging conditions of the two semispherical cameras may be the same. As a result, when the images captured by the two semi-spherical cameras are combined, an effect is obtained in which the boundary between the two images becomes difficult to discern.

The processing means preferably encloses a partial area within the imaging range with a closed curve and defines the inner area surrounded by the closed curve as one area when allowing the user to divide the area. When the user divides the inner area surrounded by the closed curve into two by the curved line, each of the divided areas may be regarded as one section. Preferably, the processing means causes the user to input a numerical weight for each set section. The numerical value to be input should be selected from a pull-down or pop-up list.

The processing means preferably has a function of inputting the type of the partition instead of or in addition to the function of inputting the weight numerically. A weight is previously associated with the type of partition, and when the type of partition is input, the processing means preferably associates the weight associated with the type with the partition. As the types of sections, for example, types such as driver, meter panel, car navigation screen, windshield, rear seat, front passenger's seat, etc. may be provided.
The processing means preferably has a function that allows the user to change the correspondence relationship between the type of partition and the weight. It is preferable that the list of partition types can be registered in advance by the user.

The processing means preferably has a function of setting a section by allowing the user to designate a desired area within the imaging range as a function of weighting each section. It is preferable to set the weight of the section designated by the user to be viewed higher than the weight of the other areas. This function corresponds to an example of classifying a section into two types, that is, a section that the user wants to see and other sections, in the function that allows the user to input the type of section.

The processing means preferably allows the user to segment the imaging range in an interactive format (wizard format).
For example, the processing means outputs the message "Please enclose the unnecessary area." or "Please fill in the unnecessary area." It should be recognized as In addition, the processing means outputs messages such as "Please paint the area of the driver" and "Please paint the part of the road." It should be recognized as a section with the type of Enclose with a closed curve or fill with
For example, it is preferable to use a pointing device such as a touch panel or a mouse.

The processing means preferably realizes a function of dividing the imaging range into a plurality of sections based on the captured image by artificial intelligence (AI). For example, it is preferable to automatically distinguish between the inside and outside of the vehicle, the seats, the dashboard, the instrument panel, etc. from the image, and divide the imaging range for each automatically distinguished component. The processing means preferably displays the result of classification on a display screen and allows the user to determine whether or not there is any problem with this classification. If the user determines that the classification made by the AI is incorrect, the processing means may have the function of allowing the user to correct the classification. If the user determines that the parcel and the weight or type corresponding to the parcel are incorrect, the processing means preferably prompts the user to select the correct type from a pull-down or popup list.

(23) The processing means has a function of allowing the user to divide the imaging range captured by the camera into a plurality of sections, and a function of assigning a weight to each section. The system may perform a process of compressing a relatively small partition with a larger block when compressing than a partition with a relatively higher weight when compressing.

The user can increase the compression of the less important partitions by weighting the more important partitions heavier and the less important partitions lighter weights. As a result, it is possible to reduce the volume of image data while maintaining high image quality of the image of the section with high importance, compared to the case where the entire image is uniformly compressed. For example, the processing means may perform compression using 8×8 pixels as one block for weighted partitions and 16×16 pixels as one block for lightly weighted partitions.

The image of the barrel portion of the lens may appear in the area outside the circular image formed on the rectangular imaging surface of the camera. Since the image outside the circle image is unnecessary, it is better to paint this part black. As a result, the amount of image data including pixels outside the circle image can be reduced. Further, it is preferable to obtain image data by raster scanning only the pixels inside the circle image, excluding the pixels outside the circle image. As a result, the volume of image data can be reduced.

(24) The processing means has a function of detecting the following vehicle and a function of detecting the distance to the following vehicle as the function of the image analysis, and performs processing corresponding to the analysis result based on the distance to the following vehicle. The system may be configured to execute processing for detecting tailgate driving of the following vehicle.

By executing a process according to the determination result, the processing means can take appropriate measures against tailgate driving. For example, when the same following vehicle continues to exist within a predetermined judgment distance for a predetermined judgment time or more even though the speed of the vehicle is equal to or higher than a predetermined reference value, the following vehicle is driving in the wind. It should be judged that

A camera having a function of measuring the distance to an object for each pixel may be used as the camera. For example, a back-illuminated time-of-flight distance image sensor may be used as the image sensor of the camera.

When the processing means determines that the following vehicle is driving on the road, as processing according to the determination result, for example, notification to the driver, recording of the rear image, advice to the driver, traffic enforcement agency (e.g. (Police) should be processed. When reporting to a traffic enforcement agency, it is advisable to attach an image of the vehicle and its driver who is driving tailgate. When the processing means detects, through image analysis, that the vehicle that is driving tailgate has overtaken the own vehicle and has come around in front of the own vehicle, it notifies the driver of the possibility of being harassed by a sudden stop. do it.

The processing means preferably detects people and vehicles around the vehicle and controls the vehicle according to the distance to the people and vehicles. For example, the processing means determines that there is a high possibility of contact with a person based on the speed of the vehicle, the distance to the person ahead, and the determination result as to whether or not the person is on the planned travel route of the vehicle. When it is determined that the vehicle is stopped, it is preferable to perform control to stop the vehicle.

(25) The program may be a program for causing a computer to implement the function of the processing means according to any one of (1) to (24) above.

The inventions shown in (1) to (24) above can be combined arbitrarily. For example, a configuration may be adopted in which at least part of the configuration of at least one of the following inventions (2) is added to all or part of the configuration of the invention shown in (1). In particular, in the invention shown in (1), (2
) It is preferable to add at least a part of the configuration of at least one of the following inventions. Also, arbitrary configurations may be extracted from the inventions shown in (1) to (24) and the extracted configurations may be combined. The applicant of this application intends to acquire rights to inventions including these configurations.

It is possible not only to record the image captured by the camera attached to the vehicle, but also to use the image more effectively.

The effect of the invention of the present application is not limited to this, and the effect exerted by the parts of the configuration disclosed in the specification and drawings etc. is also disclosed.
I have the intention to acquire the right by amendment, etc. For example, in this specification, the description of "can be done" is a description that clearly shows the effect, and there are parts showing the effect even if there is no description of "can be done". Moreover, even without such a description, there is an effect that can be grasped by the configuration.

FIG. 1 is a diagram showing a windshield, a dashboard, etc. in a vehicle in which a drive recorder as a system according to the first embodiment is mounted in the vehicle. FIG. 2 is a block diagram of a drive recorder as a system according to the first embodiment. FIG. 3 is a block diagram of a system according to a modification in which the functions of the processing unit are realized by two devices, an in-vehicle device and a server. FIG. 4 is a block diagram of a system according to a modification in which the function of the processing unit is realized by an in-vehicle device and an external device such as a personal computer. FIG. 5 is a diagram of an imaging plane for explaining a method of dividing an imaging range into a plurality of sections by a method according to a comparative example. FIG. 6 is a diagram of an imaging plane for explaining a method of dividing the imaging range into a plurality of sections in the system according to the 22nd embodiment.

[First embodiment]
A system according to a first embodiment of the present invention will now be described with reference to FIGS. 1 and 2. FIG. The function of this system is realized by a drive recorder attached to the vehicle.

FIG. 1 is a view showing a windshield, a dashboard, etc. inside a vehicle with the drive recorder 1 mounted therein. A vehicle equipped with the drive recorder 1 is a passenger car. The drive recorder 1 is mounted on the upper part of the windshield 2 of the vehicle and on the passenger seat side adjacent to the rearview mirror 3 near the center in the left-right direction. The drive recorder 1 is attached and fixed to the inner surface of the windshield 2 with an attachment member such as double-sided tape. A drive recorder 1 is connected to a cigarette lighter socket 5 of a vehicle via a power cable 4 . When the accessory power source of the vehicle is turned on, power is supplied from the cigar socket 5 to the drive recorder 1 . The drive recorder 1 includes a main body 10 and a 360-degree omnidirectional camera 11. - 特許庁The camera 11 is fixed in relative position and orientation with respect to the vehicle in normal use.

FIG. 2 is a block diagram of the drive recorder 1 as a system according to the first embodiment.
The drive recorder 1 includes a processing unit 20, a camera 11, a GPS receiver 12, an SD card reader 13, an acceleration sensor 14, a speaker 15, a display 16, and operation buttons 17.
Strictly speaking, the SD card reader 13 should be called "SD card reader/writer", but in this specification, it is simply called "SD card reader".

The speaker 15 and the display 16 function as output means for notifying the user of various information. The speaker 15 issues warnings and various information by sounds and voices under the control of the processing unit 20 . The display 16 displays various information as images under the control of the processing unit 20 . The operation button 17 functions as input means for the user to give various commands to the drive recorder 1 .

The camera 11 is a 360-degree omnidirectional camera, and captures the entire area of the vehicle in the front-back direction, the up-down direction, and the left-right direction as a moving image. Therefore, the camera 11 images the outside of the vehicle through the windshield, the rear glass, and the side windows, and also images the inside of the vehicle including the driver's seat, front passenger's seat, rear seats, instrument panel, center console, and the like. A moving image captured by the camera 11 is taken into the processing unit 20 as image data. The processing unit 20 is the camera 1
Based on the image data input from 1, an image is recorded in an SD card connected to an SD card reader 13.

The GPS receiver 12 detects positional information of the vehicle at the current time based on instructions from the processing unit 20 . The position information includes the time obtained based on the signal from the GPS satellite, the speed of the vehicle,
Contains information such as longitude, latitude and altitude. The processing unit 20 performs processing for recording the history of the position information.

The SD card reader 13 reads data from an SD card inserted into the SD card slot or records data on the SD card under the control of the processing unit 20 .

As the acceleration sensor 14, a 3-axis type sensor that detects acceleration and tilt in each of 3 axes (x-axis, y-axis, z-axis) is used. The acceleration sensor 14 always outputs the detected value to the processing unit 20 . The processing unit 20 acquires triaxial acceleration information, for example, every 10 ms.

The processing unit 20 has a well-known CPU 20a, memories such as a ROM 20b and a RAM 20c, a timer 20d, and other peripheral circuits. Various programs are stored in the ROM 20 b of the processing unit 20 . The processing unit 20 implements various functions by executing these programs. Various programs include an operating system (OS), a GPS information processing program, an image analysis program, and the like.

The processing unit 20 stores the GPS information received by the GPS receiver 12 in the SD card connected to the SD card reader 13 by executing the GPS information processing program. By executing the image processing program, the processing unit 20 always records the moving image captured by the camera 11 in a predetermined recording area of the SD card as image data associated with time information and position information.

The processing unit 20 performs image analysis of the image captured by the camera 11 by executing an image analysis program. Furthermore, the processing unit 20 executes various processes corresponding to the analysis results.

By installing the system according to the first embodiment in a vehicle, it is possible not only to record the image captured by the camera attached to the vehicle, but also to utilize the image more effectively. If the images captured by the camera 11 are converted into moving images, it becomes possible to easily determine the movement of the subject, the presence or absence of the subject before and after a certain time, etc. by analyzing the images of the moving images.

Next, a modification of the first embodiment will be described.
In the first embodiment, the drive recorder 1 is attached to a passenger car, but in a modification, it is attached to other types of vehicles such as trucks and special vehicles. Moreover, in the first embodiment, the camera 1
1 images both the interior and exterior of the vehicle, in the modified example, the camera 11 images at least one of the interior and exterior of the vehicle. "Inside the vehicle" and "outside the vehicle" may be interpreted as follows, for example.

When the interior space and the exterior space are clearly separated by the body of the vehicle, the interior of the body may be referred to as the "vehicle interior" and the exterior of the body may be referred to as the "exterior." For example, in a van with a contiguous passenger space and cargo space, the contiguous passenger space and cargo space may be referred to as the interior. For sedans with separate passenger and cargo spaces, both the passenger and cargo spaces may be referred to as the interior. In a covered truck having a cargo bed with a closed structure, the passenger space and the inside of the closed cargo bed may be referred to as the vehicle interior. In open trucks (flat-body trucks) with an open cargo bed, it is difficult to clearly separate the space for storing the cargo on the cargo bed from the space around it. The space in which it is housed can also be called the inside of the vehicle. It should be noted that the occupant space of an open truck is also referred to as the inside of the vehicle.
In some specialized vehicles, such as forklifts, both the passenger space and the cargo space are open, and it may be difficult to clearly separate the passenger space and the cargo space from the surrounding space. Even in such cases, both the passenger space and the cargo space may be referred to as the interior. For example, the space above the seat where the occupant sits and the space above the forks that hold and lift the cargo may be referred to as the vehicle interior. In this way, the space that can be considered to move with the vehicle as it moves should be called the inside of the vehicle, and the surrounding space should be called the outside of the vehicle.

In the first embodiment, a 360-degree omnidirectional camera is used as the camera 11, but a 360-degree hemispherical camera may be used, or an ordinary wide-angle camera may be used. When using a wide-angle camera, it is preferable to attach a plurality of cameras to the vehicle so that a wide area can be captured.

In the first embodiment, the function of recording the image captured by the camera 11 is implemented by the in-vehicle device, but it may be implemented by a server that communicates with the in-vehicle device. Also, in the first embodiment, the function of analyzing the image captured by the camera is realized by the in-vehicle device, but it may be realized by a server that communicates with the in-vehicle device.

As described above, the processing unit 20 of this system may be composed of one in-vehicle device, or may be composed of a plurality of devices. As in the first embodiment, the function of the processing unit 20 is that of the drive recorder 1
It may be realized by a microcomputer built in and an application program stored in a memory. In addition, for example, the function of the processing unit 20 can be divided into the in-vehicle device and the server.
It is preferable that the image is recorded by the in-vehicle device and the image is analyzed by the server. Further, it is preferable that the functions of the processing unit 20 are realized by an in-vehicle device and an external device such as a personal computer, the in-vehicle device records an image, and the external device analyzes and displays the image.

FIG. 3 is a block diagram of a system according to a modification in which the functions of the processing unit are realized by two devices, an in-vehicle device and a server. A vehicle-mounted drive recorder 1 includes a communication device 21 . A server 30 includes a processing unit 31 , a communication device 32 , an input device 33 and a display 34 . The communication device 21 of the drive recorder 1 and the communication device 32 of the server 30 are both connected to the communication network 40 and transmit and receive data to and from other devices via the communication network 40 .
The processing unit 20 of the drive recorder 1 and the processing unit 31 of the server 30
perform mutual data communication via

The input device 33 inputs various commands and data to the processing unit 31 by being operated by the user. The processing unit 31 causes the display 34 to display processing results and various messages to the user.

FIG. 4 is a block diagram of a system according to a modification in which the function of the processing unit is realized by an in-vehicle device and an external device 50 such as a personal computer. The external device 50 includes a processing unit 51, an SD card reader 52
, an input device 53 and a display 54 . The SD card 60 with image data recorded by the SD card reader 13 of the drive recorder 1 is transferred to the SD card reader 52 of the external device 50 .
, the processing unit 51 of the external device 50 can access the image data recorded on the SD card 60 .

Based on a command input from the input device 53, the processing unit 51 reads image data recorded on the SD card 60, and performs image analysis or image processing. The processing unit 51 causes the display 54 to display the analysis result or the processing result.

Second to twenty-fourth embodiments will now be described. Duplicated description of the configuration common to the system according to the first embodiment will be omitted. Also, the system according to the various embodiments described below may be implemented in any of the block diagrams shown in FIGS. 2, 3, and 4, by way of example.

[Second embodiment]
Next, a second embodiment will be described. In the second embodiment, the processing unit of the system has, as an image analysis function, a function of determining whether or not the movement of a person in an image corresponds to a specific movement registered in advance. When the movement of the person corresponds to a specific movement, the processing unit performs processing corresponding to the analysis result, recording an image of the specific movement of the person, and storing the image of the specific movement on the server 30 ( FIG. 3), reporting that the person has made a specific movement, and issuing an alarm.

The recorded image or the image sent to the server can be used as evidence that the person was making a specific movement. By reporting that a person has made a specific movement, it is possible to inform an outside person that there is a person who makes a specific movement. The issued alert motivates the person performing the specific motion to stop the specific motion by noticing that the alert has been issued to him/herself.

Next, an example of "a person who performs a specific movement" will be described.
A person who makes a specific movement is, for example, a driver of a vehicle. The specific motion is, for example, a motion of the driver other than driving. For example, the action of looking at a mobile terminal such as a smartphone, the action of eating food, the action of looking away or looking aside, and dozing off. For example, whether or not there is an inattentive action is determined based on the detection result of the position of the eyes of the driver. The recorded images can be used to improve driver awareness, analyze the causes of accidents, and the like. When the processing unit detects that the driver is operating a mobile terminal such as a smart phone, the processing unit issues an alarm warning the driver not to hold the mobile terminal while driving. If the driver notices the warning and stops the specific movement, the danger can be avoided in advance. The data transmission destination server 30 is, for example, a vehicle administrator's server or a police server. A vehicle manager can evaluate the driver's driving behavior based on the transmitted data. The police can evaluate the driving attitude of the driver, analyze the cause of the accident, etc., based on the transmitted image.

For example, a suspicious person outside the vehicle is assumed to be a person who makes a specific movement. Specific movements are, for example, trying to break into a car, breaking glass windows with a hammer,
It is an operation of looking into the inside of the vehicle, and the like. By issuing an alarm, it is possible to dissuade a suspicious person who intends to illegally enter the vehicle from entering the vehicle. The recorded images or the images sent to the server 30 can be used as useful information for the arrest of thieves after an on-board crime has occurred. By notifying the owner or driver of the vehicle that a person has made a specific movement, the owner or driver can immediately return to the vehicle and prevent damage aimed at on the vehicle. In addition, the driver or owner who receives the notification can also report to the police. Thus, this function can be used for crime prevention.

For example, a suspicious person in a car is assumed to be a person who makes a specific movement. The specific actions include various on-vehicle tactics, such as illegal entry into a parked car, movement of a suspicious person inside a parked car, and removal of devices in the car. When the thief flees in a hurry by issuing the alarm, the effect of reducing the damage aimed at on the vehicle can be obtained. The recorded images or the images sent to the server 30 can be used as useful information for the arrest of thieves after an on-board crime has occurred. By notifying the owner or driver of the vehicle that a person has made a specific movement, the owner or driver can immediately return to the vehicle or report to the police, thereby reducing the damage targeted at the vehicle. It becomes possible to let An example of a report to the outside includes transmission of an e-mail attached with an image of a suspicious person. E-mail should be sent to
For example, a vehicle manager, driver, owner, police, and the like.

A person who makes a specific movement is, for example, a passenger of an unspecified person riding in a commercial vehicle.
The specific motions are various robbery tricks, such as the motion of flicking a knife, the motion of holding a pistol, the motion of committing violence, and the like. By executing the process of issuing an alarm by the processing unit, damage can be prevented in advance if the robber stops thinking about committing a crime. The recorded images or the images sent to the server can be used as useful information for arresting robbers.
By notifying the manager of the vehicle that the person has made a specific movement, the manager can notice the occurrence of the robbery and report it to the police. The processing unit also has a function of reporting the occurrence of a robbery to the police using a communication function.

For example, let us assume that a person who makes a specific movement is an infant in a car. The specific movement is, for example, the movement when waking up from sleep. When the image analysis detects that the infant has woken up, the processing unit notifies the guardian by e-mail or the like. There are cases in which an infant falls asleep in the vehicle while the vehicle is running, and the user wants to go shopping or do some errands with the engine running. At this time, there is a demand to keep the infant asleep, because if the infant is forced to wake up, he or she becomes in a bad mood. However, parents are concerned that their infants are awake when they are away from the vehicle. Using this system, when an infant wakes up, an e-mail is sent to the parent's mobile terminal, so the parent can immediately rush to the vehicle and deal with the awakening of the infant.

A person who makes a specific movement is, for example, a person outside the vehicle requiring care or a person requiring rescue. The specific movement is, for example, a movement peculiar to the care-requiring person and a movement corresponding to a situation in which rescue is considered necessary. An example of a motion peculiar to a person requiring assistance is, for example, a wobbling motion with a cane bent and the waist bent. An example of a motion that corresponds to a situation in which rescue is considered to be necessary includes a motion that corresponds to a situation in which a person has collapsed and is blowing foam from his or her mouth.

The processing unit selects at least one image from among an image in front of the vehicle, an image behind the vehicle, and an image inside the vehicle as a function of detecting a person making a specific movement, and selects a corresponding image from the selected images. It has a function to detect people. Further, the processing unit has a function of allowing the user to designate at least one image. As a result, the processing unit does not analyze unnecessary or less important images, so there is no need to give the processing unit excessive processing power.

The processing unit has a function of detecting drowsiness of the driver as a function of detecting a person making a specific movement. For example, the processing unit measures the pulse from the variation in redness of the driver's skin, and determines whether or not the driver is drowsy based on the pulse variation. Furthermore, the processing unit has a function of increasing the detection accuracy of drowsiness by coordinating body temperature measurement with a thermo camera, pulse measurement with microwaves, and the like. When the processing unit of this system detects drowsiness, processing to lower the set temperature of the air conditioner,
Alternatively, execute processing to issue a warning by voice or sound.

[Third embodiment]
Next, a third embodiment will be described. The processing unit of the system according to the third embodiment has, as an image analysis function, a function of determining whether or not a person in an image has a specific feature registered in advance. Execute processing according to the specific characteristics of the person inside.

The system according to the third embodiment can perform appropriate processing according to the characteristics of the person appearing in the image.

The processing unit has a face authentication function as a function of determining whether or not a person in an image has specific features. A processing unit performs face authentication of a person sitting in a driver's seat, and if the person sitting in the driver's seat does not correspond to a pre-registered person, accesses the vehicle so that the engine cannot be started. For example, the processing unit transmits a signal for prohibiting engine starting to an engine control unit (ECU) via an in-vehicle network such as CAN. Accordingly, it is possible to register persons who can drive the vehicle in advance and to prohibit unregistered persons from driving the vehicle. As a result, the security function of the vehicle can be enhanced. It is also effective as a countermeasure against children's pranks.

According to a modification of the third embodiment, the processing unit prevents the accelerator from being stepped on, prevents the side brake (parking brake) from being released, prevents the shift lever from entering the drive position, and sounds an alarm. , to send an e-mail together with an image to a registered e-mail address.

According to another modification of the third embodiment, the processing unit performs face recognition of a person outside the vehicle, and if the person outside the vehicle corresponds to a pre-registered person, controls the vehicle and unlocks the doors. have. For example, when the vehicle is provided for a car sharing service, registered members of the car sharing service are registered in advance in the processing unit. The registered member can use the vehicle without having a card key or the like by approaching the vehicle and moving his or her face within the angle of view of the camera of the vehicle. This will improve the usability of vehicles for car sharing services.

According to still another modification of the third embodiment, the vehicle is a commercial vehicle in which an unspecified person rides, and the processing unit determines whether or not the passenger is a criminal being sought by the face authentication function. have a function. The face photo of the criminal being sought is transmitted from the police database to the processing unit of this system by communication via LTE, wireless LAN, or the like. The processing unit has a function of registering the received face photograph. When the processing unit determines that the passenger is a criminal being sought, the processing unit notifies the vehicle manager, notifies the police, etc., through the communication device 21 (FIG. 3) and the communication network (FIG. 3). do through. In addition, the processing unit has the function of notifying the driver in a way that is transparent to the passengers.
The vehicle manager, police, driver, etc. can recognize that the criminal being wanted has boarded the vehicle, and can quickly take appropriate measures.

For example, taxis are in a closed room where only the crew and passengers board, so there is a lot of demand for this system in terms of ensuring the safety of the crew. When the processing unit detects that a criminal has boarded a taxi, it notifies the taxi company and the police of the criminal's boarding. Also, according to this system, it is possible to detect the criminal's boarding when the criminal gets into the vehicle, before the criminal actually commits a criminal act in the vehicle. Therefore, it becomes possible to prevent crimes from occurring.

According to still another modification of the third embodiment, the processing unit uses the appearance of the person in the image as a function of determining whether the person in the image has specific features. It has a function (hereinafter referred to as "appearance authentication function") to compare the appearance and determine whether the two match or not. As specific appearances, appearances peculiar to wanderers, appearances peculiar to criminals, etc. are registered in advance in the processing unit. Examples of appearances peculiar to criminals include, for example, carrying knives on the street, carrying dangerous objects, and the like. In addition, the processing unit has a function of detecting a person who possesses illegal drugs by combining an image from an infrared camera, an ultraviolet camera, etc., and an image from a normal camera. When the processing unit detects a person with a specific appearance in the image, the processing unit notifies the police and the vehicle manager via the communication device 21 (FIG. 3) and the communication network 40 (FIG. 3).

According to still another modification of the third embodiment, the processing unit has a function of individually registering the appearance of specific missing persons, loitering persons, etc. at the time of their disappearance as search information. For example, it has a function of registering individual search information such as gender, height, type of clothing, color of clothing, whether or not a person is wearing a hat, whether or not he/she has a cane, and whether or not the waist is bent. Furthermore, the processing unit has a function of registering an area (search target area) where it is thought to be wandering. The processing unit executes a process of detecting a person who matches the individual search information when the vehicle is located in the search target area. Such individual search information is received by the processing unit of this system from a database (server) of the police, local support center, or the like. In this system, the processing unit of the in-vehicle device performs the function of determining whether or not the person in the image has specific characteristics. Download search information to

According to still another modification of the third embodiment, the processing unit has a function of determining whether or not a person in the image has specific features, and is configured to determine whether the person sitting in the front passenger seat or the rear seat is wearing a seatbelt. It has a function (hereinafter referred to as an appearance authentication function) to determine whether or not it is worn. When the processing unit of this system detects a person who is not wearing a seatbelt, it issues a message prompting the person to wear the seatbelt.

Furthermore, the processing unit has a function of issuing a message prompting the person in the backseat to fasten the seatbelt when approaching the entrance of the expressway without wearing the seatbelt. When the distance from the current position of the vehicle to the entrance of the highway becomes equal to or less than a reference distance, the processing unit determines that the vehicle has approached the entrance of the highway. As another example, when the processing unit receives a signal for checking whether the ETC card has been forgotten to be inserted, the processing unit determines that the vehicle has approached the entrance of the expressway. As yet another example,
The processing unit analyzes the image in front of the vehicle, and when detecting the entrance of the expressway in the image, determines that the entrance of the expressway is approached.

[Fourth embodiment]
Next, a fourth embodiment will be described. The processing unit of the system according to the fourth embodiment has a function of counting the number of people in an image as an image analysis function, and executes a process of recording or transmitting the number of people data in the image as a process corresponding to the analysis result.

The system according to the fourth embodiment makes it possible to count the number of people without using a special device for counting the number of people.

According to a modification of the fourth embodiment, the camera is attached to the bus, and the processing unit adopts the image inside the vehicle as the original image for counting the number of people. The processing unit displays the counting result on the display 16 (FIG. 2) to notify the crew. The crew can easily grasp the number of passengers (passengers) on board the vehicle. This function can be used in place of the tour roll call. When the number of passengers does not reach zero when all the passengers should get off, the crew can know that there are people left behind in the car. For example, it is possible to detect lingering persons such as babies who cannot express their own intentions and persons who cannot get off voluntarily such as disabled persons.

According to another modification of the fourth embodiment, the vehicle to which the camera is attached is a type of vehicle in which the number of passengers is predetermined, such as a sightseeing bus, a long-distance highway bus, etc., and the processing unit of this system It has a function to register the number of passengers decided. The processing unit compares the result of counting the number of people with the registered number of people, and if there is an excess or deficiency, warns the crew of the excess or deficiency.
This warning is given by generating a warning sound from the speaker 15 (FIG. 2) or displaying a warning message on the display 16 (FIG. 2). This prevents the train from departing with an insufficient number of passengers. In addition, it is possible to easily recognize that an unqualified person has boarded the vehicle, and to encourage the person to get off the vehicle.

According to yet another modification of the fourth embodiment, a camera (for example, a semi-spherical camera on the ceiling) is installed inside the bus, and the processing unit analyzes the images to determine the number of passengers, the number of standing passengers, and the number of standing passengers. It has a function to count the number of seated passengers. Furthermore, the processing unit acquires the current position information of the bus, the open/close signal of the bus door, the buzzer when getting off the bus, the touch of the IC card when getting on and off the bus, and the signal of the fare box. And it has a function to associate and record the image at the time of event occurrence. The server 30 (FIG. 3) accumulates this information in the data lake. Using this data lake, it will be possible to perform big data analysis on where people got on and where they got off, and how the seating situation was in the car during that time. Such data
It can be valuable information for consulting companies that carry out city planning and the like. Furthermore, the processing unit has a function of acquiring together information transmitted through an in-vehicle network such as CAN and vital information of the driver. These pieces of information can be used to identify causal relationships such as whether the abnormality is due to the driver, the driver, or the vehicle when some abnormality occurs in the vehicle.

According to yet another modified example of the fourth embodiment, the processing unit employs an image outside the vehicle as the original image for counting the number of people. The processing unit of this system has a function of recording the counting result as the number of people data. Based on the recorded number of people data, it is possible to easily extract places where many people were present. The processing unit has a function of recording the image that is the basis of the counting, using the counting result as a trigger. For example, it has a function of recording an image when the number of people exceeds a predetermined threshold.

[Fifth embodiment]
Next, a fifth embodiment will be described. The processing unit of the system according to the fifth embodiment has, as an image analysis function, a function of detecting the display content of the display device of the vehicle or the operation status of the operation device of the vehicle from the image inside the vehicle, and as a process corresponding to the analysis result, , and executes processing according to the display content of the display device or the operation status of the operation device.

These data can be acquired by the present system without retrofitting a device for acquiring data that is the basis of the display content of the display device or a device for acquiring data according to the operation status of the operation device. For example, these data can be acquired without connecting an aftermarket device to the OBD connector. In particular, even if the vehicle manufacturer prohibits connecting third-party products to the OBD connector, the system allows these data to be obtained. As a display device for vehicles, various instruments, warning lights (warning lamps) in the car
, a head-up display that projects an image on a windshield or the like.

In the modified example of the fifth embodiment, the processing unit of the system has a function of detecting the vehicle speed displayed on the speedometer as the display content of the display device. The processing unit has the function of pattern recognition of the position of the needle of the analog speedometer, character recognition of the numbers on the dial, and acquisition of vehicle speed data. Furthermore, the processing unit has a function of recognizing characters displayed on the digital speedometer to obtain vehicle speed data.
The acquired vehicle speed data can be used for various processes. An example of how the vehicle speed data is used will be described below.

The processing unit has a function of automatically detecting the speedometer by image recognition. This feature allows
The processing unit can detect speedometers in various vehicles, regardless of vehicle type. In addition, the processing unit has a function of preliminarily registering the position and shape in the image of the speedometer of the vehicle in which the processing unit is mounted. This allows the processing unit to easily detect the speedometer.

A typical stand-alone car navigation device uses G to acquire the current position.
Positional information by PS is used. If GPS signals cannot be received due to tunnels, etc.,
The vehicle's current position must be determined by some other method. As an example, the current position of the vehicle within the map is moved for 5 minutes at a constant speed of about 80% of the speed immediately before the GPS signal is lost.
A mode in which such processing is performed is called a tunnel mode. Tunnel mode is effective not only when entering a tunnel, but also when driving under an overpass. However, if you encounter a traffic jam while processing in tunnel mode, the method that assumes that you are traveling at a constant speed will result in an excessively large error in the current position, making it practically useless. By attaching an acceleration sensor to the vehicle and using the acceleration data from the acceleration sensor, the accuracy of the current position information in the tunnel mode can be improved. However, it is difficult for the user to attach the acceleration sensor, and it is expensive, so it is not realistic.

This system has a stand-alone car navigation function. The processing unit has a function of reading the vehicle speed from the display contents of the speedometer and updating the current position information of the vehicle using a map matching function based on the read speed information. As a result, the accuracy of the current position information of the own vehicle can be improved.

In a place where the road in the tunnel and the ground road run side by side, and where the elevated road and the ground road under the elevated road run side by side, the processing unit selects which mode based on whether or not the tunnel mode is activated. It has a function to select whether to match with the road. For example, when the tunnel mode is activated, the vehicle position is mapped on the road in the tunnel or on the ground road under the overpass, and when the tunnel mode is not activated, the vehicle position is mapped on the road in the tunnel. It has a function to map on the running ground road or elevated road.

In another modification of the fifth embodiment, the processing unit uses, as the display content of the display device to be detected,
It has a function to detect the remaining amount of fuel displayed on the fuel gauge. The processing unit detects the remaining amount of fuel by recognizing the image of the fuel gauge. When the processing unit detects that the remaining amount of fuel has fallen below a predetermined reference value, the processing unit notifies the driver of a message prompting him to refuel. For example, in conjunction with the current location information of your own vehicle, when you return from an outing to your home or company vehicle storage area,
If the remaining amount of fuel is less than a reference value, for example, 1/4 of the full tank, the processing unit issues a warning to prompt refueling. In addition, the car navigation device and this system are interlocked, and the processing unit compares the distance to the destination and the remaining amount of fuel, and refuels at the time of departure or during the route to the destination based on the comparison result. prompt. For example, a message prompting for refueling at a gas station near the current location is output near a gas station on the route.

The function of extracting the speedometer and fuel gauge from the image is realized by the artificial intelligence (AI) installed in the processing section of this system. The processing unit further has a function of registering in advance the positions and image information of the speedometer and fuel gauge in the image. If it is difficult for artificial intelligence to extract the speedometer and fuel gauge from the image, the processing unit uses the registered image information to extract the speedometer and fuel gauge from the image.

The processing unit reads the fuel gauge of the vehicle provided for the car sharing service, detects that the remaining amount is less than a predetermined reference value, and determines that if the fuel is refueled, the benefits of the car sharing service can be received. It has a function to notify the driver. For example, a message such as "If you refuel now, you can receive a discount equivalent to 15 minutes."

In still another modification of the fifth embodiment, the processing section has a function of determining whether or not the warning light in the instrument panel is lit. The warning lights to be determined whether or not they are lit include headlight lighting indicators, winker indicators, high beam indicators (high beam warning lights), parking brake indicators, ajar door warning lights, fuel warning lights, and the like. The processing means of this system can detect the state of the vehicle corresponding to the warning lights by determining whether or not these warning lights are lit.

In still another modification of the fifth embodiment, the processing unit has a function of detecting warning information displayed by the head-up display. The processing unit detects changes in the image displayed by the head-up display to detect the content of the warning, and executes predetermined processing according to the content of the warning. For example, a warning is issued by sound or voice. By changing the visual warning to auditory information and issuing the warning, the driver can easily notice the warning.

In still another modification of the fifth embodiment, the processing unit cooperates with outside brightness information and current position information of the vehicle to turn on the headlights even though the headlights should be turned on. It has a function to notify the driver to turn on the headlights when it detects that the lighting indicator is not lit. Conditions in which the headlights should be turned on include, for example, a condition in which the outside of the vehicle is darker than a certain level of brightness, a condition in which the current position of the vehicle is in a tunnel or a culvert, and the like. The processing unit has a function of notifying the driver to turn off the headlights when the headlight lighting indicator remains on even though the situation to turn on the headlights has been resolved. Because there are tunnels with bright interiors due to lighting,
If it is determined whether or not the situation should turn on the headlights based only on the brightness information outside the vehicle, a situation may occur in which it is erroneously determined that the situation is not for turning on the headlights even though the vehicle is in a tunnel. . The processing unit can avoid such an erroneous determination by determining whether the vehicle is in a tunnel based on the current position information of the vehicle. Furthermore, the processing unit
Not only the headlights, but also the side lights (small lights) should be turned on, but if it detects that the side lights are not lit, it will drive so that the side lights are turned on. It has a function to notify the person.

In still another modification of the fifth embodiment, the processing unit determines whether or not there is an oncoming vehicle from an image in front of the vehicle, and if it detects that the high beam indicator is lit despite the presence of an oncoming vehicle, the low beam is switched on. It has a function to warn the driver to change to Since the beam high/low switching operation differs depending on the vehicle type, when the driver is driving a vehicle of a vehicle type different from the vehicle type he or she normally uses, there is a tendency for the driver to turn on the low beam but remain on the high beam. In such a case, the function of issuing this warning is particularly effective.

In still another modification of the fifth embodiment, the processing unit detects blinking of the blinker from an image outside the vehicle when the outside of the vehicle is dark, and blinks inside the vehicle in synchronization with the blinking cycle of the blinker. It has a function to recognize the indicator as a blinker indicator. By doing so, the processing unit can easily find out the blinker indicator in the instrument panel.

In yet another modified example of the fifth embodiment, when the processing unit detects that the headlights and sidelights are on even though it is detected from the image inside the vehicle that no one is inside the vehicle, It has a function to notify the driver outside the vehicle and the vehicle manager that the lights have been forgotten. This notification is made, for example, by sending an e-mail to a pre-registered e-mail address. In addition, the processing unit has a function of notifying by e-mail or the like that the hazard lamps and room lights have been forgotten to be turned off. The lighting state of the hazard lamps is detected from the blinking state of the blinker indicator. The lighting state of the room light is detected from the brightness of the room light reflected in the image.

There are generally three types of room light settings: a constant lighting mode, a constant lighting mode, and a door open/close state interlocking mode (hereinafter simply referred to as an interlocking mode). In many vehicles the
These three types of modes are cyclically updated each time the room light on/off switch is pressed. Some vehicles have a function in which the room light turns on when the engine is stopped in a dark place (at night) and turns off automatically after a while (for example, after several seconds have passed). In some cases, it may be desirable to keep the interior lights on for a longer period of time to prepare for exiting the vehicle after the engine has been stopped. For example, after stopping the engine, it is desirable to turn on the room light during the period in which purchased items are sorted and taken out, portable terminals are taken out, and the like. In such a case, the driver switches the setting of the room light to the interlock mode or the constant lighting mode in order to prevent the room light from automatically turning off after a certain period of time has passed since the engine was stopped. However,
When the door is open, it is difficult to determine whether the room light is currently set to the constant lighting mode or the interlocking mode. Even though the room light is currently set to the constant lighting mode, if you misunderstand that it is in the interlocking mode and get out of the car, the room light will remain on. By using this system, the driver or manager can notice that the room light has been left on and turn off the room light, thereby preventing the vehicle's battery from running out due to the room light being on for a long period of time. A vehicle with a structure in which it is not possible to tell which mode the room light is currently set to just by looking at the operation switch. This feature of the system is particularly useful in.

When the room light setting is in the linked mode, the room light will remain on if any door is open or the door is ajar. A driver or manager who is notified that the room light is on can notice that the door may be ajar or the door on the side where it is difficult to see may be open.

Some vehicles have a function that keeps the room light on until a certain period of time (turn-off delay time) elapses after the door is closed when the room light is set to interlock mode. In such a vehicle, the processing unit of this system has a function of not determining whether or not the lights have been forgotten until the light-off delay time has elapsed after detecting that the door was closed. The processing unit has a function of setting a turn-off delay time based on an input from the user.

In addition, when the processing unit detects that the room light is still on even after a certain period of time has elapsed since no one was inside the vehicle, it sends a signal to the vehicle controller to turn off the room light. have.

In still another modification of the fifth embodiment, when the processing unit detects that the door of the vehicle has been opened while the headlights, side lights, hazard lights, room lights, etc. are on or blinking, It has a function to issue an alarm notifying that the light remains on.
As a result, the driver can notice that he has forgotten to turn off the light before he leaves the vehicle, and can prevent him from forgetting to turn off the light.

In still another modification of the fifth embodiment, when the processing unit detects that the parking brake is being applied based on the lighting of the parking brake indicator, the driver is dozed off from the image of the driver. It has a function that does not issue an alarm even if it detects that In addition, when the parking brake is applied, even if it detects that the driver is doing something that is considered dangerous while driving, such as looking at the mobile terminal,
It has a function that does not issue an alarm.

In yet another modification of the fifth embodiment, when the processing unit detects that the warning light indicating the need for early inspection or repair remains lit for a certain period of time, the vehicle is It has a function to notify dealers and registered repairers of the vehicle status. for example,
Users who are not interested in automobiles often leave the vehicle unattended without noticing the lighting of the warning light when the vehicle can be driven even if the warning light is lit. For example, if a warning light for the engine itself lights up, the vehicle will normally be unable to run, but even if the warning light that indicates an abnormality in the engine oil is lit, the vehicle will still be able to run. Further, in the case of an O2 sensor abnormality, the fuel consumption becomes worse, but the vehicle itself can be driven. If you continue to run the vehicle without checking and repairing it, even if it is possible to run the vehicle, minor failures may become more severe. For dealers and repairers, the notification of the lighting of the warning light is useful information for carrying out sales activities for inspection and repair to the owner of the vehicle on which the warning light is lit. again,
The owner of the vehicle can prevent the failure from becoming serious by performing inspection and maintenance at an early stage.

In still another modification of the fifth embodiment, the processing unit has a function of detecting the temperature displayed in the temperature display area within the instrument panel that displays the temperature measurement value measured by the temperature sensor that measures the temperature outside the vehicle. have. As a function of image analysis, the processing unit has a function of reading the temperature by character recognition from the image displayed in the temperature display area of the instrument panel. As a process corresponding to the analysis result, the processing unit executes a process of transmitting the temperature information to an external server via the communication device and the communication network.

For example, in order to obtain data on pollen dispersal, sensors are installed in the homes of general users, and by collecting measurement data from the sensors, detailed predictions of pollen dispersal conditions and distribution of pollen dispersal information are provided. Provided by Weathernews, etc. In addition, weather forecasts incorporate outdoor photographs posted by general users and images collected from outdoor live cameras installed in ordinary homes. Also, a service for distributing these images (for example, Weather News SOLiVE24, etc.) is provided. A system that provides these weather information services can use many vehicles as temperature measurement probes by collecting outside air temperature information measured by many vehicles. When many vehicles transmitting temperature information are distributed over a wide area, the temperature information collected from many vehicles can be useful as spot information. For example, it is possible to issue a spot heatstroke warning that cannot be expressed by general weather information.

A company that provides such a weather information service can obtain the effect of being able to collect temperature information over a wide range without installing a temperature sensor in-house. The company that receives the temperature information should provide free or discounted paid weather information services provided by the company in return for the owner of the vehicle that provided the temperature information.

The processing unit further performs character recognition not only on the temperature information read from the temperature display area in the instrument panel, but also on the numbers displayed on the temperature display signs on the side of the road that are reflected in the images acquired by the camera. It has a function to acquire outside temperature information. Detection of the temperature display sign is performed by recognizing characters such as "current temperature" in the sign. The acquired outside air temperature information is transmitted to an external server by the processing unit in the same manner as the temperature information acquired from the temperature display area in the instrument panel.

In still another modified example of the fifth embodiment, the processing unit has a function of detecting the position of the driver's foot with respect to the accelerator pedal and the brake pedal as the function of detecting the operating state of the operating device by image analysis. As a process according to the operation status of the operating device, the processing unit notifies the driver whether the driver's foot is on the accelerator pedal or on the brake pedal so that the driver can easily recognize. Execute the processing to be performed. For example, when the foot is on the accelerator pedal and when the foot is on the brake pedal, the driver is notified by changing the light emission color of the light emitting element.

In still another modification of the fifth embodiment, a vehicle having a function of detecting whether the foot is resting on the accelerator pedal or the brake pedal and displaying the detection result as visual information on a display is provided. This system is installed. The processing unit of this system has the function of analyzing the image displayed on the display and detecting the position of the driver's foot relative to the accelerator and brake pedals.

In still another modification of the fifth embodiment, the processing unit issues an alarm by sound or the like when the foot is at the position of the accelerator pedal even though an obstacle is detected in front by image analysis. have a function. By the driver noticing the warning, it is possible to suppress the occurrence of erroneous depression of the accelerator pedal and the brake pedal.

In still another modification of the fifth embodiment, the processing unit has a function of detecting the rotation angle of the steering wheel (steering wheel) as a function of detecting the operation status of the operation device by image analysis. The processing unit, as a process corresponding to the operation state of the operation device, executes a process of obtaining the traveling direction and the traveling distance of the vehicle based on the rotation angle of the steering wheel and the vehicle speed. The processing unit
The processing result is given to the stand-alone car navigation device. The stand-alone car navigation device can calculate the travel route of the vehicle based on the information about the direction of travel and travel distance given from this system even in situations where GPS signals cannot be received.

In yet another modification of the fifth embodiment, the processing unit has a function of performing image analysis, whether the vehicle is in automatic driving (including automatic driving levels 1 to 5) or manual driving (automatic driving level 0). It has a function to detect This detection is performed by the processing unit recognizing from an image the display state of an indicator that indicates whether or not various functions of automatic driving displayed on the instrument panel are in operation. As a process corresponding to the analysis result, when detecting that the automatic driving function has been canceled, the processing unit executes a process of notifying the driver that the automatic driving function has been canceled. In addition, even though the automatic driving function has been deactivated, if it detects that the driver is not holding the steering wheel, the processing unit issues a warning to the driver to operate the steering wheel. do. Furthermore, triggered by the cancellation of the automatic driving function and the fact that the driver does not put his hand on the steering wheel, the processing unit records the image captured by the camera as an event record separately from the constant recording. have a function. The recorded image can be used as evidence that the driver's operation was inappropriate.

The processing unit has a function of disabling the function of the radar detector when the automatic driving function is effective in the vehicle equipped with the radar detector. This eliminates unnecessary alarms from radar detectors. When the automatic driving state is disabled, the processing unit enables the function of the radar detector.

Whether the vehicle is driving automatically or manually, and what level (mode) of automatic driving Judging from the direction in which the head is facing.

[Sixth embodiment]
Next, a sixth embodiment will be described. The processing unit of the system according to the sixth embodiment has a function of recognizing the characters displayed on the sticker attached to the vehicle from the image inside the vehicle as an image analysis function. A process of calling attention is executed according to the information indicated by the character.

By being alerted, the driver, manager, or owner of the vehicle can pay attention to the character information on the sticker attached to the vehicle and perform appropriate processing according to the character information.

In the modification of the sixth embodiment, the processing unit has a function of recognizing characters displayed on the inspection mark (car inspection sticker). The date on which the valid period of the vehicle inspection expires is displayed on the inside surface of the inspection mark. The processing unit reads information on the expiration date of the validity period of the vehicle inspection by character recognition, and when the period from the current time to the expiration date of the validity period of the vehicle inspection (the expiration date of the vehicle inspection) becomes shorter than a predetermined period, the vehicle inspection is performed. Reminder to receive. For example, the processing unit alerts by issuing a voice saying "The expiration date of the vehicle inspection valid period is approaching."
If the period until the expiration date of the validity period of the vehicle inspection becomes shorter, for example, the processing unit issues a warning with a voice saying, "Please undergo a vehicle inspection."

In another modification of the sixth embodiment, the processing unit has the function of recognizing the characters displayed on the maintenance check sticker. The due date for the next periodic inspection is displayed on the maintenance check sticker in terms of date and time. The processing unit reads the date information indicating the deadline of the periodic inspection by character recognition, and if the period from the current time to the deadline of the periodic inspection becomes shorter than a predetermined period, alerts the user to undergo the periodic inspection. For example, the processing unit alerts the user by issuing a voice saying, "The periodical inspection deadline is approaching." If the period until the periodical inspection deadline becomes shorter, for example, the processing unit issues a warning with a voice saying, "Please undergo a periodical inspection."

In still another modification of the sixth embodiment, the processing unit has, as the image analysis function, a function of determining whether or not an inspection mark is included in an image captured by a camera. As a process corresponding to the analysis result, if the inspection mark is not captured in the image, the processing unit executes a process of calling attention to adjust the mounting position and posture of the camera so that the inspection mark is captured. do.
For example, the processing unit may issue a voice saying "The inspection mark is not captured. Please adjust the position and posture of the camera so that the inspection mark is captured." This makes it possible to easily adjust the position and orientation of the camera when it is installed. The processing unit includes a display device in the vehicle,
For example, it has a function to read the contents displayed on the speedometer, fuel gauge, etc., and has a function to call attention so that the display equipment in the vehicle is reflected in the image.

In still another modification of the sixth embodiment, the processing unit, as a function of image analysis, determines whether stickers other than those permitted by law are affixed to the windshield, the side windows of the driver's seat and the passenger's seat. It has a function to judge whether or not As a process corresponding to the analysis result, the processing unit executes a process of notifying the driver or the like to remove the sticker when it is determined that a sticker other than the sticker permitted by law is attached.

[Seventh embodiment]
Next, a seventh embodiment will be described. The processing unit of the system according to the seventh embodiment has, as an image analysis function, the function of analyzing the image of the area in which the front passenger seat is reflected, and as processing corresponding to the analysis result, executes processing according to the situation of the front passenger seat. do.

For example, appropriate processing can be executed depending on whether a person is in the front passenger seat or not, or whether an object is placed in the front passenger seat.

In a modification of the seventh embodiment, a drive recorder 1 (FIG. 1) is installed in a vehicle having a function of issuing a warning when a person sitting in the front passenger seat is not wearing a seatbelt. As a function of image analysis, the processing unit has a function of determining whether or not a person is sitting in the passenger seat from the image of the area in which the passenger seat is reflected. As a process corresponding to the analysis result, when the processing unit detects that no one is sitting in the front passenger seat, it disables the function of issuing a warning that the front passenger seat belt is not fastened. The processing unit further has, as an image analysis function, a function of determining whether or not an object is placed on the front passenger seat. As a process corresponding to the analysis result, the processing unit executes a process of invalidating the function of issuing a warning that the passenger's seat belt is not fastened when an object is placed on the passenger's seat.

A vehicle may erroneously determine that a person is in the vehicle when an object equivalent to the weight of a person is placed on the front passenger seat, and issue a warning that the seat belt is not fastened. When this system is installed in a vehicle, the vehicle will not issue an unfastened seatbelt warning if an object is placed in the front passenger seat. Therefore, the driver will not be misled by the warning. In particular, this function is effective in vehicles with specifications that continue to sound a warning buzzer when a heavy object is placed on the front passenger seat. To prevent the vehicle from issuing an unfastened seatbelt warning, the processing unit sends a signal to the vehicle's controller through the in-vehicle network (CAN) to disable the issuing of the warning.

In a vehicle having a similar function for the rear seats, the processing means cancels the seatbelt non-fastening warning function when it detects that an object is placed on the rear seats.

[Eighth embodiment]
Next, an eighth embodiment will be described. The processing unit of the system according to the eighth embodiment has, as an image analysis function, a function of comparing an image of the interior of the vehicle at the present time with an image of the interior of the vehicle in the past. Then, when the degree of disagreement with the image of the inside of the vehicle in the past exceeds a predetermined level, a notification process is executed.

By receiving the notification, the owner or manager of the vehicle can know that the situation inside the vehicle has changed beyond a certain level.

In the modified example of the eighth embodiment, the processing unit uses, as the image of the past interior of the vehicle, an image of a state in which no objects other than those previously attached to the vehicle were brought into the vehicle, and this image and the current image are used. It has a function of determining whether or not there is an object brought into the vehicle by comparing it with the image of the interior of the vehicle.

The processing unit further has a function of calculating the amount of time that an object brought into the vehicle remains in the vehicle by comparing images taken at a plurality of times in the past with the current image. The processing unit has a function of notifying the user of an object that has been left in the vehicle for a long time.

Next, specific processing of the processing unit will be described. First, with the vehicle unloaded, preferably with only the driver on board, the initial setting button of the system is pressed to drive the vehicle. The processing unit recognizes a portion in the image having an optical flow corresponding to the vehicle speed as a window. Other moving parts are recognized as the driver. A part other than the window and the part recognized as the driver is recognized as an area where an object may be placed, and is recorded as a no-luggage state image (no-luggage state video) in which no luggage is loaded on the vehicle. The processing up to this point completes the initial setting processing.

After that, when the engine is started, when the engine is stopped, or the like, the processing unit finds the difference between the no-luggage state image and the current image, and identifies the presence of an object brought into the vehicle. If the object identified as being brought into the vehicle does not change position for a certain period of time, the processing unit issues a warning. This will
To allow a driver or the like to recognize an object that has been left in the vehicle for a long time. This function is especially useful for people whose car interiors are quickly filled with garbage.

[Ninth embodiment]
Next, a ninth embodiment will be described. The processing unit of the system according to the ninth embodiment has, as an image analysis function, a function of determining whether or not a pre-registered object appears in the image inside the vehicle. When an object is captured in the image inside the vehicle, an alarm is issued.

A driver or the like can be alerted to the presence of pre-registered items in the vehicle.

In the modified example of the ninth embodiment, the processing unit has a function of registering an image of an item (a specific bag, wallet, etc.) to be checked for lost items. When the processing unit detects that the registered object is reflected in the image while no one is inside the vehicle, the processing unit issues an alarm or notifies of the forgotten item. As an example of issuing an alarm, for example, a buzzer or the like installed in the vehicle is sounded. As an example of the notification of a forgotten item, for example, the driver's smart phone, mobile phone, or the like is notified wirelessly by e-mail or the like.

In another modified example of the ninth embodiment, the processing unit has a function of issuing an alarm when it detects an item left behind in the vehicle and also detects a motion of opening the door. By the warning, the driver or the like can notice the forgotten item before leaving the vehicle. The processing unit further has a function of notifying the driver or the like of the forgotten item by e-mail or the like when it detects an item left behind in the vehicle and detects the motion of closing the door. A driver or the like can notice that there is an item left behind after leaving the vehicle. The detection of the door opening/closing movement is performed by detecting optical flow, for example.

In still another modification of the ninth embodiment, the processing unit has a function of detecting other movements inside the vehicle by detecting optical flow. For example, the processing unit has, as an image analysis function, a function of detecting opening/closing movements of doors, opening/closing movements of windows, blinker operations, steering wheel operations, sun visor raising/lowering operations, and the like. This system is
This information can be obtained without connecting to a connector (such as an OBD connector) of the vehicle's in-vehicle network.

When the sun visor is down, especially when a 360-degree camera attached to the windshield is used to take an image of the driver's side, the sun visor becomes an obstacle and an image in that direction cannot be obtained at all. Therefore, when the processing unit detects that the sun visor is down even though the brightness in front of the vehicle is such that the sun visor is not necessary, a warning is issued. Whether or not the sun visor is down is determined by the processing unit based on the ratio of the area of the sun visor to the image.

[Tenth embodiment]
Next, a tenth embodiment will be described. The processing unit of the system according to the tenth embodiment has, as an image analysis function, a function of capturing an image of an object brought close to the camera by the user and recognizing the image. Execute the processing according to the information described in .

By bringing an object that the user is carrying close to the camera, the user can execute processing according to the information written on the object.

In a modification of the tenth embodiment, the processing unit has a function of performing image recognition of the driver's driver's license. The processing unit performs character recognition of the valid date written on the driver's license, and if the period from the current time to the valid date is shorter than a predetermined period, the processor notifies the user that the driver's license is about to be renewed. let me know.

In another modification of the tenth embodiment, the processing unit has a function of recognizing images of gas station receipts. The processing unit performs character recognition on the date, amount of fuel, and amount of money displayed on the receipt, and records this information into a database.

In still another modification of the tenth embodiment, the processing unit has a function of performing image recognition of an identification card (for example, driver's license, employee ID card, resident registry card, etc.). The processing unit performs character recognition of the name, etc., written on the identification card, and determines whether or not the person is a pre-registered person. If the person on the identification card is not registered, the processing unit performs processing to prohibit the engine from starting. Thereby, the person who can drive the vehicle can be limited to the person registered in advance. Furthermore, the processing unit records the name of the person on the identification card. The recorded information can be used to identify the driver when managing vehicle operations.

In still another modification of the tenth embodiment, the processing unit has, as image recognition functions, a function of recognizing an object that the user has brought close to the camera outside the vehicle, and a function of recognizing an object that the user has brought close to the camera inside the vehicle. have a function. For example, the processing unit compares with a normal image,
When an object that is not in a normal image occupies a wider area than the specified range within the angle of view of the camera,
Recognize that there is an object brought close to the camera. In particular, when the processing unit recognizes that the user is holding an object that occupies a wider area than a predetermined range within the angle of view of the camera, the processing unit determines that the user has brought the object closer to the camera.

[11th embodiment]
Next, an eleventh embodiment will be described. The processing unit of the system according to the eleventh embodiment has, as an image analysis function, a function of performing image analysis of an object reflected in the front image, and as processing corresponding to the analysis result, information to be noted regarding vehicle travel is processed. to the driver.

By calling the driver's attention according to the situation ahead, it is possible to suppress the occurrence of an accident that may occur in the future due to the vehicle traveling forward.

In the modified example of the eleventh embodiment, the processing unit has a function of calculating at least one of the inter-vehicle distance and the relative speed to the preceding vehicle as the image analysis function. The processing unit detects a situation in which the risk of collision is high based on the inter-vehicle distance or the relative speed, and notifies the driver of the high risk of collision by issuing an alarm or the like. The inter-vehicle distance is calculated from information such as the size of the forward vehicle in the image, such as the size of the license plate, the size of the tread (tire gap), and the width of the tail lamp. The relative speed is calculated from the rate of change in inter-vehicle distance.

In another modified example of the eleventh embodiment, the processing unit has a function of detecting the color of the traffic signal in front as the image analysis function. The processing unit notifies the driver to stop the vehicle when the traffic light is lit in red.

In still another modification of the eleventh embodiment, the processing unit has a function of recognizing a road sign ahead as an image analysis function. The processor calls the driver's attention according to the information indicated by the road sign. For example, if the road sign indicates a speed limit, the processing unit acquires the speed limit at the current location through character recognition and notifies the driver of the speed limit. If the road sign indicates that entry is prohibited, the processing unit issues an alarm to notify the driver when the driver attempts to enter a road on which entry is prohibited. When the processing unit detects that the vehicle is about to run in violation of the traffic rules indicated by the road sign or is running in violation of the traffic rules, the processing unit notifies the driver of the violation of the rules. As a result, the occurrence of traffic rule violations can be prevented or suppressed. By recognizing the character strings of road signs, the processing unit warns the driver of excessive speed when it detects a sign indicating that the section in which the vehicle is currently traveling is a section where an automatic speed control device is installed. Notice.

In still another modification of the eleventh embodiment, the processing unit has, as the image analysis function, a function of detecting traffic congestion conditions in lanes adjacent to the lane in which the vehicle is traveling. Whether or not there is traffic congestion is determined by the processing unit based on the average traveling speed of vehicles in adjacent lanes and the inter-vehicle distance. When the traveling speed of vehicles in adjacent lanes is equal to or less than a predetermined reference value, the distance between vehicles is equal to or less than a predetermined reference value, and a predetermined number of vehicles are lined up, the processing unit determines that traffic congestion is occurring. do. As a process corresponding to the analysis result, when the processing unit detects that a traffic jam is occurring in a lane adjacent to the lane in which the vehicle is traveling, the processing unit instructs the driver to be careful not to jump out from between vehicles in the traffic jam. Execute the processing to call attention. The lanes adjacent to the own vehicle traveling lane include oncoming lanes and adjacent lanes traveling in the same direction as the own vehicle. In particular, in places where people gather, such as shopping malls, theme parks, and train stations, and in places where vehicles come and go, such as gas stations, there is a high possibility that the lanes in the same direction adjacent to the lane in which the vehicle is traveling are also congested. In such a case, it is effective to detect congestion in the adjacent lane traveling in the same direction as the own vehicle. As a result, it is possible to suppress the occurrence of accidents due to jumping out. The processing unit determines that the running speed of the own vehicle is less than a predetermined speed, for example, 30k/h.
When the number is less than m, a process of not emitting a warning sound for caution against jumping out is executed. This makes it possible to avoid unnecessary warning sounds. In addition, the processing unit has a function of detecting a situation in which there is a high risk of jumping out from the front image and calling attention. For example, when there are vehicles parked continuously on the side of the road, when there are vehicles parked in a parking lot where a parking meter is installed, etc., the processing unit calls attention to jumping out.

For example, some car navigation devices are equipped with a function that calls the driver's attention while driving based on traffic congestion probe information, general road high-speed travel information, and information on surrounding facilities (residential areas, schools, etc.). By using this system, even in places where traffic congestion probe information cannot be obtained, it is possible to call attention due to traffic congestion.

In still another modified example of the eleventh embodiment, the processing unit has a function of detecting animals (dogs, cats, raccoon dogs, wild boars, etc.) and people around the vehicle before starting the vehicle as an image analysis function. have When the processing unit detects an animal or person around the vehicle, the processing unit notifies the driver of the presence of the animal or the like around the vehicle. As a result, it is possible to suppress the occurrence of an accident immediately after starting the vehicle.

In still another modification of the eleventh embodiment, the processing unit has, as the image analysis function, a function of detecting that the vehicle is slowly moving forward from the flow of surrounding scenery. When the processing unit detects that the vehicle is moving slowly forward, it issues an alarm. For example, if the vehicle starts to move little by little due to a creep phenomenon due to the loosening of the force applied to the brake pedal after the vehicle has stopped temporarily, the driver is notified that the vehicle is moving and that there is a risk of a rear-end collision. be able to. The processing unit determines that the vehicle is slowly moving forward when the vehicle is moving at a speed that allows the vehicle to move forward due to a creep phenomenon. The flow speed of the surrounding scenery is determined by the processing unit detecting optical flow.

In still another modified example of the eleventh embodiment, the processing unit determines whether or not the preceding vehicle is a large vehicle, such as a bus, and the loading platform is an aluminum box-shaped truck or the like, as a function of image analysis. have a function. When the processing unit detects that the preceding vehicle is a large vehicle and approaches a traffic light,
Inform the driver of the presence of traffic lights. For example, a voice such as "Caution for signal" is emitted. The presence or absence of a traffic light is registered in the processing unit in advance as POI information. For example, when a large vehicle is running ahead, it becomes difficult to see traffic signals in the direction of travel. It is preferable to anticipate the existence of traffic signals by looking at traffic signals and pedestrian traffic lights in the opposite lane, and to keep a sufficient inter-vehicle distance so that traffic lights in the direction of travel can be easily seen. However, if such attention is neglected, there is a danger of overlooking traffic signals in the direction of travel. Also, in rural areas, traffic signals for opposite lanes and pedestrian traffic signals may not be installed. By notifying the presence of a traffic signal when the preceding vehicle is a large vehicle, it is possible to reduce the risk of the driver overlooking the traffic signal. In addition to the function of directly notifying the presence of a traffic signal, the processing unit also has a function of calling attention to drive so as to reduce the risk of overlooking a traffic signal. For example, the processing unit
Please keep enough following distance. ” is emitted. For example, when the distance to the intersection where the traffic light is installed is longer than the safe stopping distance of the own vehicle, the attention is called. The safe stopping distance of the own vehicle varies depending on the speed of the own vehicle and the weather (rain or not). The processing unit sets an appropriate distance as the safe stopping distance based on the speed of the vehicle and the weather (rain or not). for example,
If the safe stopping distance in fine weather is 50 m, the processing unit determines that the current weather is fine, the distance to the preceding large vehicle is less than 50 m, and the direction of travel is outside what is called a car navigation device's urban area. When there is an intersection with a traffic light within 100m, alert the driver.

In still another modification of the eleventh embodiment, the processing unit, as a function of image analysis, determines whether or not the road ahead is sloped, and if it is sloped, the degree of inclination It has a function to detect Depending on the degree of inclination of the slope, the processing unit prompts attention such as excessive speed, slow speed, setting to hill climb mode, and the like.

In still another modification of the eleventh embodiment, the vehicle is a vehicle that handles packages, such as a forklift or a truck, and the processing unit extracts a character string, barcode, or the like displayed on the package from the captured image. It has the function of recognizing the signs and determining whether or not the cargo should be handled by the own vehicle. Information indicated by a label attached to a package to be handled by the own vehicle is registered in the processing unit in advance. When the processing unit detects that the own vehicle is loading a load that should not be handled, the processing unit issues an alarm.

In still another modification of the eleventh embodiment, the processing unit has a function of detecting the window area in the image from the optical flow. Furthermore, the processing unit has a function of determining whether the vehicle has entered a tunnel, whether the vehicle has exited the tunnel, or whether the vehicle is traveling in the tunnel, from the optical flow. In addition, the processing unit detects from the optical flow that the vehicle is under the elevated (
It has a function to determine whether the vehicle is traveling under a parallel road). Furthermore, the processing unit has a function of determining whether or not there is a guardrail and whether or not caution is required in the ditches from the optical flow.

[Twelfth embodiment]
Next, a twelfth embodiment will be described. The processing unit of the system according to the twelfth embodiment has, as a function of image analysis, a function of recognizing characters written on license plates of other vehicles reflected in the image outside the vehicle, and as processing corresponding to the analysis results, At least one of a process of recording license plate information identified by character recognition and a process of transmitting it via a network is executed.

It is possible to identify the license plate information of other vehicles that are running or parked near a vehicle equipped with an on-board device that has the function of the processing unit of this system. The processing unit transmits the current position information of the vehicle equipped with the vehicle-mounted device and the date and time information when the license plate was imaged in association with the specified license plate information. License plate information acquired along with location information and date and time information is used together with license plate information acquired by the automatic number plate reader (N system) installed on the road to know the driving routes of many vehicles. Useful information. In this way, the vehicle number and GP
By linking the S information, it is possible to construct a system that can be called a wide area mobile N system.

In a modification of the twelfth embodiment, a center system is constructed that collects position information, date and time information, and license plate information in association with each other from the present system installed in each of a plurality of vehicles via a network. This center system has a function of extracting position information and date/time information of a vehicle having specific license plate information and identifying the travel route of the vehicle.

[Thirteenth embodiment]
Next, a thirteenth embodiment will be described. The processing unit of the system according to the thirteenth embodiment has, as an image analysis function, a function of determining whether or not the edge of the window glass exists in the window portion of the vehicle. When the edge of the window glass is detected, the notification process is executed.

The presence of the edge of the windowpane in the window portion means that the windowpane has been left unclosed. A driver or the like can notice that he or she has forgotten to close the window glass by receiving the notification from this system. The processing unit detects discontinuity in the image due to the difference in refractive index between the window glass and the atmosphere,
Detect the edge of the window glass. As notification processing, the processing unit performs processing such as wireless notification to the driver's portable terminal or the like, notification by e-mail, and the like. In addition, the processing unit processes notification by sounding a buzzer attached to the vehicle. When a smoke film is attached to the window of a vehicle in which the processing unit is installed, the processing unit detects the window based on the difference in brightness between the window open area and the window glass smoke area. detect that is open.

[14th embodiment]
Next, a 14th embodiment will be described. The processing unit of the system according to the fourteenth embodiment has, as an image analysis function, a function to determine whether it is raining by detecting the presence or absence of wiper operation or the presence or absence of an increase or decrease in raindrops adhering to the windshield. As a process corresponding to the analysis result, when it is determined that it is raining, a process of transmitting information notifying that it is raining from the communication device is executed.

By receiving the information transmitted from this system, it is possible to know whether it is raining in the area where the vehicle is traveling. The processing unit has a function of notifying an external device such as a server that it is raining and notifying the current position information of the vehicle. The external device can create a map of areas where it is raining by collecting information from multiple vehicles. Furthermore, the processing unit has a function of notifying that it is raining and notifying the current time information. As a result, the external device can acquire information about temporal changes in areas where it is raining.

In a modification of the fourteenth embodiment, the external device is a server of an organization that provides weather information. The processing unit has a function of uploading to the server an image at a timing when the wiper is not reflected, for example, at a timing when the wiper is lowered. The server has a function of obtaining information on rainfall, such as the amount of rainfall, from the uploaded images. The server has a function of utilizing data such as rainfall collected from vehicles as big data in various application fields. For example, the server is used for prediction of localized torrential rain. The processing unit determines that the rain has stopped when there is no increase or decrease in the number (number) of the raindrops adhering to the windshield. When the processing unit determines that the rain has stopped, it notifies the server that the rain has stopped. The server can obtain more detailed information about rainfall.

[15th embodiment]
Next, a fifteenth embodiment will be described. The processing unit of the system according to the fifteenth embodiment has, as an image analysis function, a function of judging the weather from the sky conditions reflected in the image, and as a process corresponding to the analysis results, information on the weather is transmitted from the communication device. Execute the processing to be performed.

Weather-related information collected from multiple vehicles can be used as weather-related big data. The server that has collected these pieces of information determines whether the weather is clear, sunny, cloudy, etc., based on the color of the sky reflected in the image, the presence or absence of clouds, and the like. Furthermore, the processing unit has a function of detecting the strength of the morning sun and the afternoon sun from the height of the sun and the brightness of the sky, and transmitting information that the morning sun and the afternoon sun are strong. The processing unit transmits the current location information of the vehicle together with the weather information.
The server that collects this information has a function of providing the information that the morning sun and afternoon sun are strong and the current position information of the vehicle to the drivers of other vehicles heading to that point. Drivers who have been provided with this information can notice that they need to be careful in the morning sun or late afternoon sun.

[16th embodiment]
Next, a 16th embodiment will be described. The processing unit of the system according to the sixteenth embodiment has a function of detecting sound information from a microphone mounted on the vehicle, associates the detection result of the sound information with the analysis result of the image analysis function, and obtains the result of the association. Execute the corresponding process.

Compared to executing processing based only on sound information detection results or image analysis results,
By associating the two, more appropriate processing can be performed.

In the modification of the sixteenth embodiment, the processing unit has, as an image analysis function, a function of detecting a change in the state of the display unit of equipment having a display unit installed in the vehicle, and processing corresponding to the analysis result. , when the detected sound information and the change in the state of the display unit are synchronized, a process of associating and storing the detected sound information and the device in which the state of the display unit has changed is executed.

In another modification of the sixteenth embodiment, when the processing unit detects sound information from the microphone,
It has a function of identifying a device that emits a sound based on information in which sound information and changes in the state of the display unit are associated and stored. For example, the processing unit has a voice recognition function, determines whether the sound information from the microphone is emitted from a specific device, and if it is emitted from a specific device, recognizes the voice. Cancel function. This makes it possible to avoid executing unnecessary speech recognition processing. Further, when the processing unit determines that the sound information from the microphone is emitted from a specific device, the processing unit adds information capable of distinguishing the region of the device that generated the sound from other regions in the image of the vehicle interior. It has a function to record as For example, an image of a rectangle or an arbitrary shape of a closed curve surrounding an area in which the device that generated the sound is captured is superimposed on the original image and recorded.
Alternatively, a translucent geometric pattern is superimposed and recorded on the area where the device that generated the sound is shown.

In still another modification of the sixteenth embodiment, the processing unit has, as the image analysis function, a function of determining whether or not an emergency vehicle such as an ambulance, fire engine, or police car is captured in the image. When the processing unit detects the sound of the siren of the emergency vehicle from the microphone and also detects the emergency vehicle in the image, the processing unit associates both and determines that the emergency vehicle is in the vicinity of the own vehicle. The processing unit
If it determines that there is an emergency vehicle, it will notify the driver with a message to alert the driver.

[17th embodiment]
Next, a 17th embodiment will be described. In the 17th embodiment, there are a plurality of cameras installed in the vehicle, and the processing unit has a function of detecting the positions of other cameras reflected in the image as an image analysis function, and processes corresponding to the analysis results. , the process of specifying the positional relationship of a plurality of cameras is executed.

It is possible to obtain an image of a wide area inside the vehicle, which is difficult to capture with a single camera. Once the positional relationship of the cameras is specified, the positional relationship of the images captured by each camera can be specified.

According to the modification of the seventeenth embodiment, the processing unit has a function of identifying the model of another camera included in the image and identifying the angle of view from the model. The processing unit further has a function of synthesizing images captured by a plurality of cameras. By synthesizing the images, it is possible to grasp the situation inside the vehicle more intuitively than when displaying the individual images of the cameras.

According to another modified example of the seventeenth embodiment, at least part of the functions of the processing section are realized by a viewer such as a personal computer. The processing unit displays a marker such as a camera icon in the image to be displayed so that the position of the camera can be identified. It is difficult to specify the position of the camera in the vehicle from the image captured by the camera (self). The processing unit detects the position of the camera in the vehicle from images of other cameras in which the camera is captured. For example, when synthesizing and displaying images captured by three cameras, the processing unit displays the positions of the three cameras by icons or the like in the synthesized image. Furthermore, the icon is made different according to the model or angle of view of the camera.

According to still another modified example of the seventeenth embodiment, the processing unit displays the image captured by each camera, and displays the position of the camera and the angle of view of the camera (the image captured by the camera) in the image. It has a function to display the area where the For example, depending on the function of a 360-degree spherical camera, a 360-degree semispherical camera, a wide-angle camera, or the like, the angle of view of the camera is displayed in the image. A user who sees the image can intuitively understand whether each camera is capturing an image of the area.

[Eighteenth embodiment]
Next, an eighteenth embodiment will be described. In the 18th embodiment, the camera mounted on the vehicle is a 360-degree camera having an image circle smaller than the length of the diagonal line of the imaging surface, and the processing unit calculates the feature amount of the image as a function of image analysis. It has a function of deforming the basic pattern for the image according to the position within the image circle.

The processing unit has a function of calculating the feature amount of the image using the basic pattern without correcting the distortion of the image within the image circle. Thereby, the load on the processing unit can be reduced. A basic pattern is defined for an image obtained by imaging with a standard lens (a general wide-angle lens). Projection methods for obtaining an image circle include, for example, orthogonal projection, equidistant projection, stereographic projection, equisolid angle projection, etc. The processing unit transforms the basic pattern according to these projection methods. As a camera lens, a circular fisheye lens is used to obtain an image circle. The imaging surface of the camera is rectangular, and the diameter of the image circle is shorter than the long side and equal to or shorter than the short side.

A template pattern of an image to be detected is registered in the processing unit in advance as a basic pattern. The processing unit deforms the template pattern according to its position within the image circle. In addition, the processing unit has, for example, a function of recognizing an image using a Har-like feature, and a function of deforming each of a plurality of rectangular patterns for obtaining a Har-like feature quantity according to the position within the image circle. .

In the modification of the 18th embodiment, the processing unit has a function of limiting a region in which a specific object to be detected is included in the image circle when performing image recognition. Image recognition is performed on the inside. This function is realized by the drive recorder. In general, the recommended installation position of the drive recorder is determined in advance, so the area in the image circle where the driver, the traffic light, and the people in the back seat are reflected is almost fixed. . Therefore, depending on the person or object to be detected, the area where the image analysis is performed to detect the person or object is limited to the area in which the person or object is captured. For example, when analyzing an image in front of the vehicle, the processing unit analyzes an area in which the windshield is reflected.

In another modified example of the eighteenth embodiment, the processing unit converts a specific area of the image circle into a plane image (parallel projection image), and performs matching on the plane image after conversion using a normal basic pattern. .

In yet another modification of the eighteenth embodiment, the processing unit maps the image within the image circle onto a spherical surface using a predetermined transformation function, zooms to a portion of the image mapped onto the spherical surface, and ignores distortion. Then, image analysis is performed as a planar image. When it is desired to detect a specific object such as a road sign, image analysis is performed by zooming in on the area in which the specific object is captured. When it is desired to detect a road sign, image analysis is performed by zooming in on a region in which the road sign is reflected in the front image.
[19th embodiment]
Next, a 19th embodiment will be described. In the nineteenth embodiment, as an image analysis function, the processing unit has a function of specifying an area of an image that should not be disclosed to the public from the viewpoint of privacy protection as a privacy protection area. To execute processing for generating an image in which a person captured in a privacy protection area cannot be identified by appearance from an image captured by a camera.

By making a person unidentifiable, the person's privacy can be protected. The processing unit has a function of determining an area in which the interior of the vehicle is reflected and an area that should not be disclosed.
In particular, when an image is acquired using an omnidirectional camera, the interior of the vehicle is imaged, but there may be cases where the image of the interior of the vehicle should not be retained. In such a case, the processing unit has a function of painting the area inside the vehicle black, or applying a mosaic to the area inside the vehicle and recording the image outside the vehicle as it is. The processing unit has a function of distinguishing between an area showing the interior of the vehicle and an area showing the exterior of the vehicle by performing image analysis. As an example, the processing unit has a function of determining an area where optical flow occurs at a speed corresponding to the vehicle speed as being outside the vehicle. The processing unit has a function that allows the user to preset the privacy protection area.

In the modification of the 19th embodiment, the processing unit has a normal mode for recording an image that can identify a person in the privacy protection area, and a normal mode for recording an image that can identify a person in the privacy protection area, and a processing unit that identifies the person in the privacy protection area by appearance. It has a function that allows the user to select a privacy mode that records images that cannot be recorded. The user can switch between the normal mode and the privacy mode according to the purpose of using the vehicle and the usage conditions.

According to another modification of the nineteenth embodiment, the processing unit has a function of setting the area in which a person is photographed as a privacy area. The processing unit has a function of detecting a person by performing image analysis and setting an area surrounded by outlines of the detected person as a privacy area. Furthermore, the processing unit
It has a function to set only the detected person's face as a privacy area. At this time, the processing unit has a function of displaying the person as a silhouette. Alternatively, the processing unit has a function of displaying an image when only the outline of the person is displayed and there is no person inside the outline. At this time, the person appears transparent. The processing unit has a function of displaying the edges of a moving object as the outline of a person.

As still another modified example of the nineteenth embodiment, the processing unit has a function of displaying an avatar instead of a person in an area in which the person is shown. The displayed avatar should resemble the original person. For example, an avatar that reflects the features is displayed according to gender, body shape, facial outline, and the like.
A person and an avatar are associated in advance, and the processing unit identifies the person in the image,
It has the ability to display the avatar associated with the identified person. Avatar is
It is created in advance for each person and registered in the processing unit. The processing unit has a function of moving the avatar according to the movement of the person. For example, when a person raises his hand, the avatar raises his hand, and when the person turns sideways, the avatar also turns sideways. The processing unit has a function of changing the facial expression of the avatar according to the facial expression of the person. For example, the processing unit may determine that the person is laughing,
When a sad or angry facial expression is detected, it has a function to change the facial expression of the avatar into a smiling facial expression, a sad facial expression, or an angry facial expression. The processing unit has a function of reading a person's emotion and changing the color of the avatar according to the person's emotion. For example, when it is detected that a person is angry, the color of the avatar is changed to a color representing being angry, such as red.

As still another modified example of the nineteenth embodiment, the processing unit, before recording the image, selects an image from the image captured by the camera in which the person in the privacy protection area cannot be specified by appearance. It has the ability to generate and record the generated image. Further, the processing unit has a function of converting the person appearing in the privacy protection area into an image that cannot be identified by appearance and displaying the image when reproducing the image captured by the camera.

[Twentieth embodiment]
Next, a twentieth embodiment will be described. In the twentieth embodiment, as processing corresponding to the analysis result, the processing unit executes processing of converting the result of image analysis into text data or encoding and recording it.

Converting to text data or coding makes it possible to easily find a desired image from the recorded images. For example, the processing unit generates "human detection" as text data when a person is detected in the image, and generates "red light" as text data when a red light is detected in the image. The processing unit has a function of recording the generated text data in association with the image from which the text data was generated. In still another modification, the processing unit has a function of associating the image analysis result with a code instead of text data, and encoding the image analysis result (binary data conversion).

[21st embodiment]
Next, a 21st embodiment will be described. In the twenty-first embodiment, the processing unit has a function of detecting blocked black areas in an image as an image analysis function, and detects blocked black areas of an image as a process corresponding to the analysis result. Execute processing to restore brightness and display on the display screen.

A user who sees the image can easily recognize an object or a person in the blocked-up area. In particular, when an image is acquired using an omnidirectional camera, the bright scenery outside the vehicle and the dark interior of the vehicle are simultaneously captured in the image. The process of detecting a blocked-up area and restoring and displaying it brightly is particularly effective when acquiring an image using an omnidirectional camera. As processing for brightening restoration, the processing unit performs, for example, gamma correction, color correction, conversion to a black and white image, and contrast enhancement.

[22nd embodiment]
Next, a 22nd embodiment will be described. In the twenty-second embodiment, the processing unit has a function of allowing the user to divide the imaging range captured by the camera into a plurality of sections, and a function of weighting each section. It has a function of determining imaging conditions based on the assigned weights and causing the camera to perform imaging under the determined imaging conditions.

The user can obtain an image under imaging conditions suitable for the region desired to be viewed by increasing the weight of the section desired to be viewed in the image and lightening the weight of the section not required to be viewed. The user can divide the imaging range into an arbitrary shape according to the shape of the area that the user wants to see or the area that does not need to be seen.

FIG. 5 shows a diagram of an imaging plane for explaining a method of dividing an imaging range into a plurality of sections by a method according to a comparative example. An image circle 73 is formed on the rectangular or square imaging plane 71 of the 360-degree camera for the front, and an image circle 74 is formed on the rectangular or square imaging plane 72 of the 360-degree camera for the rear. In an image circle 73 that captures an image of the front, the front view is reflected through the front window, as well as the dashboard, instrument panel, display screen of the car navigation system, etc. inside the vehicle. In an image circle 74 that captures an image of the rear, the driver, front passenger seat, rear seat, and ceiling in the vehicle are reflected, and the side and rear views of the vehicle are reflected through the side and rear windows.

The imaging planes 71 and 72 are divided into a plurality of sections by square grids. The user can weight each partition.

FIG. 6 shows a diagram of an imaging plane for explaining a method of dividing the imaging range into a plurality of sections in the system according to the twenty-second embodiment. As in the example shown in FIG. 5, an image circle 73 is formed on the imaging plane 71 of the 360-degree camera for the front, and an image circle 74 is formed on the imaging plane 72 of the 360-degree camera for the rear.

The processing unit 51 (FIG. 4) divides a part of the imaging range into a closed curve 7 when the user is allowed to divide the area.
5, and the inner area surrounded by the closed curve 75 is set as one section. The user
A plurality of sections are set by drawing a plurality of closed curves 75 . When the user divides the inner region surrounded by the closed curve into two by the curved line, each of the divided regions becomes one partition.

As compared with the comparative example shown in FIG. 5, weighting can be performed more appropriately for the user than in the case of uniformly weighting square grid-shaped sections regardless of the content of the image.

In particular, when a 360-degree camera is used for imaging, the difference in the positions within the imaging range in which parts of the vehicle are captured increases for each vehicle type compared to when a general camera is used. For this reason, there is an increased possibility that the relation between the general-purpose weighting table set in advance in the firmware of the processing unit and the division of the imaging range that is actually expected does not match. A function that enables determination of weighting using PC browser software, etc., rather than using a general-purpose weighting table, is particularly effective when using a 360-degree camera.

The processing unit determines, for example, gain, exposure time, etc. as imaging conditions. In a configuration in which the back surfaces of two semi-spherical cameras face each other to form an omnidirectional camera, the processing unit sets the imaging conditions of the two semi-spherical cameras to be the same. As a result, when the images captured by the two semi-spherical cameras are combined, an effect is obtained in which the boundary between the two images becomes difficult to discern.

In the modification of the twenty-second embodiment, a processing unit (for example, AI software for a PC) analyzes an image captured by a camera without weighting, and estimates what is captured in which part of the imaging range. have the function to For example, the processing unit assigns a weight of "5" to the road portion, a weight of "3" to the face portion, a weight of "1" to the instrument panel portion, and a weight of "1" to the area outside the circle image. A weight of "0" is attached.

The processing means preferably encloses a partial area within the imaging range with a closed curve and defines the inner area surrounded by the closed curve as one area when allowing the user to divide the area. Preferably, the processing means causes the user to input a numerical weight for each set section. The numerical value to be input should be selected from a pull-down or pop-up list.

In another modification of the twenty-second embodiment, instead of the function of inputting the weight numerically, the processing unit
Alternatively, in addition to the function of inputting numerical values, it has the function of inputting the type of partition. A weight is associated in advance with the type of partition, and when the type of partition is input, the processing unit associates the weight associated with the type with the partition. As types of sections, for example, types such as driver, meter panel, car navigation screen, windshield, rear seat, passenger seat, etc. are provided. The processing unit has a function that allows the user to change the correspondence relationship between the division type and the weight. A list of partition types can be registered in the processing unit in advance by the user.

In still another modified example of the twenty-second embodiment, the processing unit has a function of setting a section by allowing the user to designate a desired area within the imaging range as a function of weighting each section. The weight of the section that the user wants to see is set higher than the weight of other areas. This function is a function that prompts the user to input the type of partition, and includes a partition that the user wants to see,
This corresponds to an example of dividing into two types, ie, other divisions.

In still another modification of the twenty-second embodiment, the processing unit has a function of allowing the user to segment the imaging range in an interactive manner (wizard format). The processing part outputs the message "Please enclose the unnecessary area" or "Please fill in the unnecessary area." recognize. Furthermore, the processing unit outputs messages such as "Please fill in the area of the driver" and "Please fill in the part of the road." Recognize as a partition with a type. The processing unit has a function of allowing the user to perform an operation of encircling with a closed curve or an operation of painting using a pointing device such as a touch panel or a mouse.

In still another modified example of the twenty-second embodiment, the processing unit uses artificial intelligence (AI) to realize a function of dividing the imaging range into a plurality of sections based on the captured image. The processing unit has a function of automatically distinguishing between the inside and outside of the vehicle, the seats, the dashboard, the instrument panel, etc. from the image, and dividing the imaging range for each automatically distinguished component. The processing unit displays the result of classification on the display screen, and allows the user to determine whether the classification is satisfactory. The processing unit has the function of allowing the user to modify the divisions. If the user determines that the segmentation made by the AI is incorrect, the user can correct the segmentation. The processing unit has a function of allowing the user to select the correct type from a pull-down or popup list. If the user determines that the weight or class associated with a parcel is incorrect, the user can easily correct the weight or class.

[23rd embodiment]
Next, a twenty-third embodiment will be described. In the twenty-third embodiment, the processing unit has a function of allowing the user to divide the imaging range captured by the camera into a plurality of sections, and a function of assigning a weight to each section. A process is executed in which the blocks of relatively small partitions are compressed larger than the blocks of relatively large partitions when compressed.

The user can increase the compression of the less important partitions by weighting the more important partitions heavier and the less important partitions lighter weights. As a result, it is possible to reduce the volume of image data while maintaining high image quality of the image of the section with high importance, compared to the case where the entire image is uniformly compressed. For example, the processing unit performs compression using 8×8 pixels as one block for weighted partitions, and 16×16 pixels as one block for lightly weighted partitions.

In the modification of the 23rd embodiment, the processing section has the function of blacking out the area outside the image circle. An image of the barrel portion of the lens may appear in the area outside the image circle formed on the rectangular imaging surface of the camera. By blacking out the unnecessary image in the area outside the image circle, it is possible to reduce the amount of image data including pixels outside the image circle. In another modification, the processing unit has a function of obtaining image data by raster scanning only pixels inside the image circle, excluding pixels outside the image circle. As a result, the volume of image data can be reduced.

[24th embodiment]
Next, a twenty-fourth embodiment will be described. In the twenty-fourth embodiment, the processing unit has a function of detecting the following vehicle and a function of detecting the distance to the following vehicle as image analysis functions, and performs processing corresponding to the analysis result based on the distance to the following vehicle. Then, a process for detecting tailgate driving of the following vehicle is executed.

The processing unit can take appropriate measures against tailgate driving by executing processing according to the determination result. For example, when the same following vehicle continues to exist within a predetermined determination distance for a predetermined determination time or more even though the speed of the vehicle is equal to or higher than a predetermined reference value, the processing unit determines that the following vehicle will continue to drive on the road. It is determined that the

In the modification of the twenty-fourth embodiment, a camera having a function of measuring the distance to an object for each pixel is used. For example, a back-illuminated time-of-flight range image sensor is used as the image sensor of the camera.

In another modification of the twenty-fourth embodiment, when the processing unit determines that the following vehicle is driving on the road, the processing according to the determination result includes, for example, notifying the driver and recording the rear image. , give advice to the driver, report to the traffic enforcement agency (for example, the police), and so on. When notifying the traffic enforcement agency, the processing unit attaches an image of the vehicle and its driver driving tailgate. The processing unit uses image analysis to detect that a vehicle that is driving tailgate has overtaken the own vehicle and has come around in front of the own vehicle. do.

In still another modification of the twenty-fourth embodiment, the processing unit has a function of detecting people and vehicles around the vehicle and controlling the vehicle according to the distance to the people and the vehicle. For example, based on the speed of the vehicle, the distance to the person ahead, and the determination result of whether or not the person is on the planned travel route of the vehicle, the processing unit determines that there is a high possibility of contact with the person. When it is determined that the vehicle is stopped, control is performed to stop the vehicle.

The scope of the invention is not limited to the configurations or limitations expressly set forth herein, but rather includes within its scope any combination of the various aspects of the invention disclosed herein. Although the patentable configuration of the present invention is specified in the appended claims, any configuration not currently specified in the claims is disclosed herein. It is our intention to claim such configurations in the future.

The present invention is not limited to the configurations described in the above-described embodiments. The constituent elements of the above-described embodiments and modifications may be arbitrarily selected and combined. In addition, arbitrary constituent elements of each embodiment and modifications, arbitrary constituent elements described in Means for Solving the Invention, or constituent elements embodying arbitrary constituent elements described in Means for Solving the Invention and may be configured in any combination. We intend to acquire the rights for these as well in the amendment of the present application or in a divisional application.

I also intend to acquire rights to the whole design or partial design by filing a conversion application to a design application. Although the drawing shows the entire device in solid lines, the drawing includes not only the overall design but also the partial design claimed for a part of the device. For example, it is a drawing that includes a part of the device as a partial design regardless of the member, as well as a partial design of a member of the device. The part of the device may be a part of the device or a part of the member. We intend to acquire rights not only for the overall design, but also for the partial design in which any part of the solid line part of the drawing is the broken line part.

1 drive recorder 2 windshield 3 room mirror 4 power cable 5 cigar socket 10 main body of drive recorder 11 camera 12 GPS receiver 13 SD card reader 14 acceleration sensor 15 speaker 16 display 17 operation button 20 processing unit 21 communication device 30 server 31 processing Unit 32 Communication device 33 Input device 34 Display 40 Communication network 50 External device 52 SD card reader 53 Input device 54 Display 60 SD card 71, 72 Imaging surfaces 73, 74 Image circle 75 Closed curve

Claims (6)

A system having processing means having a function of executing processing based on an image captured by a camera mounted on a vehicle,
The processing means has a function of issuing an alarm when there is no change in the position of the object brought into the vehicle identified based on the current image of the vehicle interior and the past image of the vehicle interior for a certain period of time. matter
A system characterized by The processing means further comprises a function of calculating the amount of time that an object brought into the vehicle remains in the vehicle by comparing images taken at a plurality of points in the past with the current image.
The system of claim 1, characterized by: Run the vehicle with only the driver on board,
The processing means recognizes a portion in the image where there is an optical flow corresponding to the vehicle speed as a window, recognizes a portion where there is other movement as a driver, and recognizes a portion other than the window and the portion recognized as the driver. A part is recognized as an area where an object may be placed, and the existence of objects brought into the vehicle is identified based on the no-luggage state image and the current image as the no-luggage state image in which no luggage is loaded in the vehicle. have the function to
3. A system according to claim 1 or 2, characterized in that: The processing means has a function of determining whether or not a pre-registered object appears in the image inside the vehicle. Provide a function to execute processing to issue an alarm
4. A system according to any one of claims 1 to 3, characterized in that: The processing means has a function of issuing an alarm when an item left in the vehicle is detected and a movement of opening the door is detected, and an e-mail is sent to the driver or the like when an item left in the vehicle is detected and a movement of closing the door is detected. Equipped with a function to notify that there is something left behind
5. A system according to any one of claims 1 to 4, characterized in that: A program for causing a computer to implement the functions of the system according to any one of claims 1 to 5.

Images