JP7471024B2 - DISPLAY CONTROL SYSTEM, DISPLAY CONTROL METHOD, AND PROGRAM - Google Patents

Description

The present invention relates to a display control system, a display control method, and a program that controls the display of thumbnail images related to event-unit videos recorded by a drive recorder installed in a vehicle.

Conventionally, in-vehicle video recording devices, so-called drive recorders, have become widespread, which capture video of the vehicle's surroundings using a camera installed in the vehicle and record the surrounding video and vehicle speed when the vehicle is subjected to an impact due to an event such as sudden braking, sudden steering, or a collision. By equipping a vehicle with such a drive recorder, it is possible to verify the cause of an accident by analyzing the recorded information in the event of an accident. In addition, this can improve the driver's awareness of safe driving, and the video recorded on a daily driving situation can be used to provide safe driving guidance, etc.

Patent Documents 1 and 2 disclose a driving support device that collects driving information such as camera images, vehicle speed, and GPS (Global Positioning System) location information recorded by drive recorders installed in commercial vehicles such as trucks and taxis, and allows an operations manager or the like to check the contents of the driving information, analyze the driving style of each driver, and use the information in driving education for drivers.

JP 2010-257483 A JP 2010-257484 A

In the above driving assistance device, the operation manager or the like can narrow down and select event unit images of specific events such as sudden braking from a large amount of driving information (hereinafter referred to as event unit images) recorded for each event such as sudden braking, abrupt steering, and a collision, by using related data such as the magnitude and waveform of acceleration recorded in each event unit image. In other words, this related data is recognized as a tag for narrowing down the event unit images of specific events, and the event unit images are classified and narrowed down. However, to ultimately identify the event unit images of the desired event, the operation manager or the like must view and check the contents of the camera images contained in each event unit image in order, and identify the event unit images.

In addition, the driving assistance devices disclosed in Patent Documents 1 and 2 display thumbnail images that are reduced in size from the camera images recorded at the time when a trigger (e.g., a timing signal indicating that a recording condition by a drive recorder has been met) is detected from among the multiple camera images contained in one event-unit video. However, since this thumbnail image is a camera image recorded at the time when the trigger was detected, which is after the time when an event such as sudden braking or steering occurred, it is difficult to find the event-unit video of the desired event from this thumbnail image.

The present invention has been made in consideration of the above-mentioned circumstances, and aims to provide a display control system, a display control method, and a program that enable a user to easily find a desired event-unit video from a thumbnail image.

In a first aspect, the present invention provides a display control system that controls the display of thumbnail images related to event-unit video recorded by a drive recorder mounted on a vehicle, the display control system being characterized by comprising a display control means that displays multiple thumbnail images corresponding to different time points within the range of the event-unit video.

In the display control system according to the first aspect, the content of the event-unit video can be confirmed from multiple thumbnail images corresponding to different time points within the range of the event-unit video. This makes it easier to find the desired event-unit video from the thumbnail images compared to the case where the content of the event-unit video is confirmed from a single thumbnail image.

An event may be, for example, an occurrence specified by a predetermined rule. An event unit may be, for example, a time range in units that can distinguish and manage a certain event from other events. For example, an event unit may be a predetermined period before and after the occurrence of an event, as in the configuration of the second aspect. For example, the time ranges of a certain event unit and another event unit may be set so that they do not overlap, or may be set so that they partially overlap.
The event-unit image may be, for example, an image of a time range that allows a certain event to be managed by distinguishing it from other events. The event-unit image may be, for example, recorded as separate images for each event, or a certain range within one image may be treated as an event-unit image. In other words, "event-unit images recorded by a drive recorder mounted on a vehicle" do not necessarily have to be images recorded for each event.

When a certain range in one video is set as an event-unit video, for example, the ranges of a plurality of event-unit videos may be set so as to allow overlapping of the ranges, or the ranges of each event-unit video may be set so as not to overlap. An event-unit video may be, for example, a range in a video that may be related to an event.
Furthermore, "multiple thumbnail images corresponding to different time points within the range of the event-unit video" may be, for example, "multiple thumbnail images corresponding to video other than the event-unit video at different time points within the range of the event-unit video", but it is better to say "multiple thumbnail images corresponding to the event-unit video at different time points within the range of the event-unit video". A specific example of "corresponding to" in "multiple thumbnail images corresponding to the event-unit video" is "representing". Other examples include "created from", "image-processed with", "created by image processing with", "created by applying a specified image processing", etc.

In a second aspect, the present invention is characterized in that in the display control system according to the first aspect, the event-unit video is video spanning a predetermined period of time before and after the occurrence of the event, and the thumbnail images include images recorded either before or after the occurrence of the event, or both.

In the display control system according to the second aspect, the contents of the event-unit video can be confirmed from thumbnail images corresponding to images recorded either before or after the event occurrence, or both. This makes it easier to find event-unit videos related to events occurring before and after the event occurrence.

The time when an event occurs may be, for example, the time when a predetermined condition is satisfied. The time when an event occurs may be, for example, the time when a predetermined condition representative of the event is satisfied. For example, the time when a predetermined condition is satisfied may be set as the time when the event occurs, and the range of the event-unit image may be determined according to the type of event. The event may be, for example, the time when an incident equivalent to an accident occurs. For example, the point where sudden deceleration equivalent to a collision occurs may be set as the time when the event occurs.

There are types of drive recorders, for example, that record video before and after a "trigger indicating that a recording condition has been met," and types that record video at all times (for example, from engine on to off) and store information that makes it possible to identify the position of the video where the trigger has been activated. This "information that makes it possible to identify" is, for example, information on the time when the trigger was activated. For example, the position in which the trigger was activated can be identified from information on the start time of recording the video and information on the time when the trigger was activated.

In the case of the former type of drive recorder described above, the "event-unit video" may be, for example, video recorded over a predetermined period of time before and after the occurrence of a trigger indicating that a recording condition by the drive recorder has been met.
In addition, in the case of the latter type of drive recorder mentioned above, the "event unit video" may be, for example, a portion of the video that is recorded over a predetermined period of time before and after the trigger occurs, within the video that is the recording unit (in the above example, one unit is from engine on to engine off).

The specified period may be, for example, the entire range of the recorded video (e.g., the entire period from engine on to off), but is particularly preferably a limited time range within the video. The limited time range within the video may particularly be a time range that includes the time point at which the event occurs. The limited time range within the video may be, for example, a fixed time range, but may also be a variable time. Furthermore, the period before the time point at which the event occurs and the period after the time point at which the event occurs that constitute the specified period may be the same time period, but are preferably different time periods. In particular, the specified period should be a period sufficient to enable identification of the cause of the occurrence of the event. In particular, the specified period should be a range that differs depending on the type of event.

In a third aspect, the present invention is characterized in that, in the display control system according to the second aspect, the thumbnail image is an image recorded at a time within the specified period that is not the time when the event occurred, but is determined according to a specified rule that uses information used to determine that the event occurred.

In the display control system according to the third aspect, it becomes possible to check the contents of the event-unit video from a thumbnail image corresponding to an image recorded at a time within a specified period that is not the time when the event occurred, but is determined by a specified rule using information used to determine that the event occurred (e.g., the acceleration applied to the vehicle) (e.g., the time when the acceleration exceeds a specified value). This makes it easy to find the event-unit video related to the incident at the time determined by the above-mentioned specified rule.

In a fourth aspect, the present invention is characterized in that in the display control system according to the second or third aspect, the thumbnail image is an image recorded within the specified period at a time determined by a specified rule using information different from the information used to determine that the event has occurred.

In the display control system according to the fourth aspect, the contents of the event-unit video can be confirmed from a thumbnail image corresponding to an image recorded at a time point (e.g., the time point of the maximum value of the vehicle's longitudinal acceleration within a specified period of time) that is determined by a specified rule using information (e.g., the vehicle's longitudinal acceleration due to sudden braking) different from the information used to determine that an event has occurred (e.g., the vehicle's lateral acceleration due to sudden steering). This makes it easy to find event-unit videos related to the events at the time points determined by the above-mentioned specified rules.

In a fifth aspect, the present invention is characterized in that in the display control system according to the third or fourth aspect, the predetermined rule is a rule for estimating the occurrence time of a sub-event that caused the occurrence of the event.

In the display control system according to the fifth aspect, it is possible to easily find an event unit video related to an incident at a time point determined by a rule for estimating the occurrence time point of a sub-event that caused the occurrence of an event.
The predetermined rule is a rule for estimating a characteristic point related to the occurrence of the event by tracing back from the time when the event is determined to have occurred, and may be, for example, a maximum point or an inflection point of the acceleration of the vehicle within a predetermined period of time, or another characteristic point of the acceleration. Also, different rules may be used depending on the type of the event.
The predetermined period of time may be set to be long enough to consider the possibility of a sub-event causing an event occurring to be low.

In a sixth aspect, the present invention is characterized in that in the display control system according to any one of the second to fifth aspects, the time point before or after the occurrence of the event, or both, are different times depending on the type of the event.

In the display control system according to the sixth aspect, the contents of the event-unit video can be confirmed from thumbnail images corresponding to images recorded at different times depending on the type of event, either before or after the event occurrence, or both. This makes it easy to find event-unit videos related to events at different times depending on the type of event.

In a seventh aspect, the present invention is characterized in that in the display control system according to any one of the second to sixth aspects, the thumbnail image is an image recorded either immediately before or immediately after the occurrence of the event, or both.

In the display control system according to the seventh aspect, the contents of the event-unit video can be confirmed from thumbnail images corresponding to images recorded either immediately before or immediately after the event occurs, or both. This makes it easier and more accurate to find event-unit videos related to events occurring immediately before or immediately after the event occurs.

In an eighth aspect, the present invention is characterized in that in the display control system according to any one of the second to seventh aspects, the thumbnail images include images recorded both immediately before and immediately after the occurrence of the event, the acquisition interval of the information used to determine that the event has occurred is set to an interval shorter than the recording interval of the frames of the video, the image recorded immediately before the occurrence of the event is an image of the frame immediately before the determination that the event has occurred, and the image recorded immediately after the occurrence of the event is an image of the frame immediately after the determination that the event has occurred.

In the display control system according to the eighth aspect, the contents of the event-unit video can be confirmed from thumbnail images corresponding to images recorded both immediately before and after the occurrence of the event. This makes it easier and more accurate to find event-unit videos related to events both immediately before and after the occurrence of the event.

In a ninth aspect, the present invention is characterized in that in a display control system according to any one of the first to eighth aspects, the thumbnail image is an image recorded at either the beginning or the end of the event-unit video, or at both points in time.

In the display control system according to the ninth aspect, the contents of the event-unit video can be confirmed from a thumbnail image corresponding to an image recorded at either the beginning or the end, or both, of the event-unit video. This makes it easier and more accurate to find the event-unit video related to the events at the beginning or end of the event-unit video.

In a tenth aspect, the present invention is characterized in that in the display control system according to any one of the second to ninth aspects, the thumbnail image is an image recorded at the time when the event occurred.

In the display control system according to the tenth aspect, the contents of the event-unit video can be confirmed from a thumbnail image corresponding to an image recorded at the time the event occurred. This makes it easier and more accurate to find the event-unit video related to the event at the time the event occurred.

In an eleventh aspect, the present invention is characterized in that in the display control system according to any one of the first to tenth aspects, the thumbnail image is an image of a specific period within the range of the event-unit image, depending on the state of the vehicle when the event-unit image was recorded.

In the display control system according to the eleventh aspect, the contents of the event-unit video can be confirmed from thumbnail images corresponding to images of a specific period according to the state of the vehicle when the event-unit video was recorded. This makes it easier to find event-unit videos related to the state of the vehicle when the event-unit video was recorded.

The "state of the vehicle when the event-unit video was recorded" refers to, for example, the state of the vehicle when (1) sudden braking, (2) sudden steering, or (3) the vehicle collides with something.
"Images of a specific period selected from event-unit images" are, for example, (1) in the case of sudden braking, images recorded from when the brake is applied until it is released, or images recorded from when an acceleration event applied to the vehicle occurs until it ends, etc. Also, (2) in the case of sudden steering, images recorded from when the steering wheel starts to be turned until it is turned to completion, etc. Furthermore, (3) in the case of a vehicle colliding with something, images recorded from when the vehicle's acceleration behavior occurs due to the reaction of the bumper, guardrail, etc., until the vehicle stops and the acceleration behavior converges, etc.

In a twelfth aspect, the present invention is characterized in that in the display control system according to any one of the first to eleventh aspects, the thumbnail image is an image recorded at the time when the acceleration applied to the vehicle exceeded a predetermined value.

In the display control system according to the twelfth aspect, the contents of the event-unit video can be confirmed from the thumbnail image corresponding to the image recorded at the time when the acceleration applied to the vehicle exceeded a predetermined value. This makes it easier to find the event-unit video related to the time when the acceleration applied to the vehicle exceeded a predetermined value.

In a thirteenth aspect, the present invention is characterized in that, in the display control system according to any one of the second to twelfth aspects, the occurrence of the event is specific behavior information based on information that detects the behavior of the vehicle.

In the display control system according to the thirteenth aspect, the contents of the event-unit video can be confirmed from a thumbnail image corresponding to an image in which specific behavior information based on information detecting the behavior of a vehicle is recorded as an event. This makes it easier to find event-unit videos in which specific behavior information is recorded as an event.

"Information on detected vehicle behavior" refers to, for example, information obtained from the detection results of an acceleration sensor, gyro sensor, or air pressure sensor (or altitude sensor) mounted on the vehicle, or the results of analysis of recorded video.

In a fourteenth aspect, the present invention is characterized in that in the display control system according to any one of the second to thirteenth aspects, the occurrence of the event is specific vehicle information based on information about the functional state of the vehicle.

In the display control system according to the fourteenth aspect, the contents of the event-unit video can be confirmed from a thumbnail image corresponding to an image in which specific vehicle information based on information about the functional status of the vehicle is recorded as an event. This makes it easier to find event-unit videos in which specific vehicle information is recorded as an event.

"Information regarding the functional status of the vehicle" includes, for example, the operation of the brake, turn signal, accelerator, steering wheel rotation, shift lever position, or handbrake while driving the vehicle, as well as information obtained from the OBD (On Board Diagnosis) vehicle fault diagnosis system and CAN (Controller Area Network), an in-vehicle LAN.

In a fifteenth aspect, the present invention is characterized in that in the display control system according to any one of the first to fourteenth aspects, the plurality of thumbnail images are displayed in a list on the same screen in a matrix structure in which thumbnail images of different event-unit videos form different rows and thumbnail images of the same point in time form different columns.

In the display control system according to the fifteenth aspect, a plurality of thumbnail images are displayed in a list that is visually easy to understand, organized as thumbnail images for different event-unit videos or thumbnail images for the same point in time. This makes it easy to distinguish the contents of the thumbnail images.

In a sixteenth aspect, the present invention is characterized in that in the display control system according to any one of the first to fifteenth aspects, the plurality of thumbnail images are displayed in a list on the same screen in a matrix structure in which thumbnail images for the same event unit video are arranged in chronological order from left to right or from right to left in the same row.

In the display control system according to the sixteenth aspect, a plurality of thumbnail images are displayed in a visually easy-to-understand list as thumbnail images arranged in chronological order in the same row for each event unit video. This makes it easy to distinguish the contents of the thumbnail images.

It can be considered as a method invention having steps corresponding to components of the display control means according to any one of the first to sixteenth aspects. For example, in the seventeenth aspect, as a method invention corresponding to the first aspect, the present invention provides a display control method for controlling the display of thumbnail images related to event-unit video recorded by a drive recorder mounted on a vehicle, the display control method being characterized in having a step of displaying multiple thumbnail images corresponding to different time points within the range of the event-unit video.

The display control method according to the seventeenth aspect makes it possible to check the contents of an event-unit video from multiple thumbnail images corresponding to different time points within the range of the event-unit video, so that the desired event-unit video can be easily found from the thumbnail images.

In an eighteenth aspect, the present invention provides a program for causing a computer to realize the functions of a display control system according to any one of the first to sixteenth aspects.

The display control system of the present invention makes it possible to easily find the desired event unit video from thumbnail images.

1 is a diagram showing a schematic configuration of a driving assistance system according to a first embodiment; 2 is a diagram conceptually showing the data structure of the contents recorded on a memory card 25. FIG. 10A and 10B are diagrams illustrating a state of acceleration change when an event occurs and a method of storing images and the like according to the first embodiment. 6 is a flowchart showing a process of creating and displaying thumbnail images according to the first embodiment. FIG. 4 is a screen display diagram showing an example of a display mode of thumbnail images according to the first embodiment. FIG. 11 is a diagram showing a change in acceleration in the case of sudden braking according to the second embodiment. 13 is a flowchart showing a process of creating and displaying thumbnail images according to the second embodiment. FIG. 11 is a diagram showing a change in acceleration in the case of abrupt steering according to the third embodiment. 13 is a flowchart showing a process of creating and displaying thumbnail images according to a third embodiment. 13 is a diagram showing a change in acceleration in the event of a vehicle collision according to the fourth embodiment. FIG. 13 is a flowchart showing a process of creating and displaying thumbnail images according to a fourth embodiment. FIG. 13 is a diagram showing an experimental example of acceleration change in the case of a car-to-car collision. FIG. 13 is a diagram showing a display example of a thumbnail screen related to the experimental example shown in FIG. 12 .

The following describes in detail a display control system according to one embodiment of the present invention with reference to the drawings. Note that the present invention should not be interpreted as being limited to this, and various changes, modifications, and improvements may be made based on the knowledge of those skilled in the art without departing from the scope of the present invention.

[Example 1]
<Configuration of driving assistance system>
FIG. 1 is a diagram showing a schematic configuration of a driving assistance system according to a first embodiment of a display control system of the present invention.

As shown in FIG. 1, the driving assistance system is composed of a drive recorder 10 mounted on a vehicle, a memory card 25 that stores images when an event occurs in the drive recorder 10, and a driving assistance device 30 for playing back the images stored in the memory card 25.

The drive recorder 10 includes a first control unit 11, an acceleration sensor (G sensor) 12, a gyro sensor 13, an air pressure sensor 14, a GPS antenna 15, a GPS (Global Positioning System) receiver 16 that measures the GPS position information of the vehicle from the GPS signal received by the GPS antenna 15, a first operation unit 17 including switches, buttons, etc., a first memory 18 for temporarily storing images, a first IF (interface) 19 that has a slot for inserting a memory card 25 and is used for writing images to the memory card 25, a CCD camera 20 installed in the vehicle to continuously capture images of the area ahead of the vehicle at a predetermined rate, a vehicle signal acquisition unit 21, and a power supply unit 22 for supplying power to each element of the drive recorder 10.

The drive recorder 10 may also be configured to work in conjunction with a navigation device (not shown) so that images can be played back on the display of the navigation device. Furthermore, the GPS receiver 16 is not necessarily a required component, but may be an external device, such as an AVM (Automatic Vehicle Monitoring) system, or a configuration that acquires location information received by the navigation device.

The first control unit 11 is composed of a microcomputer including a CPU, ROM, RAM, etc., and is responsible for controlling the overall operation of the drive recorder 10.

The G sensor 12 detects gravitational acceleration (hereinafter simply referred to as acceleration) in two mutually perpendicular axial directions (X-axis and Y-axis) and transmits the detected acceleration (also referred to as G data) together with time data to the first control unit 11. The first control unit 11 can obtain the G data (Gx, Gy) output from the G sensor 12 at intervals of, for example, 10 milliseconds, and determine the value (magnitude) of the acceleration from the G data.

Note that Gx indicates acceleration in the X-axis direction (front-to-back direction of the vehicle), and Gy indicates acceleration in the Y-axis direction (left-to-right direction of the vehicle). The G sensor 12 may be a three-axis sensor that also includes the Z-axis direction (up-down direction of the vehicle), or it may not be necessary to use data on the acceleration Gz in the Z-axis direction in a three-axis sensor.

The gyro sensor 13 detects the angular acceleration (rotation) of the vehicle, mainly detecting changes in the vehicle's direction of travel (i.e., rotation of the yaw (Z-axis), which is the vertical axis), and transmits the detected angular acceleration data together with time data to the first control unit 11. The first control unit 11 can acquire the angular acceleration data output from the gyro sensor 13 at intervals of, for example, 10 milliseconds, and determine the value (magnitude) of the angular acceleration from the angular acceleration data.

The atmospheric pressure sensor 14 detects the atmospheric pressure outside the vehicle, calculates the altitude based on the detected atmospheric pressure value, and transmits the atmospheric pressure value and altitude value (atmospheric pressure/altitude data) together with time data to the first control unit 11. The first control unit 11 can acquire the atmospheric pressure/altitude data output from the atmospheric pressure sensor 14 at intervals of, for example, 10 milliseconds, and determine the values (magnitudes) of the atmospheric pressure and altitude from the atmospheric pressure/altitude data.

The first control unit 11 controls the CCD camera 20 to capture one image every 100 milliseconds, and to store the captured images in a sequential and cyclical manner in the first memory 18 together with time data (primary storage for storage in the memory card 25 described below). For example, the CCD camera 20 is configured to always store 24 seconds' worth of image data (240 images) captured by the CCD camera 20 in the first memory 18. Note that the CCD camera 20 may capture one image every 50 milliseconds, for example, rather than one image every 100 milliseconds. It may also be configured to store video captured by the CCD camera 20.

The vehicle signal acquisition unit 21 is connected to the vehicle's fault diagnosis connector, acquires various vehicle signals and information (vehicle data) at intervals of, for example, 100 milliseconds, and transmits the acquired vehicle data together with time data to the first control unit 11. Examples of vehicle data include: [1] a signal that detects braking operation while driving the vehicle; [2] a signal that detects turn signal operation; [3] a signal that detects accelerator operation; [4] a signal that detects steering wheel rotation; [5] a signal that detects shift lever position; [6] a signal that detects handbrake operation; [7] vehicle fault diagnosis information (vehicle speed, engine RPM, coolant temperature, etc.) obtained from the vehicle fault diagnosis system (OBD) in the vehicle; and [8] various vehicle information obtained via the on-board LAN (CAN) (function control data from the ECU (Electronic Control Unit) of the vehicle, communication data from an external network or other vehicles, etc.).

As an example, when the vehicle signal acquisition unit 21 acquires a signal that detects a brake operation, it transmits brake operation data indicating whether or not the vehicle brakes have been operated, brake operation start time data indicating the time when the brake operation started, and brake operation end time data indicating the time when the brake operation ended to the first control unit 11. In other words, based on the brake operation data, brake operation start time data, and brake operation end time data from the vehicle signal acquisition unit 21, the first control unit 11 can determine whether or not a brake operation has been performed, the time when the brake operation started, and the time when the brake operation ended.

As another example, when the vehicle signal acquisition unit 21 acquires a signal that detects steering wheel rotation, it transmits steering wheel rotation data indicating whether the steering wheel of the vehicle has been rotated, steering wheel rotation start time data indicating the time when steering wheel rotation started, and steering wheel rotation end time data indicating the time when steering wheel rotation ended to the first control unit 11. In other words, based on the steering wheel rotation data, steering wheel rotation start time data, and steering wheel rotation end time data from the vehicle signal acquisition unit 21, the first control unit 11 can determine whether the steering wheel has been rotated, the time when steering wheel rotation started, and the time when steering wheel rotation ended.

In addition, some or all of the GPS antenna 15, GPS receiver 16, first IF 19, CCD camera 20, vehicle signal acquisition unit 21, etc. may be configured separately from the drive recorder 10. For example, when linked to an existing navigation device with a GPS function, the drive recorder 10 does not need functions such as the GPS receiver 16, so the cost of the drive recorder 10 can be kept low.

The driving support device 30 has a second control unit 31, a display unit 32 consisting of a display or the like, an output unit 33 consisting of a printer or the like that prints the display screen on paper, a second operation unit 34 consisting of input devices such as a keyboard and a mouse, a second memory 35 for temporarily storing images, a second IF (interface) 36 that includes a slot for inserting a memory card 25 and for reading images from the memory card 25, and an image processing unit 37 for performing predetermined image processing on the input image.

The second control unit 31 is a personal computer (PC) that includes a CPU, ROM, RAM, etc., and has driving assistance software installed. It interprets and executes the programs included in the driving assistance software to operate each component of the driving assistance device 30, thereby realizing various functions related to the characteristic parts of the present invention.

The driving assistance device 30 is one aspect of a display control system that controls the display of thumbnail images related to event-unit images recorded by the drive recorder 10. In this embodiment, the event-unit images are images recorded during a predetermined period before and after the time point at which the event occurs. The time point at which the event occurs is, for example, the time point at which an event equivalent to an accident occurs, for example, the time point at which an impact is applied to the vehicle due to sudden deceleration such as sudden braking.

The memory card 25 is a rewritable non-volatile semiconductor memory card with a storage capacity of 2 GB. This memory card 25 is inserted into a slot constituting the first IF 19, and stores predetermined images (e.g., images for 12 seconds before and 12 seconds after the time point at which an event occurs) that are always temporarily stored in a cyclical manner in the first memory 18 during a predetermined period before and after the time point at which an event occurs. The memory card 25 is then ejected from the slot constituting the first IF 19 and reinserted into the slot constituting the second IF 36, and outputs the stored images to the driving assistance device 30.

Note that although a semiconductor memory card is used as the memory card 25, the present invention is not limited to this, and semiconductor memory other than card type, a portable hard disk (HD), other storage media, etc. can be used. Also, the storage capacity of the memory card, etc. is arbitrary and can be freely determined according to the application.

When a predetermined condition is met (i.e., when an event that should be recorded by the drive recorder 10 occurs), specifically when (a) an impact is applied to the vehicle due to sudden braking, sudden steering, a collision, or other event (phenomenon) equivalent to the occurrence of an accident, causing the value (magnitude) of the G data output from the G sensor 12 installed inside to exceed a predetermined threshold, or (b) when the emergency switch of the first operation unit 17 is pressed, the drive recorder 10 uses the detection as a trigger to obtain various data for several to tens of seconds before and after (for example, 12 seconds before and 12 seconds after) along with data on the time of the event occurrence, and records this data on the memory card 25 as a single piece of driving information.

The driving information recorded on the memory card 25 includes G data output from the G sensor 12, angular acceleration data output from the gyro sensor 13, air pressure and altitude data output from the air pressure sensor 14, GPS position information measured by the GPS receiver 16, camera image data captured by the CCD camera 20, and various vehicle signals and information acquired by the vehicle signal acquisition unit 21 (detection data such as braking operations and steering wheel rotation, vehicle speed data, etc.).

The drive recorder 10 records various driving information data on the memory card 25 in association with each of a number of camera images taken from a period of several to several tens of seconds (e.g., 12 seconds before and 12 seconds after) before and after the occurrence of an event that should be recorded and a trigger is detected.

<Memory card data structure>
Next, the data structure of the contents recorded on the memory card 25 will be described with reference to Fig. 2. Fig. 2 is a diagram conceptually showing the data structure of the contents recorded on the memory card 25.

As shown in FIG. 2, memory card 25 records driving information corresponding to each event, divided into driving information 1, driving information 2, and driving information 3. Driving information 1 is made up of 240 camera images 1-240 corresponding to 24 seconds before and after the event (12 seconds before and 12 seconds after), and image-related information including other driving information other than the camera images. The image-related information includes main item data including G data and time data at the time the event occurred, and related data 1-240 corresponding to each of the 240 camera images 1-240. The same is true for driving information 2 and 3.

The main item data includes G data, angular acceleration data, air pressure and altitude data, and the respective time data, while the image-related data includes the date and time the camera image was taken, various vehicle signals and information at the time the camera image was taken (detection data for braking operations, steering wheel rotation, etc., vehicle speed data, etc.), and the vehicle's GPS position information.

<Method of storing changes in acceleration and images when an event occurs>
Next, referring to Figure 3, we will explain how the acceleration (G data) detected by the G sensor 12 (see Figure 1) of the drive recorder 10 changes when an event occurs, such as an impact to the vehicle, and how images, etc. are stored in the drive recorder 10.

Figure 3(a) shows the time series change in the G data (composite value of Gx and Gy) output from the G sensor 12, Figure 3(b) shows an example of the image period stored in the memory card 25, and Figure 3(c) shows an example of the selection timing of the image that will be the basis for the thumbnail image to be created. The creation and display of thumbnail images will be described later.

From the moment the vehicle is subjected to an impact, the value (magnitude) of the G data output from the G sensor 12 increases rapidly from nearly zero. Then, as shown in FIG. 3(a), at time tp, when the value (magnitude) of the G data from the G sensor 12 reaches the threshold acceleration Gs (e.g., 0.3 G) (time tp), the first control unit 101 recognizes this as a trigger, determines (judges) that an event has occurred, and operates to store the image captured by the CCD camera 20, which is always temporarily stored in the first memory 18 in a circular manner, in the memory card 25, and also to store the time tp in the memory card 25 as event occurrence determination time data.

Specifically, as shown in FIG. 3(b), the image PA for TA seconds before time tp and the image PB for TB seconds after time tp are stored in the memory card 25. For example, TA can be 12 seconds and TB can be 12 seconds. Note that the periods TA and TB can be freely set. For example, while the set time (seconds) is displayed on the setting screen (not shown) of the drive recorder 10, the time can be set by operating the time setting button of the first operation unit 17 of the drive recorder 10, and the set information can be saved, allowing the time to be set to any desired time.

When an event such as an impact to the vehicle occurs and the value (magnitude) of the detected G data of the G sensor 12 exceeds a predetermined threshold value as described above, the first control unit 11 stores, for each event, the event occurrence time data (tp), the detected G data (Gx, Gy) of the G sensor 12 and its time data, the detected angular acceleration data of the gyro sensor 13 and its time data, the atmospheric pressure and altitude data of the atmospheric pressure sensor 14 and its time data, and the vehicle data and its time data for TA seconds before the time tp and TB seconds after the time tp acquired by the vehicle signal acquisition unit 21 (for example, brake operation data and its time data, steering wheel rotation data and its time data, vehicle speed data and its time data, etc.) together with image data in the memory card 25. Note that it may be configured to store only a portion of the above-mentioned data in the memory card 25.

The number of events stored in the memory card 25 can be varied depending on the storage capacity of the memory card 25, but it is preferable to store data for, for example, 500 or more events. In addition, it is preferable for the first control unit 11 to create a folder in the memory card 25 for each event, and store image data, event occurrence time data (tp), G data (Gx, Gy) from the G sensor 12 and its time data, angular acceleration data and its time data, air pressure/altitude data and its time data, and vehicle data and its time data.

Note that, when the above-mentioned events occur, exceeding the threshold acceleration Gs is set as the trigger for the occurrence of the event, but, for example, it may be determined that an event has occurred when the amount of change in acceleration per unit time exceeds a certain value (when the value (magnitude) of the G data changes by 0.1 G or more in 0.1 seconds).

<Creating and Displaying Thumbnail Images (Example 1)>
Next, the creation and display processing of thumbnail images in the driving support device 30 will be described with reference to Fig. 4. In this example, the creation and display processing operation of a thumbnail image related to one event will be described.
4 is a flowchart showing thumbnail image creation and display processing according to the first embodiment, and this thumbnail image creation and display processing is realized by the CPU executing a control program loaded from an HDD or the like into the RAM of the second control unit 31 of the driving assistance device 30, and is started based on an operation to start the driving assistance software by the second operation unit 34. Also, it is assumed that the memory card 25 on which images are stored in the drive recorder 10 is inserted into the second IF 36 constituting a slot at the time when the processing of the flowchart shown in FIG.

In step S11, the second control unit 31 acquires event occurrence determination time data (tp) from the data stored in the memory card 25.

Next, in step S12, the second control unit 31 selects multiple images taken at multiple different times from the image PA taken TA seconds before the time tp shown in FIG. 3(b) and the image PB taken TB seconds after the time tp.

Specifically, the control program includes a control process in which, from among the image PA taken TA seconds before the time tp shown in FIG. 3(b) and the image PB taken TB seconds after the time tp shown in FIG. 3(c), image P1 taken at time ts TA seconds (e.g., 12 seconds) before the time tp shown in FIG. 3(c) is selected as the "first image", image P2 taken at time tp1 just before the time tp (a predetermined time TC seconds before, e.g., 0.2 seconds before) is selected as the "second image", image P3 taken at time tp2 just after the time tp (after the predetermined time TC, e.g., 0.2 seconds after) is selected as the "third image", and image P4 taken at time te TB seconds (e.g., 12 seconds) after the time tp is selected as the "fourth image". The second control unit 31 controls the selection of the first to fourth images from the image PA and image PB in accordance with this control program.

Note that time ts, TA seconds before time tp, is the time when image storage for one event starts, and the "first image" is also the image captured when the event occurs. Also, time te, TB seconds after time tp, is the time when image storage for one event ends, and the "fourth image" is the image captured when the event detection ends.
The predetermined time TC can be set to, for example, 0.2 seconds, but an optimal value including 0 seconds can be selected depending on the application. For example, while the set time (seconds) is displayed on a setting screen (not shown) of the driving support device 30, the time can be set by operating the time setting button of the second operation unit 34 of the driving support device 30, and the set information can be saved, thereby setting an arbitrary time.

Next, in step S13, the second control unit 201 causes the image processing unit 37 to perform image processing in accordance with a control program stored in the ROM within the second control unit 201 to create first to fourth thumbnail images from the "first to fourth images" (P1 to P4) selected in step S12.

To create a thumbnail image from the selected first to fourth images, for example, the original first to fourth images are each subjected to image processing to reduce the size of the images, to create the thumbnail images. In addition to image processing to reduce the size of the images, thumbnail images may be created by performing other predetermined image processing, such as image processing to enlarge the selected first to fourth images or image processing to cut out a part of the images. Furthermore, the selected first to fourth images may not be subjected to any image processing, and may be used as the thumbnail image as is.

In the next step S14, the second control unit 31 causes the display processing unit 38 to perform a display process for displaying the first to fourth thumbnail images created in step S13 on the display unit 32 in accordance with a control program stored in the ROM within the second control unit 31. Specifically, the thumbnail images are arranged in a matrix structure and displayed as a list on the same screen, as will be described later.
This completes the thumbnail image creation and display process for one event.

<Thumbnail image display example>
Next, one aspect of a display image displayed on the display unit 32 of the driving support device 30 will be described with reference to Fig. 5. Fig. 5 is a screen display diagram showing an example of a display aspect of thumbnail images according to the first embodiment.
The display mode of thumbnail images illustrated in FIG. 5 shows a state in which thumbnail images relating to a plurality of events are displayed on the display unit 32 of the driving support device 30.

In FIG. 5, 16 thumbnail images A1-D4 are arranged in a matrix structure of 4 rows vertically and 4 columns horizontally and displayed on the same screen, with each row constituting an event unit image 51, 52, 53, 54 relating to one event. That is, each of the event images 51, 52, 53, 54 constitutes four sets of thumbnail images A1-A4, B1-B4, C1-C4, D1-D4 in which four thumbnail images relating to one event are arranged and displayed in one row, and four thumbnail images relating to different events are arranged and displayed in different rows.

The shooting time of the image that is the basis for each thumbnail image is displayed below thumbnail images A1 to D4. The example shown in the figure displays the shooting time of the image that is the basis for each thumbnail image when the event occurrence time tp in event unit image 51 is "11:46:14:60".

The 16 thumbnail images shown in FIG. 5 constitute time-unit images 61, 62, 63, and 64, each created from images taken at the same time (more precisely, at a time before or after the event occurrence time tp for each event) for each column. That is, each of the time-unit images 61, 62, 63, and 64 constitutes four sets of thumbnail images A1-D1, A2-D2, A3-D3, and A4-D4, in which four thumbnail images relating to the same time are arranged and displayed in one column, and four thumbnail images relating to different time points are arranged and displayed in different columns.

Furthermore, of the 16 thumbnail images shown in FIG. 5, the first column from the left is a thumbnail image created from image P1 (first image) taken at the time the event occurs (time ts), the second column is a thumbnail image created from image P2 (second image) taken just before the event occurrence determination time (time tp1), the third column is a thumbnail image created from image P3 (third image) taken just after the event occurrence determination time (time tp2), and the fourth column is a thumbnail image created from image P4 (fourth image) taken at the end of event detection (time te).

Furthermore, in FIG. 5, the four thumbnail images arranged in one row are displayed in the order of X1 → X2 → X3 → X4 (X is A, B, C, or D), with thumbnail images for the same event image arranged in chronological order from left to right within the same row. Note that the display order may be from right to left within the same row, rather than from left to right.

Note that while four events are displayed in Figure 5, the number of events displayed is not limited to this, and it is possible to select a different number of events to display, or to display them together with other elements (G data, vehicle data, etc.).

As described above, the driving assistance device 30 of the first embodiment has the following advantages.
(1) When an event occurs, such as when an impact is applied to the vehicle, a first image P1 captured TA seconds before the event occurrence (time tp), a second image P2 captured just before time tp, a third image P3 captured just after time tp, and a fourth image P4 captured TB seconds after time tp are selected, and first to fourth thumbnail images are created from the selected first to fourth images P1 to P4 and displayed on the display unit 32. As a result, when an event occurs, the contents of the event unit images stored for each event occurrence can be easily confirmed by checking the first to fourth thumbnail images corresponding to the first to fourth images P1 to P4 captured at different times among the event unit images captured for a predetermined period before and after the event, and the cause and circumstances of the event occurrence can be easily understood.

For example, when searching for an event unit image related to a specific event from among a large number of event unit images stored for each event occurrence, the specific event unit image desired can be easily found from the thumbnail image, compared to searching for the specific event unit image by checking the contents of the event unit image from a thumbnail image of a single point in time related to the event unit image.

(2) As shown in the example of the display mode of thumbnail images in FIG. 5, four thumbnail images relating to one event are displayed in one row, and four thumbnail images relating to different events are displayed in different rows, so that the four thumbnail images relating to one event can be clearly distinguished at a glance.
In addition, the four thumbnail images arranged in one row are displayed arranged in chronological order from left to right in the same row for each event, so that the chronological order of the four thumbnail images related to one event can be clearly determined at a glance.

Furthermore, by looking at the thumbnail image in the first row, you can see the state in front of the vehicle at the time the event occurs, by looking at the thumbnail image in the second row, you can see the state in front of the vehicle just before the event occurrence determination time, by looking at the thumbnail image in the third row, you can see the state in front of the vehicle just after the event occurrence determination time, and by looking at the thumbnail image in the fourth row, you can see the state in front of the vehicle at the end of event detection, so for example, by comparing thumbnail images in the same row, you can compare and verify the state in front of the vehicle at each of the same times when an event occurs.

[Example 2]
In the second embodiment, an event example in which an impact is applied to a vehicle is caused by sudden braking.
FIG. 6 is a diagram showing the change in acceleration when sudden braking is applied, and shows the time-series change in the G data (a composite value of Gx and Gy) output from the G sensor 12.

As shown in FIG. 6, when the brakes are suddenly applied (ON) at time t1, the G sensor 12 mainly detects the acceleration (Gx) in the longitudinal direction of the vehicle, and the acceleration changes. The acceleration (G data) of the G sensor 12 gradually increases from time t1 (i.e., the acceleration in the direction opposite to the direction in which the vehicle moves increases), and when it reaches a threshold acceleration Gs (e.g., 0.3 G) (time tp), the first control unit 11 recognizes this as a trigger, determines (judges) that an event has occurred, and stores the image captured by the CCD camera 20 and constantly stored in the first memory 108 in the memory card 150, and also operates to store the time tp in the memory card 150 as event occurrence judgment time data.

Specifically, in this second embodiment, as in the first embodiment, the image PA for TA seconds before time tp and the image PB for TB seconds after time tp are stored in the memory card 25.

After time tp, the acceleration continues to increase, reaches a maximum value, then starts to decrease gradually (i.e. the acceleration in the direction opposite to the direction in which the vehicle is moving becomes smaller), then suddenly decreases, passes through zero, and continues to decrease (i.e. the acceleration in the direction in which the vehicle is moving increases), reaches a minimum value, then starts to increase again, suddenly increases (i.e. the acceleration in the direction in which the vehicle is moving decreases), and when the vehicle stops and the brake is released (OFF) (time t2), it reaches zero.

In FIG. 6, the period Td from time t1 to time tp is the "period from the beginning of dangerous driving to immediately before the trigger occurs," and the period Te from time tp to time t2 is the period of "dangerous driving behavior."

<Creating and displaying thumbnail images (Example 2)>
Fig. 7 is a flowchart showing thumbnail image creation and display processing according to Example 2, which, like Example 1, is realized by the CPU executing a control program loaded from an HDD or the like into the RAM of the second control unit 31 of the driving assistance device 30, and is started based on an operation to start the driving assistance software by the second operation unit 34. Also, it is assumed that the memory card 25 on which images are stored in the drive recorder 10 is inserted into the second IF 36 constituting a slot at the time the processing of the flowchart shown in Fig. 7 is started.

In step S21, the second control unit 31 acquires event occurrence determination time data (tp) from the data stored in the memory card 25. Then, in step S22, the second control unit 31 acquires the brake operation detection signal (brake operation data) output from the vehicle signal acquisition unit 21 from the data stored in the memory card 25.

Next, in step S23, the second control unit 31 selects multiple images captured at multiple different times related to the braking operation from among the images PA taken for TA seconds before the time tp shown in FIG. 3(b) and the images PB taken for TB seconds after the time tp.

Specifically, the control program is programmed to select, from among the image PA for TA seconds before the time tp shown in FIG. 3(b) and the image PB for TB seconds after the time tp, the image captured at time t1 when the brake operation is started (i.e., the start time of the occurrence of the acceleration event) based on the brake operation start time data included in the acquired brake operation data as the "first image", the image captured just before the time tp (before the predetermined time TC, for example, 0.2 seconds before) as the "second image", the image captured just after the time tp (after the predetermined time TC, for example, 0.2 seconds after) as the "third image", and the image captured at time t2 when the brake operation is ended (i.e., the end time of the occurrence of the acceleration event) based on the brake operation end time data included in the acquired brake operation data as the "fourth image". The second control unit 31 controls the selection of the first to fourth images from the image PA and the image PB according to such a control program.

The predetermined time TC can be set to, for example, 0.2 seconds, but an optimal value, including 0 seconds, can be selected depending on the application.
In addition to the above-mentioned times t1, just before or just after time tp, and time t2, images may be selected that are captured at the time when the acceleration reaches a maximum value (maximum value), just before or just after that, or at the time when the acceleration reaches a minimum value (minimum value), or just before or just after that.

Next, in step S24, the second control unit 31 causes the image processing unit 37 to perform image processing in accordance with a control program stored in the ROM within the second control unit 31 to create first to fourth thumbnail images from the "first to fourth images" selected in step S23.

In the next step S25, the second control unit 31 causes the display processing unit 38 to perform a display process for displaying the first to fourth thumbnail images created in step S24 on the display unit 32 in accordance with a control program stored in the ROM within the second control unit 31. Specifically, the thumbnail images are arranged in a matrix structure as shown in Fig. 5 and are displayed in a list on the same screen.
This completes the thumbnail image creation and display process for one event.

As described above, the driving assistance device 30 of the second embodiment has the following advantages.
In the case of sudden braking as an event example such as an impact on a vehicle, a first image captured at time t1 when a braking operation is started, a second image captured just before time tp, a third image captured just after time tp, and a fourth image captured at time t2 when the braking operation is ended are selected, and first to fourth thumbnail images are created from the selected first to fourth images and displayed on the display unit 32. In this way, in the case of sudden braking, by checking the first to fourth thumbnail images corresponding to the first to fourth images captured at different times among the event unit images captured during a predetermined period before and after the event occurrence time (time tp), particularly the time points including the time t1 when the braking operation is started and the time t2 when the braking operation is ended, it is possible to easily check the contents of the event unit images stored in the case of sudden braking as an event example, and it is easy to grasp the cause and situation of the sudden braking.

For example, when searching for an event unit image related to sudden braking from among a large number of event unit images stored for each event occurrence, the event unit image related to sudden braking can be easily found from the thumbnail image compared to searching for an event unit image related to sudden braking by checking the contents of the event unit image from a thumbnail image of a single point in time related to the event unit image.

Also, by looking vertically at the thumbnail images in the first row from the left of the 16 thumbnail images shown in Figure 5, it is possible to compare images at the time when the brakes were applied suddenly (when braking operation began). By looking vertically at the thumbnail images in the second row, it is possible to compare images just before the time when it was determined that an event had occurred due to the application of the brakes, and by looking vertically at the thumbnail images in the third row, it is possible to compare images just after that. Furthermore, by looking vertically at the thumbnail images in the fourth row, it is possible to compare images at the time when the brakes were released (when braking operation ended).

[Example 3]
In the third embodiment, an event example in which an impact is applied to a vehicle is caused by sudden steering.
FIG. 8 is a diagram showing the change in acceleration when the steering wheel is turned suddenly, and shows the time-series change in the G data (composite value of Gx and Gy) output from the G sensor 12.

As shown in FIG. 8, when the steering wheel starts to be turned suddenly at time t3, the G sensor 12 mainly detects the acceleration (Gy) in the left-right direction of the vehicle, and the acceleration changes. The acceleration (G data) from the G sensor 12 gradually increases from time t3 (i.e., the acceleration in the direction opposite to the direction in which the vehicle is turning increases), and when it reaches a threshold acceleration Gs (e.g., 0.3 G) (time tp), the first control unit 11 recognizes this as a trigger, determines (judges) that an event has occurred, and stores the image captured by the CCD camera 20 and always stored in the first memory 108 in the memory card 25, and also operates to store the time tp in the memory card 25 as event occurrence judgment time data.

Specifically, in this third embodiment, as in the first and second embodiments, the image PA for TA seconds before time tp and the image PB for TB seconds after time tp are stored in the memory card 25.

After time tp, the acceleration continues to increase, reaches a maximum value, then starts to decrease gradually (i.e. the acceleration in the direction opposite to the direction in which the vehicle is turning becomes smaller), then suddenly decreases, passes through zero, decreases again (i.e. the acceleration in the direction in which the vehicle is turning becomes larger), reaches a minimum value (minimum value), then starts to increase again and suddenly increases (i.e. the acceleration in the direction in which the vehicle is turning becomes smaller), and when the vehicle has finished turning and the steering wheel has finished being turned (i.e. the steering wheel is returned to its original position) (time t4), it goes to zero.

In FIG. 8, the period Tf from time t3 to time tp is the "period from the beginning of dangerous driving to immediately before the trigger occurs," and the period Tg from time tp to time t4 is the period of "dangerous driving behavior."

<Creating and displaying thumbnail images (third embodiment)>
Fig. 9 is a flowchart showing thumbnail image creation and display processing according to Example 3, which, like Examples 1 and 2, is realized by the CPU executing a control program loaded from an HDD or the like into the RAM of the second control unit 31 of the driving assistance device 30, and is started based on an operation to start the driving assistance software by the second operation unit 34. Also, it is assumed that the memory card 25 on which images are stored in the drive recorder 10 is inserted into the second IF 36 constituting a slot at the time the processing of the flowchart shown in Fig. 9 is started.

In step S31, the second control unit 31 acquires event occurrence determination time data (tp) from the data stored in the memory card 25. Then, in step S32, the second control unit 31 acquires the steering wheel rotation detection signal (steering wheel rotation data) output from the vehicle signal acquisition unit 21 from the data stored in the memory card 25.

Next, in step S33, the second control unit 31 selects multiple images captured at multiple different times related to the rotation of the steering wheel from among the images PA captured for TA seconds before the time tp shown in FIG. 3(b) and the images PB captured for TB seconds after the time tp.

Specifically, the control program is programmed to select, from among the image PA for TA seconds before time tp and the image PB for TB seconds after time tp shown in FIG. 3(b), the image captured at time t3 when the steering wheel rotation started based on the steering wheel rotation start time data included in the acquired steering wheel rotation data as the "first image", the image captured just before time tp (before a predetermined time TC, for example, 0.2 seconds before) as the "second image", the image captured just after time tp (after the predetermined time TC, for example, 0.2 seconds after) as the "third image", and the image captured at time t4 when the steering wheel rotation ended based on the steering wheel rotation end time data included in the acquired steering wheel rotation data as the "fourth image". The second control unit 31 controls the selection of the first to fourth images from the image PA and image PB according to this control program.

The predetermined time TC can be set to, for example, 0.2 seconds, but an optimal value, including 0 seconds, can be selected depending on the application.
In addition to the above-mentioned times t3, just before or just after time tp, and time t4, images may be selected that are captured at the time when the acceleration reaches a maximum value (maximum value), just before or just after that, or at the time when the acceleration reaches a minimum value (minimum value), or just before or just after that.

Next, in step S34, the second control unit 31 causes the image processing unit 37 to perform image processing in accordance with a control program stored in the ROM within the second control unit 31 to create first to fourth thumbnail images from the "first to fourth images" selected in step S33.

In the next step S35, the second control unit 31 causes the display processing unit 38 to perform a display process for displaying the first to fourth thumbnail images created in step S34 on the display unit 32 in accordance with a control program stored in the ROM within the second control unit 31. Specifically, the thumbnail images are arranged in a matrix structure as shown in Fig. 5 and are displayed in a list on the same screen.
This completes the thumbnail image creation and display process for one event.

As described above, the driving support device 30 of the third embodiment has the following advantages.
In the case of an event example caused by abrupt steering, such as an impact on a vehicle, the first image captured at time t3 when steering wheel rotation is started, the second image captured just before time tp, the third image captured just after time tp, and the fourth image captured at time t4 when steering wheel rotation is completed are selected, and first to fourth thumbnail images are created from the selected first to fourth images and displayed on the display unit 32. In this way, in the case of abrupt steering, by checking the first to fourth thumbnail images corresponding to the first to fourth images captured at different times among the event unit images captured during a predetermined period before and after the event occurrence time (time tp), particularly at times including time t3 when steering wheel rotation is started and time t4 when steering wheel rotation is completed, it is possible to easily check the contents of the event unit images stored in the case of abrupt steering as an event example, and it is easy to grasp the cause and circumstances of the occurrence of abrupt steering.

For example, when searching for an event unit image related to abrupt steering from among a large number of event unit images stored for each event occurrence, the event unit image related to the desired abrupt steering can be easily found from the thumbnail image, compared to searching for an event unit image related to abrupt steering by checking the contents of the event unit image from a thumbnail image of a single point in time related to the event unit image.

Also, by looking vertically at the thumbnail images in the first row from the left of the 16 thumbnail images shown in Figure 5, you can compare images from the time when steering operation began (when steering rotation began). By looking vertically at the thumbnail images in the second row, you can compare images from just before an event was determined to have occurred due to steering operation, and by looking vertically at the thumbnail images in the third row, you can compare images from just after that. Furthermore, by looking vertically at the thumbnail images in the fourth row, you can compare images from the time when steering operation stopped (when steering rotation ended).

Furthermore, creating a thumbnail image at the time when the steering wheel rotation starts as in the third embodiment is advantageous when the steering wheel operation and the acceleration event are not linked, as described below.
For example, there are cases where acceleration occurs regardless of steering operation, such as when the vehicle collides or travels up a slope, and cases where acceleration occurs due to steering operation such as sudden steering. The first thumbnail image in the third embodiment (first row from the left in FIG. 5 above) is of the time when the steering wheel starts to turn, but there are cases where the steering wheel operation and the acceleration event are not linked.
According to the third embodiment, it is possible to display a thumbnail image of the time when the steering wheel rotation started, which has a predetermined relationship with the occurrence of an event. Therefore, particularly in cases where the time when the steering wheel rotation started does not coincide with the time when the acceleration event started, it is possible to confirm the relationship between the time when the steering wheel rotation started and the occurrence of an event from the thumbnail image of the time when the steering wheel rotation started, rather than the thumbnail image of the time when the acceleration event started to appear, and this can be useful in understanding the cause of the event and the situation.

[Example 4]
In the fourth embodiment, a vehicle collision is taken as an example of an event in which an impact is applied to a vehicle.
FIG. 10 is a diagram showing the change in acceleration in the event of a vehicle collision, and shows the time-series change in the G data (a composite value of Gx and Gy) output from the G sensor 12. In FIG.

As shown in FIG. 10, when the vehicle collides with something at time t5, the G sensor 12 detects and changes the vehicle's longitudinal acceleration (Gx) and lateral acceleration (Gy). The acceleration (G data) from the G sensor 12 increases rapidly immediately after time t5, then decreases rapidly again, and after repeating this increase and decrease, when it increases rapidly and reaches a threshold acceleration Gs (e.g., 0.3 G) (time tp), the first control unit 11 recognizes this as a trigger, determines (judges) that an event has occurred, and stores the image captured by the CCD camera 20, which is always stored in the first memory 108 in a circular manner, in the memory card 25, and also operates to store the time tp in the memory card 25 as event occurrence determination time data.

Specifically, in this fourth embodiment, as in the first, second, and third embodiments, the image PA for TA seconds before time tp and the image PB for TB seconds after time tp are stored in the memory card 25.

After time tp, the acceleration continues to increase, reaches a maximum value, then suddenly decreases, passes through zero, reaches a minimum value, then suddenly increases, passes through zero again, reaches a maximum value, then suddenly decreases, passes through zero again, and after repeating this rapid decrease, the vehicle stops and the acceleration converges (t6), becoming zero.

In FIG. 10, the period Th from time t5 to time tp is the "period of impact before the big impact", during which a shock waveform appears due to the impact reduction effect of the vehicle's bumper, guardrails, etc. Also, the period Ti from time tp to time t6 is the period of "shock behavior".

<Creating and displaying thumbnail images (Example 4)>
Fig. 11 is a flowchart showing thumbnail image creation and display processing according to Example 4, which, like Examples 1, 2, and 3, is realized by the CPU executing a control program loaded from an HDD or the like into the RAM of the second control unit 31 of the driving assistance device 30, and is started based on an operation to start the driving assistance software by the second operation unit 34. Also, it is assumed that the memory card 25 on which images are stored in the drive recorder 10 is inserted into the second IF 36 constituting a slot at the time the processing of the flowchart shown in Fig. 11 is started.

In step S41, the second control unit 31 acquires event occurrence determination time data (tp) from the data stored in the memory card 25. Then, in step S42, the second control unit 31 acquires the vehicle speed (vehicle speed data) output from the vehicle signal acquisition unit 21 from the data stored in the memory card 25. Furthermore, in step S43, the second control unit 31 acquires the detected G data (Gx, Gy) of the G sensor 12 from the data stored in the memory card 25.

Next, in step S44, the second control unit 31 selects multiple images taken at multiple different timings related to the vehicle collision from among images PA taken TA seconds before time tp shown in Figure 3 (b) and images PB taken TB seconds after time tp.
Specifically, the control program includes a control process in which, from among the image PA for TA seconds before the time tp shown in FIG. 3B and the image PB for TB seconds after the time tp, the image captured at the time tp3 (i.e., the time when the acceleration behavior of the vehicle occurs due to the reaction of the bumper, guard rail, etc.) when the magnitude (value) of the G data immediately after the time t5 of the vehicle collision is rapidly decreased and reaches the second maximum value is selected as the "first image", the image captured immediately before the time tp (before a predetermined time TC, for example, 0.2 seconds before) is selected as the "second image", the image captured immediately after the time tp (after the predetermined time TC, for example, 0.2 seconds after) is selected as the "third image", and the image captured at the time t6 when the vehicle speed becomes zero (i.e., the time when the acceleration behavior converges) is selected as the "fourth image". The second control unit 31 performs control to select the first to fourth images from the image PA and the image PB according to such a control program.

The predetermined time TC can be set to, for example, 0.2 seconds, but an optimal value, including 0 seconds, can be selected depending on the application.
In addition to the above-mentioned images taken immediately after time t5, immediately before or immediately after time tp, and time t6, images may be selected that are taken at the time when the acceleration reaches a maximum value (maximum value), or immediately before or immediately after that, or at the time when the acceleration reaches a minimum value (minimum value), or immediately before or immediately after that.

Next, in step S45, the second control unit 31 causes the image processing unit 37 to perform image processing in accordance with a control program stored in the ROM within the second control unit 31 to create first to fourth thumbnail images from the "first to fourth images" selected in step S44.

In the next step S46, the second control unit 31 causes the display processing unit 38 to perform a display process for displaying the first to fourth thumbnail images created in step S45 on the display unit 32 in accordance with a control program stored in the ROM within the second control unit 31. Specifically, the thumbnail images are arranged in a matrix structure as shown in Fig. 5 and are displayed in a list on the same screen.
This completes the thumbnail image creation and display process for one event.

As described above, the driving assistance device 30 of the fourth embodiment has the following advantages.
In the case of a vehicle collision as an event example in which an impact is applied to a vehicle, the first image captured at the time when the G data repeatedly rapidly decreased immediately after the time t5 of the vehicle collision (the time of the reaction acceleration behavior), the second image captured immediately before the time tp, the third image captured immediately after the time tp, and the fourth image captured at the time t6 when the vehicle speed became zero (the time when the acceleration converged), are selected, and first to fourth thumbnail images are created from the selected first to fourth images and displayed on the display unit 32. In this way, in the case of a vehicle collision, by checking the first to fourth thumbnail images corresponding to the first to fourth images captured at different times among the event unit images captured during a predetermined period before and after the event occurrence time (time tp), particularly the first to fourth thumbnail images captured at the time when the G data repeatedly rapidly decreased immediately after the time t5 of the vehicle collision (the time of the reaction acceleration behavior) and the time when the vehicle speed became zero (the time when the acceleration converged), the contents of the event unit images stored in the case of a vehicle collision as an event example can be easily checked, and the cause and circumstances of the vehicle collision can be easily understood.

For example, when searching for an event unit image related to a vehicle collision from among a large number of event unit images stored for each event occurrence, the event unit image related to the desired vehicle collision can be easily found from the thumbnail image, compared to searching for an event unit image related to a vehicle collision by checking the contents of the event unit image from a thumbnail image of a single point in time related to the event unit image.

Also, by looking vertically at the thumbnail images in the first row from the left of the 16 thumbnail images shown in Figure 5, it is possible to compare images taken at the time of a vehicle collision. By looking vertically at the thumbnail images in the second row, it is possible to compare images taken just before an event was determined to have occurred due to a vehicle collision, and by looking vertically at the thumbnail images in the third row, it is possible to compare images taken just after that. Furthermore, by looking vertically at the thumbnail images in the fourth row, it is possible to compare images taken at the time when the vehicle speed became zero.

<Experimental Example>
FIG. 12 is a diagram showing an experimental example of acceleration change in the event of a vehicle-to-vehicle collision, in relation to acceleration change in the event of a vehicle collision according to Example 4 shown in FIG. 10, and shows the value of detected acceleration Gx in the X-axis direction (front-to-rear direction of the vehicle), the value of detected acceleration Gy in the Y-axis direction (left-right direction of the vehicle), and the value of detected acceleration Gz in the Z-axis direction (up-down direction of the vehicle) output from the G sensor 12 as their respective time-series changes.

In this experimental example, as shown in FIG. 12, when a vehicle collides with something at time t7, the values of Gx, Gy, and Gz begin to rapidly decrease and increase in a relatively short period immediately after time t7, and when the value of Gx reaches a threshold acceleration Gs (e.g., 1.0) (time tp), a trigger is recognized and it is determined that an event has occurred.
After time tp, the value of Gx may become slightly larger, but eventually converges to near zero (time t8). Also, after time tp, the values of Gy and Gz become relatively small, and eventually converge.

In Figure 12, the period Tj from time t7 to time tp is the "period of impact before the big impact" like the period Th in Figure 10, and during this period Tj, a shock waveform appears due to the impact reduction effect of the vehicle's bumper, guard rails, etc. Also, the period Ti from time tp to time t8 is the "shock behavior" period like the period Ti in Figure 10, and during this period Tk, the vehicle moves in reaction to the impact.

<Display example>
FIG. 13 is a diagram showing a display example of the thumbnail screen related to the experimental example shown in FIG.
In this display example, the memory card 25 that is inserted into the drive recorder 10 and records driving information including camera images and image-related information is removed and displayed on the display unit 32 of the driving support device 30.
In Figure 13, 16 thumbnail images are arranged in a matrix structure of 4 rows and 4 columns and displayed on the same screen, with each row displaying event-unit images 71, 72, 73, and 74 relating to one event, and each column displaying time-unit images 71, 72, 73, and 74 created from images taken at the same point in time (more precisely, the same point in time for each event relative to the event occurrence time tp).

Next, a method for displaying a list of thumbnail images arranged in a matrix structure as described above will be described below.
First, the user selects files 91, 92, 93, and 94 of a plurality of events (four in this example) that he or she wishes to display from the playlist 90 in the upper right corner of the display image shown in Fig. 13 using the second operation unit 204. Next, event unit images corresponding to the files 91, 92, 93, and 94 of the selected events are loaded, and thumbnail images are created and displayed based on the loaded files in accordance with the processing procedure in the flowchart of Fig. 4, and the thumbnail images are arranged in a matrix structure and displayed in a list on the same screen.

In the display example of FIG. 13 , the event unit images 71 displayed on each row display thumbnail images created from images captured at four points in time for an event in file 91 (i.e., [1] the event occurrence point, which is TA seconds before the event occurrence time tp, [2] just before time tp (just before the event occurrence), [3] just after time tp (just after the event occurrence), and [4] the event detection end point, which is TB seconds after time tp).
Similarly, event-unit image 72 displays thumbnail images of the above-mentioned four points in time for an event in file 92, event-unit image 73 displays thumbnail images of the above-mentioned four points in time for an event in file 93, and event-unit image 74 displays thumbnail images of the above-mentioned four points in time for an event in file 94.

In the display example of Figure 13, the time-point unit images 81 displayed in each column display thumbnail images created from an image taken at the event occurrence time, which is TA seconds before the event occurrence time tp, out of the four thumbnail images of event unit images 71, 72, 73, and 74 [1].
Similarly, the time-point unit image 82 displays [2] a thumbnail image created from an image taken just before the event occurrence time tp (just before the event occurs) out of the four thumbnail images of each of the event unit images 71, 72, 73, and 74, the thumbnail image created from an image taken just after the event occurrence time tp (just after the event occurs) out of the four thumbnail images of each of the event unit images 71, 72, 73, and 74, and the time-point unit image 84 displays [4] a thumbnail image created from an image taken at the end of the event detection, which is TB seconds after the event occurrence time tp, out of the four thumbnail images of each of the event unit images 71, 72, 73, and 74.

<Modification>
The present invention is not limited to the above-described embodiment of the present invention, and various modifications are possible. For example, the following modifications may be made.

(1) In the above embodiment, the time (time tp) when the value (magnitude) of the G data output from the G sensor 12 reaches the threshold acceleration Gs is recognized as a trigger and it is determined that an event has occurred, but this is not limiting, and for example, the time when the values (magnitudes) of the angular acceleration data output from the gyro sensor 13 or the atmospheric pressure/altitude data output from the atmospheric pressure sensor 14 reach a predetermined value may be recognized as a trigger and it may be determined that an event has occurred. Also, it may be determined that an event has occurred based on the results of analyzing recorded video.
In addition, an event may occur when the emergency switch of the first operating unit 17 is pressed, when a signal is input from an external port, or when a command is received via communication (e.g., RS-232C) from an external device such as an AVM-ECU (Automatic Vehicle Monitoring - Engine Control Unit).

(2) In the above embodiment, the first to fourth images taken at four time points are selected for each event such as sudden braking, sudden steering, and vehicle collision from the images stored for a predetermined period for each event, and the first to fourth thumbnail images are created, but this is not limited to the above, and multiple images taken at multiple different time points other than the four time points may be selected, and thumbnail images corresponding to the selected images may be created. Furthermore, various combinations of multiple different time points may be used depending on the type of event, but it is preferable to use a combination of time points at which the characteristics of the event appear (a specific period within the predetermined period).
The combination of times at which the characteristics of an event appear can be determined by using, alone or in combination, various data such as "data detecting vehicle behavior" and "data relating to the functional status of the vehicle" stored on memory card 25 together with the image data, and based on that data, multiple images taken at appropriate times are selected and thumbnail images corresponding to the selected images are created.

"Data that detects vehicle behavior" may be, for example, acceleration data, angular acceleration data, air pressure/altitude data, video analysis data, etc. Furthermore, "data related to the functional status of the vehicle" may be, for example, data related to the operation of the brake, turn signal, accelerator, steering wheel rotation, shift lever position, or handbrake while driving the vehicle, or data obtained from the OBD (On Board Diagnosis) vehicle fault diagnosis system or the CAN (Controller Area Network) in the vehicle's LAN.

(3) In the above embodiment, the occurrence of an event is determined using the composite value of the component acceleration (G data) Gx in the X-axis direction (front-rear direction of the vehicle) and the component acceleration (G data) Gy in the Y-axis direction (left-right direction of the vehicle) as the G data, but this is not limited to the above, and the occurrence of an event may be determined using each component acceleration (Gx, Gy) individually. For example, when the change pattern of the value (magnitude) of each component takes a predetermined pattern, such as when Gx exceeds a predetermined threshold value and then Gy exceeds a predetermined threshold value within a predetermined time, a trigger may be recognized and it may be determined that an event has occurred. Furthermore, the occurrence of an event may be determined by taking into account the component acceleration Gz in the Z-axis direction (up-down direction of the vehicle) in addition to Gx and Gy.

(4) Not limited to the above-described embodiment, an image captured at the time when information (e.g., G data (Gx) in the longitudinal direction of the vehicle due to sudden braking) different from information used to determine that an event has occurred (e.g., G data (Gy) in the lateral direction of the vehicle due to sudden steering) reaches a predetermined value (e.g., a maximum value) may be selected, and a thumbnail image may be created. This makes it possible to easily find an event-unit video related to the occurrence of an event at the time when information (e.g., G data (Gx) due to sudden braking) different from information used to determine that an event has occurred (e.g., G data (Gy) due to sudden steering) reaches a predetermined value.

(5) Not limited to the above-described embodiment, a thumbnail image may be created by going back from the time (time tp) when the value (magnitude) of the G data reaches the threshold acceleration Gs and it is determined that an event has occurred, and selecting an image captured at a time point when the sub-event that caused the event to occur, such as the time point when the G data reaches a maximum or minimum value, the time point when the G data changes over time, or the time point when the G data changes characteristically, such as by repeatedly increasing and decreasing rapidly in a short period of time. This makes it possible to easily find an event-unit video related to an event at a time point when the sub-event that caused the event to occur, such as the time point when the G data changes characteristically, such as by a rapid increase and decrease, can occur.

(6) In the above embodiment, the drive recorder 10 is a trigger recording type that records video before and after a "trigger indicating that a recording condition has been met," but the present invention is not limited to this. For example, the drive recorder may be a continuous recording type that records continuously from engine on to engine off and stores information that enables identification of the position of the video where the trigger has been activated. In this continuous recording type drive recorder, for example, it is possible to identify which position in which video the trigger has been activated from the video recording start time information and the trigger time information.
In the case of the above-mentioned trigger recording type drive recorder, for example, the video recorded over a predetermined period before and after the occurrence of a trigger indicating that a recording condition by the drive recorder has been satisfied may be regarded as the "event-unit video." Also, in the case of the above-mentioned continuous recording type drive recorder, for example, the "event-unit video" may be a portion of the video recorded over a predetermined period before and after the occurrence of a trigger, in which one recording unit is from engine on to engine off.

(7) In the above embodiment, the image PA for TA seconds and the image PB for TB seconds before and after the time tp at which it is determined that an event has occurred are stored in the memory card 25. However, this is not limiting, and for example, images for the entire time from engine on to engine off, or images for a limited period of time within that entire time, may be stored in the memory card 25. The limited period of time within the entire time may be the time including the time tp at which it is determined that an event has occurred.
The total time of TA seconds and TB seconds before and after the time tp may be a fixed time for each event occurrence, or may be a variable time. The total time of TA seconds and TB seconds should be a period sufficient to enable identification of the cause of the event occurrence. In particular, the total time of TA seconds and TB seconds should be different depending on the type of event.

(8) In the above embodiment, the TA seconds and TB seconds before and after the time tp at which it is determined that an event has occurred are the same (in the embodiment, 12 seconds before and 12 seconds after), but this is not limiting and, for example, they may be different times. In particular, it is preferable to lengthen the TA seconds before time tp and shorten the TB seconds after (for example, 15 seconds before and 5 seconds after).

(9) In the above-described embodiment, from among the images PA within TA seconds and the images PB within TB seconds before and after the time tp determined to be an event occurrence, the first to fourth images taken TA seconds before, just before (TC seconds before), just after (TC after), and TB seconds after are selected, and the first to fourth thumbnail images are created from the selected first to fourth images. However, this is not limited to the above, and for example, two or three images from the first to fourth images may be selected in combination, or any number of images from one to four of the first to fourth images may be selected in combination with one or more images from other times, and a thumbnail image may be created from the selected images.

(10) In the above-described embodiment, an image taken before or after the time tp at which it is determined that an event has occurred is selected, but this is not limited to the above. An image taken at the time tp may be selected and combined with images taken before or after the time tp.

(11) In the above-mentioned embodiment, as in the example of the display mode of thumbnail images shown in FIG. 5, 16 thumbnail images are arranged to form an event unit image for one event for each row, and a time unit image for the same time for each column. However, this is not limited to this, and the rows and columns may be interchanged to form an event unit image for each column, and a time unit image for each row. Furthermore, when an event unit image for one event is formed for each row (or column), the event unit images in one row (or column) may be arranged in chronological order, and images may be formed according to the time of the event, rather than the same time for each column. Furthermore, the images may be arranged in a matrix structure with other combinations of the number of rows and columns, such as 5 rows x 5 columns or 3 rows x 3 columns, and displayed on the same screen. Furthermore, when the images cannot be displayed on the screen, it is preferable to scroll the matrix as a whole, rather than scrolling left and right for each event.

(12) In the above embodiment, the display control system of the present invention has been described as an example of a driving assistance system (driving assistance device 30), but it can be implemented as a function of various electronic devices. For example, it may be incorporated as a function of a navigation device, a drive recorder, a radar detector, and a car audio device.

(13) In addition to the above-described embodiment, a control program pre-stored in the ROM of the second control unit 31 of the driving assistance device 30 may be downloaded to a portable terminal such as a general-purpose personal computer, a mobile phone, a smartphone, or a portable game console, and a computer mounted on the portable terminal may execute the control process of the second control unit 31, read images and various data stored in the memory card 25, and display thumbnail images created from the read images and various data.

(14) Although each function of the above embodiment is realized by describing the function in a computer-readable programming language and having the computer of the second control unit 31 execute the program, this is not limiting, and for example, the program may be distributed among multiple computers for distributed processing.

10 Drive recorder 11 First control unit 12 Acceleration sensor (G sensor)
13 gyro sensor 14 atmospheric pressure sensor 15 GPS antenna 16 GPS receiver 17 first operation unit 18 first memory 19 first IF (interface)
20 CCD camera 21 Vehicle signal acquisition unit 22 Power supply unit 25 Memory card 30 Driving support device 31 Second control unit 32 Display unit 33 Output unit 34 Second operation unit 35 Second memory 36 Second IF (interface)
37 Image processing unit 38 Display processing unit 51 to 54, 71 to 74 Event unit images 61 to 64, 81 to 84 Time unit images 90 Playlist 91 to 94 Files

Claims (5)

A display control system for controlling display of thumbnail images related to video recorded by an electronic device mounted on a vehicle, comprising:
A function for displaying a plurality of thumbnail images in a matrix structure on the same screen based on the recorded video,
A display control system having a function of displaying thumbnail images created based on the video relating to different events for each of a plurality of rows, and displaying thumbnail images created based on the video taken at a time that is relatively the same with respect to the event occurrence time of the event corresponding to the row for each of a plurality of columns. The display control system of claim 1, wherein the multiple columns of thumbnail images include an image showing the state in front of the vehicle at the time an event occurs, an image showing the state in front of the vehicle just before the event occurrence determination time, an image showing the state in front of the vehicle just after the event occurrence determination time, and an image showing the state in front of the vehicle at the time event detection ends. A display control system for controlling display of thumbnail images related to video recorded by an electronic device mounted on a vehicle, comprising:
A function for displaying a plurality of thumbnail images in a matrix structure on the same screen based on the recorded video,
a function of displaying thumbnail images created based on the video images relating to different events for each of a plurality of columns, and displaying thumbnail images created based on the video images taken at relatively the same time points with respect to the event occurrence times of the events corresponding to the respective rows for each of a plurality of rows;
A display control system comprising: The display control system according to claim 1 , wherein the function of displaying creates and displays the thumbnail image based on video analysis data of the recorded video. A program for causing a computer to realize the functions of the display control system according to any one of claims 1 to 4.

Images