JP7446605B2 - Systems, programs, trained models, learning model generation methods, generation devices, etc. - Google Patents

Description

The present invention relates to, for example, a system, a program, a trained model, a learning model generation method, a generation device, and the like.

In conventional technology, event recording information is created using the occurrence of a predetermined event in a photographing device such as a drive recorder as a trigger. Among such devices, one has been considered that efficiently collects photographic data obtained by photographing under abnormal conditions (Patent Document 1).

Japanese Patent Application Publication No. 2019-016227

It would be desirable to have a technique for efficiently collecting images of a specific scene, such as a scene desired by a user, in addition to photographic data obtained by photographing under abnormal conditions.

In view of the above-mentioned problems, one object of the present invention is to provide a technique for detecting a specific image scene that can be captured by a vehicle.

The purpose of the present invention is not limited thereto, and the present invention intends to acquire rights through divisional applications, amendments, etc. for structures that aim to obtain effects from the parts of the structures disclosed in this specification, drawings, etc. For example, the present specification discloses a problem in which passages such as ``can be done'' or ``possible'' are read as ``the problem is.'' Each issue is described as an independent entity, and we intend to acquire rights to the structure for solving each issue individually through divisional applications, amendments, etc. Even if the problem is implicitly understood from the description of the specification, the present applicant has the intention to claim a part of the structure described in the specification in an amendment or divisional application. The company also discloses a structure that solves a problem that combines these independent problems, and has the intention to acquire the rights to it.

(1) The system according to the present invention detects an image representing the scene from images taken by a vehicle at each of a plurality of time points, based on data defining a scene of a specific image that can be taken by the vehicle. Has a function.

In this way, it is possible to provide a technique for detecting a specific image scene that can be captured by a vehicle.

(2) It has a function of taking pictures with a camera installed in the vehicle, and a function of recording images taken by the camera at each of a plurality of points in time on a recording medium, and the detecting function has the function of taking pictures with a camera installed in the vehicle. It is preferable to detect an image representing a scene whose degree of coincidence with the scene of the specific image is equal to or higher than a threshold value among the images recorded on the medium at each of the above-mentioned points in time.

In this way, it is possible to detect an image representing a scene whose degree of coincidence with the scene of a specific image is equal to or higher than a threshold value from images taken by a camera installed in a vehicle.

(3) The above data may be an image at one point in time.

If the specific image is at one point in time, the time required to detect a scene can be shortened.

(4) The above data may be images at two or more points in time.

If the specific images are images at two or more points in time, the accuracy of the scene to be detected can be improved. Images taken at two or more time points are preferably temporally continuous images (images in which the main subject in the image moves continuously) like a moving image.

(5) The data may be generated by deep learning by inputting an image that is the same as the specific image or a plurality of images that are similar to the specific image.

In this way, a specific image scene that can be captured by a vehicle can be detected using deep learning technology.

(6) The above data is a trained first learning model that has been subjected to machine learning using the above specific images as training data, and the above detection function uses images taken at each of the above points to use the trained first learning model. It is preferable to detect an image representing the above-mentioned scene by inputting the following.

In this way, a specific frame can be detected relatively quickly.

(7) The specific scene may include, for example, at least one of a scene that may lead to an accident or disaster, or a scene that has led to an accident or disaster.

In this way, a scene that is likely to lead to an accident or disaster, or a scene that is close to a scene that leads to an accident or disaster can be detected from the video.

(8) The apparatus may further include warning control means for controlling the warning device to issue a warning in response to detection of an image representing the scene.

For example, if the above system is installed in a vehicle and records a video while shooting, and the scene that is similar to a specific image is found in the video, the vehicle It is possible to warn drivers of

(9) The warning control means may control the warning device to change the content of the warning depending on the type or degree of danger of the scene.

In this way, the driver of the vehicle can easily understand the degree to which he or she should be careful.

(10) It is preferable to further include a first recording control means for controlling the recording device to record the image of the detected scene on a recording medium.

In this way, it is possible to record an image of a frame of a specific scene, and for example, it is possible to easily refer to the image of that frame later.

(11) The detecting function preferably detects an image representing the scene based on the data and a physical quantity indicating the driving state of the vehicle.

In this way, a desired scene can be detected with higher accuracy by taking into consideration the driving situation of the vehicle.

(12) Images taken by the vehicle at each of the above points are stored in the recording medium, and a playback device is controlled to play back the images recorded in the recording medium and taken by the vehicle at each of the above points. Further comprising a reproduction control means, and a notification control means for controlling the notification device to notify the detected image representing the scene in association with the image reproduced on the reproduction device under the control of the reproduction control means. good.

This makes it easier to detect a scene that resembles a specific image when reproducing an image recorded on a recording medium.

(13) The above-mentioned recording medium stores images taken by the vehicle at each of the above-mentioned times, with the period from when a recording start command is given to when a recording stop command is given as one period, and the above-mentioned one period and a second recording control means for controlling a recording device to record the image generated by the generation means in the defective portion. It's good to be prepared.

In this way, the missing portion of the video can be restored.

(14) The generating means inputs at least one second image before or after the first portion corresponding to the missing portion in images taken at each of a plurality of times in another vehicle, An image corresponding to the missing part is generated by inputting at least one image before or after the missing data to a trained second learning model that has undergone machine learning using the first part as an output. It's good to do that.

In this way, a moving image of the missing portion can be generated relatively accurately.

(15) A computer may be provided with a program for realizing the functions of the above system.

In this way, by installing the program into the device, the device can detect a specific image scene that can be photographed by the vehicle.

(16) The trained model according to the present invention uses specific images that can be taken by a vehicle as training data, inputs images taken by the vehicle at each of a plurality of points in time, and outputs the above images from the input images. Let us detect an image representing a specific image scene.

In this way, it is possible to provide a trained model for detecting a specific image scene that can be captured by a vehicle.

(17) The learning model generation method according to the present invention uses specific images that can be taken by a vehicle as training data, inputs images taken by the vehicle at each of a plurality of time points, and outputs the input images. A learning model is generated from which the image represents the scene of the specific image.

In this way, it is possible to generate a learning model that can detect a relatively large number of images of a specific scene or detect images of a specific scene with high accuracy.

(18) The above-mentioned specific image may be an image photographed by a plurality of photographing devices.

In this way, it is possible to generate a learning model that can detect a relatively large number of images of a specific scene or detect images of a specific scene with high accuracy.

(19) The photographing device is a drive recorder installed in a vehicle, and the specific image is a scene when a recording command is given to the drive recorder, or a scene of the vehicle to which the drive recorder is installed. It is best to describe the scene when a shock is applied to the scene.

In this way, it is possible to generate a learning model that detects an image of a scene when a recording command is given to the drive recorder or when an impact is applied to the vehicle to which the drive recorder is attached.

(20) It is preferable to detect an image representing a scene in which a target object is moving at a speed or acceleration higher than a certain level.

In this way, it is possible to generate a learning model that detects images that include objects that have moved at a speed or acceleration above a certain level.

(21) It is preferable to generate a learning model for detecting an image representing the scene based on the data and a physical quantity indicating the driving state of the vehicle.

In this way, a learning model that can detect a desired scene with higher accuracy can be generated by taking into consideration the driving situation of the vehicle.

(22) A first image that is an image of a predetermined period among images taken at each of a plurality of time points, and a second image that is at least one image before or after the first image. is used as training data, the first image is used as an input, the second image is used as an output to learn, and a third image taken at each of a plurality of time points including the missing part during a predetermined period is acquired. It is preferable to estimate the fourth image, which is the image of the missing portion, from the input third image.

In this way, a learning model for generating an image of the missing portion can be generated.

(23) The trained model generation device according to the present invention generates a learning model using the learning model generation method of any one of (17) to (22).

In this way, a learning model that can provide a trained model for detecting a specific image scene that can be captured by a vehicle can be generated.

The inventions shown in (1) to (22) above can be combined arbitrarily. For example, at least a part of the structure of at least one of the inventions (2) to (23) may be added to all or part of the structure of the invention shown in (1). In particular, it is preferable to create an invention in which at least a part of the structure of at least one of the inventions (2) to (23) is added to the invention shown in (1). Further, arbitrary configurations may be extracted from the inventions shown in (1) to (23) and the extracted configurations may be combined.

The applicant of this application intends to acquire rights to inventions containing these structures. Furthermore, even if there is a description of "in the case of" or "at the time of", the description is not intended to be limited to those cases or times. These are examples of better configurations, and we intend to acquire rights to these cases and other configurations as well. Furthermore, the sections described in order are not limited to this order. It also discloses a configuration in which some parts have been deleted or the order has been changed, and we have the intention to acquire the rights.

According to the present invention, it is possible to provide a technique for detecting a specific image scene that can be captured by a vehicle.

Note that the effects of the invention of the present application are not limited to these, and effects obtained from the parts of the configuration disclosed in the present specification, drawings, etc. are also disclosed, and the configurations that provide the effects are also disclosed by divisional applications, amendments, etc. Have the intention to acquire the rights. For example, in this specification, passages such as "can be done," "is possible," etc. are descriptions that clearly indicate the effect to be achieved, and even if there is no description such as "can be done," or "it is possible," the effect can be obtained. There is a part shown. Further, even without such a description, there are effects that can be understood from the configuration.

This is an example of a drive recorder. This shows a drive recorder installed inside the car. FIG. 2 is a block diagram showing the electrical configuration of a drive recorder. FIG. 2 is a block diagram showing the electrical configuration of a server. This is an example of a learning model. This is an example of a target image. This is an example of a flowchart for generating a learning model. This is an example of a learning model that generates multiple target images. (A) to (C) are examples of target images. FIG. 2 is a block diagram showing the electrical configuration of a generative adversarial network that generates a target image. This is an example of a learning model. 3 is a flowchart showing a processing procedure for event recording. This is an example of a learning model. It is a flowchart which shows a matching degree calculation process procedure. This is an example of a frame. This is an example of frames that make up a video. This is an example of a learning model. This is an example of a personal computer. This is an example of a recording format. This is an example of a playback window. 3 is a flowchart showing a reproduction processing procedure. This is an example of a playback window. This is an example of a playback window. (A) and (B) show the relationship between frame numbers and address position information. 5 is a flowchart showing a repair processing procedure. This is an example of a recording format. This is an example of a learning model. This shows how the learning model is trained. It is a flowchart which shows a learning model generation process procedure. 3 is a flowchart illustrating a procedure for generating a missing portion of a moving image.

Hereinafter, embodiments for carrying out the present invention will be described with reference to the drawings. In addition, the embodiment shown below is one embodiment which provided this invention, and the content of this invention is not limited and interpreted based on the following description.

[How the invention of the present application was conceived]
In current drive recorders, event recording is automatically (passively) performed according to the detection results of sensors such as acceleration sensors, but the only way for the user to actively record events is by switching. There were situations in which video recording could not be performed if the sensors did not respond and the user was unable to press the switch (for example, the driver was not inside the vehicle while driving or parking the vehicle). . Therefore, we use methods such as deep learning to learn about accidents and dangerous situations in which sensors do not respond, using video data, and automatically record events when the same or similar situations occur. The inventor thought of this solution (first and second embodiments, etc.). In this way, this solution supplements event recording with machine learning, and learns from images of situations in which sensors do not respond. In addition, instead of recording events that capture situations such as accidents, etc., the scenery while driving or the point the user wants to take is learned in advance as a teacher image, and similar situations are automatically determined and recorded using the drive recorder. The inventor thought that it could also be recorded. The inventor thought that the use of the drive recorder would increase if it could be used as a photographing device rather than for capturing images of accidents, etc. (Third Example, etc.).

[Summary of each example]
Each of the embodiments described below detects an image representing the scene from images taken by the vehicle at each of a plurality of time points, based on data defining a scene of a specific image that can be taken by the vehicle. Regarding technology. In each embodiment, the specific image is an image that defines a desired scene (also referred to as a desired image). "Detecting an image representing the scene" includes generating a trigger to perform predetermined processing. The predetermined process is event recording in the first and second embodiments, and in the third embodiment, it is displaying information about an image representing the scene, and when the displayed information is selected, a corresponding action is taken. displaying an event-recorded video including an image representing the scene based on video data representing the video; fast-forwarding to the point where the image of the scene is displayed in response to this selection during video playback; This includes jumping to the relevant point during playback. The process is not limited to these, and "detecting an image representing the above scene" may include processing such as associating identification information indicating detection with the image. In each embodiment, a method will be described in which a frame representing a scene whose degree of match with the scene of a desired image is equal to or higher than a threshold value is found among a plurality of frames constituting a video represented by video data. . A moving image is constructed by arranging images (here, still images, and also referred to as frames in the embodiment) at each of a plurality of time points on a time axis. When a video is played back based on video data, images are played back sequentially along the time axis. The specific image may be an image at one time point (that is, a still image), an image at two or more time points, or a temporally continuous image representing frames that make up a video. The images may be taken at two points in time or at three or more points in time. The desired image is an image of an accident such as a car accident (not limited to a car accident), a scene that may lead to a disaster such as a natural disaster (not limited to a natural disaster), or a scene that leads to an accident or disaster. In addition to this, any image may be used, such as an image of a desired landscape. Furthermore, as will be described later, the image may be an image of a scene in which a recording command is given to the drive recorder using an operation button or the like, or a scene in which a shock is applied to the vehicle in which the drive recorder is mounted.

[First example]
1 to 11 show a first embodiment. The first embodiment uses a plurality of images that are the same as a desired image or an image that approximates the desired image, or a plurality of images that are the same as the desired image and an image that approximates the desired image. A target image is generated using deep learning, and the generated target image is used as a desired image. Also, generation of a trained model will be explained.

FIG. 1 is a perspective view of a drive recorder 1 (which is an example of a system) viewed diagonally from the rear. In the following description, the drive recorder 1 is a device that is installed later on the vehicle in this embodiment. The vehicle is, for example, a private car or a business car (passenger car). However, the vehicle may be a special vehicle such as a bus, truck, or forklift, or a public transportation vehicle such as a train, monorail, or linear motor car.

An SD card insertion slot 10 is formed on one side of the housing of the drive recorder 1. A display 11 is formed on almost the entire back surface of the housing of the drive recorder 1, and a plurality of operation buttons 12 are provided on both left and right sides with the display 11 in between. A joint rail 13 is provided on the top surface of the drive recorder 1. Although not shown in FIG. 1, a camera lens is provided on the front of the drive recorder. A DC jack of the drive recorder 1 is formed on the side surface not shown in FIG. 1, and a speaker and an HD (high definition) output terminal are formed on the bottom surface of the drive recorder 1.

A camera including a lens captures an image in front of a vehicle, for example. Preferably, the interior of the vehicle and the exterior of the vehicle (lateral direction, rear of the vehicle, etc.) may be photographed. The DC jack is a jack for connecting to a DC power source via a power cable. The SD card insertion slot 10 is used to insert an SD card [this is an example of a memory card, and may be any other memory card such as an xD picture card, or may be a memory other than a memory card such as a USB (Universal Serial Bus) memory]. It is an insertion port for A speaker outputs sound or audio. The HD output terminal is a terminal for connecting to other information equipment via a cable. The joint rail 13 is used to attach a joint for mounting the drive recorder 1 on a vehicle. The display 11 displays various images such as images taken by the camera of the drive recorder 1. The operation button 12 is used by the user to input various commands to the drive recorder 1.

FIG. 2 shows a view looking forward from the interior of a vehicle in which the drive recorder 1 is mounted.

A room mirror 4 is provided near the center of the upper part of a windshield 3 of a vehicle. A drive recorder 1 is fixed to the vehicle by a joint at a position on the passenger seat side (on the left side when facing the front in FIG. 2) adjacent to the rearview mirror 4.

A DC jack of a drive recorder 1 is connected to a cigarette lighter socket 5 via a power cable 6. When the accessory power source of the vehicle is turned on, power is supplied from the cigarette lighter socket 5 to the drive recorder 1. Note that the drive recorder 1 is not limited to the configuration shown in FIGS. 1 and 2, and may be equipped with a spherical camera or a hemispherical camera, for example.

FIG. 3 is a block diagram showing the electrical configuration of the drive recorder 1. As shown in FIG.

The drive recorder 1 includes a controller 20. This controller 20 has a CPU (Central Processing Unit) 20a that controls the entire operation of the drive recorder 1, a ROM (Read Only Memory) 20b that stores the operation programs of the drive recorder 1, and data etc. temporarily. A RAM (Random Access Memory) 20c for storage and a timer 20d are included. The controller 20 has functions such as a GPS information processing program, a video processing program, and a communication processing program.

The drive recorder 1 also includes a GPS (Global Positioning System) receiver 16, a camera 17, an SD card reader/writer 14, and an acceleration sensor 18. The GPS receiver 16 detects the position of the drive recorder 1, and the position of the vehicle in which the drive recorder 1 is mounted can be known. The camera 17 photographs the front of the vehicle in which the drive recorder 1 is mounted as described above. When the SD card 23 is inserted into the SD card insertion slot 10 as described above, the SD card reader/writer 14 reads the data recorded on the inserted SD card 23 and transfers the data to the SD card 23. Write. The acceleration sensor 18 detects the vertical, horizontal, and longitudinal acceleration applied to the drive recorder 1, and can detect the vertical, horizontal, and longitudinal acceleration of the vehicle in which the drive recorder 1 is installed. becomes.

GPS data representing the position of the drive recorder 1 outputted from the GPS receiver 16, video data representing the image taken by the camera 17, and a signal representing the acceleration detected by the acceleration sensor 18 are input to the controller 20, respectively. . By executing the GPS information processing program, the controller 20 can record the GPS data output from the GPS receiver 16 onto the SD card 23 inserted into the SD card insertion slot 10. Furthermore, by executing the video processing program, the controller 20 can record the video data obtained from the camera 17 in the SD card 23 in association with the time.

Furthermore, the drive recorder 1 includes a speaker 15 that outputs the above-mentioned audio and the like, a display 11 that displays images and the like, operation buttons 12, a communication circuit 19, and an LTE module 21.

The controller 20 outputs audio data, etc. to the speaker 15, so that the speaker 15 outputs the audio, and the controller 20 outputs the video data, etc. to the display 11, so that the display 11 displays images. Various commands from the operation button 12 are input to the controller 20.

The communication circuit 19 functions as a communication means for wirelessly communicating with external devices such as servers, personal computers, smartphones, tablet terminals, and the like. The controller 20 has a function of transmitting data such as video to an external device via the communication circuit 19 by executing a communication processing program. As the communication circuit 19, it is preferable to use one that complies with short-range wireless communication standards such as the WiFi standard and Bluetooth (registered trademark). The short-range wireless communication standard can be applied, for example, to communication between work vehicles (forklifts, etc.) operating within a premises and external devices. Mobile communication system standards can be applied, for example, to communications between vehicles that move within a wider area and external equipment.

The LTE (Long Term Evolution) module 21 is a communication circuit that complies with the communication standard of mobile phones. Instead of the LTE module 21, a communication circuit compliant with other mobile communication system standards such as 4G may be used. The LTE module 21 can transmit video data captured by the camera 17 to an external server or the like.

The operation program of the drive recorder 1 may be recorded in advance on the SD card 23, or may be received by the LTE module 21 via the Internet and recorded on the SD card 23, or the SD card 23 may be stored on the personal computer. The data may be inserted into a personal computer, connected to the Internet via a personal computer, downloaded from the personal computer, and recorded on the SD card 23.

FIG. 4 is a block diagram showing the electrical configuration of a server 30 that communicates with the drive recorder 1.

The server 30 includes a control device 31 that controls the entire operation.

Connected to the control device 31 of the server 30 are a communication device for communicating with the drive recorder 1 and other client computers, a memory 33 for storing data, and a hard disk drive . The hard disk drive 34 accesses the hard disk 35 that stores moving image data and the like, reads the moving image data recorded on the hard disk 35, and records the moving image data and the like on the hard disk 35.

FIG. 5 is an example of the learning model 40.

The learning model 40 shown in FIG. 5 generates a target image by inputting a plurality of images by deep learning. The learning model 40 includes an input layer 41, an intermediate layer (hidden layer) 42, and an output layer 43.

Input each of the pixels P1 to PN constituting an arbitrary image (including not only a desired image and an image approximated to the desired image but also other images) from the input layer 41 of the learning model 40, Learning by deep learning through the intermediate layer 42 and output layer 43 is performed on a huge number of arbitrary images. After that, when a desired image or an arbitrary image that includes many images that approximate the desired image is input from the input layer 41 of the learning model 40, neurons 43a that respond to the desired image or images that approximate the desired image are searched for. be able to. When an unknown image is input from the input layer 41 of the learning model 40 and the neuron 43a responds, it can be determined that the input unknown image is a desired image or an image that approximates the desired image. Thereby, image recognition can be performed using the learning model 40.

Furthermore, by tracing back from the responsive neuron 43a to the input layer 41 side, it is possible to generate a target image that is most likely to be the desired image. The target image generated in this way becomes a desired image generated by deep learning (unsupervised learning).

For example, images that are likely to lead to an accident, images that are likely to lead to a disaster, images that have led to an accident, images that have led to a disaster, and a large number of images unrelated to other accidents or disasters are input to the learning model 40, and the deep When learning is performed, a neuron that responds when an image that is likely to lead to an accident is input to the learning model 40 (referred to as the first neuron), and a neuron that responds when an image that is likely to lead to a disaster is input to the learning model (referred to as the first neuron). A neuron that reacts when the image that led to the accident is input to the learning model (selected as the second neuron), a neuron that responds when the image that led to the disaster is input to the learning model (selected as the third neuron), The fourth neuron) etc. can be understood.

By tracing back from the first neuron that reacted to the input layer 41 side, it is possible to generate a target image that is likely to lead to an accident, and by tracing the image backward from the second neuron that responded to the input layer 421. , it is possible to generate a target image that looks like an image that is likely to lead to a disaster, and by tracing back from the reacted third neuron to the input layer 41 side, it is possible to generate a target image that looks like an image that is likely to lead to an accident. By tracing back from the reacted fourth neuron to the input layer 41 side, it is possible to generate a target image that is similar to the image that led to the disaster. The generated target image can be used as a desired image.

FIG. 6 is an example of the target image 50 generated as described above, and is an example of an image that is likely to lead to an accident.

The target image 50 is an image generated based on deep learning, and is different from an actually photographed image. However, it is possible that the image may be almost the same as the image that was actually photographed by chance.

As an example of this embodiment, a frame of a scene whose degree of coincidence with the scene of the target image 50 is equal to or higher than a threshold value is found among the frames constituting the moving image.

FIG. 7 is a flowchart showing the target image generation processing procedure.

The target image generation processing procedure shown in FIG. 7 may be performed at any timing as long as a large number of images are obtained. Further, if image data representing a large number of images is recorded in the SD card 23 or database 22, the process may be performed in the drive recorder 1, or if image data representing a large number of images is stored in the hard disk 35 of the server 30. If it is recorded, it may be performed on the server 30. When collecting images from drive recorders (multiple drive recorders) installed in a plurality of vehicles, it is desirable that the server 30 executes a target image generation processing procedure. Furthermore, if you have access to the Internet, even if many images are not recorded on the SD card 23, database 22, or hard disk 35, by reading many images from the Internet, you can set the target on either the drive recorder 1 or the server 30. Image generation processing may also be performed. Furthermore, target image generation processing may be performed in a client computer (personal computer).

A large number of arbitrary images are input to the learning model 40 (step 61), and deep learning is performed in the learning model 40 (step 62). A target image is obtained as described above (step 63).

Needless to say, the learning model 40 program is installed in the controller 20 of the drive recorder 1 or the control device 31 of the server 30.

FIG. 8 shows a method for generating multiple target images.

The first learning model 40A, the second learning model 40B, and the third learning model 40C include an input layer, a middle layer, and an output layer, similar to the learning model shown in FIG.

The image input to the first learning model 40A is referred to as the first image, the image input to the second learning model 40B is referred to as the second image, and the image input to the third learning model 40C is referred to as the third image. do. As explained with reference to FIG. 5, when a huge number of images are input to the first learning model 40A and it is trained by deep learning, neurons that respond to specific images (for example, images that are likely to lead to an accident) is found, and by tracing that neuron to the input layer, an image representing the characteristics of a particular image is obtained. For example, a first target image is obtained that represents features that are likely to lead to an accident. Similarly, when a large number of images are input to the second learning model 40B and it is trained by deep learning, neurons that respond to a specific image (for example, an image that is likely to lead to a disaster) are found, and that neuron By tracing the image to the input layer, an image representing the characteristics of a specific image can be obtained. For example, a second target image is obtained that represents features that are likely to lead to a disaster. When a huge number of images are input to the third learning model 40C and it is trained by deep learning, neurons that respond to a specific image (for example, the image that led to the accident) are found, and those neurons are traced back to the input layer. An image representing specific image characteristics is obtained. For example, a third target image is obtained that represents the characteristics of the image that led to the accident. In this way, multiple types of target images are obtained.

In FIG. 8, multiple types of target images are obtained using multiple learning models 40A to 40C, but even in one learning model 40 as shown in FIG. By searching for neurons that respond to the second target image, neurons that respond to the third target image, etc., the first target image, second target image, third target image, etc. are traced to the input layer 41. can be found.

In the example shown in FIG. 8, it is preferable that the first image, the second image, and the third image be images that are sequentially photographed in chronological order. As a result, the obtained first target image, second target image, and third target image also represent continuous images in chronological order. For example, if the first target image, the second target image, and the third target image represent characteristics of images that are likely to lead to an accident, then the first target image, the second target image, and If frames close to the third target image appear one after another in the video, it can be determined that there is a higher possibility of an accident in such a situation, and that only one target image is likely to lead to an accident. The accuracy will be higher than when making a judgment.

FIGS. 9A, 9B, and 9C are examples of the first target image, second target image, and third target image generated as described above.

It can be seen that as the first target image 51, second target image 52, and third target image 53 change, the child appearing as the subject gets closer to the car, and the possibility of an accident increases. These first target image 51, second target image 52, and third target image 53 are set as desired images, and these first target image 51, second target image 52, and third target image 53 are set as desired images. If a similar scene can be found in a video, it can be determined that there is a high probability that an accident will occur in that scene.

In the above embodiment, the three first target images 51, the second target image 52, and the third target image 53 are the desired images, but two or four or more target images may be generated, It may be a desired image.

FIG. 10 shows another method of generating a target image, and is a block diagram showing the electrical configuration of an adversarial generation network.

Feature amount data representing the feature amount of a desired image is input to the generator 71. The generator 71 generates pseudo image data representing a pseudo image of a desired image from the input feature data. The generated pseudo image data is input from the generator 71 to the discriminator 72. Image data representing a desired image is also input to the discriminator 72. The classifier 72 identifies whether the desired image represented by the image data representing the desired image and the pseudo image represented by the pseudo image data are the same or similar images. Data representing the identification result is input to each of the generator 71 and the weighting changer 73. The generator 71 is adjusted based on the data representing the classification result, and the weighting changer 73 changes the weighting of the feature amount of the classifier 72.

When the generation of pseudo image data in the generator 71, the identification in the discriminator 72, the adjustment based on the discrimination result in the generator 71, and the change in the weighting of the discriminator 72 in the weighting changer 73 are repeated, the generator 71 generates The discriminator 72 determines that the pseudo image represented by the generated pseudo image data and the desired image are the same or similar. The pseudo image determined in this way is set as the target image. A plurality of images of the same type may be used as desired images, and a pseudo image obtained by repeating the above-described process may be used as the target image. The target image obtained in this manner becomes a desired image found from the frames constituting the moving image.

In the above embodiment, a target image is generated using an arbitrary image and the generated target image is used as the desired image, or a pseudo image is generated using the desired image and the generated pseudo image is used as the desired image. However, if you already know the images that are likely to lead to an accident or disaster, the images that led to the accident or disaster, or the images of the scene that the user wants to search, you can use target images, pseudo images, etc. These images themselves may be used as desired images without generating them using deep learning or the like. For example, if there is a video of an event record (for example, a video recorded when a drive recorder installed in a vehicle records a video when the vehicle is subjected to an impact, etc.), the event record One or more frames constituting a moving image may be used as the desired image.

FIG. 11 shows a method for training a learning model to generate a trained model.

This is another example of generating a target image, and shows an example of a learning model.

The learning model 80 includes an input layer 81, an intermediate layer (hidden layer) 82, and an output layer 83.

Using a desired image as teacher data, data representing pixels P1 to PN constituting the desired image are input to the input layer 81 in the order of constituting the image. The input layer 81, the intermediate layer 82, and the output layer 83 are configured such that the same data as pixels P1 to PN inputted to the input layer 81 via the intermediate layer 82 is outputted from the output layer 83 in the same order as the desired image. Adjust the feature values, weighting, etc. as necessary. That is, machine learning is performed so that the output of the learning model 80 is almost the same as the input, and the feature amounts, weighting, etc. of the learning model 80 are adjusted as necessary.

The number of data representing pixels input to the input layer 81 may be reduced, or the number of data representing pixels output from the output layer 83 may be reduced. The feature amounts, weighting, etc. of the learning model 80 may be adjusted as necessary so that the data representing pixels at the same position in the image is almost the same between input and output.

When data representing an arbitrary image (for example, each frame constituting a video) is input to a learning model that has been trained in this way, the data of each pixel output from the output layer 83 will be The data is close to that of pixels P1 to PN that constitute the desired image.

According to the first embodiment, a trained model can be obtained that can generate a desired image and can be used to detect a frame of a scene that approximates the scene of the desired image from among a plurality of frames constituting a moving image.

[Second example]
12 to 16 show a second embodiment. In the second embodiment, a trained learning model 80 generated with reference to FIG. 11 is used to search for a frame that has a high degree of matching with a desired image from among a plurality of frames constituting video data. . In the second embodiment, video data shot by a photographing device is recorded, the recorded video data is read, and a frame that has a high degree of matching with a desired image is searched from among multiple frames that make up the read video data. ing. In the following, it will be explained that the acceleration detected by the acceleration sensor 18 is used to detect a desired image, and that there is a process for determining whether there is an object moving at a speed or acceleration above a certain level in the video. Although used in combination, any desired image may be detected by at least applying the image photographed by the drive recorder 1 to the trained model generated by the method described in the first embodiment. .

FIG. 12 is a flowchart showing an event recording process performed by the drive recorder 1 shown in FIGS. 1 and 2 during continuous recording.

A program for implementing the processing procedure shown in FIG. 12 may be installed in the controller 20 of the drive recorder, or may be stored in the SD card 23 (which is an example of a recording medium storing the program). However, the information transmitted via the network may be received by the communication circuit 19 and installed in the drive recorder 1.

When the drive recorder 1 is installed in a vehicle, when the engine of the vehicle is started, the camera 17 of the drive recorder 1 starts shooting, and the video data obtained by shooting is transferred to the SD card reader/writer 14. Recording to the SD card 23 (constant recording) is started. Continuous recording starts when the vehicle engine is started and ends when the vehicle engine is turned off. Further, even when the engine of the vehicle is not running, when a recording start command is given to the controller 20 from the operation button 12, recording is always started, and when a recording end command (which is an example of a recording stop command) is given from the operation button 12, the controller 20 receives a recording start command from the operation button 12. When given, continuous recording ends. The period from the start to the end of continuous recording becomes one moving image of continuous recording. Further, in this embodiment, moving image data obtained by photographing with the camera 17 is also given to the RAM 20c and recorded therein. The moving image data recorded in the RAM 20c is periodically recorded only for a certain period of time, and is repeatedly overwritten after the certain period has elapsed.

In this embodiment, when an event is detected by the acceleration sensor 18 or by a recording command given by the operation button 12 (YES in step 91), the event is recorded in the event recording area of the SD card 23. is performed (step 98).

If the event is not detected by the acceleration sensor 18 or the recording command (NO in step 91), the event is recorded in the RAM 20c (an example of a recording medium) by the controller 20 (an example of a first recording control means). The video data represented by the read video data is read, and the controller 20 determines whether there is an object moving at a speed or acceleration above a certain level in the video represented by the read video data (step 92). This object may or may not be determined in advance. It may be the relative speed or acceleration of the object and the vehicle in which the drive recorder 1 is mounted, or it may be the absolute speed or acceleration of the object.

If there is an object moving at a speed or acceleration higher than a certain level in the video represented by the video data read (YES in step 92), the video data read from the RAM 20c is used as a trained object. The model 80 is input one frame at a time (step 93). Learning processing is performed in the learning model 80, and image data representing feature images is obtained frame by frame from the output layer 83 of the learning model 80 (step 94). Although learning model 80 is described as being hardware, it is actually implemented in software in controller 20.

The degree of coincidence between the obtained feature image and the desired image is calculated by the controller 20 (step 95).

FIG. 13 shows how video data is input frame by frame to the trained learning model 80.

Pixels P11 to P1N constituting one frame are input to an input layer 81 according to the arrangement of one frame, and theta of pixels P21 to P2N is output as image data representing a characteristic image via an intermediate layer 82 and an output layer 83. .

The data of these pixels P21 to P2N representing the characteristic image are compared with the data of each pixel constituting the desired image, and the controller 20 performs a matching degree calculation process between the characteristic image and the desired image.

FIG. 14 is a flowchart showing the processing procedure for calculating the degree of matching.

First, the degree of coincidence between the desired image and the feature image is calculated as a first degree of coincidence (step 131). Next, when the desired image is obtained from the drive recorder 1 mounted on the vehicle (which is an example of a second vehicle), vehicle information (which is an example of traveling information) and characteristic images are obtained. The degree of agreement between the frame input to the learning model 80 and the vehicle information obtained from the drive recorder 1 mounted on the vehicle (which is an example of the first vehicle) is calculated as the second degree of agreement. (Step 132). The driving information includes, for example, physical quantities indicating the state of the vehicle, such as the speed of the vehicle, the attitude of the vehicle body, and the acceleration of the vehicle. Such a physical quantity is a physical quantity that changes as the vehicle travels. In addition to this, the traveling information may also include information that changes as the vehicle travels, such as the location of the vehicle (position information). Further, an overall degree of coincidence is calculated from the first degree of coincidence and the second degree of coincidence (step 133).

In the processing procedure shown in Fig. 14, vehicle information at the time when the desired image was obtained is used, so if the desired image is a virtual target image generated using a trained learning model, will be determined as the overall degree of coincidence using the first degree of coincidence without calculating the second degree of coincidence. However, even if the desired image is a virtual target image, if vehicle information about the image used to obtain the target image is known, that vehicle information can be used to calculate average vehicle information. The second matching degree may be calculated by calculating .

Returning to FIG. 12, if the calculated matching degree (total matching degree) is equal to or higher than the threshold value (YES in step 96), among the frames constituting the moving image represented by the moving image data recorded in the RAM 20c, When a frame representing a scene whose degree of coincidence with the desired image scene is equal to or higher than a threshold value is detected, an event recording command is issued from the controller 20 (an example of a warning control means), and a warning is issued (step 97). The warning may be an audio warning output from the speaker 15 of the drive recorder 1 (which is an example of a warning device), a warning by text, an image, etc. displayed on the display 11 (which is an example of a warning device), A warning may be provided by emitting or blinking an LED (light emitting diode). Further, the warning may be issued using a speaker, a display screen, etc. provided in the vehicle in which the drive recorder 1 is mounted. The warning allows the driver of the vehicle to know that the vehicle is in a driving situation (being in a driving situation) of a scene close to the scene of the desired image. For example, if the desired image is an image that is likely to lead to an accident, the user can drive more carefully. Furthermore, when an event recording command is issued, event recording is performed in the event recording area of the SD card 23 (step 98).

In the above embodiment, when the target object is moving at a speed or acceleration above a certain level (YES at step 92), the photographic data is input to the trained learning model. You can input photographic data into a trained learning model regardless of whether the object is moving at a speed or acceleration of The data may be input into a trained learning model.

FIG. 15 is an example of one of a plurality of frames constituting a moving image represented by moving image data read from the RAM 20c. FIG. 16 shows a plurality of frames constituting a moving image represented by moving image data read from the RAM 20c.

As shown in FIG. 16, a video represented by video data is composed of a plurality of frames FR1 to FRe, and the degree of matching (overall matching) obtained when frames FRn shown in FIGS. degree) is greater than or equal to the threshold value. Then, a frame representing a video shot from several seconds to several tens of seconds (any other time is fine) before that frame FRn, and a frame representing a video taken from several seconds to several tens of seconds (any other time is fine) after frame FRn. Data each representing a frame representing a moving image is recorded in the event recording area of the SD card 23 by the SD card reader/writer 14.

The content of the warning in the above-described embodiment may be changed depending on the type of scene or the degree of danger. For example, in a highly dangerous scene, a warning is given both audibly and visually, or the volume is increased so that the driver can recognize it. If a low-risk scene is found, for example, a scene similar to a favorite place, the system will notify the driver with a relatively quiet voice.

FIG. 17 shows a modification of the learning model 80 shown in FIG. 13.

The learning model 105 shown in FIG. 17 includes an input layer 106, an intermediate layer 107, and an output layer 108, and the output layer 108 is a logistic regression layer.

A plurality of frames constituting a video are input into the learning model 105 one frame at a time. For example, data representing pixels P31 to P3N constituting one frame is input to the input layer 106 of the learning model 105 according to the pixel array. In such a case, if the frame has a high degree of matching with the desired image, identification data is output from the output layer 83, which is a logistic regression layer. By outputting the identification data, it is determined that the degree of coincidence between the desired image and the frame input to the learning model 105 is greater than or equal to the threshold value, and an event recording command is issued from the controller 20. The learning model 105 also receives image data representing a large number of desired images as input from the input layer 106, and is trained to output identification data from the output layer 108 when image data representing the desired image is input from the input layer 106. This is a trained learning model generated by

In addition, when the identification data is output, the degree of coincidence between the vehicle information when the frame input to the learning model 105 is obtained and the vehicle information corresponding to the desired image (the second degree of coincidence described above) is the threshold. An event recording command may be generated when the value is greater than or equal to the value.

Even without an event recording command from the acceleration sensor 18 or an event recording command from the operation button 12, event recording can be performed based on the captured video. Furthermore, when calculating the overall matching degree using vehicle information, not only the captured video but also the vehicle information is used, so it is possible to record events more appropriately depending on the vehicle situation.

In the embodiment described above, when a frame similar to a desired image is detected in a video, an event is recorded and video data is recorded on the SD card 23. Identification information for identifying a frame similar to the image may be recorded on the SD card 23 or other recording medium.

In the embodiment described above, the presence or absence of event recording is determined using the overall degree of coincidence; , the presence or absence of event recording may be determined. However, if physical quantities indicating the driving state of the vehicle are combined, it can be expected that frames of a desired scene can be detected with higher accuracy. .

[Third example]
This embodiment proposes an automatic playback support function in viewer software of recorded data. Video data from a drive recorder for management purposes is often long-term and basically has no problems, and if it is an event, the point of occurrence will be displayed on the viewer software, but the point of interest that has not become an event will be displayed. It was sometimes difficult for administrators to discover and confirm the information. Therefore, the inventor learned the desired video situation using deep learning, etc., and when there is a video in the recorded data that is the same or similar to the desired situation, it is identified and detected as a gaze point separately from the event. The inventor has devised a solution that makes it easier for administrators to check the content by displaying it on viewer software, etc., or returning it to normal speed at event occurrence points and gaze points during fast-forward playback.

Additionally, this embodiment proposes a recovery function for data such as video recorded on a recording medium.
For example, when using the recording format described below, data is written sequentially to the storage medium rather than the FAT method used in Windows (registered trademark), so even if the video is in an abnormal state (deleted Although the data itself is recorded (even if some sectors exist), if a special abnormality occurs, a certain sector of the video data may be erased with zeros or other information may be written, causing the viewer to There was a problem with the software not being able to play the video. In this case, the contents of the recording medium may be checked sector by sector and restored manually, but methods such as deep learning are used to learn the status of the recording medium and the erased data can be recovered from the data before and after the missing part. The inventor has devised a solution that not only restores the data, but also restores the recording format to a normal state so that it can be confirmed using viewer software or the like. The details of this embodiment will be explained below.

FIGS. 18 to 29 show a third embodiment, in which a moving image recorded by a recording device such as the drive recorder 1 is played back. In the second embodiment, when a recording device such as the drive recorder 1 is recording moving image data, event recording is performed when a frame with a high degree of coincidence with a desired image is found, but in the third embodiment In this example, when reproducing constantly recorded moving image data recorded on the SD card 23 (other recording media may be used), a frame having a high degree of coincidence with a desired image is found and event recording is performed. In the third embodiment, constantly recorded video data is played back using a personal computer, but if a recording device such as the drive recorder 1 has a playback function, such a recording device You can also perform playback and perform the following processing.

FIG. 18 is a block diagram showing the electrical configuration of a personal computer 110 that plays moving images.

The entire operation of personal computer 110 is supervised by control device 115.

The personal computer 110 is provided with a display device 111, and images and the like are displayed on the display screen of the display device 111 under the control of a display control device 112. A communication device 113 and a memory 114 for communicating with the Internet and the like are connected to the control device 115. The personal computer 110 is provided with an input device 116 such as a keyboard and a mouse, and commands given from the input device 116 are input to the control device 115.

Furthermore, the control device 115 includes a hard disk drive 117 that reads data recorded on the hard disk 118, reads data recorded on the SD card, and reads data recorded on the SD card 23. An SD card reader/writer 119 for recording data is also connected.

FIG. 19 shows the recording format of the SD card 23.

A file system area 121, a constant recording area 122, and an event recording area 125 are formed in the recording area of the SD card 23.

Dedicated software for reproducing data recorded in the constant recording area 122 and the event recording area 125 is recorded in the file system area 121.

A management area 123 and a recording area 124 are formed in the permanent recording area 122. Various setting information is recorded in the management area 123 by dedicated software recorded in the file system area 121. In the recording area 124, image data representing frames constituting a moving image is recorded in the order of frame numbers. In the recording area 124, a header recording area 131, a frame image data recording area 132, and a footer recording area 133 are formed for each frame. Additional information such as a frame number, address position, and shooting time is recorded in the header, and additional information other than the additional information recorded in the header is recorded in the footer.

Frame 1 to frame E recorded in the recording area 124 of the constant recording area 122 represents one moving image period of constant recording.

Similar to the constant recording area 122, the event recording area 125 also includes a management area 126 and a recording area 127. Various setting information is also recorded in the management area 126 by the dedicated software recorded in the file system area 121. In the recording area 127, image data representing frames constituting a moving image is recorded in the order of frame numbers. Also in the recording area 127, a header recording area 131, a frame image data recording area 132, and a footer recording area 133 are formed for each frame. Additional information such as a frame number, address position, and shooting time is recorded in the header, and additional information other than the additional information recorded in the header is recorded in the footer.

Frame 1 to frame E recorded in the recording area 127 of the event recording area 125 represents one moving image of event recording.

FIG. 20 is an example of the playback window 150.

When the SD card 23 is loaded into the personal computer 110 and the icon of the SD card 23 appearing on the display screen of the display device 111 is clicked, an icon of dedicated software appears. When the dedicated software icon is clicked, the dedicated software is started and a playback window 150 is displayed on the display screen of the display device 111.

The playback window 150 includes a video display area 151 for displaying the selected video, an event record list display area 152, a map image display area 153, a vehicle information display area 160, a constant record list display area 161, and an acceleration display area 163. It is formed.

The video display area 151 is an area for displaying video recorded by the drive recorder 1 or the like (constantly recorded video, event recorded video). The event record list display area 152 is an area for displaying an event record list in a format of recording time (a format other than recording time may be used). In the event record list displayed in the event record list display area 152, all event records recorded in the SD card 23 are displayed. When a constantly recorded video is selected, event recorded videos recorded during the shooting of the selected constantly recorded video may be displayed in the event record list display area 152.

The map image display area 153 is an area that displays an image of a map near the shooting location of the video displayed in the video display area 151. By transmitting data representing the vehicle's position from the personal computer 110 to a map server existing on the Internet, and transmitting data representing a map near the vehicle's position from the map server to the personal computer 110, the map image display area An image of the map is displayed on 153. The vehicle information display area 160 is an area that displays information such as the speed of the vehicle at the time of video recording. The vehicle information display area 160 includes a vehicle status display area 158 that displays the status of the vehicle to which the drive recorder 1 that recorded the video displayed in the video display area 151 is attached, and a vehicle status display area 158 that displays the status of the vehicle, such as video playback, stop, etc. an operation button 157 that gives an operation command to display an image inside the vehicle interior, an interior emphasis button 154 that gives an instruction to display a bright image inside the vehicle interior, and an instruction to stop the process of displaying a bright image inside the vehicle interior and display a normal display It includes a normal display button 155, a time display area 156 that displays the recording time of the video displayed in the video display area 151, an acceleration graph display area 159 that displays a graph of the acceleration of the vehicle, and the like.

The constantly recorded list display area 161 is an area where constantly recorded moving images recorded on the SD card 23 are displayed in a list. An image of a car is displayed in the acceleration display area 162. Before posting an image of a car, add the letters "+X" with an arrow pointing forward, the letters "+Y" with an arrow pointing horizontally next to the image of the car, and the letters "+Z" with an arrow pointing upwards above the image of the car. ' is displayed. The front or rear acceleration of the car is displayed in front of the car image, the lateral acceleration of the car is displayed in the lateral direction of the car image, and the vertical acceleration of the car is displayed above the car image. Ru. Since the acceleration of the car is displayed together with the exterior of the car, it is relatively easy to understand in which direction the acceleration is being applied to the car.

FIG. 21 is a flowchart showing a playback processing procedure, and FIGS. 22 and 23 show an example of the playback window 150.

When a desired event recording video or constant recording video is selected from the event recording list or the constant recording list displayed in the event recording list display area 152 or the constant recording list display area 161 of the playback window 150, the selected event recording video or constant recording video is displayed. The moving image data representing the captured moving image is read from the permanent recording area 122 of the SD card 23 by the SD card reader/writer 119 (step 141). In this embodiment, it is assumed that a constantly recorded moving image is selected.

When a constantly recorded video is selected, a representative image of the selected constantly recorded video is displayed in the video display area 151 of the playback window 150, as shown in FIG. 22 (step 142). When the playback button among the operation buttons 157 is pressed (YES in step 143), the control device 115 (playback device, an example of playback control means) performs playback processing of the read video data (step 144). . The selected constantly recorded video comes to be displayed in the video display area 151.

When the stop button among the operation buttons 157 is pressed during playback of the constantly recorded moving image (YES at step 145), the moving image being played is stopped by the control device 115 (step 146).

Even when a desired event recording video is selected from the event recording list, video data representing the selected event recording video is read from the event recording area 125 of the SD card 23 by the SD card reader/writer 119 (step 141).

When an event recorded video is selected, a representative image of the selected event recorded video is displayed in the video display area 151 of the playback window 150, as shown in FIG. 23 (step 142). The representative image of the event recording video may be an image taken at the time when the event recording command is issued, or may be the first image of the event recording video.

When the playback button among the operation buttons 157 is pressed (YES at step 143), the control device 115 performs playback processing of the read video data (step 144). The selected event recording video comes to be displayed in the video display area 151. When the stop button among the operation buttons 157 is pressed during playback of the event recorded video (YES in step 145), the control device 115 stops the video being played (step 146).

Even during the above-described reproduction process of the constantly recorded moving image, the processes from steps 92 to 98 in FIG. 12 may be performed, and when a desired image is detected, the event may be recorded in the event recording area 125 by the control device 115. When an event is recorded, it is additionally displayed in the event record list display area 152. Preferably, information indicating that the image is related to the desired image is recorded in the header of the frame constituting the constantly recorded moving image in which the desired image is detected during playback, so that when the moving image is played back, it can be found. The control device 115 or the display control device 112 (an example of a notification control means) is configured to display (an example of a notification) the time of the recorded event, etc. on the playback window 150 of the display device 111 (an example of a notification device). The display device 111 may be controlled by

24(A) and 24(B) are examples of tables showing the relationship between frame numbers of frames constituting one moving image and address position information.

As mentioned above, the frame number and address position information are recorded in the header recording area 131 regardless of whether it is a constantly recorded video or an event recorded video, so there is no need to create the tables shown in FIGS. 24(A) and 24(B). However, a table as shown in FIGS. 24(A) and 24(B) may be generated and recorded in the management area 123 of the constant recording area 122 or the management area 126 of the event recording area 125.

As described above, in the header recording area 131, frame numbers and address position information (information indicating the address of the beginning of the frame) are recorded in correspondence with the frames constituting the constantly recorded moving image or the event recorded moving image. The moving image data representing a constantly recorded moving image or an event recorded moving image according to this embodiment is continuously recorded in the recording area 124 of the constantly recording area 122 or the recording area 127 of the event recording area 125 in the order of frame numbers. For example, as shown in FIG. 24(A), one moving image composed of frames from frame 1 to frame E is recorded frame by frame in the recording area 124 of the constant recording area 122 or the recording area 127 of the event recording area 125. be done.

In such a case, at least one of the frame number and address position information may not be recorded properly, resulting in an abnormal state. For example, the frame numbers should be recorded consecutively but they are not, the address position information is an address position that is extremely larger than the previous address position information, or the address position information is incorrect. For example, it indicates an address position earlier than the previous address position information. In such a case, even though the video data itself is normal, that part of the video may not be played back.

For example, in Figure 24(A), all frame numbers and address position information are recorded normally, but in Figure 24(B), frame numbers and address position information for frames 11 to 20 are recorded as indicated by hatching. has an abnormal value, and the video represented by these frames 11 to 20 often cannot be played normally.

In this embodiment, if at least one of the frame number and address position information is not recorded normally, they are rewritten so that they become normal values. Furthermore, if the moving image data is not recorded normally, such as due to missing moving image data, the missing moving image data is generated and written to the frame where the moving image data is not normally recorded.

FIG. 25 is a flowchart showing the repair processing procedure, and FIG. 26 shows the recording format of the SD card 23.

The restoration process may be started when the SD card 23 is loaded into the personal computer 110 and the playback software is started, or when a predetermined video is specified for video playback, the video The repair process may be started for the playback window 150, or a repair mode button or the like may be provided in the playback window 150, and the repair process may be started when the repair mode button is pressed.

When the restoration process for a desired moving image is started, the control device 115 reads the frame number recorded in the header recording area 131 of the frames constituting the moving image (step 171). If the read frame numbers are not consecutive, the control device 115 determines that the non-consecutive frame numbers are not recorded normally (YES in step 172). Furthermore, as shown in Equation 1, the address position information of a frame plus the video data amount of that frame is the address position information of the next frame, so such a relationship may be broken. If so, the control device 115 determines that the address position information is not recorded normally (YES in step 172).

Address position information of a frame + video data amount of that frame = address position information of the next frame...Formula 1

If at least one of the frame number or address position information is not recorded normally (NO in step 172), the incorrectly recorded frame number or address position information (or frame number and address position information) is recorded in the control device. 115 (step 173). As described above, if the frame numbers are not recorded normally, the control device 115 rewrites the frame numbers so that they are consecutive in the order of the frames that make up the moving image. If the address position information is not recorded normally, the control device 115 rewrites the address position information to a normal value such that the relationship shown in Equation 1 is satisfied.

Next, the control device 115 determines whether the moving image data recorded in the recording area 124 or 127 is recorded normally (step 174). If the scenes represented by the frames are very different, for example, a car may be visible in one frame, but the next frame may be completely black or completely white, making the object unrecognizable. If so, it is determined that the frame was not recorded correctly. However, even if the scene suddenly changes, if the frame is determined to be a frame constituting a moving image of the same photographed scene based on the photographing position information, it is determined that the frame has been recorded normally. For example, when an accident occurs, the scene changes suddenly, and this is to prevent it from being determined that the image has not been recorded properly in such a case. The user may visually check and determine which frames are not being recorded normally.

If the moving image data is not recorded normally (NO in step 174), the incorrectly recorded moving image data is generated and the missing portion (missing data) of the moving image is repaired (step 175). A method for restoring missing parts of a video will be described later. Referring to FIG. 26, as shown by hatching, among the video data recorded in the recording area 124 of the continuous recording area 122, the video data recorded in frames 10 to 20 is not recorded normally. If not, the video data recorded in those frames 10 to 20 will be repaired. When a missing part of a video is repaired, information indicating that the repaired frame is a repaired frame rather than a frame obtained by shooting is recorded as additional information in the header recording area 131 or footer recording area 131, etc. It is preferable to leave it there. This is to make it possible to distinguish between a real scene actually obtained by photographing and a pseudo scene generated later.

When a missing portion of a video is repaired, the amount of data in the frame may change, so there is a high possibility that the address position information of the repaired frame will also change. To this end, the address location information is modified according to Equation 1 (step 176).

In this embodiment, a learning model is used to repair video data, but video data can also be generated by image interpolation using frames before and after the frame to be repaired without using a learning model. good.

Figures 27 and 28 show a method of generating a learning model. FIG. 27 is an example of a learning model, and FIG. 28 shows how the learning model is trained. FIG. 29 is a flowchart showing the learning model generation processing procedure.

In Figures 27 and 28, 10 frames from frames 21 to 30 (an example of the second video data, teacher data, and first part) are compared to frames 1 to 20 before frame 21. It is generated by machine learning using frames 31 to 50 after frame 30 (which are examples of first video data, teacher data, and second images). Frames 1 to 50 are frames actually obtained by photographing other vehicles. For example, a frame actually captured by another vehicle may be a frame captured by a drive recorder installed in another vehicle that traveled at the same location or another location where the same or similar scene (for example, a landscape) can be captured. This is the resulting frame. , Machine learning is supervised learning in which frames 1 to 20 and frames 31 to 50 are input, and frames 21 to 30 are output. Frames other than frames 21 to 30 can be generated in the same way.

The learning model 180 includes an input layer 181, a middle layer 182, and an output layer 183. Inputs are input to each neuron in the input layer 181 in the order in which frames 1 to 20 and frames 31 to 50 are photographed in the order of frame numbers. As shown in FIG. 28, the input layer 181 of the learning model 180 is inputted in accordance with the order of pixel arrangement for each pixel of each frame, such as data of pixel E1 and data of pixel E2. In this way, video data (frames 1 to 20, frames 31 to 50) excluding some video parts (frames 21 to 30) is input to the input layer 181 (step 191).

Data input from the input layer 181 is output from the output layer 183 via the intermediate layer 182. The learning model 180 is trained to output data corresponding to frames 21 to 29 from each neuron of the output layer 183 according to the pixel array. As shown in FIG. 28, frames 1 to 20 and frames 31 to 50 are input in the order of frame numbers, and the learning of the learning model ends when frames 21 to 30 are output from the output layer 183. . In this way, learning is performed so that the output is a constant video portion (frames 21 to 30). By similarly performing a large amount of learning on other video data (step 192), it becomes possible to generate a learning model for the video of the defective video portion by using video frames before and after the defective video portion.

In FIGS. 27 to 29, the input is 40 frames and the output is 10 frames, but the number of input frames and the number of output frames can be changed without being limited to these numbers. Also, keep the number of input frames and output frames constant, and if the number of frames of the defective video you are trying to generate is less than the number of output frames, generate video data that represents the output frames. You can thin out the frames so that the number of defective video frames is equal to the number of frames of the defective video you want to generate, or if the number of frames of the defective video you are trying to generate is greater than the number of output frames, use video data representing the output frames. The number of frames can be adjusted by interpolation etc.

Preferably, a plurality of learning models 180 suitable for various scenes are generated, and the defective video portion is generated using the learning model according to the scene. For example, a learning model 180 is used to generate a defect video when a car in front causes an accident, and a learning model is used to generate a defect video when a bird, bicycle, pedestrian, etc. suddenly appears in front. A learning model 180 such as 180 corresponding to a scene before or after a defective video portion is prepared in advance, and the learning model 180 corresponding to the scene is used to generate the defective video portion.

Furthermore, when generating defective videos, if the video is from a drive recorder installed in a vehicle, we use a video with a high degree of matching of the vehicle information of the vehicle installed in the drive recorder that captured the defective video. The generated learning model 180 may be used.

FIG. 30 is a flowchart showing a procedure for generating a missing portion of a moving image.

When the video data of the defective part is found among the video data recorded on the SD card 23, the video data before and after the defective part is input into the trained learning model 180 frame by frame in accordance with the shooting order (step 201).

The input video data is trained by the trained learning model 180 (an example of a second learning model) (step 202), and the video data including missing parts is trained by the control device 115 (an example of video data generation means). (which is an example of the third image) is input to the learning model 180, and video data (which is an example of the fourth image) representing the video data of the missing portion is outputted from the learning model 180 in frame order (step 203). . The video data of the missing portion obtained in this manner is recorded in the missing portion of the recording area 124 or 127 of the control device 115 (which is an example of the second recording control means) (step 204).

Even if there is a recording error such as a frame number recorded on the memory card 23, it can be repaired, and even if there is a missing part in the video, it can be repaired.

In the above description, in response to "detecting an image representing the above scene," information regarding the image representing the scene is displayed in the event record list display area 152, and when the displayed information is selected, the corresponding information is displayed. Although the event-recorded video including the image representing the scene is displayed based on the video data representing the video, the selection is not limited to this. Depending on the situation, it may be possible to fast-forward to the point in time when the image of the scene is displayed, or to jump to the point in time while the image is being played back (to cue up).

In each of the embodiments described above, various processes are performed based on video data, but the processes may also be performed based on data that is not in video format. The non-video format data may be constituted by a set of data in a plurality of still image formats representing still images taken in chronological order.

In the embodiment described above, the drive recorder 1 has the acceleration sensor 18 to detect the state of the vehicle (driving state, etc.), but the acceleration sensor 18 is not used to generate a learning model or detect a scene. In this case, it may be omitted. In the second embodiment, the drive recorder 1 may not execute the process of step 91, or may not execute the process of detecting an event by the acceleration sensor 18 among the processes of step 91. In this case, the image of the specific scene is detected based on whether the first degree of coincidence is equal to or higher than the threshold value, without using the second degree of coincidence. Even in such a case, the drive recorder 1 does not need to have the acceleration sensor 18. Even when the drive recorder 1 detects the state of the vehicle using a sensor other than the acceleration sensor 18, it can be configured without the sensor.

In the embodiments described above, the functions realized using deep learning technology may be realized using other machine learning or other alternative means, if any. For example, a pattern matching technique may be used as a means for detecting images with a high degree of matching.

Note that the scope of the present invention is not limited to the configuration explicitly described in the specification, but also includes combinations of various aspects of the invention disclosed in this specification. Of the present invention, the structure for which a patent is sought has been specified in the attached claims, but at present, even if the structure is not specified in the claims, it is not disclosed in this specification. We intend to include such configurations in the scope of claims in the future.

The present invention is not limited to the configuration described in the embodiments described above. The components of each of the embodiments and modifications described above may be arbitrarily selected and combined. Also, any component of each embodiment or modification, any component described in the means for solving the invention, or a component that embodies any component described in the means for solving the invention. It may be configured in any combination. The applicant intends to acquire rights to these matters through amendments to the application or divisional applications. Even if there is a description of ``in the case of'' or ``in the case of'', the description is not intended to be limited to that case or at that time. We have also disclosed these cases and other configurations, and we intend to acquire the rights. Furthermore, the sections described in order are not limited to this order. It also discloses a configuration in which some parts have been deleted or the order has been changed, and we have the intention to acquire the rights.

In addition, the applicant intends to acquire rights to the entire design or partial design by converting the application to a design registration application. Although the drawing depicts the entire device using solid lines, the drawing includes not only the overall design but also the partial design claimed for some parts of the device. For example, it is a drawing that not only includes some members of the device as a partial design, but also includes some parts of the device as a partial design regardless of the members. The part of the device may be a part of the device or a part of the device. We intend to obtain rights not only for the entire design, but also for partial designs in which any part of the solid line part of the drawing is a broken line part. In addition, the modules, members, parts, etc. inside the device housing shown in the drawings are all subject to independent transactions, and similarly, they are included in the design registration application. There is an intention to make changes and obtain rights.

1: Drive recorder, 3: Windshield, 4: Room mirror, 5: Cigarette socket, 6: Power cable, 10: SD card slot, 11: Display, 12: Operation buttons, 13: Joint rail , 14: SD card reader/writer, 15: speaker, 16: GPS receiver, 17: camera, 18: acceleration sensor, 19: communication circuit, 20: controller, 20c: RAM, 20d: timer, 21: LTE module , 22: database, 23: SD card, 30: server, 31: control device, 33: memory, 34: hard disk drive, 35: hard disk, 40: learning model, 40A: first learning model, 40B: Second learning model, 40C: Third learning model, 41: Input layer, 42: Middle layer, 43: Output layer, 43a: Neuron, 50: Target image, 51: First target image, 52: Second target image, 53: Third target image, 71: Generator, 72: Discriminator, 73: Weighting changer, 80: Learning model, 81: Input layer, 82: Middle layer, 83: Output layer , 105: Learning model, 106: Input layer, 107: Middle layer, 108: Output layer, 110: Personal computer, 111: Display device, 112: Display control device, 113: Communication device, 114: Memory, 115: Control Device, 116: Input device, 117: Hard disk drive, 118: Hard disk, 119: SD card reader/writer, 121: File system area, 122: Constant recording area, 123: Management area, 124: Recording area, 125: Event recording area, 126: Management area, 127: Recording area, 131: Header recording area, 132: Frame image data recording area, 133: Footer recording area, 150: Playback window, 151: Video display area, 152: Event record list display area, 153: Map image display area, 154: Indoor emphasis button, 155: Normal display button, 156: Time display area, 157: Operation button, 158: Vehicle status display area, 159: Acceleration graph display Area, 160: Vehicle information display area, 161: Record list display area, 162: Acceleration display area, 163: Acceleration display area, 180: Learning model, 181: Input layer, 182: Middle layer, 183: Output layer, 421 : input layer, E1: pixel, E2: pixel, FR1: frame, FRn: frame, P1: pixel, P21: pixel

Claims (5)

Photographing with the vehicle at each of a plurality of time points based on an image that defines a scene of a specific image that can be photographed with the vehicle, and the physical quantity detected by a function that detects a physical quantity indicating the driving state of the vehicle. a function of detecting an image representing a scene of the specific image from the captured image ;
Equipped with
The function of detecting an image representing the scene of the specific image is
each of the plurality of time points based on the degree of coincidence between the image defining the scene of the specific image and the image taken by the vehicle, and the physical quantity when the image taken by the vehicle was obtained. detecting an image representing the scene of the specific image from images taken with the vehicle in
system. The function of detecting an image representing a scene of the specific image is
a first degree of coincidence, which is a degree of coincidence between an image that defines the scene of the specific image and an image taken by the vehicle; and a physical quantity when the image that defines the scene of the specific image is obtained; Detecting an image representing the scene of the specific image based on a second degree of coincidence that is a degree of coincidence with the physical quantity when the image photographed by the vehicle is obtained.
The system according to claim 1. 3. The system according to claim 1 , further comprising a function of controlling a warning device to issue a warning in response to detection of an image representing a scene of the specific image . The system according to any one of claims 1 to 3, further comprising a function of controlling a recording device to record an image representing a scene of the specific image on a recording medium. A program for realizing in a computer a function of detecting an image representing a scene of the specific image of the system according to any one of claims 1 to 4 .

Images