JP2023176441A - System, control method, and program or the like - Google Patents

Abstract

To provide a system that makes an inference while image capturing, a control method thereof, and a program, which can substantially shorten the time required to obtain an inference result.SOLUTION: A system 5 includes an imaging device 1 installed in a vehicle 4 such as a car to image a traveling direction, a rear direction, etc., of the car, and infers captured video data obtained by image capturing, with an inference model, while image capturing. By inferring the captured video data while image capturing, the time required to obtain an inference result is substantially shortened, and the time from imaging end to inference end is shortened.SELECTED DRAWING: Figure 1

Description

The present invention relates to, for example, a system, a control method, a program, and the like.

In the conventional technology, for example, video data obtained by photographing a subject using a vehicle-mounted photographing device or the like is input to a trained model, and characteristic images are obtained from the trained model (Patent Document 1).

Japanese Patent Application Publication No. 2021-164034

For example, when inference is performed using an inference model using photographed video data obtained by photographing with an in-vehicle photographing device, the time required for inference may become longer as the amount of photographed video data increases. In particular, when performing inference using an inference model using continuously recorded video data, it often takes several times the recording time. It may take a long time to complete.

In view of the above-mentioned problems, one of the objects of the present invention is to provide a technique different from the conventional one, such as making inferences while photographing.

The purpose of the present invention is not limited thereto, and the present invention intends to acquire rights through divisional applications, amendments, etc. for structures that aim to obtain effects from the parts of the structures disclosed in this specification, drawings, etc. For example, the present specification discloses a problem in which passages such as ``can be done'' or ``possible'' are read as ``the problem is.'' Each issue is described as an independent entity, and we intend to acquire rights to the structure for solving each issue individually through divisional applications, amendments, etc. Even if the problem is implicitly understood from the description of the specification, the present applicant has the intention to claim a part of the structure described in the specification in an amendment or divisional application. The company also discloses a structure that solves a problem that combines these independent problems, and has the intention to acquire the rights to it.

(1) The system according to the present invention preferably has a function of performing inference using an inference model using captured video data obtained by shooting with an in-vehicle photographing device.

In this way, the time from the end of photography to the end of inference can be shortened. The inference model is, for example, a trained model trained using teacher data, and inference is, for example, inputting an arbitrary image to the trained model to obtain an output. In-vehicle photographing devices may include those mounted on vehicles such as cars, buses, trucks, forklifts, and two-wheeled vehicles such as motorcycles and bicycles.

(2) The above inference model may be determined from among a plurality of types.

In this way, appropriate inferences can be made. The plurality of types may include those with different versions of the inference model. It is preferable to determine the inference model according to the selection command. The selection command may be a command from the user or may be automatically determined by the system.

(3) The inference model may be determined to be suitable for the shooting environment.

In this way, inferences suitable for the shooting environment can be made. The photographing environment includes, for example, the situation around the photographing device (brightness, darkness, etc.), the condition of the vehicle in which the photographing device is mounted (vehicle speed, etc.), and the condition of the photographing device itself. (The position where the photographing device is installed, whether the photographing device is installed inside the vehicle or exposed outside the vehicle, whether to photograph the front, right or left side, It is preferable to include one or more of the following: whether to photograph the rear, or to photograph the entire front, rear, left, and right sides such as a celestial sphere image or a circumferential image.

(4) The inference model may be determined to be suitable for the shooting location, the weather at the shooting location, or the brightness of the shooting location.

In this way, inferences can be made that are appropriate for the photographing location, the weather at the photographing location, or the brightness of the photographing location. For example, the shooting location can be known from the video shooting data obtained from the shooting (city area, suburbs, mountain road, etc.), and the shooting location can be determined using the GPS (Global Positioning System) function of the shooting device. You can also tell from the latitude and longitude of the location. The weather at the shooting location can be determined, for example, by accessing a weather server from the shooting location, and the brightness can be determined from an illuminance sensor.

(5) It is preferable to determine the above-mentioned inference model to be used so that the darker the brightness of the shooting location, the more accurate the inference model is used for inference than when it is brighter. For example, if an object is to be detected, it is preferable to use a high-precision inference model that detects objects more accurately when it is dark than an inference model used when it is bright.

In this way, the darker the shooting location, the more accurate the inference can be made.

(6) It is preferable to determine the inference model to be used for inference using the highly accurate inference model at night.

In this way, inferences suitable for nighttime can be made. Whether it is nighttime or not can be determined, for example, from the time at which the image was taken. It may be determined that it is nighttime when the illuminance sensor determines that it is dark.

(7) The inference result of the inference model may be corrected in accordance with disturbances applied to the in-vehicle photographing device.

In this way, it is possible to obtain inference results that can withstand disturbances applied to the vehicle-mounted imaging device. The disturbance may be, for example, a factor from outside the photographing device that disturbs the operation of the vehicle-mounted photographing device. For example, the disturbance may be determined when the brightness of the place photographed by the photographing device is below a threshold value, or the disturbance may be determined when the photographing device is irradiated with light. Further, the disturbance may include, for example, disturbance to a vehicle on which the photographing device is mounted. For example, a G-sensor installed in a vehicle, etc. on which a photographing device is installed may apply an acceleration of more than a predetermined threshold to the vehicle, or the speed of the vehicle given by the vehicle's speedometer may exceed a predetermined threshold. It is best to consider this as a disturbance.

(8) Disturbance imparted from the vehicle to the vehicle-mounted photographic device due to movement of the vehicle in which the vehicle-mounted photographic device is installed, or disturbance imparted to the vehicle-mounted photographic device due to light being irradiated to the vehicle-mounted photographic device. It is preferable to correct the inference result of the above inference model according to the disturbance.

In this way, it is possible to obtain inference results that can withstand disturbances caused by the movement of the vehicle to the in-vehicle imaging device, or disturbances caused to the in-vehicle imaging device due to light irradiation on the in-vehicle imaging device. be able to. The disturbance imparted from the vehicle to the vehicle-mounted imaging device due to the movement of the vehicle may be, for example, vibration of the vehicle (for example, vibration due to a change in vehicle speed, vibration due to impact on the vehicle), or the like. Irradiation of light onto the vehicle-mounted imaging device includes, for example, reflection of sunlight, irradiation of sunlight, backlighting, irradiation of vehicle lights, irradiation by reflection of light, and the like. Whether or not light has been irradiated may be detected using, for example, a photometer.

(9) It is preferable to notify whether a state in which inference using the inference model is possible or not is possible due to a disturbance applied to the in-vehicle photographing device.

In this way, it can be determined whether or not the inference result cannot be obtained because the state is such that inference is not possible. Whether or not inference is not possible is determined in advance for each type of disturbance and the magnitude of the disturbance, and it is preferable to determine and notify whether inference is not possible according to these types and magnitudes.

(10) The inference result of the inference model may be adjusted depending on the vibration or speed of the vehicle in which the in-vehicle photographing device is installed.

In this way, it is possible to obtain inference results that are adjusted according to the sway or speed of the vehicle. It is preferable that the shaking of the vehicle is detected by, for example, a G sensor of the photographing device, and the speed of the vehicle is determined by the speedometer of the vehicle, the speed sensor of the photographing device, or the like.

(11) Adjusting the system based on the height at which the in-vehicle camera is installed so that it more closely resembles the video obtained by shooting with the in-vehicle camera installed at a standard height. good.

In this way, even if the vehicle-mounted photographing devices are installed at different heights, it is possible to obtain a video that looks like it was shot by the vehicle-mounted photographing device installed at the standard height. The height of the reference may vary depending on the vehicle or the like in which the photographing device is installed. The reference height for each vehicle or the like may be determined by trial and error, for example, based on a video obtained by shooting that is considered to capture the subject relatively well. It is preferable to obtain an image obtained by photographing at each height and adjust the system so that it resembles a video obtained by photographing with a vehicle-mounted photographing device installed at a reference height.

(12) The vehicle-mounted photographing device preferably photographs a circumferential image with the photographing direction set below the reference height.

In this way, it is possible to obtain a circumferential image and a celestial sphere image that are below the reference height.

(13) It is preferable to input the height at which the in-vehicle photographing device is installed, and adjust the inference in the inference model according to the input height.

In this way, it is possible to obtain inference results that are adjusted according to the height.

(14) The above inference model preferably performs inference to detect a person whose moving speed is less than or equal to a predetermined speed from the video.

In this way, a person moving faster than a predetermined speed can be prevented from being detected. To determine whether the moving speed is below a predetermined speed, for example, the speed of the person included in the video obtained by shooting may be calculated.

(15) The above inference model preferably performs inference that excludes the person inside the vehicle.

In this way, the person inside the vehicle can be excluded from the inference results. For example, if a vehicle is detected from a video obtained by shooting and a person is inside the detected vehicle, that person may be excluded.

(16) The above inference model performs inference to detect objects from a moving image, and it is preferable to exclude objects reflected on a reflective surface.

In this way, the object to be detected can be something other than what is reflected on the reflective surface. Whether or not an object is reflected on a reflective surface may be determined by, for example, whether the image around the object has a mirror surface.

(17) The above inference model performs inference to detect an object from each image that makes up a video, and has a function that displays a mark that identifies the detected object part on each image that makes up the video. It is preferable to interpolate the mark to the image where the mark is interrupted when the time when the mark is interrupted in the moving image is within a predetermined time.

In this way, marks can be displayed even in images where marks specifying parts of the detected object are interrupted. For example, it is sufficient to display a mark on at least one of the images constituting the moving image, which is an image in which a target object has been detected.

(18) Event recording of video data is performed in response to the detection of an object by inference using the inference model using the captured video data obtained by photographing with the in-vehicle imaging device. good.

In this way, event recording can be performed in response to detection of a target object. For example, it would be good to be able to record events not only when an impact is applied to the car, but also when an object is detected.

(19) The threshold value used for detecting the object may be lowered in response to the detection of the object.

In this way, the probability of detecting the target object can be increased depending on the detection of the target object. For example, if there is an algorithm that determines that an object has been detected if the parameter used to detect the object is greater than or equal to a threshold value, by lowering the threshold value, you can detect an object that was not detected before lowering the threshold value. Can detect objects.

(20) It is preferable to display the location where the event recording was performed on a map.

In this way, the location where the event recording took place can be known by looking at the map. Data representing a map may be received from a map server on the Internet, for example. The location where the event was recorded may be determined using, for example, a GPS function.

(21) It is preferable to display an image of the place where the above event recording was performed on a map.

In this way, it is possible to easily understand the state of the place where the event recording took place. Since you can see the actual image, it can be easier to understand.

(22) It is preferable to embed a link to the video data of the event record at the location where the event record was performed, which is displayed on the map.

In this way, you can view a video of the location where the event was recorded. For example, it is preferable to embed the link by associating data representing the location where the event was recorded with the link and recording it on the medium in which the photographed video data is recorded.

(23) The above-mentioned event recording of video data is performed in response to the detection of a target object by the inference using the above-mentioned inference model, and attention is paid to the detected target object in the image constituting the video in which the above-mentioned event has been recorded. It is a good idea to display a mark to indicate the location.

In this way, it is possible to know the location of interest for detecting the target object. The mark may be used as long as it can specify, for example, a part of the object or a part of interest to be specified as the object. The mark may be a heat map.

(24) Obtain the inference results obtained by inference using the inference model in each of the plurality of in-vehicle imaging devices, and obtain the inference results for each in-vehicle imaging device. It is better to display the degree of similarity.

In this way, it is possible to compare the inference results for each vehicle-mounted photographing device.

(25) The above inference model was conducted using captured video data obtained when the location, information, speed, shooting time, etc. of the vehicle in which the vehicle camera camera is installed are similar. It is preferable to express the degree of similarity of inference results.

In this way, it is possible to understand the relationship between the approximation of the situation and the similarity of the inference results.

(26) The method according to the present invention performs inference using the inference model using the captured video data obtained while shooting with the in-vehicle camera.

In this way, it is possible to provide a technology different from the conventional technology, such as a shorter time from the end of imaging to the end of inference.

(27) The program according to the present invention causes a computer to realize the functions of the above system.

In this way, it is possible to provide a technology different from the conventional technology, such as a shorter time from the end of imaging to the end of inference.

The inventions shown in (1) to (25) above can be combined arbitrarily. For example, at least a part of the structure of at least one of the inventions (2) to (25) may be added to all or part of the structure of the invention shown in (1). In particular, it is preferable to create an invention in which at least a part of the structure of at least one of the inventions (2) to (25) is added to the invention shown in (1). Further, arbitrary configurations may be extracted from the inventions shown in (1) to (25) and the extracted configurations may be combined. Further, arbitrary configurations may be extracted from the inventions shown in (2) to (25) and the extracted configurations may be combined with the invention shown in (26). Furthermore, arbitrary configurations may be extracted from the inventions shown in (2) to (25) and the extracted configurations may be combined with the invention shown in (27).

The applicant of this application intends to acquire rights to inventions containing these structures. Furthermore, even if there is a description of "in the case of" or "at the time of", the description is not intended to be limited to those cases or times. These are examples of better configurations, and we intend to acquire rights to these cases and other configurations as well. Furthermore, the sections described in order are not limited to this order. It also discloses a configuration in which some parts have been deleted or the order has been changed, and we have the intention to acquire the rights.

According to the present invention, it is possible to provide a technique different from the conventional technology, such as shortening the time from the end of imaging to the end of inference.

Note that the effects of the invention of the present application are not limited to these, and effects obtained from the parts of the configuration disclosed in the present specification, drawings, etc. are also disclosed, and the configurations that provide the effects are also disclosed by divisional applications, amendments, etc. Have the intention to acquire the rights. For example, in this specification, passages such as "can be done," "is possible," etc. are descriptions that clearly indicate the effect to be achieved, and even if there is no description such as "can be done," or "it is possible," the effect can be obtained. There is a part shown. Further, even without such a description, there are effects that can be understood from the configuration.

An example of the system is shown. (A) is a perspective view of the photographing device seen from the front, and (B) is a perspective view of the photographing device seen from the back. (A) is a view of the bracket 40 viewed from the front diagonally upper right direction, (B) is a view of the bracket viewed from the side, and (C) is a view of the bracket viewed from the front diagonally lower right direction. be. The electrical configuration of the photographing device is shown. This is an example of a trained model. It is a flowchart which shows the processing procedure of an imaging device. It is a flowchart which shows the processing procedure of an imaging device. It is a flowchart which shows the processing procedure of an imaging device. This is an example of a display screen of a photographing device. This is an example of a photographed image. An example of a file structure of a storage medium is shown. This is an example of a photographed image. This is an example of an image of a person. This is an example of an image of a person. This is an example of a photographed image. It is a flowchart which shows the processing procedure of an imaging device. It is a flowchart which shows the processing procedure of an imaging device. An example of a file structure of a storage medium is shown. An example of a file structure of a storage medium is shown. This is an example of a system including an AI module. FIG. 2 is a block diagram showing the electrical configuration of an AI module. This is an example of a display screen of a photographing device. It is a flowchart which shows the processing procedure of an imaging device. It is a flowchart which shows the processing procedure of an imaging device. It shows the relationship between the imaging device and the inference server. FIG. 2 is a block diagram showing the electrical configuration of an inference server. It is a flowchart which shows the processing procedure of an imaging device and an inference server. It is a flowchart which shows the processing procedure of an imaging device. It is a flowchart which shows the processing procedure of an imaging device. This is an example of an image taken by a photographing device. This is an example of an image taken by a photographing device. It is a flowchart which shows the processing procedure of an imaging device. It is a flowchart which shows the processing procedure of an imaging device. This is a view of the photographing device from the back. FIG. 1 is a block diagram showing the electrical configuration of a personal computer. 3 is a flowchart showing a reproduction processing procedure. 3 is a flowchart showing a reproduction processing procedure. This is an example of a display screen of a personal computer. 3 is a flowchart showing a reproduction processing procedure. This is an example of an image portion of a person. 3 is a flowchart illustrating a procedure for displaying similarity of inference results. It shows the similarity of the inference results. It shows the similarity of the inference results. It shows a photographing device installed on a forklift. This is an example of a photographed image. This is an example of a photographed image. This is an example of a photographed image. This is an example of a photographed image that has been subjected to transformation processing. This is an example of a photographed image that has been subjected to transformation processing.

[1. Overall system configuration]
FIG. 1 is a diagram illustrating the configuration of a system according to this embodiment. FIG. 1 shows a schematic diagram of the vehicle 4 viewed from the side. The system 5 includes a photographing device 1 as a first photographing device and a photographing device 2 as a second photographing device, which are arranged in a vehicle 4. The vehicle 4 is, for example, a four-wheeled vehicle, but is not limited to a four-wheeled vehicle, and may be any vehicle as long as the photographing device 1 and the photographing device 2 can be installed therein. The vehicle may be, for example, a large transport vehicle with four or more wheels such as an automobile, a bus, or a truck, a two-wheeled vehicle such as a motorcycle or a bicycle, or other vehicle. The vehicle may be, for example, a transportation vehicle such as a train, monorail, or linear motor car.

The photographing device 1 is a front camera arranged on the front side of the vehicle 4. The photographing device 1 is attached, for example, to a predetermined position on the front side of the cabin of the vehicle 4, and photographs the front of the vehicle 4 through the windshield. The photographing device 1 is, for example, a drive recorder. Specifically, the photographing device 1 has a function of photographing, a function of recording image data representing a photographed image, and a function of recording image data obtained from the photographing device 2, but at least the function of photographing. It is sufficient to have

The photographing device 2 is a rear camera arranged on the rear side of the vehicle 4. The photographing device 2 is attached, for example, to a predetermined position on the rear side of the cabin of the vehicle 4, and photographs the rear of the vehicle through the rear window. The photographing device 2 has a function of photographing and a function of outputting image data representing a photographed image to the photographing device 1, but it is sufficient if it has at least a function of photographing.

The photographing device 1 and the photographing device 2 are connected via a cable 3. The cable 3 is a wired communication path that connects the photographing device 1 and the photographing device 2. The cable 3 includes, for example, a power line for supplying operational power from the imaging device 1 to the imaging device 2, and a signal line for transmitting various signals between the imaging device 1 and the imaging device 2. . The photographing device 2 operates by receiving power from the photographing device 1 via the cable 3 . Note that the photographing device 1 and the photographing device 2 may be connected not by a wired communication path but by a wireless communication path of Wi-Fi (registered trademark), Bluetooth (registered trademark), or other standards. Furthermore, the photographing device 1 may be used without being connected to the photographing device 2 through communication.

[2. External configuration of photographing device 1]
FIG. 2 is a diagram showing an example of the external configuration of the photographing device 1. As shown in FIG. FIG. 2(A) is a diagram of the imaging device 1 viewed from the front diagonally upper right direction. FIG. 2(B) is a diagram of the photographing device 1 viewed from the diagonally upper right direction on the rear side. The photographing device 1 has a housing 11. The housing 11 has a rectangular parallelepiped shape that is longer in the horizontal direction than in the vertical direction and has a relatively small thickness. The housing 11 has an upper surface 16 that faces upward when attached to the vehicle 4, a first side surface 21, a second side surface 32, and a third side surface 20 located on the opposite side of the second side surface 32. and a fourth side surface 22 located on the opposite side of the first side surface 21.

A joint rail 12 and a camera jack 17 are provided on the top surface 16. A bracket (for example, a bracket 40 described later) for attaching the photographing device 1 to a predetermined mounting position of the vehicle 4 can be attached to and detached from the joint rail 12. The mounting position may be, for example, the windshield of the vehicle 4 (for example, near the upper end of the windshield), the room mirror of the vehicle 4, the ceiling of the vehicle interior, or the like. When the photographing device 1 is attached to the vehicle 4, the first side surface 21 faces the front side of the vehicle 4. At this time, the second side surface 32 faces to the right when viewed from the rear side of the vehicle 4. The third side surface 20 faces to the left when viewed from the rear side of the vehicle 4. The fourth side surface 22 faces toward the rear of the vehicle 4.

The camera jack 17 is a terminal to which one end of the cable 3 is connected. The camera jack 17 complies with the USB Type C standard, for example, and may be a terminal for the photographing device 1 to communicate with the photographing device 2 according to the Ethernet standard.

The first side surface 21 is provided with an imaging lens 15, a sound emitting hole 13, and a microphone hole 14. The imaging lens 15 is a light condensing lens included in the imaging unit (the imaging unit 67 described later) included in the imaging device 1. The sound emitting hole 13 is a hole that is provided above the imaging lens 15 and allows the sound output from the sound output section (sound output section 66 described later) of the photographing device 1 to be transmitted from the inside of the housing 11 to the outside. . The microphone hole 14 is provided below the imaging lens 15 and is a hole that allows external sound to pass from the outside of the housing 11 to the inside. The sound transmitted into the interior of the housing 11 is input to a microphone (microphone 61 to be described later) included in the photographing device 1.

An event recording button 31 is provided on the second side 32. The event recording button 31 is an operation means for instructing the start or end of recording of images photographed by the photographing section 67. If the user operates the event recording button 31 while event recording is not being performed, the photographing device 1 starts event recording. Event recording will be described in detail later. The event recording button 31 is provided on the second side surface 32 facing the driver's seat so that the driver of the right-hand drive vehicle 4 can easily operate it.

The third side surface 20 is provided with a terminal 18 and a storage medium insertion slot 19. Terminal 18 is a terminal for receiving power supply from an external device. The terminal 18 is, for example, a DC jack. A connector at one end of a power cord (for example, a cigarette plug cord) is connected to the terminal 18. The connector at the other end of the power cord is connected, for example, to a power supply terminal (for example, a cigarette lighter socket) provided on the vehicle 4 side.

An OBD II adapter connectable to the OBD II ("II" is a Roman numeral for "2") connector of the vehicle 4 may be connected to the terminal 18. The OBD II connector is also called a failure diagnosis connector, and is a terminal that is connected to a vehicle's ECU (Engine Control Unit) and outputs various vehicle information every predetermined period (for example, every 0.5 seconds). By connecting the terminal 18 to the OBDII connector using the OBDII adapter, the photographing device 1 can receive power for operation and acquire vehicle information.

The vehicle information is information regarding the state of the vehicle 4. Vehicle information includes, for example, the speed of the vehicle 4 (vehicle speed), engine rotation speed, engine load factor, throttle degree, ignition timing, remaining fuel percentage, intake manifold pressure, intake air amount (MAF), injection opening time, engine Temperature of cooling water (cooling water temperature), temperature of air taken into the engine (intake temperature), air temperature outside the vehicle (outdoor temperature), amount of remaining fuel in the fuel tank (remaining fuel amount), fuel flow rate, instantaneous fuel efficiency, The information may be at least one of the following: accelerator opening, blinker information (operation (ON/OFF) of left and right blinkers), brake opening, steering angle of steering wheel, gear position, door opening/closing state, etc.

The storage medium insertion port 19 is an insertion port for inserting a storage medium 71 as an external storage means into the interior of the photographing device 1. The storage medium 71 is a storage medium in which images photographed by the photographing device 1 or the photographing device 2 are recorded, and is, for example, an SD card. The SD card includes, for example, any shape such as an SD memory card, a miniSD card, and a microSD card. The storage medium 71 may further store a program for a viewer (for example, a dedicated viewer) for reproducing the stored images on an information display terminal such as a personal computer.

The fourth side surface 22 is provided with an operation section 26, a display surface 23, and a light emitting section 25. The operation unit 26 has a first button 27, a second button 28, a third button 29, and a fourth button 30. The first button 27, the second button 28, the third button 29, and the fourth button 30 are arranged vertically along one right side of the display surface 23. Examples of functions assigned to each of these buttons include the following functions.

The first button 27 functions as a button for switching images when pressed for a long time, and functions as a button for instructing the format of the storage medium 71 when pressed for a short time. The image displayed on the display surface 23 is, for example, one or both of the image currently being photographed by the photographing device 1 and the image currently being photographed by the photographing device 2. Formatting the storage medium 71 means initializing the storage medium 71, for example, erasing data such as images stored in the storage medium 71, or putting the storage medium 71 into a state where the photographing device 1 can use it. writing setting information indicating the contents of the operation settings to the storage medium 71 in order to do so (for example, placing the storage medium 71 in a state where images can be recorded and read); and setting the storage medium 71 in a specific file state; It is understood as at least one of the following.

The second button 28 is a button for displaying a selection screen for selecting an image to be reproduced by the photographing device 1. The third button 29 is a button for displaying a menu regarding the settings of the photographing device 1 and the photographing device 2. The fourth button 30 is a button for instructing to start and stop image recording. For example, if the fourth button 30 is pressed briefly during recording using the constant recording function described below, the recording is temporarily stopped. If the fourth button 30 is pressed briefly during the pause, image recording by the constant recording function is resumed. If the fourth button 30 is held down, the frame rate at which images are recorded can be changed.

The display surface 23 is an area where an image displayed by a display unit (display unit 65 described later) included in the photographing device 1 is displayed. The display surface 23 is, for example, a rectangular or square area. A touch sensor 24 is provided to overlap the display surface 23 and detect a touch operation by a user.

The light emitting section 25 is provided above the first button 27 and emits light in a predetermined color.

Note that the photographing device 2 may also have a function as a drive recorder, and may have the same configuration as the photographing device 1, for example. Furthermore, as the photographing device connected to the photographing device 1 through communication, one or more photographing devices that photograph in other directions may be used instead of or in addition to the photographing device 2. Other directions include directions such as diagonally to the right rear of the vehicle 4, diagonally to the left rear, and in the vehicle width direction (sideways).

[3. Bracket configuration]
FIG. 3 is a diagram showing the configuration of the bracket 40. FIG. 3(A) is a diagram of the bracket 40 viewed from the front side diagonally upward to the right. FIG. 3(B) is a side view of the bracket 40. FIG. 3(C) is a diagram of the bracket 40 viewed from diagonally lower right on the front side. The bracket 40 is an example of a mounting member for mounting the photographing device 1 on the vehicle 4.

Bracket 40 is a mounting member using a ball joint mechanism. In the bracket 40, a flat base portion 41 has a mounting surface 42 that is attached to the windshield of the vehicle 4. The base portion 41 is inclined at a predetermined angle with respect to the column of the ball stud 43. An adhesive member such as double-sided tape is attached to the mounting surface 42, and the device is attached to the windshield or the like of the vehicle 4 via the adhesive member.

The ball stud 43 is a portion of the base portion 41 that stands up from the surface opposite to the mounting surface 42. The ball portion 44 of the ball stud 43 is attached to the socket portion 45. The nut 46 is detachably attached around the socket portion 45. A thread groove is formed on the outer periphery of the socket portion 45. A nut 46 that fits into the thread groove is attached to the outer periphery of the socket portion 45. When the ball part 44 of the ball stud 43 is installed in the socket part 45 and before the nut 46 is tightened, the socket part 45 rotates in a desired direction along the circumferential surface of the ball part 44, and the base part It is possible to change 47 postures and positions. When the nut 46 is tightened, the posture and position of the base portion 47 are fixed.

The base portion 47 is formed as a pair with the socket portion 45, and is a portion for mounting on the imaging device 1. The base portion 47 has a pair of guide rails 48 on the lower surface. The pair of guide rails 48 are configured to be slidable along the joint rail 12 of the photographing device 1. A claw-shaped tip 49 is provided at the tip of the base portion 47. The bracket 40 is attached to the photographing device 1 by hooking the tip portion 49 onto the joint rail 12 near the front end of the photographing device 1.

[4. Electrical configuration of photographing device 1]
FIG. 4 is a block diagram showing the electrical configuration of the photographing device 1. As shown in FIG. The control unit 50 controls each part of the photographing device 1. The control unit 50 is, for example, a computer including a processor 51 and a memory 52. The processor 51 is, for example, a CPU (Central Processing Unit), an MPU (Micro Processing Unit), a GPU (Graphics Processing Unit), or an ASIC (Application-Specific Input). integrated circuit), and FPGA (Field Programmable Gate Array). The memory 52 is a main storage device including, for example, RAM (Random Access Memory) and ROM (Read Only Memory). The processor 51 temporarily stores the program read from the ROM in the memory 52 in the RAM. The RAM of memory 52 provides a work area for processor 51. The processor 51 performs various controls by performing arithmetic processing while temporarily storing data generated during program execution in the RAM. The control unit 50 further includes a clock unit 53 that measures time. The clock section 53 is, for example, a real-time clock. The clock section 53 may be mounted on the motherboard of the processor 51, or may be externally attached to the processor 51.

The input unit 60 receives information input from the user. The input unit 60 includes, for example, the above-mentioned microphone 61, event recording button 31, operation unit 26, and touch sensor 24. The microphone 61 converts sound incident through the microphone hole 14 etc. into an electrical signal. The microphone 61 is, for example, a condenser microphone. Touch sensor 24 detects the position touched by the user on display surface 23. The touch sensor 24 is, for example, of a capacitive type.

The display unit 65 displays an image on the display surface 23. The display unit 65 is, for example, a liquid crystal display (LCD).

The audio output unit 66 outputs audio. Examples of this sound include notification sounds, BGM, voice messages, and the like. The audio output unit 66 includes, for example, an audio processing circuit and a speaker.

The imaging unit 67 takes an image and generates image data obtained by the imaging. The imaging unit 67 includes, for example, an imaging lens 15 and an imaging element that images the light collected by the imaging lens 15. The image sensor is, for example, a CMOS (Complementary MOS) or a CCD (Charge Coupled Device). The photographing unit 67 generates a color (multicolor) image made up of color components of red (R), green (G), and blue (B), for example.

The communication unit 68 communicates with external devices. The communication unit 68 includes a communication circuit for wirelessly communicating with an external device using, for example, Wi-Fi (registered trademark), Bluetooth (registered trademark), other wireless LAN (Local Area Network) communication, or short-range wireless communication. . The communication unit 68 may include a communication circuit for performing communication based on mobile communication system standards such as LTE (Long Term Evolution), 4G, and 5G, for example.

The sensor section 69 includes various sensors. The sensor section 69 includes, for example, at least one of an acceleration sensor, a gyro sensor, an atmospheric pressure sensor, and an illuminance sensor. The acceleration sensor is a three-axis acceleration sensor that detects, for example, longitudinal, lateral, and vertical acceleration of the vehicle. The gyro sensor is a sensor that detects the tilt of the photographing device 1. The acceleration sensor and the gyro sensor may be used, for example, to estimate the position of the vehicle 4 by autonomous navigation when signals from GNSS satellites cannot be received. The atmospheric pressure sensor measures atmospheric pressure. The atmospheric pressure sensor is used, for example, to detect differences in elevation and determine whether a highway is a highway or a general road. The illuminance sensor is, for example, a sensor that detects illuminance indicating the brightness in the vehicle interior surrounding the photographing device 1. The illuminance sensor is used, for example, to adjust the brightness of the display on the display unit 65.

The reader/writer 70 functions as a medium holding section that holds a storage medium 71 inserted into the photographing device 1 from the storage medium insertion port 19. The reader/writer 70 writes data to and reads data from the storage medium 71. The reader/writer 70 may hold only one storage medium 71, but may be configured to hold two or more storage media 71 at the same time.

The terminal section 72 has a terminal for electrically connecting to an external device. The terminal section 72 includes the camera jack 17 and the terminal 18 described above. As a device connected to the terminal portion 72, an external battery may be used so that the photographing device 1 and the photographing device 2 can operate even without power being supplied from the vehicle 4 side. The device connected to the terminal section 72 may be, for example, a device having a function of supporting safe driving of the user. Examples of such a device include a device that has a function of photographing the driver (for example, his face) and detecting and notifying the driver's condition, such as inattentive driving and drowsy driving, and There is a device (for example, a device used for vehicle detection for a forward vehicle collision warning system (FCWS)) that has a function of detecting and giving an alarm. The device connected to the terminal section 72 may also be an in-vehicle device such as a radar detector, a laser detector, a car navigation device, a display device, or the like.

The position information acquisition unit 73 acquires position information indicating the position of the imaging device 1 (more specifically, the current position). The position of the photographing device 1 is the same as the position of the vehicle 4 in which the photographing device 1 is placed, the position of the photographing device 2 placed in the vehicle 4, and the position of the driver and other persons riding in the vehicle 4. I can do it. The position information acquisition unit 73 acquires the position information (latitude information and longitude information) of the photographing device 1 based on, for example, a signal from a GPS (Global Positioning System), which is one of the GNSS (Global Navigation Satellite System). ) to obtain. The position information acquisition unit 73 may also utilize Michibiki as QZSS (Quasi-Zenith Satellite System).

The light emitting section 21 emits light in a predetermined color. The light emitting section 21 includes, for example, a light emitting diode.

The power supply control unit 75 controls the supply of power to each part of the photographing device 1 and the photographing device 2. The power control unit 75 includes, for example, a power switch and a power control circuit. The power supply control section 75 supplies electric power supplied from the vehicle 4 side via the terminal section 72 to each part of the photographing device 1 and the photographing device 2. The power supply control unit 75 may further include a secondary battery, a button battery, or an electric double layer capacitor (also called a supercapacitor) as a power storage means.

The video input/output unit 76 outputs video data obtained by imaging with the imaging device 1 to the outside of the imaging device 1, and inputs video data provided from the outside into the imaging device 1. The data input/output unit 77 also outputs desired data (including video data) from the photographing device 1 to the outside, inputs desired data (including video data) given from the outside into the photographing device 1, and so on. do.

The photographing device 1 may further include an auxiliary storage device such as a flash memory (for example, eMMC, SSD) as an internal storage means. As the auxiliary storage device, various storage media such as optical storage media, magnetic storage media, and semiconductor storage media can be used.

[5. Image recording function of photographing device 1]
The photographing device 1 has one or more of the following functions as an image recording function. The image recording function is a function of recording an image photographed by the photographing device 1 as image data in a predetermined file format. The image data is preferably image data in a moving picture format, for example in the MPEG (Moving Picture Experts Group) format (eg, MPEG2, MPEG4), AVI, MOV, WMV, etc.

<5-1. Continuous recording function>
The constant recording function (also referred to as the constant recording function) is a function that continuously (that is, always) records images photographed by one or both of the photographing device 1 and the photographing device 2 while the photographing device 1 is in operation. be. When the control unit 50 is executing the constant recording function, the control unit 50 stores video data captured from the time when the engine of the vehicle 4 is started to stopped. Starting of the engine is detected, for example, by turning on the accessory power source of the vehicle 4, and stopping the engine is detected by turning off the accessory power source.

<5-2. Event recording function>
The event recording function is a function of recording an image photographed by one or both of the photographing device 1 and the photographing device 2 in response to the occurrence of a specific event. An event is an event in which an image photographed by the photographing device 1 or the photographing device 2 is to be recorded, and for example, when the user suddenly operates the steering wheel or brakes suddenly while the vehicle 4 is running, or when the vehicle 4 collides with another object. In the event of a collision, etc. The control unit 50 determines that an event has occurred based on, for example, a value measured by an acceleration sensor of the sensor unit 69. Specifically, the control unit 50 determines that an event has occurred when the measured value of the acceleration sensor exceeds a predetermined threshold or shows a predetermined change over time. The conditions for determining the occurrence of an event are not limited to these. Based on the vehicle information, the control unit 50 may determine that an event has occurred when the state of the vehicle 4, such as the vehicle speed or steering state, satisfies a predetermined condition. The control unit 50 analyzes the image taken by the imaging unit 67 or the imaging device 2, and determines that an event has occurred when dangerous driving (for example, aggressive driving, approaching, or abnormally approaching) of the vehicle 4 or another vehicle is detected. It may be determined that The control unit 50 also determines that an event has occurred when the event recording button 31 is operated.

When the control unit 50 determines that an event has occurred, it records on the storage medium 71 images captured during a predetermined period before and after the event (hereinafter referred to as "event recording period"). For example, the control unit 50 temporarily records images photographed by the photographing unit 67 and the photographing device 2 in the memory 52 (for example, RAM), and when it is determined that an event has occurred, reads the images from the memory 52. It is preferable to record images during the event recording period on the storage medium 71. For example, the control unit 50 creates one file of images for a total of 40 seconds, 20 seconds before the occurrence of the event and 20 seconds after the occurrence of the event. The event recording period is just one example, and may vary depending on the type of event, or may be changeable by the user. The control unit 50 may record an image made up of a plurality of files on the storage medium 71 for each occurrence of one event. The control unit 50 further associates the values measured by the sensor unit 69 during the event recording period (for example, acceleration in each direction of three axes) and the position information acquired by the position information acquisition unit 73 with the image, It may also be recorded on the storage medium 71.

<5-3. Parking monitoring function＞
The parking monitoring function is a function of recording an image photographed by one or both of the photographing device 1 and the photographing device 2 while the vehicle 4 is parked. The parking monitoring function is a function for monitoring the inside of the parked vehicle 4 or the outside around the vehicle 4. When the engine of the vehicle 4 is off, the control unit 50 receives power from an external battery and records images on the storage medium 71. Regarding whether or not the vehicle 4 is parked, the control unit 50 determines, for example, that the accessory power source is turned off, that the engine is turned off, that power supply from an external battery is started, or that the vehicle speed is 0 km. /h or below a predetermined speed, and the location information acquired by the location information acquisition unit 73 is predetermined location information (for example, location information of home, work, or parking lot). Judgment based on

The parking monitoring function may have a time-lapse mode and a motion detection mode. Specifically, when the time-lapse mode is selected by the user, the control unit 50 records images by thinning out the frame rate compared to other image recording functions such as the constant recording function and the event recording function. For example, while the frame rate of other image recording functions is 20 to 30 frames/second, the frame rate of time-lapse mode is 1 frame/second. The moving object detection mode is a mode in which images are recorded in response to detection of a moving object. Specifically, when the moving object detection mode is selected by the user and a moving object is detected from a change in images captured by the photographing device 1 and the photographing device 2, the control unit 50 detects a moving object for a predetermined period before and after the detection. The image photographed is recorded on the storage medium 71. The frame rate may be the same frame rate as the constant recording function, event recording function, etc.

Note that as the photographing devices 1 and 2, photographing devices that photograph celestial sphere images such as a celestial sphere or a half-celestial sphere may be used. Further, as the photographing device 1, a photographing device without the display section 65 may be used. Further, the housing of the photographing device 1 does not have to be rectangular, and may be, for example, a cylindrical photographing device.

[6. Learning model】
FIG. 5 is an example of the learning model 80. The learning model 80 becomes a learning model during learning using teacher data and learning data, but becomes a learned model 80 when learning is completed. A trained model is an example of an inference model.

The learning model 80 shown in FIG. 5 inputs a plurality of images by deep learning and outputs the probability that an object to be detected is included in the images forming a moving image. The learning model 80 includes an input layer 81, a middle layer 82, and an output layer 83.

Each of the pixels P1 to PN constituting an image that is determined to have a 100% probability of containing the object to be detected is input from the input layer 81 of the learning model 80, and deep・Learning is performed by using a huge number of arbitrary images as teacher data, teacher images, learning data, learning images, etc. Similarly, the probability of an image being judged to be 95% to contain the object to be detected, an image to be judged to be 90% to containing the object to be detected, etc. in 5% increments (i.e. 85%, 80%, 75%, 70%, 65%, 60%, 55%, 50%, 45%, 40%, 35%, 30%, 25%, 20%, 15 %, 10%, 5%, 0%) (not necessarily in 5% increments) are trained by deep learning as teacher data, teacher images, training data, training images, etc. . When the learning is completed, the learning model 80 becomes the trained model 80, so both the learning model and the trained model are indicated by the same reference numeral 80.

When an arbitrary image is input from the input layer 81 of the trained model 80, the neuron 83a responds when the probability that the input image contains the object is 100%, and similarly the input image contains the object. When the probability that the object is included in the input image is 95%, the neuron 83b responds, and when the probability that the input image contains the object is 0%, the neuron 83n responds. The same applies to other probabilities.

The output data of the trained model 80 is input to a discriminator circuit (not shown), and when the discriminator circuit finds data representing a probability greater than or equal to a threshold value, it outputs data indicating that an object has been detected, and the probability value is less than the threshold value. When the data indicates that the object is not detected, the data indicating that the object is not detected is output or the output of the data itself is stopped. By outputting data indicating that an object has been detected from the discrimination circuit, it is known that an object, for example, a person, has been detected in the photographed video, and it becomes possible to issue a warning to the driver and other processing. In the following, the subject will be a person, but other subjects may be used. Even without providing a discrimination circuit, if the probability represented by the output data of the trained model 80 is equal to or higher than a threshold value, the image may be determined by software to be an image in which a person is present. When it is known that a subject is included in a captured video, for example, each image that makes up the video, a process is performed to detect the person by detecting landmarks of the person from the image and surround the detected person with a frame. It will be done. In this way, the trained model 80 shown in FIG. 5 outputs the probability that the image contains the object, and if the output probability is greater than or equal to the threshold, it is determined that the image contains the object. , the landmark of the target is detected from the image and the target is surrounded by a frame, but it is possible to input the captured video and each image that makes up the video and output an image that surrounds the person included in the image. A trained model 80 or process may also be used. Such processing can be achieved by using OpenCV etc. (https://jellyware.jp/aicorex/contents/out_c08_realtime.html). Alternatively, the trained model 80 may be used to detect the position of the target from the captured video, and a process may be performed to surround the detected position with a frame. In any case, it is sufficient if the target can be detected from the captured video.

By inputting photographed video data obtained by photographing with the photographing device 1 or 2 into the trained model 80 and causing it to infer and learn, an object can be detected from images constituting the video represented by the photographed video data. In addition to target detection, other inferences and learning are also performed using the learning model 80 by inputting captured video data and learning so that the desired result is output, and the trained model 80 is Can be built. For example, by learning a video in which an accident is likely to occur or a video before the accident occurs as training data, when a situation where an accident is likely to occur is photographed by inputting the captured video data into the trained model 80, It is also possible to detect this situation and warn the driver of the vehicle 4. Furthermore, when there is an object in the captured video, it is not necessarily necessary to surround the object with a frame. If it is determined that there is a target in the captured video, an event can be recorded as described later.

[7. First Example]
6 to 9 are flowcharts showing the processing procedure of the photographing device 1. The photographing device 2 may also perform similar processing.

When the fourth button 30 is pressed, the photographing device 1 starts photographing (step 91 in FIG. 6). A moving image represented by the moving image data obtained by shooting is displayed on the display screen 23 (step 92 in FIG. 6).

When the user presses the event recording button 31, the normal event recording mode (inference event recording mode in this embodiment) is set (YES in step 93 in FIG. 6). When the normal event recording mode is set and it is determined by inference that an event has occurred, moving images for a certain period before and after the event that has occurred are recorded in the event recording area (inferred event recording area) of the storage medium 71. In this embodiment, when an object is detected, it is assumed that an event has occurred, and inference event recording is performed. Inference event recording may also be performed by As will be described later, in this embodiment, there is a double event record in addition to the normal event record (inferred event record). A double event record is an event record in which an inferred event record and a sensor event record are performed as described below, and a normal event record is an inferred event record in this embodiment. The term normal event record (inferred event record) is used to distinguish it from double event record. The normal event record (inferred event record) will be explained. Even when the normal event record (inferred event record) is set, recording processing is always performed. If the normal event recording (inferred event recording) mode is not set (NO in step 93 in FIG. 6), the normal event recording is not performed in the process shown in FIG. 6, but the constant recording process is performed.

When the normal event recording mode is set (YES in step 93 in FIG. 6), the learned model selection image 120 shown in FIG. 9 is displayed on the display surface 23 of the photographing device 1.

Referring to FIG. 9, trained model selection image 120 includes an area 121 touched by the user when selecting trained model A, an area 122 touched by the user when selecting trained model B, Area 123 touched by the user when selecting the trained model C, area 124 where the trained model 80 to be recommended is displayed, area 125 touched by the user when the user decides on the trained model 80, and learning. Contains the area touched by the user when automatically determining the completed model 80.

A wide variety of learned models 80 are stored in the memory 52 of the photographing device 1, and an event occurs by learning the video data obtained by shooting using the learned model 80 according to the user's selection. It is detected whether the In this embodiment, the detection of an object is considered to be the occurrence of an event, but the event may also be caused by another trigger. If an event occurs due to another trigger, the trained model 80 that infers that the event has occurred according to that trigger will be used.

For example, trained model A does not detect people of a specific age group with high accuracy, but uses training data that can uniformly detect people of all age groups, regardless of whether they are children, adults, or the elderly. This is a trained model 80 obtained by learning. In addition, trained model A does not detect people with high accuracy when taking pictures during the day or when taking pictures at night, and it does not detect people with high accuracy when taking pictures at night. This is a trained model 80 obtained by learning using teacher data that can be detected. Since trained model A has average human detection accuracy, it may be difficult to detect elderly people and children with high accuracy compared to, for example, trained model 80, which detects elderly people and children with high accuracy. . Furthermore, compared to the trained model 80, which can detect people at night with high accuracy, the trained model A may have difficulty detecting people with high accuracy from video data obtained by shooting at night. The trained model 80B is trained using video data that includes elderly people and children as teacher data, and is a trained model 80 that can detect elderly people and children with high accuracy. The trained model C is a trained model 80 that is trained using video data that is photographed at night and includes a person as teacher data, and that detects a person at night with high accuracy.

In area 124, for example, trained model A is recommended in general, trained model B is recommended when detecting elderly people and children with high accuracy and recording inference events, and training model B is recommended when detecting elderly people and children with high accuracy and recording inference events. The reason for recommendation and the type of trained model 80 to be recommended are displayed so that learned model C is recommended when detecting accurately and recording inference events. The user looks at the reason displayed in the area 124 and selects a desired type of trained model 80 from the trained models A to C. When the area 125 is touched by the user, the trained model 80 specified by the area touched by the user among the areas 121 to 123 is determined as the trained model 80 to be used for recording the inference event. This touch causes the processor 51 to issue a manual selection command for the inference model, and the learned model 80 is selected.

In FIG. 9, three types of trained models A to C are illustrated, but more types of trained models 80 may be selected.

In addition, FIG. 9 shows a trained model A that is commonly used for person detection, a trained model B that detects children and the elderly with high accuracy, and a trained model C that detects people at night with high accuracy. However, the user may be allowed to select from other types of trained models 80, or the model may be determined automatically. In the case of automatic determination, a selection command is output from the processor 51 to select the learned model 80 to be automatically determined. For example, the shooting environment may be detected or input, and the trained model 80 suitable for the shooting environment may be automatically determined or selected. The shooting environment includes, for example, the shooting location, the weather at the shooting location, and the brightness of the shooting location. It is preferable to determine the trained model 80 to be used that is suitable for the shooting location, the weather at the shooting location, the brightness of the shooting location, etc. If it is a shooting location, we will decide on trained model B, which can detect children and elderly people with high accuracy in places where there are many people, such as in urban areas, and we will use trained model B, which can detect children and elderly people with high accuracy in places where there are not many people, such as in the suburbs. or determine a trained model with a low value of 80. In addition, if the weather at the shooting location is bad, such as when it is raining, a trained model with high detection accuracy will be selected, or a trained model with high detection accuracy will be selected when shooting in the rain. If the weather is good, the trained model 80, which has low detection accuracy, may be determined. Depending on the brightness of the shooting location, the trained model 80 is determined so that inference is made using the trained model 80, which is relatively more accurate than when it is bright, or the trained model 80 is able to detect people with high accuracy in dark places. I decided on 80. The user may input the shooting location, the weather at the shooting location, and the brightness at the shooting location, or the learning model may be automatically determined based on the obtained information by acquiring the information. For example, if it is a shooting location, the current location can be determined using the GPS function, and the location of the shooting location can be determined by accessing a map server. The weather at the shooting location can also be determined by using the GPS function to determine the current location and accessing the weather server. The brightness of the shooting location can be determined by using an illuminance sensor. In addition, if the shooting time is night, the trained model 80 with high accuracy for detecting people at night may be used, but it is recommended to use the trained model 80 with high accuracy regardless of night or day may be determined.

When the area 126 is touched, the learned model 80 is automatically determined by the processor 51 according to the shooting environment as described above.

Returning to FIG. 6, when any of the regions 121 to 123 in FIG. 9 is touched and the region 125 is touched, the trained model 80 used for object detection is selected by the user. (YES in step 94 in Figure 6). Then, the photographing device 1 is set to detect the object using the selected trained model 80 (step 97 in FIG. 6).

When the area 126 in FIG. 9 is touched (NO in step 94 in FIG. 6), the learned model 80 is automatically determined according to the shooting environment. The photographing environment is detected as described above (step 95 in FIG. 6), and a trained model 80 suitable for the detected photographing environment is selected (step 96 in FIG. 6). For example, if there are trained models 80 suitable for multiple shooting environments, such as nighttime and bad weather, the trained models 80 suitable for multiple shooting environments are selected. The photographing device 1 is set to detect the object using the selected learned model 80 (step 97 in FIG. 6).

Then, the photographed video data obtained by photographing is input to the selected trained model 80, and the model is caused to infer (learn) while photographing (step 98 in FIG. 7). Image data representing an image representing a captured moving image is input frame by frame to the selected learned model 80 and caused to infer.

Also, it is determined whether a disturbance has occurred during the inference (step 99 in FIG. 7). Based on the output from the sensor unit 69, it can be determined whether a disturbance has occurred during inference. If a disturbance is received (YES in step 99 in FIG. 7), it is determined whether the selected learned model 80 is in a state where inference can be made even if the disturbance is received (step 100 in FIG. 7). For example, the illuminance sensor included in the sensor unit 69 may detect that the brightness of the place photographed by the photographing device 1 is below a threshold, or the illuminance sensor may detect that the photographing device 1 is irradiated with light. It is inferred that the G-sensor included in the sensor unit 69 applies an acceleration of more than a predetermined threshold to the vehicle, or that the speed of the vehicle given by the speedometer of the vehicle exceeds a predetermined threshold. It is determined that it is not possible. For example, it is determined in advance whether inference is possible for each type of disturbance or magnitude of the disturbance, and it is determined that inference is not possible for the predetermined type or magnitude of disturbance. For each trained model 80, it is determined in advance whether or not inference can be made for each type of disturbance and the size of the disturbance, and even if it is determined that inference is not possible for the type and size of the disturbance that are predetermined, good. When it is determined whether the state is such that inference is possible, the determination result is notified (step 101 in FIG. 7). The determination result may be notified by outputting a sound from the audio output unit 66, or may be notified by being displayed on the display screen 23.

If it is determined that inference is possible even if there is a disturbance (YES in step 102 in FIG. 7), the inference result is corrected (step 103 in FIG. 7). For example, when determining that a target has been detected when the probability output from the selected learning model is greater than or equal to a threshold, the threshold can be lowered and the target detected even if the probability is lower than when there is no disturbance. The inference result and the probability output from the learning model are corrected so that it is considered that the learning model has been detected (step 103 in FIG. 7). The influence on the inference result may be calculated in advance according to the type and degree of disturbance, and the amount of correction may be increased as the influence on the inference result is greater depending on the type and degree of disturbance. . In this way, disturbances caused by the vehicle 4 to the photographing device due to the movement of the vehicle 4 in which the photographing device 1 is installed, light irradiation on the photographing device 1, for example, irradiation by the lights of an oncoming vehicle, The inference result may be corrected in accordance with disturbances caused by, for example, reflected light from the lights of the vehicle 4 being driven. For example, when determining that a target has been detected when the probability output from the selected learning model is greater than or equal to a threshold, the threshold can be lowered and the target can be considered detected even if the probability is low. Correct the inference results. The disturbance may be an event that can affect the accuracy of the inference, such as movement, speed, acceleration, or shaking of the vehicle given to the photographing device from the vehicle 4.

If the inference is not possible due to disturbance (NO in step 102 in FIG. 7), the inference result is not corrected and the process returns to step 98. Whether inference is not possible due to a disturbance is determined in advance for each type of disturbance and the magnitude of the disturbance, for example. Furthermore, if no disturbance occurs during inference (NO in step 99 in FIG. 7), the processes from step 100 to step 103 are skipped.

If no target is detected by inference (NO in step 104 in FIG. 7), the process returns to step 98. When an object is detected by inference (YES in step 104 in FIG. 7), inference event recording starts (step 105 in FIG. 8).

FIG. 10 is an example of a photographed image constituting a photographed moving image obtained by photographing with the photographing device 1.

The photographed image 160 is displayed on the display screen 23 of the photographing device 1. Vehicle 4 is traveling in a built-up area, and photographed image 160 shows vehicle 170 traveling in front of vehicle 4, and people 161, 162, and 163 with families on the left side when viewed from the direction of travel of vehicle 4. include. Persons 161 and 163 are adults, and person 162 is a child. When the photographed image 160 is input to the learning model and the learning model is trained, the target persons 161, 162, and 163 are detected, so that the inference event recording of the photographed video data is performed.

FIG. 11 shows the format of the storage medium 71.

A file system area 131, a constant recording area 132, an inference event recording area 135, and a user recording area 138 are formed in the recording area of the storage medium 71.

Dedicated software for reproducing data recorded in the constant recording area 132, the inference event recording area 135, and the user recording area 138 is recorded in the file system area 131.

A management area 133 and a recording area 134 are formed in the permanent recording area 132. Various setting information is recorded in the management area 133 by dedicated software recorded in the file system area 131. In the recording area 134, image data representing frames constituting a moving image is recorded in the order of frame numbers. In the recording area 134, a header recording area 151, a frame image data recording area 152, and a footer recording area 153 are formed for each frame. The header recording area 151 records additional information such as a frame number, address position, and shooting time, and the footer recording area 153 records additional information other than the additional information recorded in the header recording area 151.

Frame 1 to frame E recorded in the recording area 134 of the constant recording area 132 represents one moving image period of constant recording. For example, continuous recording is performed from the time the engine of the vehicle 4 is started until the engine is turned off, resulting in one video period.

Similarly to the constant recording area 132, the inference event recording area 135 also includes a management area 136 and a recording area 137. Various setting information is also recorded in the management area 136 by the dedicated software recorded in the file system area 131. In the recording area 137, image data representing frames constituting a moving image is recorded in the order of frame numbers. Also in the recording area 137, a header recording area 151, a frame image data recording area 152, and a footer recording area 153 are formed for each frame. The header recording area 151 records additional information such as a frame number, address position, and shooting time, and the footer recording area 153 records additional information other than the additional information recorded in the header recording area 151.

Frame 1 to frame E recorded in the recording area 137 of the inference event recording area 135 represents one moving image of the event record. For example, one video of an event record is the period from when an object is detected to when the object is no longer detected. As will be described later, when the object is not detected even after a certain period of time has elapsed since the object was no longer detected, the object may no longer be detected, and one event record may conveniently end.

The user recording area 138 is an area where the user can freely record data. The user recording area 138 includes a management area 139 and a user information recording area 140.

The management area 139 is an area for managing data recorded in the user information recording area 140. The start address and end address of a block of data recorded in the user information recording area 140 can be determined by the management area 139, and desired data can be read and recorded.

When event recording is started, recording of photographed moving image data begins in the recording area 137 of the inferred event recording area 135 of the storage medium 71. Since constant recording is performed, the photographed moving image data is recorded in the recording area 134 of the constant recording area 132 of the storage medium 71 regardless of whether an object is detected. The captured video data may be temporarily stored in the memory 52 and updated every several frames to several tens of frames. Thereby, for event recording, captured video data before the object is detected can be read from the memory 52 and recorded in the inferred event recording area 135.

Returning to FIG. 8, when event recording is started, information such as the trained model 80 selected by the user and the inference results of the trained model 80 is stored in the user information recording area 140 of the user recording area 138 when the target is detected. The recorded image is recorded in correspondence with the image (step 106 in FIG. 8). For example, the type and version of the selected trained model 80, the probability of detecting the target as a result of inference, the location, date and time of detecting the target, event records, or storage of captured video data constantly recorded when detecting the target. Information such as an address, a link destination, a point of interest when an object is detected on the medium 71 is recorded in the user information recording area 140, which is an example of an invisible area, in correspondence with the image in which the object was detected. The storage medium 71 may be provided with an area for recording video captions, and information regarding the learned model 80, type, version, inference results, etc. may be recorded in the caption area. Information regarding the learned model 80 can now be displayed as a subtitle when playing back an event-recorded video.

When an object is detected, the detected object is surrounded by a frame (step 107 in FIG. 8).

A photographed image 160A shown in FIG. 12 corresponds to the photographed image 160 in FIG.

When objects 161, 162, and 163 are detected in photographed image 160A, frames 164, 165, and 166 surround the objects 161, 162, and 163, respectively. By looking at boxes 164, 165 and 166, the user knows that objects 161, 162 and 163 are present. Even if all of the objects 161, 162, and 163 are not detected, if at least one object is detected, the detected object is surrounded by a frame.

Image data representing an image of the next frame constituting the photographed video represented by the photographed video data is input to the trained model 80, and learning is performed again using the trained model 80 (step 108 in FIG. 8). The process from step 98 in FIG. 7 is repeated.

If the object is detected in the next frame image as well (NO in step 109 in FIG. 8), the detected object is surrounded by a frame, and image data representing the next frame image is input to the learned model 80. The inference using the learned model 80 is repeated again (step 108 in FIG. 8). In this way, inferences regarding the images forming the moving image represented by the photographed moving image data are repeated for each frame.

In this embodiment, as will be described later, even if the object is no longer detected in the image (YES in step 109 in FIG. 8), the object remains undetected until a predetermined period of time, for example, 0.2 seconds has elapsed after the object is no longer detected (step 111 in FIG. 8). (NO), a frame (which is an example of a mark that specifies a part of the object) is displayed (an example of interpolation) in the place where the frame was displayed. If the object is not detected even after a predetermined period of time, for example 0.2 seconds, has elapsed since the object was no longer detected, the frame is erased and event recording is stopped (step 113 in FIG. 8). When the photographing end command is input to the photographing device 1 (YES in step 114 in FIG. 8), the processing shown in FIGS. 6 to 8 ends. If the photographing end command is not input to the photographing device 1 (NO in step 114 in FIG. 8), the processing from step 78 in FIG. 7 is repeated.

FIG. 13 shows some of the images forming the captured video represented by the captured video data.

If there are objects 161, 162, and 163 on the other side of the fence 167 in the N-frame image, those objects 161, 162, and 163 are detected by inferring the N-frame image, and these objects 161, 162, and 163 is surrounded by frames 164, 165 and 166. However, even if the images of the (N+1) frames are inferred, the objects 161, 162, and 163 may not be detected due to the influence of the fence 167 even if the objects 161, 162, and 163 are on the other side of the fence 167. For this reason, in this embodiment, an interpolated image, at least a frame, is generated by interpolating until a predetermined period of time, for example, 0.2 seconds has passed after the target is no longer detected, and a frame 164a is created at the location where the target was detected. 165a and 166a are displayed. When objects 161, 162 and 163 are detected by inference in the (N+M) frame image, frames 164, 165 and 166 surrounding these objects 161, 162 and 163 are displayed.

FIG. 14 also shows some of the images forming the captured video represented by the captured video data.

Also in FIG. 14, if there are objects 161, 162 and 163 on the other side of the fence 167 in the N frame image, those objects 161, 162 and 163 are detected by inference, and these objects 161, 162 and 163 are Surrounded by boxes 164, 165 and 166. Similar to what is shown in FIG. 13, in the image of the (N+1) frame, even if there are objects 161, 162, and 163 on the other side of the fence 167, if the objects 161, 162, and 163 are not detected due to the influence of the fence 167, the objects will be detected. An interpolated image is generated until a predetermined time (for example, 0.2 seconds) has elapsed since the object was no longer detected, and frames 164a, 165a, and 166a are displayed at the locations where the object was detected. If no object is detected in the (N+L) frame images after a predetermined period of time (for example, 0.2 seconds), frames 164a, 165a, and 166a are deleted.

FIG. 15 is an example of one frame of an image constituting a photographed moving image obtained by photographing with the photographing device 1.

In image 160B, objects 161, 162, and 163 are present on the left side in the photographing direction, as in FIGS. 10 and 12. These objects 161, 162, and 163 are detected by inputting the image 160B to the trained model 80 and inferring them, and the objects 161, 162, and 163 are surrounded by frames 164, 165, and 166. In addition to the vehicle in front of the photographing direction, there is also a vehicle 180 on the right side toward the photographing direction, and objects 181 and 182 exist within the vehicle 180. When image 160B is input to trained model 80, these objects 181 and 182 are also detected, and frames 183 and 184 surrounding these objects 181 and 182 are also displayed.

However, the intention of detecting an object is, for example, to detect an object walking on the road and alert the driver of the vehicle 4, and to detect an object riding in another vehicle 180 etc. It is considered that it is not necessarily necessary to notify the driver. Therefore, in this embodiment, an object whose relative speed to the vehicle 4 in which the imaging device 1 is installed is less than a certain speed is detected, and an object whose relative speed is higher than the certain speed is not detected. Trained model 80 is used. For example, when generating a trained model 80 as shown in FIG. , 182, and the speed of the vehicle 180 approaching vehicle 4 is v2 km/hour, the target image whose relative speed is (v1 + V2) km/hour, which is greater than a constant speed, is excluded from the detection target. , a trained model 80 is generated using teacher data so as to detect images of objects whose relative speed is below a certain speed. When using such a trained model 80, the trained model 80 may be trained by inputting relative speed data of an object included in a photographed image. Relative speed data can be found from the sense of shooting the two images and the distance traveled by the object. In addition, as shown in FIG. 5, the trained model 80 that detects objects is given the probability of the likeness of the target image output from the output layer 83, and the probability is calculated for objects where the relative speed of the target image is greater than a certain speed. It may be inferred that the probability is 0%, or the inference result may be corrected so that the target is not detected if the relative speed is greater than a certain speed without changing the probability.

Similarly, a trained model 80 that does not detect objects inside a vehicle may be generated, and such a trained model 80 may be used to detect objects from captured images. For objects inside the vehicle, inference or correction may be made so that the probability is 0%. The trained model 80 for detecting a vehicle may also be used, and when an object inside a detected vehicle is detected, the probability of detecting the object may be 0% as an inference result, or it may be corrected so that the object is not detected.

Furthermore, when a wall of a building or other place has a mirror surface, an object may be reflected on the mirror surface, and the object reflected within the mirror surface may be detected. Inference may be made using a trained model 80 that can exclude such targets. An object reflected on a mirror surface may be used as the training data, and a trained model 80 that does not detect an object reflected on a mirror surface as an object may be generated and used.

In the above embodiment, as a result of the inference being performed, from the discrimination circuit that inputs data representing the probability output from the output layer 83 of the learned model 80 shown in FIG. 5 and determines whether or not a target has been detected, When data indicating that an object has been detected is output, the photographing device 1 may be controlled to lower or raise the threshold of the discrimination circuit. By lowering the threshold, it is possible to detect objects even if they are confusing, and alert the driver of the vehicle 4. By raising the threshold, ambiguous objects can be undetected, ensuring reliable detection. It becomes possible to detect objects.

[Second example]
16 to 19 show a second embodiment. In the second embodiment, in addition to event recording based on the result of inference, event recording can be performed based on the output of the sensor section 69 (referred to as double event recording).

16 and 17 are flowcharts showing the processing procedure of the photographing device 1.

When the photographing device 1 starts photographing (step 201 in FIG. 16), the moving image obtained by the photographing is displayed on the display screen 23 (step 202 in FIG. 16).

A menu image is displayed on the display surface 23 by pressing the third button 29, and the double event recording mode can be set using the menu image (step 203 in FIG. 16). If the double event recording mode is not set (NO in step 203 in FIG. 16), the process moves to step 93 in FIG.

When the double event recording mode is set (YES in step 203 in FIG. 16), the inference event recording process (step 204 in FIG. 16) shown in FIGS. 6 to 9 and the sensor event process are performed in parallel. Further, constant recording processing is also performed in parallel. The inference event recording process is similar to the normal event process described above, and the process starting from step 94 in FIG. 6 is performed.

Sensor event processing begins at step 205 in FIG.

Based on the output of the sensor unit 69, it is determined whether the object has been detected (YES in step 205 in FIG. 16). For example, by providing an infrared sensor in the sensor section 69, the object can be detected by detecting the presence of a heated area that matches the shape of the object based on the output from the infrared sensor during night photography. Further, the sensor section 69 may be provided with an ultrasonic sensor, and the object may be detected based on the output from the ultrasonic sensor.

When an object is detected based on the output from the sensor unit 69 (YES in step 205 in FIG. 16), sensor event recording is started (step 206 in FIG. 16). When sensor event recording is started, the position of the detected object on the image, the date and time of detection, the type and version of the detected sensor, etc. are recorded in the storage medium 71 based on the output from the sensor unit 69. .

FIG. 18 is an example of the format of the storage medium 71.

In FIG. 18, the same components as those shown in FIG. 11 are given the same reference numerals, and the description thereof will be omitted.

In the format of the storage medium 71 shown in FIG. 18, a sensor event recording area 221 is provided in addition to the above-described file system area 131, constant recording area 132, and inference event recording area 135. A user recording area 138 shown in FIG. 11 may also be provided to record information regarding the trained model 80, inference results, etc. Information regarding the learned model 80, inference results, etc. may be recorded.

In the recording area 137 of the inference event recording area 135, photographed video data (which is an example of first photographed video data) in which a target is detected by inference is recorded. The captured video data may represent a video in which the object is surrounded by a frame, or may represent a video in which the object is not surrounded by a frame. If the object is not surrounded by a frame, during playback, the object included in the video obtained by playback will be surrounded by a frame using the inference result.

Similarly to the inference event recording area 135, the sensor event recording area 221 also includes a management area 222 and a recording area 223. Various setting information is also recorded in the management area 222 by the dedicated software recorded in the file system area 131. In the recording area 223, image data representing frames constituting a moving image is recorded in the order of frame numbers. Also in the recording area 223, a header recording area 151, a frame image data recording area 152, and a footer recording area 153 are formed for each frame. The header recording area 151 records additional information such as a frame number, address position, and shooting time, and the footer recording area 153 records additional information other than the additional information recorded in the header recording area 151. Information regarding the learned model 80, inference results, etc. may be recorded in the header recording area 151 and the footer recording area 153.

Frame n to frame γ recorded in the recording area 223 of the sensor event recording area 221 represents one moving image of sensor event recording. In this way, based on the output of the sensor unit 69, photographed video data in which the object is detected (which is an example of second photographed video data) is recorded. The captured video data may represent a video in which the object is surrounded by a frame, or may represent a video in which the object is not surrounded by a frame. If the object is not surrounded by a frame, the object included in the moving image obtained by playback will be surrounded by a frame using the output of the sensor unit 69 during playback.

Referring to FIG. 17, when an object is detected based on the output from the sensor unit 69, the object included in the images constituting the photographed video represented by the photographed video data is displayed in a frame, similar to the inference event record. surrounded (Figure 17 Step 207). If the object is detected based on the output from the sensor unit 69 (YES in step 208 in FIG. 17), the process of enclosing and displaying the detected object in a frame is repeated.

Even if the target is no longer detected based on the output from the sensor unit 69 (YES in step 208 in Figure 17), the detected location in the image will continue to be displayed in a frame, as explained using Figures 13 and 14. (Figure 17 Step 209). The frame continues to be displayed even if the object (person) is no longer detected in the image until a predetermined period of time, for example, 0.2 seconds has elapsed since the object was no longer detected (NO in step 210 in FIG. 17). When a predetermined period of time, for example 0.2 seconds, has elapsed since the object was no longer detected (YES in step 210 in FIG. 17), the frame is erased (step 211 in FIG. 17). Sensor event recording to the sensor event recording area 221 is stopped (step 212 in FIG. 17). When a command to end imaging is input to the imaging device 1, the processing of the imaging device 1 ends (YES in step 213 in FIG. 17), and if the end command is not input, the sensor event processing from step 205 in FIG. 16 is repeated. (NO in step 213 of Figure 17).

In FIG. 18, the inference results are stored in the header recording area 151, footer recording area 153, header recording area 151, and footer recording area 153 of the recording area 137 of the inference event recording area 135, which are recording areas that constitute a video file. Information regarding the inference other than the inference result may be recorded in the management area 136 of the inference event recording area 135. By doing this, for example, data other than the inference results can be recorded in a file separate from the video file. However, data other than the inference results may be recorded in the user recording area 138.

FIG. 19 shows another example of the format of the storage medium 71.

In the format of the storage medium 71 shown in FIG. 18, for example, in the inference event recording area 135, images constituting a captured video are input to the trained model 80 and trained, thereby detecting an object, and displaying the detected object in a frame. Data representing an enclosed image is recorded, and data representing an image in which an object is detected based on the output from the sensor unit 69 and the detected object is surrounded by a frame is recorded in the sensor event recording area 221. In contrast, in the format of the storage medium shown in FIG. 19, in addition to the file system area 121, constant recording area 132, inference event recording area 135, and sensor event recording area 221, (This is an example of an area where the recorded inference results are recorded in a file separate from the video file) and a sensor output result recording area 227 are formed.

The inference result recording area 224 also includes a management area 225 and a recording area 226. Various setting information is also recorded in the management area 225 by the dedicated software recorded in the file system area 131. In the recording area 226, an inference result is recorded in correspondence with an image based on the inference result recorded in the inference event recording area 135, and image data representing an image surrounding an object with a frame. For example, when the first inference event recording is performed, the results of the inference for recording the inference event, such as the location of the detection target, the probability of its existence, the type of trained model 80 used, and the training used. The version of the completed model 80, the location where the object was detected, the date and time, etc. are recorded in the recording area 226 of the inference result recording area 224 in association with the inference event record. For example, an address representing the recording location of the inference event record is recorded together with the inference result.

Similarly, the sensor output result recording area 227 also includes a management area 228 and a recording area 229. Various setting information is also recorded in the management area 228 by the dedicated software recorded in the file system area 131. In the recording area 229, sensor output results are recorded in correspondence with image data representing the image recorded in the sensor event recording area 221, an image in which a person is surrounded by a frame. For example, when the first sensor event recording is performed, the sensor output results for that sensor event recording, such as the location of the detection target, the probability of existence, the type of sensor used, and the sensor used. version, location where the object was detected, date and time, etc. are recorded in the recording area 229 of the sensor output result recording area 227 in association with the sensor event record. For example, an address representing the recording location of the sensor event record is recorded together with the sensor output result.

[Third example]
20 to 24 show a third embodiment. In this embodiment, inference is performed in an AI (Artificial Intelligence) module different from the imaging device 1.

FIG. 20 shows an overview of a system 240 with an AI module 230.

The system 240 includes the imaging device 1, an AI module 230, an example of an inference module, and a display device 239. The photographing device 1 and the AI module 230 can communicate with each other, and the AI module 230 can output video data to the display device 239.

FIG. 21 is a block diagram showing the electrical configuration of the AI module 230.

The entire operation of the AI module 230 is controlled by a CPU (Central Processing Unit) 231.

AI module 230 includes a video interface 232 and a data interface 233. The video interface 232 is connected to the video input/output section 76 of the photographing device 1. The data interface 233 is connected to the data input/output section 77 of the imaging device 1.

The AI module 230 also includes a learned model storage device 234, which stores a variety of learned models 80. The AI module 230 includes a memory card reader/writer 235 that writes data to a recording medium such as a memory card 238 and stores desired data in the recording medium such as the memory card 238, and a memory card reader/writer 235 that stores predetermined data. It also includes memory 236. Furthermore, the AI module 230 also includes a video interface 237 that outputs learned captured video data and original captured video data to a display device 239.

FIG. 22 is an example of learned model information 251 displayed on the display screen 23 of the photographing device 1.

When the imaging device 1 and the AI module 230 are connected, a menu is displayed on the display screen 23 of the imaging device 1, and the learned model information display command of the AI module 230 included in the menu is touched, the AI module 230 starts learning. Information 252 about the type of trained model for the AI module stored in the trained model storage device 234, information 253 about the version of the trained model 80, etc. are read from the trained model storage device 234, and the trained model information is read from the trained model storage device 234. is displayed as shown in Figure 22. This is an example of a display as a notification of information indicating what kind of inference model is, such as the type of inference model and the version of the inference model. If the trained model 80 is stored in the trained model storage device 234 in file format, information such as the type and version of the trained model 80 is recorded in the header of each file. Able to read and display information.

FIG. 23 is a flowchart showing the processing procedure of the AI module 230.

As shown in FIG. 22, information about the trained model 80 and the name of the trained model 80 are displayed on the display screen 23 of the photographing device 1, and the user of the photographing device 1 is allowed to select the trained model 80 to be used. When a learning model is selected, a selection command is output from the data input/output unit 77 of the imaging device 1, inputted to the AI module 230, and then inputted to the CPU 231 of the AI module 230. When a command to select the trained model 80 is input to the CPU 231 of the AI module 230 (step 241 in FIG. 23), the trained model 80 is set so that the captured video data is inferred by the trained model 80 according to the selection command ( Figure 23 step 242).

Shooting is started in the shooting device 1, and shot video data (shot video data that has not been inferred by the trained model 80 is referred to as original shot video data) is sent from the video input/output unit 76 of the shooting device 1 to the AI module. be done. When the original photographed video data is input to the AI module 230 (YES in step 243 in FIG. 23), the original photographed video data is input one frame at a time to the set learned model 80, and is caused to infer (step 244 in FIG. 23). .

As described above, when an object is detected from the images constituting the captured video by inference (YES in step 245 in FIG. 23), the detected object is surrounded by a frame by superimposing or the like (step 246 in FIG. 23). Inferred captured video data representing a captured video in which the object is surrounded by a frame is output from the video interface 237 to the display device 239. This is an example of outputting to a device other than the vehicle-mounted photographing device 1. (Figure 23 Step 247). On the display screen of the display device 239, a captured video with the object surrounded by a frame is displayed. If the target has been detected (NO in step 248 in FIG. 23), the processes in steps 246 and 247 are repeated. The inferred photographed video data may be transmitted from the data interface 233 to the photographing device 1, and the inferred photographed video data may be recorded in the storage medium 71 of the photographing device 1.

When the object is no longer detected (YES in step 248 in FIG. 23), the frame is erased (step 249 in FIG. 23). When a termination command is input from the photographing device 1 to the AI module 230 via the data interface 233 (YES in step 250 in FIG. 23), the processing of the AI module 230 is terminated. If the termination command is not input (NO in step 250 in FIG. 23), the processing from step 244 is repeated.

The inferred captured video data may be recorded on the memory card 238 using the memory card reader/writer 235. Further, the inference result, the frame number of the image in which the object was detected, the shooting date and time, the probability, the position of the object in the image in which the object was detected, etc. may be recorded in the memory card 238 according to a format such as that shown in FIG. The data may be transmitted from the data interface 233 to the photographing device 1 and recorded in the storage medium 71 in the photographing device 1.

FIG. 24 is a flowchart showing part of the processing procedure of the AI module 230.

The process shown in FIG. 24 continues from the process in step 243 of FIG.

When the original photographed video data is input to the AI module 230, it is caused to perform inference using the set trained model 80 (step 244 in FIG. 24). When an object is detected (YES in step 245 in FIG. 24), the detected object is surrounded by a frame (step 246 in FIG. 24).

Further, the original photographed video data is delayed by the CPU 231 by the time necessary for inference with the selected learned model 80 and display of the frame (step 251 in FIG. 24). The delayed original shot video data and inferred shot video data are output from the video interface 237 to the display device 239 (step 252 in FIG. 24). Since the original shot video data and the inferred shot video data are input to the display device 239, the inferred shot video and the original shot video that has not been inferred can be displayed and compared.

If the target has been detected (NO in step 253 in FIG. 24), the processing from step 246 is repeated. When the target is no longer detected (YES in step 353 in FIG. 24), the frame is erased, and when a termination command is input from the imaging device 1 to the AI module 230 (YES in step 255 in FIG. 24), the processing of the AI module 230 ends. If the termination command is not input to the AI module 230 (NO in step 255 in FIG. 24), the processing from step 244 is repeated.

In addition, original photographed video data is input from the video input/output section 76 of the photographing device 1 to the video interface 232 of the AI module 230 through the first line, and similarly, from the data input/output section 77 of the photographing device 1, the original photographed video data is input to the video interface 232 of the AI module 230 through the first line. Original photographed video data may be input to the data interface 233 of the AI module 230 through. The original captured video data input to the AI module 230 through the first line is inferred using the trained model 80, and the inferred captured video data representing the captured video that displays a frame surrounding the target by detecting the target is generated. The trained model 80 is used to infer original captured video data that is output from the video interface 237 to the display device 239 or from the video interface 232 to the imaging device 1 and input to the AI module 230 through the second line. , the image may be outputted from the video interface 237 to the display device 239 or outputted from the data interface 233 to the photographing device 1 after being delayed by the time necessary for processing to surround the object with a frame. By doing so, it is possible to compare and display the photographed video represented by the original photographed video data and the photographed video represented by the inferred photographed video data.

[Fourth example]
25 to 27 show a fourth embodiment.

FIG. 26 shows a system 269 that includes an inference server 260.

Both the imaging device 1 and the inference server 260 can be connected to the Internet. The photographing device 1 can be connected to the Internet through a communication unit 68, and the inference server 260 can be connected to the Internet through a communication circuit 262, which will be described later.

FIG. 26 is a block diagram showing the electrical configuration of the inference server 260.

The entire operation of the inference server 260 is supervised by the control device 261.

As described above, the inference server 260 includes a communication circuit 262 for connecting to the Internet, a memory 263 for temporarily storing data, and a wide variety of trained models, such as models higher than the trained model of the imaging device 1. A trained model storage device 264 that stores a trained model of accuracy 80, a hard disk 266 that stores predetermined data, and a hard disk drive 265 that writes data to the hard disk 266 and reads data written to the hard disk 266. and an input device 267 for giving commands and the like to the inference server 260.

FIG. 27 is a flowchart showing the processing procedure of the imaging device 1 and the inference server 260.

As described above, when the constantly recorded video data and the inferred event recorded video data are recorded in the storage medium 71 in the imaging device 1, the constantly recorded video data and the inferred event recorded video data are inferred from the imaging device 1. server 260 (FIG. 27 step 271). The timing of transmitting these data to the inference server 260 may be when the user of the imaging device 1 gives a transmission command using a menu to the imaging device 1, or when the transmission command from the inference server 260 is sent to the imaging device 1. 1, or at any arbitrary timing.

When the constantly recorded video data and the inference event recorded video data transmitted from the imaging device 1 are received by the inference server 260 (YES in step 281 in FIG. 27), high accuracy detection of the target using the constantly recorded data is performed. Inference is performed in the inference server 260 (step 282 in FIG. 27).

When an object is detected by inference in the highly accurate trained model 80 (YES in step 283 in FIG. 27), whether the image of the object detected in the inference server 260 is also recorded in the inference event record of the imaging device 1, It is also confirmed whether the object has been detected in the inference event record of the imaging device 1 (step 284 in FIG. 27). If the object is also detected in the inference event record, it is confirmed whether the object was detected because highly accurate inference was performed in the imaging device 1 (step 285 in FIG. 27). To determine whether highly accurate inference has been performed in the imaging device 1, the imaging device 1 may check with the inference server 260 the type, version, etc. of the trained model 80 used in the inference in the imaging device 1.

If the inference server 260 was able to detect the target (YES in step 283 in FIG. 27), but the target could not be detected in the inference event record performed in the imaging device 1 (NO in step 284 in FIG. 27), the inference server 260 to the imaging device 1, a trained model 80 that is slightly more accurate than the inference performed in the imaging device 1 (a trained model that is less accurate than the inference in the inference server 260 but capable of relatively high-precision inference) 80), the relearning command and teacher data are transmitted from the inference server 260 to the imaging device 1 (step 286 in FIG. 27). The teacher data is, for example, data representing an image containing an object detected by the inference server 260. The inference server 260 was able to detect the object (YES in step 283 in FIG. 27), and the imaging device 1 also detected the object (YES in step 284 in FIG. 27). 260) is being performed (YES in step 285 in Figure 27), the retraining command and training data are used to perform inference using the trained model 80 with lower accuracy. is transmitted from the inference server 260 to the photographing device 1 (step 286 in FIG. 27).

When a relearning command and teacher data are received in the imaging device 1 so as to perform inference using the trained model 80 with slightly higher accuracy (YES in step 272 in FIG. 27), the trained model 80 that was being used is is retrained using the training data sent from the inference server 260 so that the accuracy is slightly higher, for example, the probability of detecting the target is lower than the inference in the inference server 260, but the probability of detecting the target is increased (Fig. 27 steps 273). The teacher data is, for example, data representing an image containing an object detected by the inference server 260. When a retraining command and training data are received in the imaging device 1 so as to perform inference using the trained model 80 with low accuracy (YES in step 272 in FIG. 27), the trained model 80 that was being used is is too accurate for the photographing device 1, so it is sent from the inference server 260 so that it cannot detect objects that could be detected by the inference server 260 using the highly accurate trained model 80, so that the accuracy is too low. The training data is retrained using the trained training data (Step 273 in Figure 27).

The imaging device 1 repeats the processes of steps 272 and 273 until a termination command is given (step 274 in FIG. 27), and the inference server 260 repeats the processes from step 282 until a termination command is given (step 287 in FIG. 27). is repeated. The inference in the photographing device 1 is executed with medium precision, not too high precision, and not too low precision.

In the above-described embodiment, when the object is not detected by the inference in the imaging device 1 and the object is detected in the inference server 260, the probability of the inference in the imaging device 1 is inputted to discriminate the presence or absence of the object. The probability that the object will be detected may be increased by lowering the threshold value, or the object may be detected because the inference in the photographing device 1 is highly accurate, and the inference server 260 may also detect the object through highly accurate inference. At this time, the probability of the object being detected may be lowered by inputting the inference probability in the photographing device 1 and increasing the threshold value for determining the presence or absence of the object.

[Fifth example]
28 to 31 show the fifth embodiment.

28 and 29 are flowcharts showing the processing procedure of the photographing device 1 provided in the vehicle 4.

The photographing device 1 according to this embodiment is capable of photographing 360 degrees around the entire interior of the vehicle, including front, rear, left and right. 30 is an example of a photographed image, such as a celestial sphere image and a circumferential image 310, obtained by installing the photographing device 1 at a standard height (for example, 1.3 m above the ground), and FIG. This is an example of a photographed image 320 obtained by installing at a position lower than the reference height (for example, 1 m above the ground). The photographing device 1 may photograph an image with the photographing direction below the reference height.

A photographed image 310 obtained by installing the photographing device 1 at a reference height as shown in FIG. 30 and a photographed image 310 obtained by installing the photographing device 1 at a position lower or higher than the reference height as shown in FIG. The photographed image 320 shown in FIG. In this embodiment, the height at which the photographing device 1 is installed is input, and the photographed image 320 is adjusted so that the photographed image 320 obtained is the same as the photographed image 310 obtained when the photographing device 1 is installed at the reference height. is adjusted. Photographed video data representing the adjusted photographed image 320 is input to a learned model 80 as shown in FIG. 5, and inference is performed.

When the photographing device 1 starts photographing as described above (step 291 in FIG. 28), the moving image obtained by photographing is displayed on the display screen 23 (step 292 in FIG. 28). The user of the photographing device 1 presses the third button 29 of the photographing device 1 to display a menu on the display screen 23, selects a menu for inputting the installation height from the menu, and sets the installation height of the photographing device 1. Enter the height. When the store or user of the photographing device 1 installs the photographing device 1 in the vehicle 4, the installation height may be inputted and the height may be stored in the memory 52. The memory 52 stores the relationship between the height of the photographing device 1 and the magnification of the image of the photographed video data obtained by photographing, and the magnification of the image of the photographed video data corresponding to the height input by the user etc. is read. be able to. By enlarging or reducing the image of the photographed video data using the image magnification corresponding to the input height, the obtained image can be compared to the image when the photographing device 1 is installed at the standard height. It will be the same.

Also in this embodiment, inference event recording and constant recording are performed in parallel. When the user of the imaging device 1 uses the menu to set inferred event recording, for example, normal event recording, double event recording (YES in step 294 of FIG. 28), the input height or the height previously stored in the memory 52 is set. The magnification of the image corresponding to the height is read from the memory 52, and an adjustment process is performed in which the image represented by the captured video data is enlarged or reduced by the read magnification (step 295 in FIG. 28).

The photographed video data, in which the image has been enlarged or reduced, is input to the learned model 80 as shown in FIG. 5, and is caused to make inferences (step 296 in FIG. 28).

When the object is detected (YES in step 297 of FIG. 28), inference event recording is started (step 298 of FIG. 29), and the object is displayed on the display surface 23 surrounded by a frame (step 299 of FIG. 29). When the target is no longer detected (YES in step 300 of FIG. 29), the frame is erased (step 301 of FIG. 29), and inference event recording is stopped (step 302 of FIG. 29).

If the photographing end command is given to the photographing device 1 (YES in step 303 in FIG. 29), the processes shown in FIGS. 28 and 29 will be completed, and if the end command is not given (NO in step 303 in FIG. Then, the process from step 296 in FIG. 28 to step 303 in FIG. 29 is repeated.

[Sixth Example]
32 to 34 show a sixth embodiment. In the sixth embodiment, the photographing device 1 has a playback function. By installing playback viewer software in the file system area 131 of the storage medium 71 loaded into the photographing device 1, the software can be read and played back.

As described above, photographing is started in the photographing device 1 (step 331 in FIG. 32), and a moving image is displayed on the display screen 23 of the photographing device 1 (step 332 in FIG. 32). When the third button 29 is pressed, a menu is displayed on the display screen 23, and when the playback mode is set from the menu (YES at step 333 in Figure 32), the name of the video file to be played is displayed on the display screen 23. is displayed, and the user selects a video file to play (step 334 in FIG. 32). If the playback mode is not set (NO in step 333 of FIG. 32), the process shifts to the moving image recording process shown in FIG. 6 and the like.

When a video file to be played is selected, it is checked whether the selected video file is an inference event record (step 335 in FIG. 32). It is not limited to an inference event record, but may be a normal event record. If it is not an inference event record (NO in step 335 in FIG. 32), a constant recording reproduction process is performed. When the video file of the inference event record is selected and the playback start command button displayed on the display screen 23 is touched, a playback start command is given to the photographing device 1 (YES at step 336 in FIG. 32). Then, the specified video file is played back (step 337 in FIG. 33).

FIG. 34 shows the display surface 23 of the photographing device 1. In FIG. 34, the same components as those shown in FIG. 2(B) are given the same reference numerals, and the description thereof will be omitted.

The played video is displayed on the display screen 23. At the bottom of the display screen 23, a character string 350 reading "Is the event record correct?", an OK button 351, and an NG button 352 are displayed.

The user checks whether the inference event recording is performed correctly or incorrectly while watching the moving image of the inference event record displayed on the display screen 23. Although a frame 353 is displayed on the right side of the display surface 23 in FIG. 34, no object exists within the frame 353. This is thought to be because the object was detected by inference from the captured video data and the frame 353 was displayed even though the object did not exist. Since the inference event record is incorrect, the user touches the NG button 352. If there is an object within the frame 353, this means that the existing object has been detected by inference from the captured video data, and therefore the inference event recording is correct. The user touches the OK button 351.

Returning to FIG. 33, when the NG button 352 indicating that the inference event record is incorrect is pressed while playing the video file of the inference event record (YES in step 338 in FIG. 33), the inference event record In the header of the image that makes up the video, for example, the image that was displayed when the NG button 352 was pressed, it is recorded that relearning is necessary, that the inference event record is incorrect, etc. (Step 33 in Figure 33) 339). Instead of recording the fact that relearning is necessary, etc., in the header of one image, it may be recorded in the header of one video file or the header of the first image of one video file.

When the OK button 351 indicating that the inference event record is correct is pressed while playing the video file of the inference event record (NO in step 338 in Figure 33), the message displayed when the OK button 351 is pressed is pressed. It is recorded that there is an inference event record in the header of the image (step 340 in FIG. 33).

The processes from steps 337 to 340 are repeated until the user gives a reproduction end command (step 341 in FIG. 33).

When the NG button 352 is pressed, the header of the image that was displayed when the NG button 352 was pressed indicates that relearning is required, that the inference event record is incorrect, etc., and the inference is A message indicating that relearning is necessary is recorded for the trained model 80 used for event recording, a different trained model 80, a trained model 80 other than the trained model 80 stored in the imaging device 1, etc. Images are relearned. Even when it is recorded that the inference event record is correct in the header of the image that was displayed when the OK button 351 was pressed, the trained model 80 used for the inference event record, a different trained model 80, Relearning may be performed using a trained model 80 other than the trained model 80 stored in the photographing device 1 using the image thereof.

[Seventh Example]
35 to 38 show a seventh embodiment, which concerns reproduction processing using reproduction viewer software on a personal computer.

FIG. 35 is a block diagram showing the electrical configuration of a tablet personal computer, for example, an example of a playback device.

The entire operation of the tablet personal computer (hereinafter referred to as personal computer) 360 is controlled by a CPU (Central Processing Unit) 361.

The personal computer 360 is provided with a display device 363. This display device 363 is controlled by a display control device 362 which is controlled by a CPU 361. Further, the personal computer 360 is provided with an acceleration sensor 364, and an output signal from the acceleration sensor 364 is input to the CPU 361. Furthermore, the personal computer 360 includes an SSD (solid state drive) 365 that is accessed by the CPU 361, and a memory card reader that reads data recorded in the storage medium 71 and writes data etc. to the storage medium 71. Writer 366 is included.

Furthermore, the personal computer 360 includes an input device 367 such as a keyboard and a mouse, a memory 368 such as a RAM, and a communication circuit 369 connected to a network such as the Internet.

An operating program, which will be described later, is received by the communication circuit 369 of the personal computer 360 via the Internet and installed on the personal computer 360. The CPU 361 controls each section by reading and executing the installed program. An operating program is stored in a storage medium 71 or the like, and the operating program may be read from such storage medium 71 and installed in the personal computer 360.

36 and 37 are flowcharts showing the reproduction processing procedure of the personal computer 360, and FIG. 38 is an example of a display screen of the display device 363.

When the playback software installed on the personal computer 360 is activated, the playback process shown in FIG. 36 starts.

Referring to FIG. 38, the display screen of the display device 363 includes a playback video display area 390 that displays a playback video, an area 391 that displays a video of an inference event record as described later, and a list of constantly recorded video files. A play list area 392 to display, an area 393 to display a map, an information display area 402 to display various information about the vehicle 4 obtained when shooting the playback video displayed in the playback video display area 390, and other information. It includes a region 403 and a region 404 indicating the direction of acceleration applied to the vehicle at the time of photographing.

Also displayed in the map area 393 are an arrow 394 indicating the location where the inference event recording took place and a thumbnail image 395 of the location where the inference event recording took place. The information display area 402 also includes information 398 indicating the recording date and time of the playback video, a button area 399 for giving commands such as start, stop, pause, fast rewind, fast forward, etc. of the playback video, and a location where the playback video was recorded. Information 396, vehicle speed information 401, etc. are displayed.

A storage medium 71 is loaded into the personal computer, and data stored in the storage medium 71 is read. Referring to FIG. 38, play list area 392 displays a list of file names of constantly recorded video files. The user selects a video file with a desired file name from among the file names of constantly recorded video files displayed in the play list area 392 using the input device 367 (step 371 in FIG. 36). When a constantly recorded video file is selected, an inference event record video file corresponding to the selected video file is found in the storage medium 71, and inference event record information is read from the inference event record video file. (step 372 in FIG. 36) and temporarily stored in memory 368. The information on the inference event record includes the location where the inference event record was performed, a link to the video file of the inference event record, and the storage location of images forming the video represented by the video file of the inference event record.

When the playback start button included in the button area 399 is pressed, it means that a playback start command has been input (YES in step 373 in FIG. 36), and playback of the selected constantly recorded video file is started (step 374 in FIG. 36). ). Then, the constantly recorded video is displayed in the playback video display area 390 (step 375 in FIG. 36).

From the information of the read inference event record, data indicating the location where the inference event record was made near the location displayed in the playback video display area 390 is read, and the data is sent to a map server (not shown). Ru. Map data in the vicinity of the place where the inference event was recorded is transmitted from the map server to the personal computer 360, and a map in the vicinity of the place in which the inference event was recorded is displayed in the map display area 393 (Fig. 36 step 376). In addition, an image of the location where the inference event recording was performed is read from the video file of the inference event record, and displayed as a thumbnail image 395 on the map displayed in the map display area 393, and the inference event record is An arrow 394 is displayed at the location where the action was performed. A link to the video file of the inference event record (this link is read as one of the information of the inference event record) is embedded in these arrows 394 and thumbnail images 395.

When the arrow 394 or thumbnail image 395 displayed in the map display area 393 is clicked by the user of the personal computer 360 (YES in step 377 in FIG. 37), the inference event is displayed using the link to the video file of the inference event record. A video file of the recording is read from the storage medium 71. Playback of the inference event record video represented by the inference event record video file is started (step 378 in FIG. 37), and the inference event record video is displayed in area 391 (step 379 in FIG. 37).

Until the reproduction of the video of the inference event record is completed (step 380 in FIG. 37), the video of the inference event record is displayed in the area 391, and the video of the constant recording is displayed in the area 390. When the reproduction of the moving image of the inference event record is completed (YES in step 380 in FIG. 37), it is confirmed whether the reproduction of the constantly recorded moving image has been completed (step 381 in FIG. 37). If the reproduction of the constantly recorded moving image has not been completed (NO in step 381 in FIG. 37), the processing from step 377 in FIG. 37 is repeated. When the reproduction of the constantly recorded moving image is completed (YES in step 381 in FIG. 37), the processing in FIGS. 36 and 37 ends.

In the above embodiment, the constantly recorded video file is specified, the video file of the inference event record is found, and the constantly recorded video and the video of the inference event record are displayed. The moving image of the desired inference event record may be selected from the list, the constantly recorded moving image corresponding to the selected inferential event record may be found, and the moving image of the inferential event record and the constantly recorded moving image may be displayed. Furthermore, the video of the inference event record may be displayed instead of the video of the constant record. For example, when a video file of an inference event record is selected, a map of the vicinity of the place where the inference event recording was performed is displayed in the map display area 393, and the arrow 394 or thumbnail image 395 is clicked, the area 391 is displayed. A video of the inference event record may be displayed.

[Eighth Example]
FIG. 39 and FIG. 40 show an eighth embodiment, which indicates the location of interest when an object is detected. FIG. 39 is a flowchart showing the reproduction processing procedure, and FIG. 40 is an example of an image portion representing the object.

When the storage medium 71 is loaded into the personal computer 360 and the playback software is started, a window as shown in FIG. 38 is displayed. In this state, a command is input from the input device 367 to perform a process of notifying the target location when the target is detected. Then, the playback process shown in FIG. 39 starts.

A list of video files of inference event records is displayed on the display screen of the display device 363 of the personal computer 360, and a desired inference event record file is selected by the user using the input device 367 (step 411). ). When a reproduction start command is input from the input device 367 (YES in step 412), a moving image of the inference event record is displayed on the display screen of the display device 363 (step 413).

When the event focus display command is input using the input device 367 (YES at step 414), data of the target point of interest is read from the inference event record information about the moving image of the inference event record being played. (Step 415). Then, a mark indicating the point of interest is displayed on the inference event recording video (step 416).

Referring to FIG. 40, this is an example of an image portion 424 that constitutes a moving image of an inference event record.

The facial parts of each of the objects 161, 162, and 163 are represented by a heat map 420 (which is an example of a mark indicating a point of interest). For example, the center circle 421 of the heat map 420 is red, the ring 422 around the center circle 421 is yellow, and the ring around ring 422 is blue. It may be expressed in shading instead of in color. By looking at the heat map 420, it can be seen that the detected objects 161, 162, and 163 are detected by focusing on their faces (in particular, their noses, eyes, and mouths). Data on the target point of interest can be obtained, for example, by using VQA (Visual Question Answering) and asking the question "What is the target?" (https://jellyware.jp/aicorex/contents/out_c08_realtime .html). The data of the target point of interest may be stored in the storage medium 71 as inference event recording information when recording an inference event, or may be generated during playback.

Returning to FIG. 39, if the event focus display command is not input (NO in step 414), the processes in steps 415 and 416 are skipped. If the reproduction of the moving image of the inference event record is not completed (NO in step 417), the processing from step 413 is repeated. When the reproduction of the moving image of the inference event record ends (YES in step 417), the process shown in FIG. 39 ends.

[Ninth Example]
FIGS. 41 to 43 show the ninth embodiment and display the inference results.

FIG. 41 is a flowchart showing the processing procedure for displaying the similarity of inference results. For example, it is implemented in the personal computer 360 shown in FIG.

The inference results are read for each imaging device 1 in the personal computer 360 shown in FIG. 35 (step 431). For example, the storage medium 71 loaded into the photographing device 1 and storing the inference results is removed from the photographing device 1, and the removed storage medium is loaded into the personal computer 360 for each photographing device 1. In the personal computer 360, the result of the inference is read for each photographing device 1 in association with the photographing device 1. The similarity of the inference result is calculated in the personal computer 360, and the calculated similarity of the inference result is displayed on the display screen of the display device 363 in association with the identification data of the photographing device 1 (step 432). The identification data is read from the imaging device 1 along with the result of the inference.

FIG. 42 is an example of a list of similarities of inference results displayed on the display screen of the display device 363.

For example, the identification data of the imaging device 1 that has an inference result with a degree of similarity of 90% or more to the inference result of the imaging device 1 specified by the identification data ID0001 is ID0007, ID0010, etc., and is identified by the identification data ID0001. ID0003, ID0008, and the like are ID0003, ID0008, etc., as identification data of the photographing apparatus 1 having an inference result with a degree of similarity of 80% or more to the inference result of the photographing apparatus 1. Further, identification data of the photographing apparatus 1 having an inference result having a degree of similarity of 90% or more to the inference result of the photographing apparatus 1 specified by the identification data ID0002 is ID0005, ID0006, etc.

For example, it can be seen that there is a 90% similarity between the inference result of the photographing device 1 having the identification data ID0001 and the inference result of the photographing device 1 having the identification data ID0007, ID0010, etc. To be more precise, the results of the inference using the trained model 80 used for the inference of the identification data ID0001 in the imaging device 1, and the learned models used for the inference of the identification data ID0007, ID0010, etc. in the imaging device 1. It can be said that there is a 90% similarity with the inference result at 80. Since the degree of similarity of the inference results for each photographing device 1 (trained model 80) is known, it becomes possible to understand the difference in the inference results for each photographing device 1 (trained model 80).

Returning to FIG. 41, on the personal computer 360, the inference results obtained when the situation at the time of photographing is similar are read from the storage medium 71 removed from the photographing apparatus 1 for each photographing apparatus 1 (step 433 ). Then, on the display screen of the display device 363, a list of similarities as a result of inference when the shooting times are similar is displayed as shown in FIG. When the times of photography are similar, for example, the date and time of photography, the weather at the time of photography, the brightness at the time of photography, and the location of photography, such as whether it is in the city or in the suburbs. The data representing the situation at the time of photographing is stored in the storage medium 71 in association with the video file of the inference event record.

For example, when the shooting situations are similar, the identification data of the imaging device 1 that has an inference result with a degree of similarity of 90% or more to the inference result of the imaging device 1 specified by the identification data ID0001 is ID0007, ID0023, etc., and the identification data of the photographing apparatus 1 having an inference result with a degree of similarity of 80% or more to the inference result of the photographing apparatus 1 specified by the identification data ID0001 is ID0003, ID0007, etc. Further, the identification data of the photographing device 1 having an inference result with a degree of similarity of 90% or more to the inference result of the photographing device 1 specified by the identification data ID0002 is. ID0006, ID0012, etc.

For example, if the situations at the time of shooting are similar, the inference result of imaging device 1 with identification data ID0001 and the inference result of imaging device 1 with identification data ID0007, ID0023, etc. have a 90% similarity. I understand that there is something. To be more precise, the results of the inference using the learned model 80 used for inference in the imaging device 1 for identification data ID0001 and the learned models used for inference in the imaging device 1 for identification data ID0007, ID0023, etc. It can be said that there is a 90% similarity with the inference result at 80. Since the similarity of the inference results for each imaging device 1 when the shooting situations are similar is known, it is possible to understand the difference in the inference results for each imaging device 1 when the shooting situations are similar. . Therefore, differences in the inference results for each learned model 80 stored in the photographing device 1 can be seen.

[10th Example]
44 to 49 show a tenth embodiment.

In FIG. 44, a forklift 400 is used as the vehicle. It does not necessarily have to be a forklift 400, and may be a car or the like.

A photographing device 1 is provided on the ceiling of the forklift 400 for photographing circumferential images of the front, rear, left, and right, for example, celestial sphere images in a downward direction. Further, a photographing device 2 for photographing the rear of the forklift 400 is provided at the rear of the forklift 400.

FIG. 45 is an example of a circumferential image obtained by photographing with the photographing device 1 installed on the ceiling of the forklift 400.

In the circumferential image 460 of FIG. 45, the back side is the front, the left side is the left side, the near side is the back side, and the right side is the right side.

FIG. 46 is an example of an image obtained by photographing with the photographing device 2 provided at the rear of the forklift 400.

Since the photographing device 2 is provided behind the forklift 400 and photographs the rear of the forklift 400, the image 470 shown in FIG. 46 represents the rear of the forklift 400.

When performing the above-mentioned inference when there are a circumferential image 460 shown in FIG. 45 and an image 470 shown in FIG. It is sufficient to infer the front image portion 461 and the rear image 470 of the circumferential image 460 without inference using the model 80 or the like. This is because the image of the rear part of the circumferential image 460, for example, the image of the part other than the front image part 461 and the rear image 470 are considered to overlap. In this way, by learning a part of the circumferential image 460 using the trained model 80 or the like, the time required for inference can be shortened. The circumferential image 460 is not limited to a circumferential image.

FIG. 47 is an example of a circumferential image 480 obtained by photographing using the photographing device 1.

The circumferential image 480 is photographed from the front, back, left and right around the center of the circumferential image 480, so it is a normal image where the bottom of the image represents the bottom in real space and the top of the image represents the top in real space. Unlike the image, the subject appears to be standing upside down or sideways. Therefore, if a target is detected from the circumferential image 480 by inference using the trained model 80 shown in FIG. Sometimes.

Considering the image portion 481 representing the front of the circumferential image 480, this image portion 481 is a normal image as shown in FIG. For example, if the trained model 80 shown in Figure 5 is used to detect the target from the image, the detection accuracy will decrease and the detection time will be long. You can reduce the amount of time it takes.

For this purpose, in this embodiment, a first process is performed to transform the circumferential image 480 shown in FIG. 47 into an image 500 as shown in FIG. By performing the second transformation process so that the image 490 shown in FIG. 48 is obtained, the image portion 481 of the circumferential image 480 shown in FIG. 47 has the same vertical direction as the vertical direction in real space as shown in FIG. I found out that I can create image 490. Image portion 501 corresponds to image 490 in FIG. 48. When the input image is a circumferential image 480 shown in FIG. 47 and the output image is an image 490 shown in FIG. 48, by performing the first process and the second process on the circumferential image 480 shown in FIG. An output image 490 shown in is obtained.

By performing the first deformation process and the second deformation process not only on the circumferential image 480 shown in FIG. 47 but also on other circumferential images, the vertical and vertical directions of the real space as shown in FIG. It becomes possible to obtain an image that matches the By performing inference using the trained model 80 using such images, it is possible to improve the accuracy of the inference result and shorten the inference time.

[Modified example]
Inference may be constantly performed while recording on an edge terminal or drive recorder, and the inference results may be saved as data within the video data. Possible inference results include the position of BB (bounding box), classification tag name, and probability. In the data storage area, a subtitle stream of video data may be stored together with GPS data, acceleration data, etc.

When a detection target is detected, event recording may be performed and saved in a dedicated event recording directory. There is no need to infer and extract all continuous recordings later. In addition, after data is uploaded, inference is performed on this data again on the aggregation terminal, and if the inference result of the edge terminal is incorrect, feedback can be applied to improve the model. The edge terminal may broadly capture the information, the aggregation terminal may narrow it down further, and the results may be returned to the edge model for relearning.

In this way, data aggregation/visualization becomes easier in the management system because there is almost no inference required afterwards. You can visualize it by outputting the subtitle stream data and plotting it on a map, and you can also paste the thumbnail and playback link of the target video at that time. If the administrator is interested, all he has to do is watch the video, so it's easy. In addition, by accumulating data, you can filter it later to narrow down the location information and images that match the conditions. It can be used when you want to add AI functionality to existing DVRs and surveillance cameras later.

By making it into an AI module that can be attached to existing products, it can be used for general purposes. Since it can be installed later, the required input only needs to be video data. It would be good if it could communicate with the terminal side. As for output, it is sufficient if the inference result can be outputted to an external monitor with bb (bounding box) added. The DVR's AI module does not require external output. The inference results may be returned to the terminal via communication.

By replacing the model installed in the module, the trained model and inference model can be changed according to the use case. If the device has a data upload function, data including inference results can be uploaded to the cloud.

Drive recorder or drive recorder settings app with a selection function for the user to select the model to be used for inference, record information about the model along with the inference results in the video file (data structure that can be converted to ONNX format, etc.) , information regarding the model may be entered in one file, or in a separate file when there is a change in the model. In addition, it is equipped with a function to distinguish between types of video that are based only on the inference results of image recognition AI and videos that are based on the inference results of image recognition AI and the output of other sensors, etc. A type of video based on the inference result and the output of other sensors, etc. may be provided with a function of recording information such as the output of other sensors, etc. in association with the inference result. Software may be used that has a function of extracting at least one of a history of inference results and model information from video data and recording it as a separate file. It may be software that has a function of extracting video data from which at least one of the history of inference results and model information is removed and recording it as a separate video file.

When playing back video recorded as a result of AI recognition, a GUI, such as a button or VUI, allows the user to input whether or not this recognition is correct, whether it is on the drive recorder itself, an app, PC viewer, etc. For example, it is equipped with voice recognition for "wrong" and "correct", and performs predetermined processing based on the information on whether it is correct or not, the information on the original video, the information on the model that was the basis of the judgment, the inference results, etc. For example, processing for recording these as relearning information, relearning based on these, changing thresholds, etc. may be performed.

In addition, the system is equipped with a function to acquire video data recorded according to AI inference results from multiple drive records, and the video data includes information on the inference results as well as information that uniquely identifies the drive record, ID, etc. Displays the similarity of inference results, such as correlation, among video data recorded on drive recorders with different IDs that are close in location, time, vehicle information, speed, etc. It would be good to have a function. It is preferable to have a function to display items with a high degree of similarity and items with a low degree of similarity side by side, or to compare and display these items side by side.

The viewer of the drive recorder, the drive recorder itself, etc. is equipped with a function to acquire video data recorded according to the inference result of the AI, and the inference status can be displayed along with the type of AI inference, such as the trigger type, and preferably the video. Display the image shown. In particular, it is preferable to have a function of visually displaying points of interest in images in inference. For example, it may be useful to display it as a heat map.

The person in the car opposite will be detected and displayed as a bounding box, so if you detect a person in the car, you can remove it in post-processing. For example, if the moving speed is high, it is determined to be a person in a car, and if the moving speed is slow, it is determined to be a person. Alternatively, a mirror-like object may be repelled from the target. This is because it detects the reflection of the car in front of you in the mirror.

Furthermore, since inference takes time, the bounding box and the target may be misaligned, or the person on the other side of the fence may not be recognized. This causes the bounding box to disappear in some frames. It is best to treat it as continuous. In addition, we perform post-processing to adjust the probability of the bounding box based on the vibration and speed of the car, and because it is difficult to detect objects at night, we switch models between cars during the day and cars at night. Reflection, sunlight, and backlighting can cause a decline in recognition rate. Calibrate at night as it is affected by lamps and reflections. Displays the recognizable/unrecognizable status on the drive recorder. This shows that the inability to recognize objects is due to the unrecognizable state.

Since images vary considerably depending on the height of the camera installation, for example, the inference processing may be adjusted by having the user input the installation height. For example, inference is performed using the height of the general installation position on a forklift, and during inference, the size of the input image is adjusted, for example, enlarged or reduced, using a magnification determined from the difference between the actual installation height and the general height. After that, it may be input to the inference section.

Note that the scope of the present invention is not limited to the configuration explicitly described in the specification, but also includes combinations of various aspects of the invention disclosed in this specification. Of the present invention, the structure for which a patent is sought has been specified in the attached claims, but at present, even if the structure is not specified in the claims, it is not disclosed in this specification. We intend to include such configurations in the scope of claims in the future.

The present invention is not limited to the configuration described in the embodiments described above. The components of each of the embodiments and modifications described above may be arbitrarily selected and combined. Also, any component of each embodiment or modification, any component described in the means for solving the invention, or a component that embodies any component described in the means for solving the invention. It may be configured in any combination. The applicant intends to acquire rights to these matters through amendments to the application or divisional applications. Even if there is a description of ``in the case of'' or ``in the case of'', the description is not intended to be limited to that case or at that time. We have also disclosed these cases and other configurations, and we intend to acquire the rights. Furthermore, the sections described in order are not limited to this order. It also discloses a configuration in which some parts have been deleted or the order has been changed, and we have the intention to acquire the rights.

In addition, the applicant intends to acquire rights to the entire design or partial design by converting the application to a design registration application. Although the drawing depicts the entire device using solid lines, the drawing includes not only the overall design but also the partial design claimed for some parts of the device. For example, it is a drawing that not only includes some members of the device as a partial design, but also includes some parts of the device as a partial design regardless of the members. The part of the device may be a part of the device or a part of the device. We intend to obtain rights not only for the entire design, but also for partial designs in which any part of the solid line part of the drawing is a broken line part. In addition, the modules, members, parts, etc. inside the device housing shown in the drawings are all subject to independent transactions, and similarly, they are included in the design registration application. There is an intention to make changes and obtain rights.

1: Photography device, 2: Photography device, 3: Cable, 4: Vehicle, 5: System, 11: Housing, 12: Joint rail, 13: Sound emission hole, 14: Microphone hole, 15: Imaging lens, 16: Top surface, 17: Camera jack, 18: Terminal, 19: Storage medium insertion slot, 20: Third side, 21: First side, 22: Fourth side, 23: Display surface, 24: Touch sensor, 25 : Light emitting part, 26: Operation part, 27: First button, 28: Second button, 29: Third button, 30: Fourth button, 31: Event recording button, 32: Second side, 40: Bracket, 41: Base part, 42: Mounting surface, 43: Ball stud, 44: Ball part, 45: Socket part, 46: Nut, 47: Base part, 48: Guide rail, 49: Tip part, 50: Control unit, 51: Processor, 52: Memory, 53: Time measurement unit, 60: Input unit, 61: Microphone, 65: Display unit, 66: Audio output unit, 67: Photography unit, 68: Communication unit, 69: Sensor unit , 70: Reader/writer, 71: Storage medium, 72: Terminal section, 73: Position information acquisition section, 75: Power supply control section, 76: Video input/output section, 77: Data input/output section, 80: Learning model (learned) model), 80B: Trained model, 81: Input layer, 82: Middle layer, 83: Output layer, 83a: Neuron, 83b: Neuron, 83n: Neuron, 120: Trained model selection image, 121: System area, 122 -126: Area, 131: System area, 132: Recording area, 133: Management area, 134: Recording area, 135: Inference event recording area, 136: Management area, 137: Recording area, 138: User recording area, 139: Management area, 140: User information recording area, 151: Header recording area, 152: Frame image data recording area, 153: Footer recording area, 160: Photographed image, 160A: Photographed image, 160B: Image, 161-163: Target, 164: Frame, 164a: Frame, 165: Frame, 165a: Frame, 167: Fence, 170: Vehicle, 180: Vehicle, 181: Target, 182: Target, 183: Frame, 221: Sensor event recording area, 222: Management area, 223: Recording area, 224: Inference result recording area, 225: Management area, 226: Recording area, 227: Sensor output result recording area, 228: Management area, 229: Recording area, 230: AI module, 231: CPU, 232: Video interface, 233: Data interface, 234: Learned model storage device, 235: Memory card reader/writer, 236: Memory, 237: Video interface, 238: Memory card, 239: Display device , 240: System, 251: Learned model information, 260: Inference server, 261: Control device, 262: Communication circuit, 263: Memory, 264: Learned model storage device, 265: Hard disk drive, 266: Hard disk, 267: Input device, 269: System, 310: Captured image, 320: Captured image, 350: Character string, 351: OK button, 352: NG button, 353: Frame, 360: Computer, 361: CPU, 362: Display control device, 363: display device, 364: acceleration sensor, 366: memory card reader/writer, 367: input device, 368: memory, 369: communication circuit, 390: playback video display area, 391: area, 392: list Area, 393: Map display area, 394: Arrow, 395: Thumbnail image, 396: Information, 398: Information, 399: Button area, 400: Forklift, 401: Speed information, 402: Information display area, 403: Area, 404 : area, 420: heat map, 421: circle, 422: circular ring, 424: image part, 460: circumferential image, 461: image part, 470: image, 480: circumferential image, 481: image part, 490: Output image, 500: Image, 501: Image part, ID0001, ID0002, ID0007, ID0010, ID0023: Identification data

Claims (27)

A system that has the function of performing inference using an inference model using captured video data obtained by shooting with an in-vehicle camera. The system according to claim 1, wherein the inference model has a function of determining among a plurality of types. The system according to claim 2, having a function of determining the inference model to use the inference model suitable for a shooting environment. 4. The system according to claim 3, further comprising a function of determining the inference model to use the inference model suitable for the shooting location, the weather at the shooting location, or the brightness of the shooting location. 5. The system according to claim 4, wherein the system has a function of determining the inference model to be used so that the darker the brightness of the shooting location, the more accurate the inference model is compared to when the shooting location is brighter. The system according to claim 5, having a function of determining the inference model to be used for inference using the highly accurate inference model at night. The system according to claim 6, having a function of correcting the inference result of the inference model in accordance with disturbances applied to the in-vehicle photographing device. In response to disturbances imparted to the in-vehicle imaging device from the vehicle due to movement of the vehicle in which the in-vehicle imaging device is installed, or disturbances imparted to the in-vehicle imaging device due to irradiation of light to the in-vehicle imaging device. The system according to claim 7, having a function of correcting the inference result of the inference model using the inference model. 9. The system according to claim 8, having a function of notifying whether inference using the inference model is possible or not due to disturbances applied to the in-vehicle photographing device. The system according to claim 9, having a function of adjusting the inference result of the inference model according to the shaking or speed of the vehicle in which the in-vehicle photographing device is installed. The system has a function of adjusting the system based on the height at which the vehicle-mounted photographing device is installed so as to more closely resemble the video obtained by photographing with the vehicle-mounted photographing device installed at a standard height. , the system of claim 10. 12. The system according to claim 11, wherein the vehicle-mounted photographing device has a function of photographing a circumferential image with a photographing direction below a reference height. 13. The method according to claim 1, further comprising a function of inputting a height at which the in-vehicle photographing device is installed and adjusting inference in the inference model according to the input height. system. 14. The system according to claim 13, wherein the inference model has a function of performing inference to detect a person whose moving speed is less than or equal to a predetermined speed from a video. 15. The system according to claim 14, wherein the inference model has a function of making an inference that excludes a person inside the vehicle. 16. The system according to claim 15, wherein the inference model performs inference to detect an object from a moving image, and has a function of excluding an object reflected on a reflective surface. The above-mentioned inference model performs inference to detect a target object from each image that makes up a video, and has a function of displaying a mark that identifies the detected part of the object on each image that makes up the video. 17. The system according to claim 16, having a function of interpolating a mark to an image where the mark is interrupted when the time when the mark is interrupted in the moving image is within a predetermined time. having a function of recording an event of video data in response to detection of an object by inference using the inference model using photographed video data obtained by photographing with the in-vehicle photographing device; 18. The system of claim 17. 19. The system according to claim 18, having a function of lowering a threshold value used for detecting a target object in response to detecting the target object. 20. The system according to claim 19, having a function of displaying the location where the event recording was performed on a map. 21. The system according to claim 20, having a function of displaying an image of the location where the event recording was performed on a map. 22. The system according to claim 21, further comprising a function of embedding a link to video data of the event record at a location where the event record was performed, which is displayed on a map. The above-mentioned event recording of the video data is performed in response to the detection of the target object by the inference using the above-mentioned inference model, and the focused point of the detected target is shown in the image constituting the video in which the above-mentioned event has been recorded. 23. The system according to claim 22, having the function of displaying a mark. The inference results obtained by performing inference using the inference model in each of the plurality of in-vehicle imaging devices are acquired for each of the in-vehicle imaging devices, and the similarity of the inference results for each of the in-vehicle imaging devices is calculated. 24. The system of claim 23, having the capability of displaying. The inference results of the above inference model were performed using captured video data obtained when the position, information, speed, shooting time, etc. of the vehicle in which the above vehicle camera camera is installed are similar. 25. The system according to claim 24, having a function of representing similarity. A method for controlling a system having a function of performing inference using the inference model using captured video data taken while shooting with the in-vehicle imaging device. A program for causing a computer to implement the functions of the system according to claims 1 to 12.

Images