JP7186749B2 - Management system, management method, management device, program and communication terminal - Google Patents

Description

The present invention relates to a management system, management method, management device, program, and communication terminal.

2. Description of the Related Art In recent years, regarding the driving of automobiles, there have been problems of dangerous driving represented by tailgating driving such as driving with a sharply narrowed inter-vehicle distance and driving while watching the screen of a mobile phone. As a countermeasure against such dangerous driving, an imaging device such as a drive recorder is installed in a vehicle to capture images of the surroundings and interior of the vehicle. Also, a technology is being studied that enables a vehicle manager to check the surroundings and interior of the vehicle by transmitting an image generated by imaging. For example, Patent Literature 1 describes a drive recorder that transmits image data at the time of event occurrence to a preset transmission destination when an event such as an accident or crime occurs.

JP 2009-93255 A

The drive recorder described in Patent Literature 1 enables a manager to check the surroundings and interior of a vehicle after an accident has occurred. On the other hand, in order to prevent accidents, it has been desired to enable managers to confirm dangerous driving that does not lead to accidents.

SUMMARY OF THE INVENTION The present invention has been made to solve the above problems, and provides a management system, management method, management device, program, and communication terminal that enable a vehicle manager to appropriately confirm dangerous driving. for the purpose.

A management system according to the present invention is a management system having an imaging device provided in a vehicle and a management device, wherein the imaging device images another vehicle different from the vehicle and generates a first moving image. and a first transmission unit that transmits the first moving image to the management device, and the management device has a relationship between the information on the license plate and the inter-vehicle distance to the vehicle equipped with the license plate a storage unit that stores a learned model that has been learned in advance; a moving image acquiring unit that acquires a first moving image; and acquires information about license plates of other vehicles generated based on the first moving image an estimating unit for estimating the inter-vehicle distance between the vehicle and the other vehicle by inputting the acquired information on the license plate of the other vehicle into the learned model; a calculating unit that calculates the degree of risk of the vehicle, a second generating unit that generates a second moving image by extracting a part of the first moving image based on the degree of risk, and a second moving image; and a second transmitting unit for transmitting.

A management system according to the present invention includes an imaging device provided in a vehicle and a management device, wherein the imaging device captures a driver of the vehicle to generate a first moving image. a generation unit; and a first transmission unit that transmits the first moving image to the management device. A storage unit that stores a trained model that has previously learned a relationship with information indicating whether or not a person is gazing at an object, a moving image acquisition unit that acquires a first moving image, and a first moving image. An information acquisition unit that acquires information related to the posture of the driver of the vehicle and information related to the object held by the driver, and the acquired information related to the posture of the driver and the object held by the driver. An estimation unit that estimates whether or not the vehicle driver is gazing at the target by inputting information into the learned model, and a calculation unit that calculates the risk level of the vehicle driver based on the estimation results. a second generating unit for generating a second moving image by extracting a portion of the first moving image based on the degree of risk; and a second transmitting unit for transmitting the second moving image. , characterized in that

Moreover, in the management system according to the present invention, it is preferable that the calculation unit calculates the degree of risk based on a preset rule.

Further, in the management system according to the present invention, the calculation unit calculates the degree of risk as time-series data associated with the playback time of the first moving image, and the second generation unit determines that the degree of risk satisfies a predetermined condition. It is preferable to extract a portion of the first moving image so as to include a predetermined time range from the time when the condition is satisfied.

Further, in the management system according to the present invention, the calculation unit calculates the degree of risk as time-series data associated with the playback time of the first moving image, and the second generation unit determines that the degree of risk satisfies a predetermined condition. It is preferable to extract a portion of the first moving image so as to include the filling time.

A management method according to the present invention is a management method executed by a system having an imaging device provided in a vehicle, and a management device, wherein the imaging device images another vehicle different from the vehicle to obtain a first image. and transmits the first moving image to the management device, and the management device stores a learned model that has learned in advance the relationship between the information on the license plate and the inter-vehicle distance to the vehicle with the license plate. Then, by acquiring the first moving image, acquiring information about the license plate of the other vehicle, and inputting the acquired information about the license plate of the other vehicle to the trained model, the vehicle and the other vehicle A second moving image is generated by estimating the inter-vehicle distance, calculating the degree of danger of other vehicles based on the estimated inter-vehicle distance, and extracting a portion of the first moving image based on the driving danger. , and transmitting the second moving image.

A management method according to the present invention is a management method executed by a system having an imaging device provided in a vehicle and a management device, wherein the imaging device images a driver of the vehicle to produce a first moving image. and transmits the first moving image to the management device, and the management device sends information about the posture of the driver of the vehicle, information about the object held by the driver, and whether or not the driver is gazing at the object. A trained model that has learned in advance the relationship with information indicating whether or not the Obtaining information about the object to be grasped, and inputting the obtained information about the posture of the driver and the information about the object grasped by the driver into the trained model to determine whether the driver is gazing at the object. is estimated, the degree of danger of the vehicle driver is calculated based on the result of the estimation, a second moving image is generated by extracting a portion of the first moving image based on the degree of danger, and the second and transmitting the moving image.

A management device according to the present invention includes a storage unit that stores a learned model in which a relationship between information on a license plate and an inter-vehicle distance to a vehicle with a license plate is learned in advance, and an imaging device provided in the vehicle. a moving image acquiring unit that acquires a first moving image generated by imaging another vehicle different from the above; and an estimating unit that estimates the inter-vehicle distance between the vehicle and the other vehicle by inputting the obtained information about the license plate of the other vehicle into the trained model, a calculation unit that calculates the degree of risk; a second generation unit that generates a second moving image by extracting a part of the first moving image based on the degree of risk; 2 transmission units.

The management device according to the present invention learns in advance the relationship between information about the posture of the driver of the vehicle, information about the object held by the driver, and information indicating whether the driver is gazing at the object. a storage unit that stores a learned model; a moving image acquisition unit that acquires a first moving image generated by an imaging device provided in a vehicle capturing an image of a driver of the vehicle; and based on the first moving image, An information acquisition unit that acquires generated information on the posture of the driver of the vehicle and information on the object held by the driver, and learns the acquired information on the posture of the driver and the information on the object held by the driver. An estimating unit for estimating whether or not the vehicle driver is gazing at an object, a calculating unit for calculating the risk level of the vehicle driver based on the result of the estimation, and a risk a second generating unit that generates a second moving image by extracting a part of the first moving image based on the degree of motion; and a second transmitting unit that transmits the second moving image. and

A program according to the present invention is a program for a computer having a storage unit that stores a learned model in which a relationship between information related to a license plate and a distance to a vehicle having the license plate is learned in advance, and is provided in the vehicle. acquires a first moving image generated by imaging another vehicle different from the vehicle by the imaging device, and acquires information about the license plate of the other vehicle generated based on the first moving image; , by inputting the obtained information about the license plates of other vehicles into the trained model, the distance between the vehicle and other vehicles is estimated, and the degree of danger of the other vehicle is calculated based on the estimated distance between the vehicles. , generating a second moving image by extracting a part of the first moving image based on the degree of driving risk, and transmitting the second moving image.

The program according to the present invention learns in advance the relationship between information about the posture of the driver of the vehicle, information about the object held by the driver, and information indicating whether the driver is gazing at the object. A program for a computer having a storage unit for storing a model, wherein an imaging device provided in a vehicle acquires a first moving image generated by capturing an image of a driver of the vehicle, and acquires a first moving image generated by imaging a driver of the vehicle, and acquires the first moving image based on the first moving image. information about the posture of the driver of the vehicle and information about the object grasped by the driver, which are generated by By inputting, it is estimated whether or not the driver of the vehicle is gazing at the object, the degree of risk of the driver of the vehicle is calculated based on the result of the estimation, and the first moving image is generated based on the degree of risk. A second moving image is generated by extracting a part of the image, and the second moving image is transmitted.

A communication terminal according to the present invention is a communication terminal communicably connected to an imaging device and a management device of a system having an imaging device and a management device provided in a vehicle, and the imaging device is different from the vehicle. a receiving unit that receives from the imaging device a first moving image generated by imaging the vehicle, and a learned model that has learned in advance the relationship between information on the license plate and the inter-vehicle distance to the vehicle equipped with the license plate. Then, when the first moving image is received, information about the license plate of the other vehicle, which is generated based on the first moving image, is input to the learned model, thereby determining the relationship between the vehicle and the other vehicle. A second moving image is generated by estimating the inter-vehicle distance, calculating the degree of danger of other vehicles based on the estimated inter-vehicle distance, and extracting a portion of the first moving image based on the degree of danger. and a transmitting unit that transmits the first moving image to the transmitting management device.

A communication terminal according to the present invention is a communication terminal communicably connected to an imaging device and a management device of a system having an imaging device provided in a vehicle and a management device, wherein the imaging device detects a driver of the vehicle. A receiving unit that receives a first moving image captured and generated from an imaging device, information about the posture of the driver of the vehicle, information about the object held by the driver, and whether the driver is gazing at the object Information about the attitude of the driver of the vehicle, generated based on the first moving image when a trained model that has learned in advance the relationship with information indicating whether or not and by inputting information about the object grasped by the driver into the trained model, it is possible to estimate whether the vehicle driver is gazing at the object or not, and based on the result of the estimation, determine the danger of the vehicle driver. a transmitter for transmitting the first moving image to a management device that calculates the degree of risk and extracts a portion of the first moving image based on the degree of risk to generate and transmit a second moving image; characterized by having

A management system, a management method, a management device, a program, and a communication terminal according to the present invention enable a vehicle manager to appropriately check dangerous driving.

1 is a schematic diagram for explaining an overview of a management system 1; FIG. 1 is a diagram showing an example of a schematic configuration of a management system 1; FIG. 1 is a diagram showing an example of a schematic configuration of an imaging device 2; FIG. 3 is a diagram showing an example of a schematic configuration of a communication terminal 3; FIG. 3 is a diagram showing an example of a schematic configuration of a management device 4; FIG. FIG. 4 is a diagram showing an example of the data structure of a first risk table T1; FIG. FIG. 11 is a diagram showing an example of the data structure of a second risk table T2; FIG. FIG. 10 is a sequence diagram showing an example of the flow of an out-of-vehicle moving image transmission process; FIG. 10 is a flow chart showing an example of the flow of an outside-vehicle moving image extraction process; FIG. 10 is a sequence diagram showing an example of the flow of in-vehicle moving image transmission processing; It is a flowchart which shows an example of the flow of an in-vehicle moving image extraction process.

Various embodiments of the present invention will now be described with reference to the drawings. It should be noted that the technical scope of the present invention is not limited to these embodiments, but extends to the invention described in the claims and equivalents thereof.

FIG. 1 is a schematic diagram for explaining an overview of a management system 1 according to the present invention. The management system 1 has an imaging device 2 , a management device 4 and a display terminal 5 . The imaging device 2, the management device 4, and the display terminal 5 are connected so as to be able to communicate with each other. The management system 1 is used, for example, by a carrier or the like that operates a large number of vehicles and transports passengers and freight to manage the vehicles.

The imaging device 2 is a device having an imaging function and a communication function, such as a drive recorder. The imaging device 2 is provided above the windshield of the vehicle 11 or the like so as to be able to image the front of the vehicle 11 . The imaging device 2 images another vehicle running in front of the vehicle 11 and generates a first moving image 12 . The imaging device 2 transmits the first moving image 12 to the management device 4 .

The management device 4 is an information processing device such as a server or a PC (Personal Computer). The management device 4 acquires the first moving image 12 from the imaging device 2 . The management device 4 estimates the inter-vehicle distance between the vehicle 11 and the other vehicle based on the first moving image 12, and calculates the risk of the other vehicle based on the estimated inter-vehicle distance. Management device 4 generates second moving image 13 by extracting a portion of first moving image 12 based on the degree of risk of other vehicles. The risk of other vehicles is a value indicating the probability that other vehicles are driving dangerously. Dangerous driving is, for example, tailgate driving that sharply narrows the inter-vehicle distance. The second moving image 13 is generated by extracting a portion of the first moving image 12 so as to include the time during which other vehicles are driving dangerously. Management device 4 transmits second moving image 13 to display terminal 5 .

The display terminal 5 is an information processing terminal having a communication function and a display function, such as a PC, a mobile phone, a tablet terminal, or a smart phone. The display terminal 5 is used by an administrator of the vehicle 11 . The display terminal 5 displays the received second moving image 13 . This allows the administrator to appropriately confirm dangerous driving. Note that the manager is not particularly limited as long as it can manage the vehicle 11 , such as an individual who owns the vehicle 11 or a corporation (person in charge of the corporation) who operates a plurality of vehicles 11 .

FIG. 2 is a diagram showing an example of a schematic configuration of the management system 1. As shown in FIG. The management system 1 has an imaging device 2 , a communication terminal 3 , a management device 4 and a display terminal 5 . The imaging device 2 is provided in the vehicle 11 and is connected to a portable communication terminal 3 such as a smart phone or a tablet terminal carried by the driver or passenger of the vehicle 11 so as to be able to communicate with each other. The communication terminal 3, the management device 4, and the display terminal 5 are connected so as to be able to communicate with each other via a network 6 including a mobile communication network such as a 4th generation mobile communication system (4G) or a 5th generation mobile communication system (5G). be done.

FIG. 3 is a diagram showing an example of a schematic configuration of the imaging device 2. As shown in FIG. The imaging device 2 has a front imaging section 21 , a rear imaging section 22 , an in-vehicle imaging section 23 , a vehicle speed acquisition section 24 , a first storage section 25 , a first communication section 26 and a first processing section 27 .

The front imaging unit 21 is configured to capture an image of the front of the vehicle, the rear imaging unit 22 is configured to capture an image of the rear of the vehicle, and the in-vehicle imaging unit 23 is configured to capture an image of the driver's seat of the vehicle. is the configuration. The front imaging section 21, the rear imaging section 22, and the in-vehicle imaging section 23 are provided with cameras, for example. The front imaging section 21 and the in-vehicle imaging section 23 are provided near the windshield of the vehicle so as to be able to image the front of the vehicle and the driver's seat. The rear imaging unit 22 is provided near the rear glass of the vehicle so as to be able to image the rear of the vehicle.

The front imaging unit 21, the rear imaging unit 22, and the in-vehicle imaging unit 23 generate an image signal corresponding to the image of the subject formed by converging light rays from the subject using an optical lens or the like, and generate an image in a predetermined format. It is converted into data and supplied to the first processing unit 27 .

The vehicle speed acquisition unit 24 is configured to acquire the running speed of the vehicle, and includes, for example, an interface connectable to a signal line provided inside the vehicle for transmitting vehicle speed pulses. The vehicle speed acquisition unit 24 receives a vehicle speed pulse from the signal line, calculates the running speed of the vehicle according to the pulse interval of the vehicle speed pulse, and supplies data indicating the calculated running speed to the first processing unit 27 .

The first storage unit 25 is configured to store data and programs, and includes, for example, a semiconductor memory. The first storage unit 25 stores an operating system program, a driver program, an application program, data, and the like used for processing by the first processing unit 27 . The program is installed using a known setup program or the like from a computer-readable and non-temporary portable storage medium such as CD (Compact Disc)-ROM (Read Only Memory) or DVD (Digital Versatile Disc)-ROM. .

The first communication unit 26 is configured to allow the imaging device 2 to communicate with the communication terminal 3, and includes a communication interface circuit. The communication interface circuit provided in the first communication unit 26 is a communication interface circuit for a wireless LAN (Local Area Network) such as Wi-Fi. The first communication unit 26 transmits data supplied from the first processing unit 27 to the communication terminal 3 and supplies data received from the communication terminal 3 to the first processing unit 27 .

The first processing unit 27 is configured to collectively control the operation of the imaging device 2, and includes one or more processors and their peripheral circuits. The first processing unit 27 includes, for example, a CPU (Central Processing Unit), LSI (Large Scale Integration), or ASIC (Application Specific Integrated Circuit). The first processing unit 27 may include a GPU (Graphics Processing Unit), DSP (Digital Signal Processor), FPGA (Field Programmable Gate Array), and the like. Based on the programs stored in the first storage unit 25, the first processing unit 27 controls the operation of each component of the imaging device 2 so that various processes of the imaging device 2 are appropriately executed, Execute the process.

The first processing unit 27 has a moving image generation unit 271 , a vehicle speed generation unit 272 and a moving image transmission unit 273 . Each of these units is a functional module implemented by a program executed by the first processing unit 27 . Each of these units may be implemented in the imaging device 2 as firmware. Note that the moving image generation unit 271 and the moving image transmission unit 273 are examples of a first generation unit and a first transmission unit, respectively.

FIG. 4 is a diagram showing an example of a schematic configuration of the communication terminal 3. As shown in FIG. The communication terminal 3 has a second storage section 31 , a second communication section 32 , a third communication section 33 and a second processing section 34 .

The second storage unit 31 is configured to store data and programs, and includes a semiconductor memory, for example. The second storage unit 31 stores an operating system program, a driver program, an application program, data, etc. used for processing by the second processing unit 34 . The program is installed from a computer-readable and non-temporary portable storage medium using a known setup program or the like.

The second communication unit 32 is configured to allow the communication terminal 3 to communicate with the imaging device 2, and includes a communication interface circuit. The communication interface circuit included in the second communication unit 32 is a wireless LAN communication interface circuit. The second communication unit 32 transmits data supplied from the second processing unit 34 to the imaging device 2 and supplies data received from the imaging device 2 to the second processing unit 34 .

The third communication unit 33 is configured to enable the communication terminal 3 to communicate with other devices via the network 6, and includes a communication interface circuit. The communication interface circuit provided in the third communication unit 33 is a communication interface circuit of a mobile communication system such as LTE (Long Term Evolution) or NR (New Radio). The third communication unit 33 transmits data supplied from the second processing unit 34 to other devices, and supplies data received from other devices to the second processing unit 34 .

The second processing unit 34 is configured to centrally control the operation of the communication terminal 3, and includes one or more processors and their peripheral circuits. The second processing unit 34 includes, for example, a CPU, LSI, ASIC, GPU, DSP, FPGA, and the like. Based on the programs stored in the second storage unit 31, the second processing unit 34 controls the operation of each component of the communication terminal 3 so that various processes of the communication terminal 3 are appropriately executed, and various Execute the process.

The second processing section 34 has a receiving section 341 and a transmitting section 342 . Each of these units is a functional module implemented by a program executed by the second processing unit 34 . Each of these units may be implemented in the communication terminal 3 as firmware.

FIG. 5 is a diagram showing an example of a schematic configuration of the management device 4. As shown in FIG. The management device 4 has a third storage section 41 , a fourth communication section 42 and a third processing section 43 .

The third storage unit 41 is configured to store data and programs, and includes a semiconductor memory, for example. The third storage unit 41 stores an operating system program, a driver program, an application program, data, etc. used for processing by the third processing unit 43 . The program is installed from a computer-readable and non-temporary portable storage medium using a known setup program or the like.

The fourth communication unit 42 is configured to enable the management device 4 to communicate with other devices via the network 6, and includes a communication interface circuit. A communication interface circuit provided in the fourth communication unit 42 is a communication interface circuit of a mobile communication system such as LTE or NR. The communication interface circuit provided in the fourth communication unit 42 may be a communication interface circuit for wireless LAN or wired LAN. The fourth communication unit 42 transmits data supplied from the third processing unit 43 to other devices, and supplies data received from other devices to the third processing unit 43 .

The third processing unit 43 is configured to collectively control the operation of the management device 4, and includes one or more processors and their peripheral circuits. The third processing unit 43 includes, for example, CPU, LSI, ASIC, GPU, DSP, FPGA, and the like. Based on the programs stored in the third storage unit 41, the third processing unit 43 controls the operation of each component of the management device 4 so that various processes of the management device 4 are appropriately executed, Execute the process.

The third processing unit 43 has a moving image acquiring unit 431 , an information acquiring unit 432 , an estimating unit 433 , a calculating unit 434 , a moving image extracting unit 435 and an extracted moving image transmitting unit 436 . Each of these units is a functional module implemented by a program executed by the third processing unit 43 . Each of these units may be implemented in the management device 4 as firmware. Note that the moving image extraction unit 435 and the extracted moving image transmission unit 436 are examples of a second generation unit and a second transmission unit, respectively.

FIG. 6 shows an example of the data structure of the first risk level table T1 for managing information related to vehicle exterior moving images (front moving images or rear moving images) stored in the third storage unit 41 of the management device 4. FIG. 4 is a diagram showing; The first risk level table T1 stores frame identifiers, running speeds, license plates, vehicle-to-vehicle distances, and risk levels in association with each other.

A frame identifier is an example of information for specifying a frame that constitutes a moving image captured by the imaging device 2 . Instead of the frame identifier, the playback time or the like at which the corresponding frame in the moving image is displayed may be stored. The travel speed is information indicating the travel speed of the vehicle when the corresponding frame was captured.

A license plate is information indicating the size of the license plate in the corresponding frame. In the example shown in FIG. 6, the size of the license plate is indicated by the number of pixels in the vertical and horizontal directions of the bounding box containing the license plate. The inter-vehicle distance is the inter-vehicle distance to another vehicle calculated based on the size of the license plate. The degree of risk is information indicating the degree of risk of driving calculated for the corresponding frame, and is information indicating the probability that another vehicle is driving at the tail end. If the license plate is not detected in the corresponding frame, the license plate and inter-vehicle distance are not stored.

Each piece of data in the first risk level table T1 is set by the management device 4 in the vehicle-exterior moving image transmission process, which will be described later.

FIG. 7 is a diagram showing an example of the data structure of a second risk level table T2 that manages information about in-vehicle moving images and is stored in the third storage unit 41 of the management device 4. As shown in FIG. The second risk level table T2 stores frame identifiers, running speeds, postures, objects, and risk levels in association with each other. The frame identifier and running speed are the same data as the frame identifier and running speed of the first risk level table T1.

Posture is information about the posture of the driver in the corresponding frame, and includes information about joint positions and information about face orientation. The joint position information is information indicating a plurality of joint positions of the driver in the corresponding frame. In the example shown in FIG. 7, the joint positions are indicated by, for example, the X and Y coordinates of the pixels corresponding to the joint positions. The information about the orientation of the face is information indicating the inclination of the orientation of the driver's face with respect to a predetermined direction (for example, the direction from the center position of the driver's face toward the imaging device 2). In the example shown in FIG. 7, the face orientation is indicated by the pitch angle, roll angle and yaw angle of the driver's face orientation with respect to a predetermined direction.

The target object is information indicating the position in the corresponding frame of the target object such as a smartphone, mobile phone, etc. to be gazed at while driving, and is indicated by, for example, the center position of the bounding box of the target object. Note that if no object is detected in the corresponding frame, the object is not stored. The degree of risk is information indicating the degree of risk of driving in the corresponding frame, and is information indicating the probability that the driver is driving while distracted.

Each piece of data in the second risk level table T2 is set by the management device 4 in an in-vehicle moving image transmission process, which will be described later.

The management system 1 executes an in-vehicle moving image transmission process and an in-vehicle moving image transmission process. These processes are executed in parallel on a regular or irregular basis. Further, these processes are realized by executing a program by the processing unit of each device of the management system 1 and cooperating with the configuration of each device.

FIG. 8 is a sequence diagram showing an example of the flow of vehicle-exterior moving image transmission processing executed by the management system 1. As shown in FIG. Below, the flow of the vehicle-exterior moving image transmission process when the forward moving image captured by the forward imaging unit 21 is transmitted as the vehicle-exterior moving image will be described with reference to FIG. 8 . The same applies to the case where the rearward moving image captured by the rearward imaging unit 22 is transmitted as the vehicle-outside moving image.

First, the moving image generation unit 271 of the imaging device 2 images a forward vehicle traveling in front of the vehicle to generate a forward moving image (S101). The moving image generation unit 271 controls the front imaging unit 21 to image the front of the vehicle, and generates a forward moving image of a predetermined length (for example, 5 minutes). Note that the forward vehicle is an example of another vehicle different from the vehicle, and the forward moving image is an example of the first moving image.

Subsequently, the vehicle speed generator 272 generates vehicle speed data indicating the running speed of the vehicle during the time when the image in front of the vehicle is captured (S102). The vehicle speed generation unit 272 controls the vehicle speed acquisition unit 24 to acquire the running speed of the vehicle during the time when the front of the vehicle is imaged. The vehicle speed generation unit 272 generates vehicle speed data, which is time-series data, by associating the travel speed with time.

Subsequently, the moving image transmission unit 273 transmits the front moving image and the vehicle speed data to the communication terminal 3 via the first communication unit 26 (S103). The reception unit 341 of the communication terminal 3 receives the front moving image and vehicle speed data via the second communication unit 32 .

Subsequently, the transmission unit 342 transmits the front moving image and the vehicle speed data to the management device 4 via the third communication unit 33 (S104). The moving image acquisition unit 431 of the management device 4 acquires the front moving image and the vehicle speed data by receiving them via the fourth communication unit 42 (S105). The moving image acquisition unit 431 associates the frame identifier of each frame forming the front moving image with the traveling speed of the vehicle at the time when each frame was captured, and stores them in the first risk level table T1.

Subsequently, the management device 4 executes forward moving image extraction processing (S106). The forward moving image extraction process is a process of generating an extracted forward moving image by extracting a portion of the forward moving image. Note that the extracted front moving image is an example of the second moving image.

Subsequently, the extracted moving image transmission unit 436 of the management device 4 transmits the extracted forward moving image to the display terminal 5 via the fourth communication unit 42 (S107). The display terminal 5 receives and displays the extracted front moving image (S108), and ends the forward moving image transmission process.

FIG. 9 is a flow diagram showing an example of the flow of forward moving image extraction processing.

First, the information acquisition unit 432 of the management device 4 acquires information related to the license plate of the forward vehicle, which is generated based on the forward moving image (S201). The information acquisition unit 432 acquires information about the license plate using a learned model for license plate detection stored in the third storage unit 41 in advance. The information on the license plate is information indicating the size of the area corresponding to the license plate in each frame forming the front moving image.

A trained model for license plate detection is a trained model that, when an image is input, has been trained to output the size of the bounding box containing the license plate in that image. Object detection models such as SSD (Single Shot Detector), YOLO (You Only Look once), and R-CNN (Region with Convolutional Neural Network) are used as learning models. The learning model is learned by updating the parameters so that the error between the position and shape of the bounding box output by the learning model and the position and shape of the bounding box set in advance as correct data is reduced.

The information acquisition unit 432 inputs each frame constituting the forward moving image to the trained model, and acquires the position and shape of the bounding box output for each frame. The information acquisition unit 432 generates and acquires information about the license plate based on the acquired position and shape of the bounding box. The information about the license plate is, for example, the number of pixels in the vertical and horizontal directions of the bounding box containing the license plate. The information about the license plate may be other information indicating the size of the license plate, such as the positional relationship between the vertices of the bounding box and the area of the bounding box. The information acquisition unit 432 stores the license plate information acquired for each frame in the first risk level table T1.

Subsequently, the estimating unit 433 estimates the inter-vehicle distance between the vehicle and the preceding vehicle by inputting information about the license plate of the preceding vehicle into the learned model for estimating the inter-vehicle distance (S202). The learned model for estimating the inter-vehicle distance is a learned model trained to output the inter-vehicle distance when information on the license plate is input. A regression model such as a gradient boosting decision tree is used as a learning model.

The parameters of the learning model are updated so that the error between the inter-vehicle distance output by the learning model when an input image for learning is input and the inter-vehicle distance for learning set in advance in association with the input image for learning is reduced. It is learned by The learning inter-vehicle distance is set by measuring the actual inter-vehicle distance using a distance measuring sensor or the like when the learning input image is captured. The input image for learning and the inter-vehicle distance for learning may be generated using an automatic driving simulator.

The estimating unit 433 stores the inter-vehicle distance output from the learned model in the first risk level table T1 in association with the information on the license plate.

Subsequently, the calculation unit 434 calculates the degree of danger of the preceding vehicle based on the estimated inter-vehicle distance (S203). The degree of risk is calculated as time-series data associated with the frame of the forward moving image.

The calculation unit 434 calculates the degree of danger of the preceding vehicle based on preset rules. For example, the calculator 434 identifies one of the frames that form the front moving image. The calculation unit 434 extracts a predetermined number of frames (for example, 20 frames) immediately preceding the identified frame. When the inter-vehicle distance estimated for each extracted frame is equal to or less than a predetermined value (eg, 3 m) and the running speed corresponding to each frame is equal to or greater than a predetermined value (eg, 10 km/h) In other cases, the identified frame is calculated to be of high risk, and in other cases, the identified frame is calculated to be of low risk.

Note that the preset rule is not limited to the above example. For example, the calculation unit 434 may calculate that the identified frame has a high degree of risk when the number of extracted frames in which the estimated inter-vehicle distance is equal to or less than a predetermined value is equal to or greater than a predetermined ratio. .

The calculator 434 stores the risk calculated for each frame in the first risk table T1 in association with the frame identifier of each frame.

Subsequently, the moving image extraction unit 435 generates an extracted moving image by extracting a portion of the forward moving image based on the degree of risk (S204), and ends the forward moving image extraction process. For example, the moving image extraction unit 435 extracts a portion of the forward moving image so as to include a range of a predetermined time from when the degree of risk satisfies a predetermined condition.

In the example shown in FIG. 6, the risk level is calculated as "low" up to the frame with the frame identifier "101", and the risk level is calculated as "high" from the frame with the frame identifier "102". Therefore, assuming that the predetermined condition is that the degree of risk is "high", the moving image extraction unit 435 extracts a predetermined number of frames after the frame with the frame identifier "102", Generating a composed extracted video image.

FIG. 10 is a diagram illustrating an example of the flow of in-vehicle moving image transmission processing.

First, the moving image generator 271 of the imaging device 2 images the driver of the vehicle and generates an in-vehicle moving image (S301). The moving image generation unit 271 controls the in-vehicle imaging unit 23 to image the driver of the vehicle and generates an in-vehicle moving image of a predetermined length. Note that the in-vehicle moving image is an example of the first moving image.

Subsequently, the vehicle speed generator 272 generates vehicle speed data indicating the running speed of the vehicle during the time when the driver is imaged (S302). The vehicle speed generation unit 272 controls the vehicle speed acquisition unit 24 to acquire the running speed of the vehicle during the time when the front of the vehicle is imaged. The vehicle generation unit 272 generates vehicle speed data, which is time-series data, by associating the traveling speed with time.

Subsequently, the moving image transmission unit 273 transmits the in-vehicle moving image and the vehicle speed data to the communication terminal 3 via the first communication unit 26 (S303). The receiving unit 341 of the communication terminal 3 receives the in-vehicle moving image and the vehicle speed data via the second communication unit 32 .

Subsequently, the transmission unit 342 transmits the in-vehicle moving image and the vehicle speed data to the management device 4 via the third communication unit 33 (S304). The moving image acquisition unit 431 of the management device 4 acquires the in-vehicle moving image and the vehicle speed data by receiving them via the fourth communication unit 42 (S305). The moving image acquiring unit 431 associates the frame identifier of each frame constituting the in-vehicle moving image with the running speed of the vehicle at the time when each frame was captured, and stores them in the second risk level table T2.

Subsequently, the management device 4 executes an in-vehicle moving image extraction process (S306). The in-vehicle moving image extraction process is a process of generating an extracted in-vehicle moving image by extracting a part of the in-vehicle moving image. Note that the extracted in-vehicle moving image is an example of the second moving image.

Subsequently, the extracted moving image transmission unit 436 of the management device 4 transmits the extracted in-vehicle moving image to the display terminal 5 via the fourth communication unit 42 (S307). The display terminal 5 receives and displays the extracted in-vehicle moving image (S308), and ends the in-vehicle moving image transmission process.

FIG. 10 is a flow diagram showing an example of the flow of in-vehicle moving image extraction processing.

First, the information acquisition unit 432 acquires information about the posture of the driver and information about the object held by the driver, which are generated based on the in-vehicle moving image (S401). The information acquisition unit 432 acquires information about the orientation and information about the object using the trained model for orientation estimation and the trained model for object detection that are stored in advance in the third storage unit 41 . The posture information includes information on the joint positions and face orientation of the driver in each frame that constitutes the in-vehicle moving image. The information about the object is information about an object that is a so-called driving target in each frame that constitutes the in-vehicle moving image. For example, the information about the object may be information about the position and type of a radio communication device or an image display device such as a mobile phone, a smart phone, or a car navigation system. In addition, the information about the target object is not limited to the above examples as long as it is information necessary for detecting dangerous driving such as driving while distracted. In addition, the object is not particularly limited as long as it can be judged as dangerous driving such as distracted driving by being operated during driving.

The trained models for posture estimation include a trained model for joint position estimation and a trained model for face direction estimation. A learned model for estimating joint positions is a learning model that has been trained to output joint positions of a person in an image when the image is input. As a learning model, for example, PoseNet or PersonLab is used. The learning model is learned by, for example, updating the parameters so that the error, which is the sum of the squares of the distances between the joint positions output by the learning model and the joint positions set in advance as correct data, is reduced.

A trained model for estimating face orientation is a learning model trained to output the orientation of a person's face (e.g., roll angle, pitch angle, and yaw angle) in an image when an image is input. be. For example, FSA-Net or the like is used as the learning model. The learning model is learned by updating the parameters so that the error between the face direction output by the learning model and the face direction set in advance as correct data is reduced.

A trained model for object detection is trained to output the position and shape of a bounding box containing the area corresponding to the object in the image and the type of the object when an image is input. is a model. An object detection model such as SSD, YOLO or R-CNN is used as the learning model. The learning model has parameters that reduce the error between the position, shape, and type of object of the bounding box output by the learning model and the position, shape, and type of object of the bounding box set in advance as correct data. Learned by being updated.

The information acquisition unit 432 inputs each frame constituting the forward moving image to a trained model for joint position estimation, a trained model for face orientation estimation, and a trained model for object detection. The information acquisition unit 432 generates information about the joint positions of the driver based on the output of the learned model for joint position estimation. In addition, the information acquisition unit 432 generates information regarding the orientation of the driver's face based on the output of the trained model for estimating the orientation of the face. The information acquisition unit 432 associates the information on the joint positions of the driver with the information on the orientation of the face and acquires them as information on the posture of the driver. In addition, the information acquisition unit 432 generates and acquires information about the object gripped by the driver based on the output of the learned model for object detection. The information acquisition unit 432 stores the acquired information about the posture and the information about the object in the second risk level table T2.

Next, the estimating unit 433 inputs the information about the posture of the driver and the information about the object held by the driver to the trained model for estimating the state of gaze, thereby allowing the driver of the vehicle to gaze at the object. It is estimated whether or not there is (S402). The trained model for estimating the state of gaze outputs a degree of certainty that indicates the probability that the driver is gazing at an object when information on the driver's posture and information on the object held by the driver are input. It is a learning model that has been trained to As a learning model, a classification model such as a gradient boosting decision tree is used.

The learning model is a combination of the confidence level output by the learning model when the information on the posture and object for learning is input, and the confidence level for learning set in advance in association with the information on the posture and target for learning. Learning is performed by updating the parameters so that the cross-entropy error becomes small. Information about the posture and the object for learning is generated based on, for example, an image of the driver captured for learning. In addition, the certainty factor for learning is set so that in the image of the driver captured for learning, a different element is 1 depending on whether the driver is gazing at the target object, and the other elements are 0. is a one-hot vector with

Subsequently, the calculation unit 434 calculates the risk of the driver based on the estimation result (S403). The degree of risk is calculated as time-series data associated with frames of the in-vehicle moving image.

The calculation unit 434 calculates the degree of risk of the driver based on preset rules. For example, the calculation unit 434 identifies one of the frames forming the in-vehicle moving image. The calculation unit 434 extracts the identified frame and a predetermined number of frames immediately preceding the identified frame (for example, the number of frames corresponding to two seconds). The calculation unit 434 determines that the estimation result for each extracted frame indicates that the driver is gazing at the target object and the running speed corresponding to each frame is not 0 km/h (that is, the vehicle is stopped). In other cases, the identified frame is calculated to be of high risk, and in other cases, the identified frame is calculated to be of low risk.

Subsequently, the moving image extraction unit 435 generates an extracted moving image by extracting a portion of the in-vehicle moving image based on the degree of risk (S404), and ends the forward moving image extraction process. For example, the moving image extraction unit 435 extracts a portion of the forward moving image so as to include a range of a predetermined time from when the degree of risk satisfies a predetermined condition.

As described above, the management device 4 estimates the inter-vehicle distance between the vehicle and the other vehicle by inputting the license plate information of the other vehicle generated based on the front moving image into the learned model. . In addition, the management device 4 generates an extracted front moving image by extracting a part of the front moving image based on the degree of danger of other vehicles calculated based on the estimated inter-vehicle distance. As a result, the management device 4 enables the vehicle manager to appropriately confirm dangerous driving.

That is, since the management device 4 calculates the degree of risk of other vehicles based on the inter-vehicle distance, it is possible to appropriately detect tailgate driving that does not lead to an accident, such as driving with the inter-vehicle distance sharply narrowed. Then, the management device 4 extracts a portion of the front moving image based on the degree of risk, thereby enabling the manager to efficiently check the mode of tailgate driving. In addition, since the management device 4 calculates the inter-vehicle distance using the imaging device 2 that does not include a distance measuring sensor or the like, it is possible for a carrier or the like owning a large number of vehicles to introduce the management system 1 at a relatively low cost. and

In addition, the management device 4 inputs information about the posture of the driver of the vehicle generated based on the in-vehicle moving image and information about the object held by the driver into the learned model so that the driver can grasp the object. Predict whether or not you are watching. In addition, the management device 4 generates an extracted in-vehicle moving image by extracting a part of the in-vehicle moving image based on the degree of risk of the driver calculated based on the result of the estimation. As a result, the management device 4 enables the vehicle manager to appropriately confirm dangerous driving.

That is, the management device 4 estimates whether or not the driver is gazing at the target using both the information regarding the driver's posture and the information regarding the target, thereby making it difficult to track the line of sight of the driver. Even in such cases, it is possible to appropriately detect driving. Moreover, the management device 4 extracts a portion of the in-vehicle moving image using the degree of risk based on the estimation result, thereby enabling the manager to efficiently check the mode of driving.

In addition, the management device 4 calculates the degree of risk based on preset rules. This allows the management device 4 to more appropriately detect dangerous driving. That is, since the standard for determining whether or not to drive while driving is that the driver continues to look at the screen for about two seconds or more, the management device 4 determines the degree of danger based on such rules, thereby reducing dangerous driving. can be detected more appropriately. In addition, this allows the management device 4 to easily respond to changes in the criteria for dangerous driving.

Moreover, the management device 4 extracts a part of the front moving image, the rear moving image, or the in-vehicle moving image so that the range for a predetermined time period from the time when the degree of risk satisfies the predetermined condition is included. As a result, the management device 4 enables the manager to appropriately check the situation of dangerous driving and the situation before and after that.

Note that, in S204, the management device 4 may extract a portion of the forward moving image so as to include a time period during which the degree of risk satisfies a predetermined condition. In the example shown in FIG. 6, the degree of risk is calculated as "high" from the frame with the frame identifier "102" to the frame with the frame identifier "151". Therefore, assuming that the predetermined condition is that the degree of risk is "high", the moving image extraction unit 435 extracts frames from the frame with the frame identifier "102" to the frame with the frame identifier "151", An extracted moving image composed of the extracted frames is generated. Similarly, in S404, the management device 4 may extract a portion of the in-vehicle moving image so as to include the time period during which the degree of risk satisfies a predetermined condition.

Also, in this case, the management device 4 may add a predetermined time (for example, five seconds each before and after) to the time when the degree of risk satisfies the predetermined condition. part may be extracted. As a result, the management device 4 enables the manager to more appropriately confirm the situation before and after dangerous driving.

In the above description, the imaging device 2 generates vehicle speed data in S302 of the in-vehicle video transmission process, but the imaging device 2 does not have to generate vehicle speed data. In this case, the management device 4 calculates the degree of danger based on the driver's posture and the information on the object, regardless of the traveling speed of the vehicle. It may not be desirable for the driver to gaze at the image display device even while the vehicle is stopped, such as when waiting for a signal. In such a case, the management device 4 can detect that the driver has gazed at the image display device regardless of the traveling speed of the vehicle.

In the above description, the management device 4 calculates the degree of risk for each frame of the acquired moving image, but the invention is not limited to this example. For example, the management device 4 may extract one frame at predetermined time intervals (for example, one second) from among the frames that make up the moving image, and calculate the degree of risk for the extracted frame. . Thereby, the processing load on the management device 4 can be reduced.

In the above description, in S203, the management device 4 calculates the degree of risk based on a preset rule, but the present invention is not limited to this example. The management device 4 may calculate the degree of risk using a learned model for calculating the degree of risk stored in the third storage unit 41 . A trained model is a learned model, such as a gradient boosted decision tree, that has been trained to calculate the degree of risk when an inter-vehicle distance estimated for a predetermined number of consecutive frames is input.

The learning model is designed to reduce the cross-entropy error between the risk level output by the learning model when the inter-vehicle distance for learning is input and the risk level for learning that is set in advance in association with the inter-vehicle distance for learning. It is learned by updating the parameters. The inter-vehicle distance for learning is generated, for example, based on the moving image of the preceding vehicle captured for learning. Also, the learning risk level is set based on, for example, how dangerous a person who viewed the moving image of the vehicle ahead captured for learning felt to be in danger. Thereby, the management device 4 can more appropriately detect tailgate driving. In other words, it can be considered that whether or not driving another vehicle constitutes aggressive driving is also based on the subjective factors of the driver. Cage driving can be detected.

Some or all of the functions of the management device 4 described above may be implemented by the imaging device 2 or the communication terminal 3 . For example, in the vehicle-exterior moving image transmission process, the communication terminal 3 may calculate the degree of risk, generate an extracted front moving image, and transmit the extracted forward moving image to the communication terminal 3 . Thereby, the amount of communication via the network 6 can be reduced.

In the above description, the communication terminal 3 is assumed to be a smart phone or the like carried by the driver of the vehicle, but the communication terminal 3 is not limited to such an example. For example, the communication terminal 3 may be a communication module provided in the vehicle. Also, the first communication unit 25 of the imaging device 2 may be configured to be directly communicable with the management device 4 .

It should be understood by those skilled in the art that various changes, substitutions and modifications can be made thereto without departing from the spirit and scope of the present invention. For example, the processing of each unit described above may be performed in a different order as appropriate within the scope of the present invention. Also, the above-described embodiments and modifications may be implemented in appropriate combinations within the scope of the present invention.

1 Management System 2 Imaging Device 271 Moving Image Generation Part 272 Vehicle Speed Generation Part 273 Moving Image Transmission Part 3 Communication Terminal 341 Receiving Part 342 Transmission Part 4 Management Device 431 Moving Image Acquisition Part 432 Information Acquisition Part 433 Estimation Part 434 Calculation Part 435 Moving Image Extraction unit 436 Extraction moving image transmission unit

Claims (10)

A management system having an imaging device provided in a vehicle and a management device,
The imaging device is
a first generation unit that captures another vehicle different from the vehicle to generate an exterior moving image , and captures a driver of the vehicle to generate an in-vehicle moving image ;
a first transmission unit that transmits the vehicle exterior video image and the vehicle interior video image to the management device;
The management device
A first trained model that has learned in advance the relationship between information about the license plate and the inter -vehicle distance to the vehicle with the license plate, information about the joint positions of the driver of the vehicle, and information about the face orientation of the driver. a storage unit that stores a second learned model that has previously learned a relationship between information and information about an object gripped by the driver and information indicating whether the driver is gazing at the object;
a video image acquisition unit that acquires the vehicle exterior video image and the vehicle interior video image ;
information about the license plate of the other vehicle generated based on the moving image outside the vehicle ; information about joint positions of the driver of the vehicle generated based on the moving image inside the vehicle; an information acquisition unit that acquires information on orientation and information on an object held by the driver ;
By inputting the obtained information about the license plate of the other vehicle into the first learned model, the inter-vehicle distance between the vehicle and the other vehicle is estimated , and the obtained joint position of the driver is estimated. By inputting information, information about the orientation of the driver's face, and information about the object held by the driver into the second trained model, it is possible to determine whether the driver of the vehicle is gazing at the object. an estimation unit that estimates whether or not
calculating the risk of the other vehicle based on the estimated inter-vehicle distance, and calculating the risk of the driver of the vehicle based on whether or not the driver of the vehicle is gazing at the object; a calculation unit;
An extracted vehicle exterior video image is generated by extracting a portion of the vehicle exterior video image based on the degree of risk of the other vehicle , and a portion of the vehicle interior video image is extracted based on the driver's degree of risk. a second generation unit that generates an extracted in-vehicle moving image by
a second transmission unit that transmits the extracted vehicle exterior moving image and the extracted vehicle interior moving image ;
A management system characterized by: The calculation unit calculates the risk level of the other vehicle and the risk level of the driver based on preset rules.
The management system of Claim 1 . The calculation unit calculates the degree of risk of the other vehicle as time-series data associated with the reproduction time of the video image outside the vehicle , and associates the degree of risk of the driver with the reproduction time of the video image inside the vehicle. calculated as time-series data ,
The second generation unit extracts a portion of the vehicle exterior video image so as to include a range of a predetermined time from a point in time when the degree of risk of the other vehicle satisfies a predetermined condition, extracting a portion of the in-vehicle moving image so as to include a range of a predetermined time from the time when satisfies a predetermined condition;
3. Management system according to claim 1 or 2 . The calculation unit calculates the degree of risk of the other vehicle as time-series data associated with the reproduction time of the video image outside the vehicle , and associates the degree of risk of the driver with the reproduction time of the video image inside the vehicle. calculated as time-series data ,
The second generation unit extracts a portion of the vehicle exterior video image so as to include a time period during which the degree of risk of the other vehicle satisfies a predetermined condition, and extracts a portion of the vehicle exterior video image so that the degree of risk of the driver satisfies a predetermined condition. extracting a portion of the in-vehicle video so as to include the time that satisfies
Management system according to any one of claims 1-3 . The first transmission unit further transmits vehicle speed data indicating the running speed of the vehicle to the management device,
The moving image acquisition unit further acquires the vehicle speed data,
The calculation unit
For each frame that constitutes the vehicle exterior video, the vehicle-to-vehicle distances estimated for a predetermined number of frames immediately preceding the frame are all equal to or less than a predetermined value, and the travel speed corresponding to each frame is equal to or greater than a predetermined value. and calculating the risk of the other vehicle as time-series data associated with the frame of the vehicle exterior video by calculating that the frame has a high risk,
For each frame constituting the in-vehicle moving image, the results of the estimation for a predetermined number of frames immediately before including the frame indicate that the driver is gazing at the object and correspond to each frame. When the vehicle speed data indicates that the vehicle is not stopped, the driver's risk is calculated as time-series data associated with the frame of the in-vehicle moving image by calculating that the risk of the frame is high. do,
Management system according to any one of claims 1-4. further comprising a communication terminal carried by the driver or passenger of the vehicle;
The first transmission unit transmits the vehicle exterior video and the vehicle interior video to the communication terminal,
The video image acquisition unit acquires the vehicle exterior video image and the vehicle interior video image from the communication terminal.
Management system according to any one of claims 1-5. A management method executed by a system having an imaging device provided in a vehicle and a management device,
The imaging device
imaging another vehicle different from the vehicle to generate an exterior moving image , and imaging a driver of the vehicle to generate an in-vehicle moving image ;
transmitting the vehicle exterior video image and the vehicle interior video image to the management device;
The management device
A first trained model that has learned in advance the relationship between information about the license plate and the inter -vehicle distance to the vehicle with the license plate, information about the joint positions of the driver of the vehicle, and information about the face orientation of the driver. storing a second trained model that has previously learned a relationship between information and information about an object held by the driver and information indicating whether the driver is gazing at the object;
Acquiring the vehicle exterior video image and the vehicle interior video image ,
information about the license plate of the other vehicle generated based on the moving image outside the vehicle ; information about joint positions of the driver of the vehicle generated based on the moving image inside the vehicle; Acquiring information about the orientation and information about the object grasped by the driver ;
By inputting the obtained information about the license plate of the other vehicle into the first learned model, the inter-vehicle distance between the vehicle and the other vehicle is estimated, and the obtained joint position of the driver is estimated. By inputting information, information about the orientation of the driver's face, and information about the object held by the driver into the second trained model, it is possible to determine whether the driver of the vehicle is gazing at the object. infer whether or not
calculating the degree of danger of the other vehicle based on the estimated inter-vehicle distance, and calculating the degree of danger of the driver of the vehicle based on whether or not the driver of the vehicle is gazing at the object; ,
An extracted vehicle exterior video image is generated by extracting a portion of the vehicle exterior video image based on the degree of risk of the other vehicle , and a portion of the vehicle interior video image is extracted based on the driver's degree of risk. Generates an extracted in-vehicle moving image by
transmitting the extracted vehicle exterior video image and the extracted vehicle interior video image ;
A management method characterized by comprising: A first trained model that has learned in advance the relationship between information about the license plate and the inter -vehicle distance to the vehicle with the license plate, information about the joint positions of the driver of the vehicle, and information about the face orientation of the driver. a storage unit that stores a second learned model that has previously learned a relationship between information and information about an object gripped by the driver and information indicating whether the driver is gazing at the object;
A moving image obtained by acquiring an outside moving image generated by an imaging device provided in a vehicle imaging another vehicle different from the vehicle and an in-vehicle moving image generated by imaging a driver of the vehicle by the imaging device. an acquisition unit;
information about the license plate of the other vehicle generated based on the moving image outside the vehicle ; information about joint positions of the driver of the vehicle generated based on the moving image inside the vehicle; an information acquisition unit that acquires information on orientation and information on an object held by the driver ;
By inputting the obtained information about the license plate of the other vehicle into the first learned model, the inter-vehicle distance between the vehicle and the other vehicle is estimated , and the obtained joint position of the driver is estimated. By inputting information, information about the orientation of the driver's face, and information about the object held by the driver into the second trained model, it is possible to determine whether the driver of the vehicle is gazing at the object. an estimation unit that estimates whether or not
calculating the risk of the other vehicle based on the estimated inter-vehicle distance, and calculating the risk of the driver of the vehicle based on whether or not the driver of the vehicle is gazing at the object; a calculation unit;
An extracted vehicle exterior video image is generated by extracting a portion of the vehicle exterior video image based on the degree of risk of the other vehicle , and a portion of the vehicle interior video image is extracted based on the driver's degree of risk. a second generation unit that generates an extracted in-vehicle moving image by
a second transmission unit that transmits the extracted vehicle exterior moving image and the extracted vehicle interior moving image ;
A management device characterized by having: A first trained model that has learned in advance the relationship between information about the license plate and the inter -vehicle distance to the vehicle with the license plate, information about the joint positions of the driver of the vehicle, and information about the face orientation of the driver. A storage unit that stores a second learned model that has previously learned a relationship between information and information about an object grasped by the driver and information indicating whether the driver is gazing at the object. a computer program,
Acquiring an external video image generated by an imaging device provided in a vehicle imaging another vehicle different from the vehicle and an in -vehicle video image generated by imaging a driver of the vehicle by the imaging device ,
information about the license plate of the other vehicle generated based on the moving image outside the vehicle ; information about joint positions of the driver of the vehicle generated based on the moving image inside the vehicle; Acquiring information about the orientation and information about the object grasped by the driver ;
By inputting the obtained information about the license plate of the other vehicle into the first learned model, the inter-vehicle distance between the vehicle and the other vehicle is estimated, and the obtained joint position of the driver is estimated. By inputting information, information about the orientation of the driver's face, and information about the object held by the driver into the second trained model, it is possible to determine whether the driver of the vehicle is gazing at the object. infer whether or not
calculating the degree of danger of the other vehicle based on the estimated inter-vehicle distance, and calculating the degree of danger of the driver of the vehicle based on whether or not the driver of the vehicle is gazing at the object; ,
An extracted vehicle exterior video image is generated by extracting a portion of the vehicle exterior video image based on the degree of risk of the other vehicle , and a portion of the vehicle interior video image is extracted based on the driver's degree of risk. Generate an extracted in-vehicle moving image by
transmitting the extracted vehicle exterior moving image and the extracted vehicle interior moving image ;
A program characterized by causing the computer to execute: A communication terminal communicably connected to the imaging device and the management device of a system having an imaging device provided in a vehicle and the management device,
A receiving unit that receives, from the imaging device, an exterior video image generated by imaging another vehicle different from the vehicle and an in-vehicle video image generated by the imaging device imaging a driver of the vehicle, from the imaging device. When,
A first trained model that has learned in advance the relationship between information about a license plate and the inter-vehicle distance to the vehicle with the license plate , information about joint positions of the driver of the vehicle, and information about the face orientation of the driver. storing a second trained model that has previously learned a relationship between information and information about an object grasped by the driver and information indicating whether the driver is gazing at the object ; by inputting, into the first trained model, information relating to the license plate of the other vehicle, which is generated based on the outside-vehicle moving image when the image is received, thereby determining the relationship between the vehicle and the other vehicle. information about the joint positions of the driver of the vehicle, information about the direction of the driver's face, and the driving, which are generated based on the in-vehicle moving image when an inter-vehicle distance is estimated and the in-vehicle moving image is received; By inputting information about an object held by a person into the first trained model, it is estimated whether or not the driver of the vehicle is gazing at the object, and based on the estimated inter-vehicle distance calculating the degree of risk of the other vehicle, calculating the degree of risk of the driver of the vehicle based on whether or not the driver of the vehicle is gazing at the object, and calculating the degree of risk of the other vehicle ; generating and transmitting an extracted vehicle-exterior video image by extracting a portion of the vehicle-exterior video image based on the degree of risk of the driver, and extracting a portion of the vehicle-interior video image based on the degree of risk of the driver; a transmission unit configured to transmit the vehicle exterior video and the vehicle interior video to a management device that generates and transmits video images ;
A communication terminal characterized by having:

Images