WO2024038525A1

WO2024038525A1 - Processing device, processing method, and program

Info

Publication number: WO2024038525A1
Application number: PCT/JP2022/031112
Authority: WO
Inventors: 浩司山本; 浩明前田
Original assignee: 日本電信電話株式会社
Priority date: 2022-08-17
Filing date: 2022-08-17
Publication date: 2024-02-22

Abstract

A processing device 1 is provided with: a storage device that stores profile data 12 in which a range of control costs 14 indexing the influence of each type of control for safe traveling of a vehicle 3 is associated with the type of control; a detection unit 23 that detects, by means of a prescribed method, degradation of video data 13 capturing the surroundings of the vehicle 3; a selection unit 24 that selects, from the profile data 12, a type of control including, in the range thereof, the control cost 14 determined in consideration of the detection accuracy of the method; and an instruction unit 25 that issues an instruction to perform the selected type of control.

Description

Processing equipment, processing method and program

The present invention relates to a processing device, a processing method, and a program.

Automation of agricultural machinery, construction machinery, passenger cars, etc. is progressing, and consideration is being given to the development and introduction of remote monitoring and control. In remote monitoring and control, a remote observer monitors the running of a remotely monitored vehicle, robot, etc. without an operator onboard the vehicle, using images of the surroundings of the vehicle. When the supervisor recognizes a dangerous situation, he or she takes control of the vehicle by remotely commanding the vehicle to stop.

In remote monitoring and control, monitoring images around the vehicle are transmitted to a remote location via a wireless network, and instructions such as emergency stop are sent from the remote location. With current wireless communication technology, it is difficult to ensure continuous transmission of optimal video data. Temporary video stoppage or video deterioration may cause situations in which remote monitoring and control cannot be performed properly.

Automated driving of vehicles under conditions where remote monitoring and control cannot be performed appropriately is dangerous. For example, it is necessary to quickly identify situations where images around the vehicle are not being transmitted properly and take measures such as stopping the vehicle while it is driving automatically. For example, there is a method in which a remote person visually monitors images around the vehicle, and when the image stops or deteriorates, the remote person instructs the conductor to stop.

On the other hand, there are multiple methods for understanding situations where video transmission is not being performed properly. Depending on the method, characteristics differ, such as the time it takes to detect stoppage or deterioration of surveillance video and detection accuracy.

For example, there is a method of emitting a signal to confirm the normality of a communication network, such as ICMP (INTERNET CONTROL MESSAGE PROTOCOL) (Non-Patent Document 1). There is a method of periodically emitting such a signal, detecting deterioration of the surveillance video from changes in the response, specifically changes in delay time or loss rate, and instructing the vehicle to stop.

There are methods such as MDI (Proposed Media Delivery Index) (Non-Patent Document 2) that measure network characteristics such as packet delay or jitter without affecting the network. There is a method of monitoring the loss rate or jitter of video packets using means such as MDI, and instructing the vehicle to stop when these values exceed thresholds.

The method in which a remote person visually monitors the video around the vehicle and detects stoppage or deterioration of the video requires the monitor to constantly pay attention to the normality of the video, resulting in high human costs. Furthermore, since it takes a few seconds to about 10 seconds for a human to recognize image deterioration and stop the vehicle, there is a problem in the immediacy of control.

Methods such as ICMP that emit signals to confirm the normality of communication networks have a problem with low deterioration detection accuracy because video packets are not directly monitored. Since the detection of problems is limited to sections where surveillance video information is packetized and transferred, there is a problem in that problems occurring in other sections cannot be detected. Furthermore, in a narrowband wireless network, it is not suitable to emit a signal for confirming normality.

The method of measuring network characteristics such as MDI is only one standard for video quality, and there is a problem that control may be overlooked. There is also a problem that is limited to detecting problems in sections that are packetized and transferred.

As described above, there is no technology that can appropriately detect video deterioration. Remote monitoring and control that refers to images that cannot determine the presence or absence of deterioration may cause the vehicle to run unsmoothly, such as repeatedly starting and stopping, or may increase the danger to surrounding vehicles or people.

The present invention has been made in view of the above circumstances, and an object of the present invention is to provide a technology that can support safe automatic driving of a vehicle by taking into account image deterioration.

A processing device according to one aspect of the present invention includes a storage device that stores profile data that associates, for each control for safe driving of a vehicle, a control cost range that indexes the influence of the control, and a predetermined method. a detection unit that detects deterioration of video data photographing the surroundings of the vehicle; and a selection unit that selects a control that includes a control cost determined in consideration of detection accuracy in the method from the profile data within the range. , an instruction section for instructing the selected control.

In a processing method according to one aspect of the present invention, a computer stores, in a storage device, profile data that associates, for each control for safe driving of a vehicle, a control cost range that indexes the influence of the control, and a computer detects deterioration of video data photographing the surroundings of the vehicle using a predetermined method;
The computer selects, from the profile data, a control that includes a control cost determined in consideration of detection accuracy in the method, and instructs the selected control.

One aspect of the present invention is a program that causes a computer to function as the processing device.

According to the present invention, it is possible to provide a technology that can support safe automatic driving of a vehicle while taking into account image deterioration.

FIG. 1 is a diagram illustrating the system configuration of a processing system and functional blocks of a processing device according to an embodiment of the present invention. FIG. 2 is a diagram illustrating an example of the data structure and data of constant data according to the embodiment. FIG. 3 is a diagram illustrating an example of the data structure and data of profile data. FIG. 4 is a flowchart illustrating an example of detection processing by the detection unit according to the embodiment. FIG. 5 is a flowchart illustrating an example of selection processing by the selection unit according to the embodiment. FIG. 6 is a diagram illustrating an example of a data structure and data of constant data according to a modification. FIG. 7 is a flowchart illustrating an example of detection processing by the detection unit according to the first modification. FIG. 8 is a flowchart illustrating an example of selection processing by the selection unit according to the first modification. FIG. 9 is a flowchart illustrating an example of detection processing by the detection unit according to the second modification. FIG. 10 is a flowchart illustrating an example of selection processing by the selection unit according to the second modification. FIG. 11 is a diagram illustrating the hardware configuration of a computer used in the processing device.

Hereinafter, embodiments of the present invention will be described with reference to the drawings. In the description of the drawings, the same parts are denoted by the same reference numerals and explanations will be omitted. Further, the order of processing described with reference to flowcharts and the like is an example, and is not limited to this.

(processing system)
A processing system 5 according to an embodiment of the present invention includes a processing device 1 and a vehicle 3. The processing device 1 and the vehicle 3 are connected to be able to communicate with each other via a wireless network. The vehicle 3 is a target of remote monitoring by the processing device 1 . The processing system 5 shown in FIG. 1 includes a plurality of vehicles 3, and the processing device 1 may remotely monitor the plurality of vehicles 3.

The vehicle 3 is equipped with a GPS (Global Positioning System) receiver (not shown), pinpoints the vehicle's position from signals received from GPS satellites, and drives according to a pre-given program. The vehicle 3 may issue a warning, decelerate, or otherwise avoid an obstacle in accordance with the control signal received from the processing device 1 .

As shown in FIG. 1, the vehicle 3 includes a camera 31, an encoding section 32, a communication section 33, a control section 34, and a drive section 35. The camera 31 inputs video data for capturing a video of the surroundings of the vehicle 3 to the encoding unit 32. The encoding unit 32 encodes video data input from the camera 31 and transmits it to the processing device 1 via the communication unit 33. The communication unit 33 transmits encoded video data to the processing device 1 , receives a control signal for the vehicle 3 from the processing device 1 , and inputs it to the control unit 34 . The control section 34 drives the drive section 35 according to the control signal 15 received from the processing device 1 . The drive unit 35 is a device for decelerating the vehicle 3 and warning the surrounding area, and is driven under control by the control unit 34.

(processing equipment)
As shown in FIG. 1, the processing device 1 includes constant data 11, profile data 12, video data 13, control cost 14, and control signal 15, a communication section 21, a decoding section 22, a detection section 23, and a selection section 24. , an instruction section 25, a learning section 27, and a display section 26. Each data is stored in a storage device such as memory 902 or storage 903. Each function is implemented in the CPU 901.

The constant data 11 is data that is stored in advance in the processing device 1 prior to the processing of the video data 13 by the processing device 1. The constant data 11 stores constants that take into account the accuracy of a detection method by which a detection unit 23 (described later) detects deterioration in the video data 13. A high value is set to the constant of the constant data 11 when the reliability of the deterioration detection method is high, and a low value is set when the reliability is low.

For example, in the embodiment of the present invention, the constant data 11 associates a true positive value, which is a constant set for true positives, with a false positive value, which is a constant set for false positives. The true positive value is also referred to as True Positive (tp). The true positive value corresponds to the ratio at which the quality of the video data 13 actually deteriorates when the deterioration detection method used by the detection unit 23 detects the deterioration of the quality of the video data 13. False positive values are also referred to as False Positive (fp). The false positive value corresponds to the ratio at which the quality of the video data 13 is not actually degraded when the deterioration detection method used by the detection unit 23 detects deterioration in the quality of the video data 13.

The profile data 12 is data that is stored in advance in the processing device 1 prior to the processing of the video data 13 by the processing device 1. The profile data 12 is data that associates each control for safe driving of the vehicle 3 with a control cost range that indexes the influence of the control. As shown in FIG. 3, the profile data 12 associates a large control cost with a control that has a large influence on the vehicle 3 or the surroundings of the vehicle 3, and associates a small control cost with a control that has a small influence.

The control type defined by the profile data 12 is control selected by the processing device 1 for safe driving of the vehicle 3, and is not limited to control on the vehicle 3. For example, as shown in FIG. 2, the profile data 12 may define, as a control type, that no control is performed when the control cost is low, such as "no control." Further, in parallel with the detection of deterioration of the video data 13 by the processing device 1, when the video data 13 is visually monitored by a supervisor, notification to the supervisor may be defined as the control type. . For example, when the profile data 12 defines "warning to the supervisor" as the control type, it is possible to notify the supervisor of the possibility of video deterioration and urge the supervisor to pay close attention to the video data 13. It becomes possible.

The profile data 12 may set a control cost range for each environment in which the vehicle 3 travels. In the profile data 12, as shown in FIG. 3, each control cost range may be associated with each environment in which the vehicle travels, such as urban areas, rural areas, and expressways.

The video data 13 is video data photographed in the vehicle 3, and is subject to deterioration detection by the detection unit 23. In the embodiment of the present invention, the video data 13 input to the detection unit 23 may be encoded video data received from the communication unit 21 or may be video data decoded by the decoding unit 22. be.

The video data 13 is sequentially input to the detection unit 23 in a form that can be processed by the deterioration detection method of the detection unit 23. The detection unit 23, which will be described later, detects deterioration from encoded but undecoded video data 13 by a method of detecting deterioration of data before decoding, such as packet capture data. The detection unit 23 detects deterioration from the decoded video data 13 using a method of detecting deterioration of a bit stream.

The control cost 14 is data that is determined when the detection unit 23 detects deterioration of the video data 13, taking into consideration the detection accuracy of the method used to detect the deterioration. In the embodiment of the present invention, the control cost 14 is calculated from the accuracy of the deterioration detection method used by the detection unit 23 for the video data 13 to be processed.

The control signal 15 is signal data for the vehicle 3 to perform the control selected by the selection unit 24 from the control cost 14 and the profile data 12. After the control signal 15 is generated by the selection section 24 , it is transmitted to the vehicle 3 and processed by the control section 34 .

The communication unit 21 is an interface for communicating with the vehicle 3. The communication unit 21 receives encoded video data 13 from the vehicle 3 and transmits a control signal 15 to the vehicle 3.

The decoding unit 22 decodes the video data encoded in the vehicle and outputs video data 13 that can be confirmed as an image. The decoded video data 13 is input to the detection section 23 or the display section 26.

The detection unit 23 detects deterioration of the video data 13 capturing the surroundings of the vehicle 3 using a predetermined method. In the embodiment of the present invention, the detection unit 23 employs one deterioration detection method. The deterioration detection method may be a method of detecting deterioration of data before decoding such as packet capture data, a method of detecting deterioration of a bitstream, or a method of detecting deterioration from both packet capture data and bitstream. But it's okay. Information regarding the accuracy of the deterioration detection method employed by the detection unit 23, specifically, the true positive value and false positive value of the deterioration detection method, is set in the constant data 11.

When the detection unit 23 detects deterioration of the video data 13 using the deterioration detection method, it calculates the control cost 14 that is determined by taking into account the detection accuracy of the deterioration detection method that detected the deterioration. The detection unit 23 calculates the control cost 14 from the true positive value set in the constant data 11, for example, as shown in equation (1). In this case, the control cost 14 has a positive correlation with the true positive value in the deterioration detection method that detects deterioration.

As another example, the detection unit 23 may calculate the control cost 14 from a value obtained by subtracting a false positive value from a true positive value, as shown in equation (2), for example. In this case, the control cost 14 has a positive correlation with the true positive value in the deterioration detection method that detected the deterioration, and a negative correlation with the false positive value.

Detection processing by the detection unit 23 according to the embodiment of the present invention will be described with reference to FIG. 4.

In step S101, the detection unit 23 determines deterioration in the video data 13 to be processed. If it is determined in step S102 that no deterioration is detected, the detection unit 23 processes step S101 for the video data 13 to be processed next.

If it is determined in step S102 that deterioration is detected, the detection unit 23 calculates the control cost 14 from the true positive value in step S103, and ends the process. The detection unit 23 processes step S101 for the video data 13 to be processed next.

The selection unit 24 selects a control that includes the control cost 14 from the profile data 12. The control unit 34 generates a control signal 15 for performing the selected control.

For example, in the embodiment of the present invention, when the control cost 14 is a true positive value, the true positive value shown in FIG. 2 is 0.1, so the detection unit 23 sets the control cost 14 to 0.1. The selection unit 24 selects the control "deceleration (small)" whose range includes the control cost of 0.1 from the profile data 12 of FIG.

As another example, when the control cost 14 is the value obtained by subtracting the false positive value from the true positive value, the detection unit 23 subtracts the false positive value 0.2 from the true positive value 0.1 shown in FIG. , set the control cost 14 to -0.1. The selection unit 24 selects the control “none” whose range includes the control cost −0.1 from the profile data 12 of FIG. Note that "no control" means no control, so the selection unit 24 does not need to generate the control signal 15.

When the range of each control cost is provided in the profile data 12 according to the environment in which the vehicle 3 is traveling, the selection unit 24 selects the control from the range provided for the environment in which the vehicle 3 is traveling. The selection unit 24 may identify the environment in which the vehicle 3 is traveling from the images around the vehicle 3 in the video data 13 by referring to a learned model that associates surrounding images with the environment. The selection unit 24 may identify the environment from the position obtained by the GPS receiver of the vehicle 3 by referring to data that associates the position and the environment. Further, the selection unit 24 may specify the environment of the vehicle 3 while it is running using other methods.

The selection process by the selection unit 24 will be described with reference to FIG. 6. The selection process shown in FIG. 6 is executed every time the control cost 14 is set by the detection unit 23. For example, the selection unit 24 may refer to the control cost 14 at a predetermined timing, and execute the process shown in FIG. 5 when the control cost 14 is not zero. Alternatively, the selection unit 24 may receive a notification from the detection unit 23 that the control cost 14 has been set, and may execute the process shown in FIG. 5 . In FIG. 6, when selecting control from the control costs 14, the selection unit 24 initializes the control costs 14 (clears them to zero).

In step S151, the selection unit 24 identifies the environment in the video data 13 in which the detection unit 23 has detected deterioration. In step S152, the selection unit 24 selects control from the control cost range and the control cost 14 associated with the environment specified in step S151 in the profile data 12.

After generating the control signal 15 for executing the selected control, the selection unit 24 clears the control cost 14 to zero in step S153.

The instruction unit 25 instructs the control selected by the selection unit 24. When the selected control is control for the vehicle 3, the instruction section 25 transmits the control signal 15 output by the selection section 24 to the vehicle 3 via the communication section 21. Vehicle 3 is driven according to control signal 15 received from processing device 1 . On the other hand, when the control selected by the selection unit 24 is “none”, the instruction unit 25 does not perform any control. Further, when the control selected by the selection unit 24 does not include an instruction to the vehicle 3 such as “warning to the supervisor”, the instruction unit 25 instructs the vehicle 3 to perform the control without transmitting any control signal. It's okay. The instruction unit 25 warns the monitor to draw the monitor's attention, for example, by sounding a warning sound or displaying an alert.

The display unit 26 displays the decoded video data 13 to the observer. When a supervisor refers to the video data 13 and detects deterioration of the video data 13, he or she inputs a warning, deceleration, or other control to avoid trouble into the processing device 1. The processing device 1 generates a control signal 15 for controlling the vehicle as input by the supervisor, and inputs it to the instruction section 25 . The instruction unit 25 transmits the input control signal 15 to the vehicle 3.

The learning unit 27 learns the control cost range for each control in the profile data 12. The learning unit 27 refers to the learning video data and teacher data (not shown) that associates the control input by the supervisor to the learning video data, and selects the selection unit 24 for the learning video data. The control cost range for each control is calculated so that the selected control is the control input by the supervisor. The learning unit 27 generates profile data 12 by associating control cost ranges for each control. When the profile data 12 associates the control cost range for each environment, the learning unit 27 calculates the control cost range for each environment and control by referring to the teacher data prepared for each environment. The learning process by the learning unit 27 will be described in detail later.

The processing device 1 according to the embodiment of the present invention can select the control of the vehicle 3 depending on the accuracy of the method by which the detection unit 23 detects the deterioration of the video data 13. For example, when deterioration is detected using a highly accurate deterioration detection method, the processing device 1 selects a control that has a large effect, and when deterioration is detected using a low precision deterioration detection method, it selects a control that has a small effect. The processing device 1 can support safe automatic driving of a vehicle by taking into account image deterioration.

(First modification)
In the embodiment of the present invention, a case has been described in which the detection unit 23 employs one deterioration detection method. A case will be described in which the detection unit 23a detects deterioration of the video data 13 using a plurality of deterioration detection methods in the first modification.

In the first modification, constant data 11a associates true positive values, which are constants set for true positives, with respect to each of a plurality of methods for detecting deterioration of video data 13. The constant data 11a further includes, for each of the plurality of methods, a false positive value that is a constant set for false positives, a false negative value that is a constant set for false negatives, and a set for true negatives. True negative values, which are constants, may also be associated. For example, constant data 11a shown in FIG. 6 associates true positive values, false positive values, false negative values, and true negative values with respect to six deterioration detection methods.

Here, the false negative value and true negative value of the constant data 11 will be explained. False negative values are also referred to as False Negative (fn). The false negative value corresponds to the ratio at which the quality of the video data 13 actually deteriorates when the deterioration detection method used by the detection unit 23a does not detect the deterioration of the quality of the video data 13. A true negative value is also called a true negative (tn). The true negative value corresponds to the ratio at which the quality of the video data 13 is not actually degraded when the deterioration detection method used by the detection unit 23a does not detect any deterioration in the quality of the video data 13.

The detection unit 23a according to the first modification detects deterioration of the video data 13 using each of a plurality of methods. The control cost 14 is calculated, for example, as shown in Equation (3), from a value obtained by adding true positive values set for the method in which deterioration is detected among a plurality of methods. In this case, the control cost 14 has a positive correlation with the true positive value in each deterioration detection method that detects deterioration.

When the detection unit 23a detects deterioration using NR-P (1) and NR-B (5) among the six deterioration detection methods shown in FIG. It is calculated as 9. The selection unit 24a selects the control “deceleration (large)” whose range includes the control cost 14=0.9 as the control for the vehicle 3 from the profile data 12 shown in FIG.

The control cost 14 may be calculated not only from true positive values but also from false positive values. For example, the detection unit 23a may calculate the control cost 14 from a value obtained by subtracting the sum of false positive values from the sum of true positive values of the deterioration detection method that detected deterioration, as shown in equation (4). good. In this case, the control cost 14 has a positive correlation with the true positive value in each deterioration detection method that detected deterioration, and has a negative correlation with the false positive value.

When the detection unit 23a detects deterioration using NR-P(1) and NR-B(5) among the six deterioration detection methods shown in FIG. 6, the control cost 14 is (0.1+0.8)- It is calculated as 0 from (0.2+0.7). The selection unit 24a selects "none" control that includes the control cost 14=0 in the range.

The control cost 14 may be calculated not only from the true positive values and false positive values of a method that detects deterioration, but also from the false negative values and true negative values of a method that does not detect deterioration. For example, as shown in equation (5), the control cost 14 is calculated by subtracting the sum of false positive values from the sum of true positive values of each deterioration detection method that detected deterioration, plus the value of each deterioration detection method that did not detect deterioration. It is calculated from the value obtained by subtracting the sum of false negative values from the sum of true negative values of the method and adding it. In this case, the control cost 14 has a positive correlation with the true positive value of each deterioration detection method that detected deterioration and the true negative value of each deterioration detection method that did not detect deterioration. It has a negative correlation with the false positive value of the deterioration detection method and the false negative value of each deterioration detection method that did not detect deterioration.

Detection processing by the detection unit 23a according to the first modification will be described with reference to FIG. 7. In FIG. 7, a case will be described in which the control cost 14 is calculated from the sum of true positive values of each deterioration detection method that detects deterioration according to equation (3).

In step S201, the detection unit 23a determines deterioration in the video data 13 to be processed using each deterioration detection method employed by the detection unit 23a. When the determination by each deterioration detection method is completed, the process advances to step S202.

In step S202, it is determined whether deterioration has been detected using any one of the deterioration detection methods employed by the detection unit 23a. If no deterioration is detected using all of the methods, the detection unit 23a processes step S201 for the video data 13 to be processed next.

When deterioration is detected by any one of the deterioration detection methods employed by the detection unit 23a, in step S203, the detection unit 23a calculates the control cost 14 from the sum of true positive values of the deterioration detection methods that have detected deterioration. is calculated and the process ends. The detection unit 23a processes step S201 for the video data 13 to be processed next.

Detection processing by the selection unit 24a according to the first modification will be described with reference to FIG. 8. In FIG. 8, when the selection unit 24a selects control from the control costs 14, the selection unit 24a initializes the control costs 14.

In step S251, the selection unit 24a determines whether the control cost 14 is zero. If it is zero, the selection unit 24a waits for a predetermined time such as 100 ms in step S255, and then executes step S251 again.

If the control cost 14 is not zero in step S251, the selection unit 24a identifies the environment in the video data 13 in which the detection unit 23a has detected deterioration in step S252. In step S253, the selection unit 24a selects control from the control cost range and the control cost 14 associated with the environment specified in step S252 in the profile data 12.

After generating the control signal 15 for executing the selected control, the selection unit 24a clears the control cost 14 to zero in step S254. In step S255, the selection unit 24a waits for a predetermined time, such as 100 ms, and then executes step S251 again.

In the first modification, the detection unit 23a detects deterioration of the video data 13 using a plurality of deterioration detection methods, so deterioration in the video data 13 can be detected more appropriately from a plurality of viewpoints. Since control for safe driving of the vehicle 3 can be selected according to the accuracy of each of the plurality of deterioration detection methods, safe driving of the vehicle 3 can be supported more appropriately.

(Second modification)
In the embodiment of the present invention, a case has been described in which the detection unit 23 employs one deterioration detection method and each time the detection unit 23 detects deterioration of the video data 13, the control for the vehicle 3 is determined. In the second modification, the detection unit 23b detects the deterioration of the video data 13 using a plurality of deterioration detection methods, and the selection unit 24b selects the vehicle 3 The case of determining control for safety control will be explained.

The detection unit 23b detects deterioration of the video data 13 every unit time. The detection unit 23b detects deterioration of the video data 13 every unit time, such as every second. Every time the detection unit 23b detects deterioration, it updates the control cost 14 by adding a value that takes into account the accuracy of the detection method that detected the deterioration to the current control cost 14. The value added to the current control cost 14 may be the true positive value of the detection method that detected the deterioration, as shown on the right side of equation (1), or the value added to the current control cost 14 may be the true positive value of the detection method that detected the deterioration, as shown on the right side of equation It may be the value obtained by subtracting the false positive value from the true positive value of the detection method that detected. When the detection unit 23b does not detect deterioration in the video data 13, it clears the control cost 14 to zero.

The detection unit 23b according to the second modification may detect the deterioration of the video data 13 for each unit time using a plurality of detection methods, similarly to the first modification. In that case, the value added to the current control cost 14 may be the sum of the true positive values of each detection method that detects deterioration, as shown on the right side of equation (3), or the value added to the current control cost 14 may be the sum of true positive values of each detection method that detects deterioration, as shown on the right side of equation (3), or As shown on the right side, the value may be the value obtained by subtracting the false positive value from the true positive value of each detection method that detected deterioration. In addition, the value added to the current control cost 14 is the value obtained by subtracting the false positive value from the true positive value of each detection method that detected deterioration, as shown on the right side of equation (5), and the value obtained by subtracting the false positive value from the true positive value of each detection method that detected deterioration. It may be the sum of the true negative value of each detection method minus the false negative value.

Detection processing by the detection unit 23b according to the second modification will be described with reference to FIG. 9. In FIG. 9, a case will be described in which the sum of true positive values of each deterioration detection method that detects deterioration is added to the current control cost 14 according to the right side of equation (3).

In step S301, the detection unit 23b determines deterioration in the video data 13 to be processed using each deterioration detection method employed by the detection unit 23b. When the determination by each deterioration detection method is completed, the process advances to step S302.

In step S302, it is determined whether deterioration has been detected using any one of the deterioration detection methods employed by the detection unit 23b.

When deterioration is detected by any one of the deterioration detection methods adopted by the detection unit 23b, in step S303, the detection unit 23b calculates the sum of true positive values of each deterioration detection method that detected deterioration based on the current control Add cost 14. If no deterioration is detected using all of the methods, the detection unit 23b clears the current control cost 14 to zero in step S304.

When the processing in step S303 or step S304 is completed, the detection unit 23b waits for a predetermined time such as 1 s in step S305, and then processes step S301 again.

When the control cost 14 reaches the lower limit of the control cost range of the profile data 12, the selection unit 24b selects the control associated with the lower limit. For example, when the current control cost 14 is 0.06, the selection unit 24b selects the control "warning around the vehicle" as the control for the vehicle 3 from the profile data 12 shown in FIG. 3, and generates the control signal 15. do. Thereafter, when the control cost 14 is updated to 0.08, the selection unit 24b does nothing because it has already generated the control signal 15 for the control "warning around the vehicle". When the control cost 14 is updated to 1.2, the selection unit 24b selects the control "deceleration (small)" as the control for the vehicle 3 from the profile data 12 shown in FIG. 3, and generates the control signal 15.

Detection processing by the selection unit 24b according to the second modification will be described with reference to FIG. 10. In the second modification, the control cost 14 is initialized by the detection unit 23b, so even if the selection unit 24b selects control from the control costs 14 in FIG. 10, the selection unit 24b does not initialize the control cost 14.

In step S351, the selection unit 24b determines whether the control cost 14 has been updated. If it has not been updated, specifically, if the control cost 14 referenced this time is the same as the control cost referenced last time, the process of step S351 is performed again.

If it is determined in step S351 that the control cost 14 has been updated, the selection unit 24b identifies the environment in the video data 13 in which the detection unit 23b has detected deterioration in step S352. In step S353, the selection unit 24b determines whether the updated control cost 14 in the profile data 12 has reached the next range of the control cost associated with the environment specified in step S352. do. If the next range has not been reached, the process returns to step S351.

If the next range has been reached, in step S354 the selection unit 24b selects the control corresponding to the next range of the environment specified in step S352. The selection unit 24b generates a control signal 15 for executing the selected control.

In the second modification, when the deterioration of the video data 13 is continuously detected, it is possible to select a control that has a larger influence, so that even in a situation where the video data 13 has deteriorated, the vehicle 3 can support safe driving.

(Study Department)
Here, the learning process by the learning section 27 will be explained.

The learning video data to which the supervisor associates control as teaching data is a surrounding video of the vehicle 3, and is decoded data. It is preferable that the learning video data includes video quality deterioration and has as many variations as possible. Further, the video data for learning by the learning unit 27 is a peripheral video of the vehicle 3, and is video data in a format that follows the deterioration detection method adopted by the detection unit 23. For example, when detecting from packet capture data, the video data may be a packet of video data before decoding. When detected from a bitstream, video data is decoded data.

In learning by the learning unit 27, first, an initial value is set for the control cost. The initial value may be randomly set within the range of possible values of the control cost set for each control. The initial value may be determined by ranking the effects of each control on the surroundings subjectively, and then equally allocating the control costs according to the order. Specifically, when the control cost takes a value from 0 to 1, for each of the five types of control, in order of decreasing influence on the surroundings, it is 0.2, 0.4, 0.6, 0.8 and set an initial position of 1.0. The initial value may be a control cost optimized under similar circumstances.

An example of the learning process by the learning unit 27 will be explained. Here, a case will be described in which the plurality of deterioration detection methods adopted by the detection unit 23 include a method of detecting from packet capture data and a method of detecting from a bitstream. Let the video data for learning be v(1), v(2) ...v(n). Let the packet capture data corresponding to these training video data be p(1), p(2) ...p(n), and the bitstream data be b(1), b(2) ...b(n). .

(1) The learning unit 27 selects the control using the same selection method as the selection unit 24, using the training data of v(1) as a reference and inputting p(1) and b(1) and the initial value of the control cost. . The selection process is repeated until v(n). If the control of the teacher data and the control selected in the selection process by the selection unit 24 are the same, the control is set to 1, and if the control is incorrect, it is set to 0, and the sum of the results of repeating up to v(n) is calculated.

(2) The learning unit 27 performs the same process as in (1) by changing the combination of control cost values. The combination of control cost values may be changed randomly or by an integer programming problem approach.

(3) If there is a combination of control costs that completely matches the control in the teacher data and the control selected in the selection process, set that as the control cost optimization solution. The process (2) is repeated until a control cost solution in which the control of the teacher data and the control selected in the selection process completely match is obtained. Alternatively, if an optimal solution cannot be obtained even after sufficient trials such as 1 million combinations, the combination with the largest sum, specifically the combination closest to the supervisor's selection, is set as the optimal solution for control costs. do.

Once the learning unit 27 obtains an optimized solution for one environment, it repeats the same process for other environments to obtain optimized solutions.

Note that the processing by the learning unit 27 shown individually is an example and is not limited to this.

The processing device 1 of the present embodiment described above includes, for example, a CPU (Central Processing Unit, processor) 901, a memory 902, a storage 903 (HDD: Hard Disk Drive, SSD: Solid State Drive), and a communication device 904. , a general-purpose computer system including an input device 905 and an output device 906 is used. In this computer system, each function of the processing device 1 is realized by the CPU 901 executing a program loaded onto the memory 902.

Note that the processing device 1 may be implemented by one computer or by multiple computers. Further, the processing device 1 may be a virtual machine implemented in a computer.

The program of the processing unit 1 can be stored in a computer-readable recording medium such as an HDD, SSD, USB (Universal Serial Bus) memory, CD (Compact Disc), or DVD (Digital Versatile Disc), or can be distributed via a network. You can also.

Note that the present invention is not limited to the above-described embodiments, and many modifications can be made within the scope of the invention.

1 processing device 3 vehicle 5 processing system 11 constant data 12 profile data 13 video data 14 control cost 15

control signal

21, 33 communication section 22 decoding section 23 detection section 24 selection section 25 instruction section 26 display section 27 learning section 31 camera 32 encoding Section 34 Control section 35 Drive section 901 CPU
902 Memory 903 Storage 904 Communication device 905 Input device 906 Output device

Claims

a storage device that stores profile data that associates each control for safe driving of a vehicle with a control cost range that indexes the influence of the control;
a detection unit that detects deterioration of video data photographing the surroundings of the vehicle using a predetermined method;
a selection unit that selects, from the profile data, a control that includes a control cost in the range, which is determined by taking into account the detection accuracy in the method;
A processing device including an instruction section that instructs selected control.
The storage device further stores constant data that associates a true positive value, which is a constant set for true positives, with a false positive value, which is a constant set for false positives, in the method,
The processing device according to claim 1, wherein the control cost is calculated from a value obtained by subtracting a false positive value from a true positive value.
The storage device stores constant data associated with true positive values, which are constants set for true positives, for each of the plurality of methods for detecting deterioration of the video data,
The detection unit detects deterioration of the video data using each of the plurality of methods,
The processing device according to claim 1, wherein the control cost is calculated from a value obtained by adding true positive values set for a method detected as having deterioration among the plurality of methods.
The detection unit detects deterioration of video data for each unit time, and each time deterioration is detected, adds a value that takes into account the accuracy of the detection method that detected the deterioration to the control cost,
The processing device according to claim 1, wherein when the control cost reaches the lower limit of the range, the selection unit selects the control associated with the lower limit.
The profile data sets a control cost range for each environment in which the vehicle runs,
The processing device according to claim 1, wherein the selection unit selects the control from a range provided for an environment in which the vehicle travels.
The control selected by the selection unit for the learning video data is determined by the supervisor by referring to the training data that associates the learning video data with the control input by the supervisor to the learning video data. The processing device according to claim 1, further comprising: a learning unit that calculates a control cost range for each control and generates the profile data so that the input control becomes the input control.
a computer stores in a storage device profile data associating a range of control costs indexing the influence of the control with each control for safe driving of the vehicle;
the computer detects deterioration of video data photographing the surroundings of the vehicle using a predetermined method;
the computer selects, from the profile data, a control that includes a control cost determined in consideration of detection accuracy in the method;
A processing method in which the computer directs selected control.
A program for causing a computer to function as the processing device according to any one of claims 1 to 6.