WO2023119800A1

WO2023119800A1 - In-vehicle computational processing device and computational processing method

Info

Publication number: WO2023119800A1
Application number: PCT/JP2022/037977
Authority: WO
Inventors: 和香子中野; 理宇平井; 延之村中; 浩朗伊藤
Original assignee: 日立Astemo株式会社
Priority date: 2021-12-24
Filing date: 2022-10-12
Publication date: 2023-06-29
Also published as: JP2023094745A

Abstract

When switching AI models, the present invention switches to an appropriate AI model by verifying whether the switch causes any degradation in accuracy compared to before the switch. Provided is an in-vehicle computational processing device that performs computations using a trained model, the in-vehicle computational processing device including a computational unit that performs computations using a first trained model, a state estimation unit that estimates a state by acquiring the position of the vehicle or information from inside and outside the vehicle, and a model selection unit that selects a second trained model on the basis of the state, and characterized by including a model switching verification unit that compares a computational result of the first trained model to a computational result obtained by performing computations using the second trained model in parallel with the first trained model, and determines, on the basis of a result of the comparison, whether to switch from the first trained model to the second trained model, and if the model switching verification unit has determined to switch to the second trained model, the computational unit switches from the first trained model to the second trained model to perform computations.

Description

In-vehicle arithmetic processing device and arithmetic processing method

The present invention relates to an in-vehicle arithmetic processing device and an arithmetic processing method that recognize the external environment of a vehicle by processing the output of an external sensor mounted on the vehicle with a machine-learned AI model.

In recent years, vehicles equipped with autonomous driving (AD) systems and advanced driver-assistance systems (ADAS) have become widespread. To implement these systems, it is necessary to recognize the vehicle's external environment (other vehicles, pedestrians, bicycles, road surfaces, white lines, road signs, obstacles, their positions, etc.). Therefore, adopt a method of recognizing the vehicle's external environment by processing the output of an external sensor mounted on the vehicle with an AI model that has undergone machine learning (hereinafter simply referred to as "AI model" or "model"). There are many.

Here, if one AI model recognizes the external environment for all scenes while the vehicle is running, it is necessary to prepare in advance an AI model that has undergone machine learning using exhaustive sensor output data. However, such an AI model guarantees recognition accuracy in various scenes, so the model scale increases exponentially, and the ECU (Electronic Control Unit) that executes the AI model is also very expensive. Arithmetic performance is required. As a result, there is a problem that the chip cost of the ECU increases and the calculation load (power consumption) during automatic operation also increases.

On the other hand, in order to reduce the chip cost and calculation load (power consumption) of the ECU, if a small AI model trained with limited data is applied, the recognition accuracy of the external environment is high in environments that the AI model does not support. likely to deteriorate.

In consideration of these problems, by preparing multiple relatively small AI models in advance and switching to the AI model according to the driving scene and driving position, the recognition accuracy of the external environment is improved in any driving scene. A method of reducing the calculation load (power consumption) of the ECU while achieving the above is being studied.

For example, in the abstract of Patent Document 1, the problem is that "when a learned model for controlling a vehicle is set for each predetermined region, the learned model is appropriately switched according to the position of the vehicle." As a solution, "a control support device that supports vehicle control using a learned model by machine learning, based on the vehicle information and position information of the vehicle transmitted from the vehicle, in the vehicle a control unit that selects a learned model that is updatable and that corresponds to the current position of the vehicle included in the location information; and a transmission unit that transmits the selected learned model to the vehicle.” ing.

Japanese Unexamined Patent Application Publication No. 2020-93760

The control support device of Patent Document 1 is applied according to changes in the position of the vehicle, as is clear from the description in the abstract, etc., "a control unit that selects a learned model corresponding to the current position of the vehicle." It switches the AI model.

However, in Patent Document 1, it is not verified whether the recognition accuracy deteriorates when switching the AI model, and when switching to an inappropriate AI model, the recognition accuracy of the external environment deteriorates, resulting Basically, there is a possibility that the vehicle control becomes unstable due to AD or ADAS.

The present invention has been made in view of the above problems, and provides an in-vehicle arithmetic processing device and an arithmetic processing method that select an AI model that does not deteriorate the recognition accuracy of the external environment when switching between AI models. intended to provide

In order to solve the above-mentioned problems, an in-vehicle arithmetic processing device of the present invention is mounted on a vehicle and performs arithmetic operations using a learned model. , a state estimating unit that acquires and estimates the position of the vehicle or the state inside and outside the vehicle, and a model selection unit that selects a second trained model based on the state, wherein the second learning Using the trained model, the calculation result when the calculation is performed in parallel with the first trained model is compared with the calculation result of the first trained model, and based on the comparison result, the first a model switching verification unit that determines whether or not to switch from a trained model to the second trained model, wherein the computing unit switches to the second trained model in the model switching verification unit. is determined, the in-vehicle arithmetic processing device switches from the first learned model to the second learned model to perform the calculation.

According to the in-vehicle arithmetic processing device and arithmetic processing method of the present invention, even when the AI model is switched, the recognition accuracy of the external environment does not deteriorate.

FIG. 2 is a functional block diagram of an in-vehicle arithmetic processing device according to the first embodiment; 4 is a flowchart showing a processing procedure of the in-vehicle arithmetic processing device according to the first embodiment; The figure which shows an example of the driving|running|working scene state table which is a kind of state table. FIG. 4 is a diagram showing an example of a user state table, which is a type of state table; FIG. 4 is a diagram showing an example of a device load state table, which is a type of state table; The figure which shows an example of a model table. The functional block diagram of model verification which makes the output of Lider a correct value. FIG. 10 is a functional block diagram of model verification using outputs of heterogeneous sensors as correct values. Functional block diagram of model verification using the output of the current model as the correct value. Functional block diagram of model verification using the output of the current model as the correct value. FIG. 4 is a diagram showing a display panel during automatic operation according to the first embodiment; FIG. 5 is a diagram showing a setting screen during automatic operation according to the first embodiment; FIG. 10 is a functional block diagram of an in-vehicle arithmetic processing device according to a second embodiment; 8 is a flow chart showing a processing procedure of the in-vehicle arithmetic processing device according to the second embodiment; The figure which shows an example of a model difference table. FIG. 11 is a diagram showing an example of a past travel history display method according to the third embodiment;

First, the in-vehicle processing unit 1 according to the first embodiment of the present invention will be described using FIGS. 1 to 8. FIG. In the following description, the in-vehicle processing unit 1 is assumed to be built in an ECU, but the in-vehicle processing unit 1 may be built into an external sensor, and the external environment recognized by the external sensor may be output to the ECU. .

FIG. 1 is a functional block diagram of the in-vehicle arithmetic processing device 1 according to the first embodiment. As shown here, the in-vehicle arithmetic processing unit 1 has a main arithmetic unit 11 , an AI arithmetic unit 12 and a storage unit 13 . Although FIG. 1 exemplifies the structure in which the main calculation unit 11 and the AI calculation unit 12 are separated, both calculation units may be integrated.

As shown, the in-vehicle processing unit 1 is a device that receives external sensor information I1 from an external sensor and vehicle sensor information I2 from a vehicle sensor. Here, the external sensor is a sensor that acquires information around the vehicle, such as a stereo camera, monocular camera, lidar, illuminance sensor, acceleration sensor, and radar. In addition, vehicle sensors, for example, car navigation system, GNSS (Global Navigation Satellite System) receiver, speed sensor, steering angle sensor, fuel level gauge, battery level gauge, etc. Information related to the position and state of the own vehicle is a sensor that acquires The external sensor information I1 and the vehicle sensor information I2 are not necessarily input one by one, and a plurality of external sensor information I1 and vehicle sensor information I2 may be input.

The main computation unit 11 is a unit that executes a predetermined program by means of a computation device such as a CPU (Central Processing Unit) or FPGA (Field Programmable Gate Array). By executing a predetermined program, the main computation unit 11 includes a state estimation unit 11a, a candidate model selection unit 11b, a current model control unit 11c, a candidate model control unit 11d, a result reception unit 11e, and a model verification unit 11f. and other functional units. The AI calculation unit 12 is an AI accelerator or the like specialized for AI model processing, and includes a model calculation unit 12a and an AI calculation control unit 12b. The storage unit 13 is a storage device such as a semiconductor memory, and has a state table 13a, a model table 13b, and a parameter table 13c. Details of each part of the in-vehicle arithmetic processing unit 1 will be described below mainly along the flow of information.

<Details of each part of the in-vehicle arithmetic processing unit 1>
The state estimation unit 11a of the main computation unit 11 estimates the current state of the vehicle based on the input external sensor information I1 and vehicle sensor information I2. The state estimated here includes, for example, the type of the vehicle's running position (highway, general road, parking lot, etc.), the type of the current time (morning/noon, evening, night), and the type of weather (clear). , cloudiness, rain, snow, etc.). The state estimated by the state estimating unit 11a includes the resource state of the in-vehicle arithmetic processing unit 1, such as the degree of power consumption in the own vehicle, the degree of the remaining battery level, the number of objects to be recognized, who is the driver, may be Further, state estimation by the state estimation unit 11a is assumed to be rule-based estimation using environment information from a single or multiple external sensor information I1, estimation using an AI model, or the like. Also, the road conditions during travel may be estimated from the map as the vehicle sensor information I2 and the position information.

When the state estimated by the state estimation unit 11a changes, the candidate model selection unit 11b of the main calculation unit 11 refers to the state table 13a of the storage unit 13, and selects the model ID of the AI model linked to the estimated state. select. Details of the state table 13a will be described later.

The candidate model control unit 11d of the main computation unit 11 refers to the model table 13b of the storage unit 13 to acquire model information corresponding to the model ID selected by the candidate model selection unit 11b, and refers to the parameter table 13c. to acquire the model parameter corresponding to the selected model ID. Then, the acquired model information and model parameters are transmitted to the AI calculation unit 12 . Further, the candidate model control unit 11d stops the environment recognition processing using the candidate model M2 in the AI calculation unit 12 as necessary. Details of the model table 13b will be described later.

The AI calculation control unit 12b of the AI calculation unit 12 transmits the model information and model parameters received from the candidate model control unit 11d to the model calculation unit 12a. Also, the AI calculation control unit 12b starts or stops the environment recognition processing in the model calculation unit 12a based on a command from the main calculation unit 11. FIG.

The model calculation unit 12a of the AI calculation unit 12 has a structure specialized for vector calculations and tensor calculations that are frequently used during environment recognition processing using AI models, and performs environment recognition processing using the current model M1 and candidate model M2. The used environment recognition processing can be performed in parallel. The current model M1 (first trained model) is an AI model that is currently used for AD and ADAS and that has been confirmed to have a predetermined accuracy or more at the time of the previous verification. The candidate model M2 (second learned model) is an AI model constructed using model information and model parameters received from the candidate model control unit 11d, which may be used in the future in place of the current model M1. be.

The current model control unit 11c of the main calculation unit 11 stops the environment recognition processing using the current model M1 in the AI calculation unit 12 as necessary. In addition, the current model control unit 11c sets the candidate model M2 as a new current model M1 as necessary.

The result reception unit 11e of the main calculation unit 11 receives the recognition results of the current model M1 and the candidate model M2 from the AI calculation unit 12, and transmits them to the model verification unit 11f.

The model verification unit 11f of the main calculation unit 11 compares both recognition results received from the result reception unit 11e to verify the accuracy of both models, and selects an AI model to be used in AD or ADAS as a candidate from the current model M1. It is determined whether or not it is possible to switch to the model M2. When the model verification unit 11f determines that switching to the candidate model M2 is possible, the model verification unit 11f transmits a stop signal to the current model control unit 11c. send a signal. As a result, the current model control unit 11c or the candidate model control unit 11d can stop one of the AI models being processed by the model calculation unit 12a of the AI calculation unit 12 according to the determination result of the model verification unit 11f. can.

<Processing procedure of in-vehicle arithmetic processing unit 1>
Next, the processing procedure of the in-vehicle arithmetic processing unit 1 will be described using the flowchart of FIG. In FIG. 2, the processing of the main computation unit 11 and the processing of the AI computation unit 12 are shown separately, and the arrows crossing both processing flows indicate data transmission.

At the time of step S21, the model calculation unit 12a of the AI calculation unit 12 is executing only the previous current model M1 as an AI model used in AD and ADAS.

In step S11, the state estimation unit 11a of the main computation unit 11 acquires the external sensor information I1 and the vehicle sensor information I2, and estimates the current vehicle state from the environmental information. As described above, the vehicle state estimated here includes, for example, the type of vehicle travel position (highway, general road, parking lot, etc.) and the type of current time (morning, noon, evening, night, etc.). ), type of weather (clear, cloudy, rainy, snowy, etc.), level of power consumption in the vehicle, level of remaining battery level, number of objects to be recognized, and the like.

In step S12, the candidate model selection unit 11b of the main calculation unit 11 refers to the state table 13a stored in the storage unit 13, and selects the candidate model M2 corresponding to the state estimated by the state estimation unit 11a. A method for selecting the candidate model M2 with reference to the state table 13a will be specifically described below with reference to FIGS. 3A to 3C.

FIG. 3A is an example of a driving scene state table 13a1, which is a kind of state table 13a, and is a table for specifying candidate models M2 according to driving scenes. For example, if the driving scene estimated by the state estimating unit 11a is a driving scene that is a combination of the driving position type “general road”, the weather “sunny”, and the current time “morning”, the candidate model selecting unit 11b selects the model ID “ m01" as the candidate model M2. The driving scenes registered in the driving scene state table 13a1 are not limited to those illustrated in FIG. 3A, and snow, mountain roads, tunnels, etc. may be registered as driving conditions. In addition, as a method of dividing models when preparing them, models may be prepared for each combination of multiple states, as in the driving scene shown in FIG. 3A. A single model may be created by summarizing the daytime and nighttime conditions.

FIG. 3B is an example of a user state table 13a2, which is a type of state table 13a, and is a table for specifying candidate models M2 according to drivers (users). A model ID is registered for each user in this table, and when the state estimating unit 11a detects a change in the user, such as when user settings are changed, the candidate model selecting unit 11b selects the user after the change. model ID is selected as a candidate model M2.

FIG. 3C is an example of a device load state table 13a3, which is a type of state table 13a, and is a table for specifying candidate models M2 according to the load state of the in-vehicle processing device 1. FIG. In this table, an AI model is registered for each device load. For example, if the current device load is "high load", the model ID "m15" corresponds to a model with a relatively small computational load. ”, and if the current device load is “low load”, select the model ID “m16” corresponding to a model with a relatively large computational load.

In step S13, the candidate model control unit 11d of the main calculation unit 11 refers to the model table 13b and the parameter table 13c of the storage unit 13, and selects model information and model parameters associated with the model ID selected by the candidate model selection unit 11b. get. Then, in addition to the model information and model parameters, the candidate model control unit 11d transmits to the AI calculation unit 12 a process start signal for the candidate model M2 generated based on the information.

FIG. 4 is a diagram showing an example of the model table 13b. The model table 13b exemplified here holds each information of "model ID", "number of cores", "input size", "number of layers", and "each layer weight size list". A model ID that can be selected with reference to the state table 13a illustrated in FIGS. 3A to 3C is registered in the "model ID". The number of cores of the AI calculation unit 12 required to execute the AI model associated with the "model ID" is registered in the "number of cores". The input size of the recognition target data to the AI model associated with the "model ID" is registered in the "input size". The number of layers of the AI model linked to the "model ID" is registered in the "number of layers". The weight size of each layer of the AI model linked to the "model ID" is registered in the "each layer weight size list". The information stored and acquired in the model table 13b is not limited to this, and necessary information can be held depending on the computer architecture of the AI calculation unit 12. FIG.

In step S22, the model calculation unit 12a of the AI calculation unit 12 additionally executes the candidate model M2 constructed using the model information and model parameters. Since the model calculation unit 12a has already executed the current model M1 at the time of step S21, both the current model M1 and the candidate model M2 are executed in this step. The execution results of both models are transmitted to the result receiving section 11e of the main computing section 11. FIG.

At step S14, the result receiving unit 11e of the main computing unit 11 acquires the execution results of the current model M1 and the candidate model M2.

In step S15, the model verification unit 11f of the main computation unit 11 verifies the processing result of the output of the external sensor information I1 and the like in the current model M1 and the candidate model M2, and based on the verification result, the candidate model M2 is verified. Determines whether switching is possible. The details of the verification method here will be described later.

Step S16 branches according to the determination in step S15. That is, when it is determined that switching to the candidate model M2 is possible, the process proceeds to step S23, and when it is determined that the switching is not possible, the process proceeds to step S17.

In step S23, the AI calculation control unit 12b of the AI calculation unit 12 stops the execution of the current model M1 in accordance with the command received from the current model control unit 11c of the main calculation unit 11, and replaces the candidate model M2 with the current state. The model M1 is set, and the process of FIG. 2 is terminated. As a result, AD and ADAS using the new current model M1 are started in the model calculation unit 12a.

In step S17, the model verification unit 11f of the main calculation unit 11 determines whether the previous current model M1 is valid even under the current environment. Then, if the recognition accuracy of the previous current model M1 is equal to or higher than the threshold value, the process proceeds to step S24; otherwise, the process proceeds to step S25.

In step S24, the AI calculation control unit 12b of the AI calculation unit 12 stops executing the candidate model M2 according to the command received from the candidate model control unit 11d of the main calculation unit 11, and ends the processing of FIG. As a result, in the model calculation unit 12a, only the previous current model M1 is executed, and AD and ADAS using the previous current model M1 are continued.

In step S25, the AI calculation control unit 12b of the AI calculation unit 12 controls both the current model M1 and the candidate model M2 according to the commands received from the current model control unit 11c and the candidate model control unit 11d of the main calculation unit 11. Execution is stopped and the process of FIG. 2 is terminated. As a result, the AD and ADAS are stopped and the manual operation mode is entered by the user, but inappropriate execution of AD and ADAS due to misidentification of the external environment is prevented.

In the above step S12, it is assumed that one candidate model M2 is selected. You can In that case, all the selected candidate models M2 should be executed in step S22, and the most accurate candidate model M2 should be set as the new current model M1 in step S23.

<Verification Method in Step S15>
Here, the details of the AI model accuracy verification method in step S15 will be described. Accuracy verification of the AI model can be calculated by comparing the correct value and the recognition result of the AI model, assuming that the larger the error, the lower the accuracy. However, since it is difficult to obtain a completely correct value of the AI model within the own vehicle, in this embodiment, a value that is considered to be close to the correct value is compared with the output value of the AI model, and the error between the two is calculated. Suppose we want to calculate the accuracy of an AI model.

Below, the accuracy calculation method in step S15 will be described, taking as an example a case where the current model M1 and the candidate model M2 are AI models whose recognition target is image data captured by a camera. Also, the accuracy calculation method by comparison with a value close to the correct value will be described by taking two policies as examples.

In the driving scene state table 13a1 illustrated in FIG. 3, for example, when switching from the AI model for general roads to the AI model for highways, there are cases where the recognition targets are partially different before and after switching. To calculate the model accuracy in this case, a method of calculating the accuracy of each AI model using a value obtained by a sensor other than the camera as a correct value can be considered. Such a verification method will be described with reference to FIGS. 5A and 5B.

Also, in the driving scene state table 13a1 illustrated in FIG. 3, for example, when the AI model for sunny/cloudy weather is switched to the AI model for nighttime, the recognition target object may be the same before and after switching. In order to calculate the model accuracy in this case, a method of calculating the accuracy of the candidate model M2 using the recognition result of the current model M1 as the correct value can be considered. Such a verification method will be described with reference to FIGS. 6A and 6B.

<<Method of verifying that the output of another sensor other than the camera is the correct value>>
FIG. 5A is a functional block diagram illustrating a method of calculating the accuracy of each AI model using the output of Lider 3 as a correct value and performing each AI model verification. In this case, the result receiving unit 11e receives the result of recognizing the image captured by the camera 2 with the current model M1 and the candidate model M2, and also receives the result of recognizing the output of the Lider 3 with the Lider model. Then, the model verification unit 11f verifies the model accuracy of the current model M1 and the candidate model M2 using the recognition result based on the output of the Lider 3 as the correct value.

FIG. 5B is a functional block diagram illustrating a method of calculating the accuracy of each AI model and performing AI model verification using a value obtained by synthesizing the outputs of a plurality of heterogeneous sensors 4 other than cameras as a correct value. In this case, the result receiving unit 11e receives the result of recognizing the image captured by the camera 2 with the current model M1 and the candidate model M2, and also receives the peripheral image obtained by combining the outputs of the plurality of heterogeneous sensors 4 by the fusion unit 41. Receive recognition results. Then, the model verification unit 11f verifies the model accuracy of the current model M1 and the candidate model M2 using the recognition result based on the output of the heterogeneous sensor 4 as the correct value.

Here, as shown in FIGS. 5A and 5B, the model verification unit 11f of this embodiment has an accuracy calculation unit 11f1 and an accuracy comparison unit 11f2.

The accuracy calculation unit 11f1 calculates the current model accuracy from the magnitude of the error between the correct value (the recognition result based on the output of the Lider 3 or the recognition result based on the output of the heterogeneous sensor 4) and the recognition result of the current model M1, and , the accuracy of the candidate model is calculated from the magnitude of the error between the correct answer value and the recognition result of the candidate model M2.

Then, the accuracy comparison unit 11f2 compares the current model accuracy and the candidate model accuracy, and if the candidate model accuracy is higher than the current model accuracy and the candidate model accuracy reaches the reference value, the current model M1 to the candidate model It decides that switching to M2 is possible. On the other hand, when the current model accuracy is higher than the candidate model accuracy, the accuracy comparison unit 11f2 determines that the current model M1 cannot be switched to the candidate model M2. In addition, if the accuracy of the candidate model is higher than the accuracy of the current model, but the accuracy is not sufficient for the standard value, it is possible that the accuracy of the AI model itself will deteriorate, so both models can be disabled and AD and ADAS can be stopped. Conceivable.

<<Method of verification using the output of the current model M1 as the correct value>>
FIG. 6A is a functional block diagram showing an example of a method of calculating the accuracy of the candidate model M2 using the recognition result of the current model M1 as the correct value. In the figure, a result reception unit 11e receives an image from the camera 2 as an input and receives the result of recognition by the current model M1 and the candidate model M2. Then, the model verification unit 11f uses the recognition result of the current model M1 as the correct value, and compares the recognition result of the candidate model M2 with the accuracy comparison unit 11f2.

FIG. 6B is a functional block diagram showing another example of calculating the accuracy of the candidate model M2 using the recognition result of the current model M1 as the correct value. In addition to inputs from the current model M1 and the candidate model M2, vehicle speed information I2a, which is a type of vehicle sensor information I2, is also input to the result receiving unit 11e of FIG. Further, the result receiving unit 11e of FIG. 6B predicts the current recognition result after the state change from the recognition result of the current model M1 operated before the state change detected by the state estimation unit 11a and the vehicle speed information I2a. The accuracy comparison unit 11f2 compares the correct value with the recognition result of the candidate model M2, using the value predicted by the recognition prediction unit 11e1 as the correct value.

<How to notify AI model switching>
FIG. 7 is a diagram showing an example of the display panel 5 for a car navigation system or the like after switching the AI model in the in-vehicle processing unit 1. In FIG. In addition to the map information 5a, the display panel 5 displays marks representing driving scenes such as an automatic driving mode display 5b and a night driving mode display 5c. As a result, it becomes possible for the model verification unit 11f to confirm whether or not the model has been correctly switched.

FIG. 8 is a diagram showing an example of an automatic operation setting screen 5d that can be displayed on the display panel 5. This figure shows settings for enabling/disabling automatic driving for each driving scene and for enabling/disabling switching between scenes. As in this embodiment, when automatic driving on general roads is possible and automatic driving on expressways is possible, and switching between general roads and expressways is disabled, model switching during driving is If it is not possible and it is possible to switch between general roads and expressways, it can be confirmed that model verification is possible while driving and model switching is possible.

According to the in-vehicle arithmetic processing device of this embodiment described above, even when the AI model is switched, the recognition accuracy of the external environment does not deteriorate. Therefore, even when using small AI models while switching between them, there is no deterioration in recognition accuracy before and after model switching, so the cost of the ECU chip is reduced compared to using large AI models to handle various situations. In addition, it is possible to prevent erroneous control of the vehicle due to poor recognition of the external environment while reducing power consumption.

In addition, since the model verification by the in-vehicle arithmetic processing unit of the present embodiment is executed in the vehicle, unlike the case where the model verification is performed by an external server connected via a communication line, the model verification processing is delayed, and in a remote area. It is possible to avoid a situation where model verification cannot be performed due to communication failure.

Next, the in-vehicle arithmetic processing device 1 according to Embodiment 2 of the present invention will be described with reference to FIGS. 9 to 12. FIG. In the following description, overlapping descriptions of the points in common with the first embodiment will be omitted.

In the model calculation unit 12a of the first embodiment, the recognition processing using the current model M1 and the recognition processing using the candidate model M2 are performed in parallel before and after the model switching. There is also a common channel, and there is a problem that operating the common channel redundantly increases the computational load (power consumption).

Therefore, in the present embodiment, when the current model M1 and the candidate model M2 are operated in parallel by the model calculation unit 12a, the recognition result of the current model M1 and the recognition result of the candidate model M2 are obtained without consuming processing resources for two models. Both results are acquired, and using them, it is possible to determine whether the AI model can be switched.

Therefore, in this embodiment, when creating an AI model for each driving scene, one large model is used to create a model that covers multiple driving scenes using all the learning data, and the created model is Model compression is performed for each driving scene using learning data included in each driving scene. By doing so, it is possible to create a model of each driving scene in a partially shared manner. Also, by creating a large model that covers all driving scenes and then compressing the model so that it has different computational complexity, it is possible to generate multiple models with a part of the model shared. In this way, when the AI model for each driving scene shares a part, only the difference between the current model M1 and the candidate model M2 is additionally operated when the candidate model M2 is operated. Recognition results can be obtained.

This embodiment assumes an example of model switching when the AI models for each driving scene share a part in this way.

FIG. 9 is a functional block diagram of the in-vehicle arithmetic processing device 1 of this embodiment. The vehicle-mounted processing unit 1 of the present embodiment includes a main processing unit 11, an AI processing unit 12, and a storage unit 13, like the vehicle-mounted processing unit 1 of the first embodiment (see FIG. 1). As a unique configuration of this embodiment, the main computation unit 11 is added with a model difference extraction unit 11g, and the storage unit 13 is added with a model difference table 13d.

The model difference extraction unit 11g refers to the model table 13b (see FIG. 4) and the model difference table 13d based on the model ID selected by the candidate model selection unit 11b, and finds the difference channel between the candidate model M2 and the current model M1. Extract.

The candidate model control unit 11d inputs the extracted channel difference information to the AI calculation control unit 12b of the AI calculation unit 12, and the model calculation unit 12a loads only the candidate model difference D2 as a model parameter and executes it.

In the model verification unit 11f based on the execution result, if it is possible to switch to the candidate model M2, the execution of the part of the shared model M3 is left as it is, and the current model difference D1 is stopped. As a result, an effect equivalent to that of stopping the current model M1 can be obtained. In this embodiment, channels are assumed as model differences, but the difference is not limited to this, and the differences may be model layer units or model neuron units.

FIG. 10 is a diagram showing the processing flow of the in-vehicle arithmetic processing device 1 shown in FIG. Arrows indicate data transmission. It should be noted that, in the following description, redundant description of the points in common with FIG. 2 will be omitted.

At the time of step S21, the model calculation unit 12a of the AI calculation unit 12 is executing only the previous current model M1 as an AI model used in AD and ADAS. Note that the current model M1 and the candidate model M2 themselves are not registered in the model calculation unit 12a of this embodiment, but are registered in the form of the shared model M3, the current model difference D1, and the candidate model difference D2. Running the current model M1 in the example means running both the shared model M3 and the current model delta D1, and running the candidate model M2 means running both the shared model M3 and the candidate model delta D2. means that

Steps S11 and S12 are equivalent to those in FIG.

In step S12a, the model difference extraction unit 11g of the main computation unit 11 refers to the model table 13b and the model difference table 13d of the storage unit 13, and extracts the model difference between the current model M1 and the selected candidate model M2.

In step S13a, the candidate model control section 11d of the main calculation section 11 transmits a start signal for processing using the candidate model M2, model difference information, and model difference parameters to the AI calculation section 12.

In step S22a, the model calculation unit 12a of the AI calculation unit 12 additionally executes the candidate model difference D2. As described above, the model calculation unit 12a has already executed both the shared model M3 corresponding to the current model M1 and the current model difference D1 at the time of step S21. By additionally executing, an output equivalent to executing both the current model M1 and the candidate model M2 can be obtained. The execution results of both models are sent to the main computation unit 11 .

Steps S14 to S17 and steps S23 to S25 are basically the same as those in FIG. However, in this embodiment, stopping the current model M1 in step S23 corresponds to stopping only the current model difference D1, and stopping the candidate model M2 in step S24 means stopping only the candidate model difference D2. Equivalent to.

FIG. 11 is an example of a diagram showing an example of a model difference table 13d stored in the storage unit 13 of the in-vehicle arithmetic processing device 1 shown in FIG. As shown here, the model difference table 13d holds model IDs and unnecessary channel lists for each layer. In this embodiment, the unnecessary channel list of the model difference table 13d expresses the difference by how many channels in each layer of the other models are smaller than the model m01. For example, the model m02 shows that the 4th, 8th, 9th, and 30th channels are unnecessary for the first layer L1 compared to the model m01, and the 1st and 2nd channels are unnecessary for the second layer L2. there is Conversely, a location with more channels than the model m01 can be expressed in minus notation.

Next, using FIG. 12, the in-vehicle arithmetic processing device 1 according to the third embodiment of the present invention will be described. In the following description, overlapping descriptions of the points in common with the above embodiment will be omitted.

In this embodiment, an example will be described in which the verification by the model verification unit 11f in the in-vehicle arithmetic processing device 1 shown in the first and second embodiments is omitted based on the past travel history. As the frequency of model verification in the in-vehicle processing unit 1 increases, the processing load and mounting size on the chip increase. At the timing of , it is possible to reduce the number of verifications by switching to a model that is linked to a similar scene.

FIG. 12 shows an example of past travel history displayed on the display panel 5 of the vehicle. As shown here, the display of the past travel history includes the time, route name, departure point, destination, and user ID. By selecting the past travel history from this display, the user can switch to the same model at the same timing as the previous travel. Note that the past travel history is not limited to that of a single vehicle, and the travel history of another vehicle obtained from a server or the like may also be searchable.

DESCRIPTION OF SYMBOLS 1... Vehicle-mounted arithmetic processing unit 11... Main calculation part 11a... State estimation part 11b... Candidate model selection part 11c... Current model control part 11d... Candidate model control part 11e... Result receiving part 11f... Model verification Part 11f1 Accuracy calculation unit 11f2 Accuracy comparison unit 11g Model difference extraction unit 12 AI calculation unit 12a Model calculation unit M1 Current model M2 Candidate model M3 Shared model D1 Current model difference D2 Candidate model difference 12b AI calculation control unit 13 Storage unit 13a State table 13b Model table 13c Parameter table 13d Model difference table 2 Camera 3 Lider 4... Heterogeneous sensors 41... Fusion unit 5... Display panel I1... External sensor information I2... Vehicle sensor information I2a... Vehicle speed information

Claims

An in-vehicle arithmetic processing device that is mounted in a vehicle and performs arithmetic using a trained model,
a calculation unit that performs calculations using the first trained model;
a state estimating unit that acquires the position of the vehicle or the state inside and outside the vehicle and estimates the state;
a model selection unit that selects a second trained model based on the state;
Using the second trained model, the calculation result when the calculation is performed in parallel with the first trained model is compared with the calculation result of the first trained model, and based on the comparison result a model switching verification unit that determines whether to switch from the first trained model to the second trained model,
When the model switching verification unit determines to switch to the second trained model, the computing unit performs the computation by switching from the first trained model to the second trained model. An in-vehicle processing unit characterized by:
In the in-vehicle processing device according to claim 1,
The model switching verification unit includes a model accuracy calculation unit that calculates the accuracy of the first trained model and the second trained model using values obtained based on different sensor information as correct recognition results. Characterized in-vehicle processing unit.
In the in-vehicle processing device according to claim 1,
The model switching verification unit includes a model accuracy calculation unit that calculates the accuracy of the recognition result of the second trained model using the result of calculation of the first trained model as a correct recognition result,
An in-vehicle arithmetic processing device, characterized in that if the accuracy of a second learned model is equal to or higher than a threshold, switching to a more accurate model is determined.
In the in-vehicle arithmetic processing device according to claim 3,
The model accuracy calculation unit includes a result prediction unit that predicts a current recognition result from a result of calculating the first trained model in the past,
An in-vehicle arithmetic processing unit, wherein the accuracy of the second model is calculated using the recognition result predicted by the result prediction unit as the correct recognition result.
In the in-vehicle processing device according to claim 1,
a difference extraction unit that extracts a difference channel between the first trained model and the second trained model from the second trained model;
The calculation unit performs the calculation using the channel of the difference,
When the model switching verification unit determines to switch to the second trained model, the computing unit selects the second trained model among the first trained models and the first trained model. An in-vehicle processing unit, wherein the calculation is performed using a model excluding a channel of difference from the model.
In the in-vehicle processing device according to claim 1,
The model switching verification unit has a travel history reference unit that refers to travel history such as past route information and travel time,
An in-vehicle arithmetic processing unit, characterized in that, when current driving is in the same environment as past driving, model switching is similarly determined based on the past driving history.
In the in-vehicle processing device according to claim 1,
The model selection unit selects a plurality of second trained models, and the model switching verification unit determines the most accurate second trained model from the plurality of second trained models based on the comparison result. Characterized in-vehicle processing unit.
An arithmetic processing method that is mounted on a vehicle and performs arithmetic using a trained model,
an operation step of performing an operation using the first trained model;
a state estimation step of estimating the position of the vehicle or the acquired state inside and outside the vehicle;
a model selection step of selecting a second trained model based on the state;
Using the second trained model, the calculation result when the calculation is performed in parallel with the first trained model is compared with the calculation result of the first trained model, and based on the comparison result a model switching verification step of determining whether to switch from the first learned model to the second learned model;
a second calculation step of performing the calculation by switching from the first trained model to the second trained model when the model switching verification unit determines to switch to the second trained model; An arithmetic processing method characterized by: