JP2023142113A - Information processing apparatus and program - Google Patents

Abstract

To provide an information processing apparatus and a program that can support a learned model in which data acquired under a variety of environments is processed.SOLUTION: An information processing apparatus 14 acquires image data which is input data. The information processing apparatus 14 acquires a first learned model and a second learned model. The information processing apparatus 14 acquires first output data output from the first learned model by inputting the image data into the first learned model. The information processing apparatus 14 acquires second output data output from the second learned model by inputting the image data into the second learned model. The information processing apparatus 14 generates change data for changing the first output data on the basis of the second output data. The information processing apparatus 14 controls a camera 12 which is a control object, on the basis of the generated change data.SELECTED DRAWING: Figure 1

Description

The present invention relates to an information processing device and a program.

2. Description of the Related Art Conventionally, a technique is known that improves the accuracy of object identification by providing optimal data for subsequent signal processing for object identification (see, for example, Patent Document 1). The technology disclosed in Patent Document 1 changes the parameters applied to the sensor when acquiring the image data when outputting the identification result of the image data output from the sensor, and the sensor acquires the image data based on the changed parameters. Set the parameters so that

JP 2021-144689 Publication

For example, consider a case where a signal processing device is equipped with a trained model that outputs output data corresponding to input data when input data is input. In this case, the trained model installed in the signal processing device is a given model, and it may not be allowed to change the trained model.

For example, consider a case where the input data is image data captured by a camera mounted on a vehicle. In this case, the image of the front of the vehicle captured by the camera may vary depending on the environment. Therefore, for example, it is difficult to process both image data captured at night and image data captured during the day using one object identification process or learned model.

The present invention has been made in view of the above problems, and an object of the present invention is to provide an information processing device and a program that can support a trained model that processes data acquired under various environments.

In order to achieve the above object, an information processing device according to the present invention includes: a data acquisition unit that acquires input data; a model acquisition unit that acquires a first trained model and a second trained model from a storage unit; By inputting the input data acquired by the data acquisition unit to the first trained model acquired by the model acquisition unit, first output data output from the first trained model is acquired. By inputting the input data acquired by the first control unit and the data acquisition unit to the second trained model acquired by the model acquisition unit, the second trained model output from the second trained model a second control unit that acquires two output data; and a change for changing the first output data acquired by the first control unit based on the second output data acquired by the second control unit. The first learned model includes a change unit that generates data, and a third control unit that controls a controlled object based on the change data generated by the change unit, and the first learned model The model is trained in advance based on first learning data that is a combination of the input data input to the model and the first correct answer data of the first output data output from the first trained model. The second trained model is a trained model, and the second trained model has a relationship between the input data input to the first trained model and the first output data output from the first trained model. Learning that has been trained in advance so that the first output data output by the first trained model approaches the first correct data based on second learning data that is a combination of data and second correct data. This is an information processing device that is a complete model.

The controlled object according to the present invention is a camera, the input data is image data captured by the camera, and the second output data output from the second learned model is a camera parameter or a camera parameter. The related data represents identification data of a calculation method, and the third control unit sets the camera parameters corresponding to the related data for the camera that is the controlled object, and the third control unit sets the camera parameters corresponding to the related data, and By controlling to change the characteristics of the image data, it is possible to control to change the output data output from the first trained model.

The second output data output from the second trained model according to the present invention is the same type of data as the first output data output from the first trained model, and the third control unit , based on the second output data output from the second trained model, control is performed to modify the first output data output from the first trained model, which is the controlled object. can do.

The data acquisition unit according to the present invention acquires first input data acquired by a first sensor and second input data acquired by a second sensor as the input data, and the first control unit , the first output output from the first trained model by inputting the first input data acquired by the data acquisition unit to the first trained model acquired by the model acquisition unit. data, and the second control unit inputs the second input data acquired by the data acquisition unit to the second learned model acquired by the model acquisition unit. The third control unit acquires the second output data output from the trained model, and the third control unit controls the second output data that is the controlled object based on the second output data output from the second trained model. Control may be performed to modify the first output data output from the first trained model.

The program of the present invention includes a data acquisition unit that acquires input data, a model acquisition unit that acquires a first trained model and a second trained model from a storage unit, and the input data acquired by the data acquisition unit. a first control unit that acquires first output data output from the first trained model by inputting data to the first trained model acquired by the model acquisition unit; a second control unit that acquires second output data output from the second trained model by inputting the acquired input data to the second trained model acquired by the model acquisition unit; a changing unit that generates change data for changing the first output data acquired by the first control unit based on the second output data acquired by the second control unit; A program for functioning as a third control unit that controls a controlled object based on the generated change data, wherein the first trained model is input to the first trained model. The trained model is trained in advance based on first learning data that is a combination of the input data and first correct data of the first output data output from the first trained model, and The second trained model includes the input data input to the first trained model and second correct data of related data related to the first output data output from the first trained model. The program is a trained model that has been trained in advance so that the first output data outputted by the first trained model approaches the first correct data based on second learning data that is a combination of be.

According to the present invention, it is possible to support trained models that process data acquired under various environments.

FIG. 1 is a schematic diagram showing an example of the configuration of an information processing device according to a first embodiment. FIG. 2 is a diagram illustrating a configuration example of a computer of an information processing device. FIG. 1 is a diagram for explaining an overview of an information processing system. FIG. 3 is a diagram for explaining image data and camera parameters. FIG. 3 is a diagram illustrating an example of an information processing routine according to the first embodiment. FIG. 2 is a schematic diagram showing an example of the configuration of an information processing device according to a second embodiment. FIG. 2 is a diagram for explaining an overview of the first embodiment. FIG. 2 is a diagram for explaining an overview of the first embodiment. FIG. 7 is a diagram for explaining an overview of a second embodiment. FIG. 7 is a diagram for explaining an overview of a second embodiment. FIG. 7 is a diagram showing an example of an information processing routine according to the second embodiment.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

The present embodiment will be described using an example in which a camera is attached to a vehicle and the camera mounted on the vehicle sequentially captures images in front of the vehicle.

<First embodiment>
With reference to FIG. 1, an information processing system according to a first embodiment will be described. FIG. 1 shows a schematic diagram of an information processing system 10 according to the first embodiment.

As shown in FIG. 1, the information processing system 10 includes a camera 12 and an information processing device 14.

The camera 12 is installed in a vehicle, which is an example of a moving object. The camera 12 captures an image in front of the vehicle. The camera 12 includes a sensor device 12A that includes an optical system such as a lens and an image sensor that outputs a signal according to the intensity of received light, and a so-called image engine that converts the signal output by the sensor device 12A into an image file. It is equipped with a processor 12B.

The information processing device 14 is realized by a computer including a CPU 51, a memory 52 as a temporary storage area, and a nonvolatile storage section 53, as shown in FIG. The computer of the information processing device 14 also has an input/output interface (I/F) 54 to which external devices and output devices are connected, and a read/write (R/W) that controls reading and writing of data to and from a recording medium 59. ) section 55. The computer also includes a network I/F 56 connected to a network such as the Internet. The CPU 51, memory 52, storage section 53, input/output I/F 54, R/W section 55, and network I/F 56 are connected to each other via a bus 57.

The storage unit 53 can be realized by a hard disk drive (HDD), solid state drive (SSD), flash memory, or the like. The storage unit 53 as a storage medium stores a program for making the computer function. The CPU 51 reads the program from the storage unit 53, expands it into the memory 52, and sequentially executes the processes included in the program.

This information processing device 14 will be explained in terms of functional blocks divided into functional implementation means determined based on hardware and software. As shown in FIG. It includes a data storage section 142, a model acquisition section 143, a first control section 144, a second control section 145, a change section 146, and a third control section 147.

The data acquisition unit 140 acquires image data captured by the camera 12 at each time. Image data is an example of input data of the present invention.

The trained model storage unit 141 stores a first trained model and a second trained model.

FIG. 3 shows a diagram for explaining an overview of the information processing system 10 of the first embodiment. As shown in FIG. 3, in the information processing system 10 of this embodiment, camera parameters of the camera 12 are set. Specifically, in the information processing system 10 of the present embodiment, when image data captured by the camera 12 is input to the first trained model M1, the first trained model M1 receives the data as an example of the first output data. Output object identification results. The object identification result is the identification result of an object appearing in the image data. For example, the first trained model M1 outputs an object identification result as shown in FIG.

FIG. 4 shows a diagram for explaining image data and camera parameters. As shown in FIG. 4, no object is recognized in FIG. 4A, the number of detections is 0, and the reliability meter is 0. Note that the exposure time of the camera 12 in FIG. 4(A) is "exposure time 1" representing a predetermined exposure time. Further, in FIG. 4(B), the object is recognized, the number of detections: 2, and the reliability meter: 168. Note that the exposure time of the camera 12 in FIG. 4(B) is "exposure time 2". Further, in FIG. 4C, the object is recognized, the number of detections: 2, and the reliability meter: 193. Note that the exposure time of the camera 12 in FIG. 4(C) is "exposure time 3". Further, in FIG. 4(D), the object is recognized, the number of detections: 1, and the reliability meter: 93. Note that the exposure time of the camera 12 in FIG. 4(D) is "exposure time 4".

As shown in FIG. 4, if the exposure time, which is a camera parameter of the camera 12, is different, the image data captured by the camera 12 will be different, and the object identification results obtained by the first trained model M1 will also be different. becomes. Among FIGS. 4(A) to 4(D), the most preferable identification result is FIG. 4(C), and it can be said that the exposure time 3 in FIG. 4(C) is an appropriate camera parameter.

Therefore, the information processing system 10 of this embodiment sets the camera parameters of the camera 12 using the second trained model M2 so that the object identification result output from the first trained model M1 approaches the correct answer. Thereby, it is possible to appropriately support the first trained model M1 that processes image data acquired under various environments. This will be explained in detail below.

The first trained model M1 includes image data input to the first trained model M1 and correct data of object identification results (hereinafter simply referred to as first output data outputted from the first trained model M1). Machine learning has been performed in advance based on the first learning data, which is a combination of the "first correct data" (also referred to as "first correct data").

The second trained model M2 includes image data input to the first trained model M1 and correct answer data (hereinafter simply " Machine learning is performed in advance so that the object identification result output by the first trained model M1 approaches the first correct data based on the second learning data, which is a combination of the second correct data and the second correct data. . The related data related to the object identification result is identification data of the camera parameters themselves or the calculation method of the camera parameters. In this embodiment, an example will be described in which the related data is identification data of a camera parameter calculation method.

Note that the second correct answer data of the related data included in the second learning data of this embodiment is the correct answer data of the related data regarding the object identification result output from the first learned model M1.

For example, among FIGS. 4(A) to (D), (A) has a reliability meter of 0, (B) has a reliability meter of 168, and (C) has a reliability meter of 193. In (D), the reliability meter is 93. When the image data of (B) and (C) are input to the first trained model M1, the vehicle ahead is detected, while the image data of (A) and (D) are input to the first trained model M1. If it is input to M1, it can be seen that detection of the vehicle ahead has failed. In the present embodiment, the camera parameters themselves when the image data of FIGS. 4A to 4D were captured or identification data of the camera parameter calculation method are included in the second learning data as related data. Since the second learning data includes the camera parameters themselves when the image data of (A) and (D) were captured or the identification data of the camera parameter calculation method, the second learned model M2 is , the first trained model M1 outputs identification data of camera parameters themselves or camera parameter calculation methods that improve camera parameters when image data is captured, which is difficult for the first trained model M1.

The data storage unit 142 stores various data. Specifically, the data storage unit 142 stores image data captured by the camera 12, object identification results output from the first trained model M1, camera parameters themselves output from the second trained model M2, or the camera Identification data of the parameter calculation method and camera parameters based on the selected camera parameter calculation method are stored. These various data are used in each section described later.

The model acquisition unit 143 reads out the first trained model M1 and the second trained model M2 stored in the trained model storage unit 141.

The first control unit 144 inputs the image data acquired by the data acquisition unit 140 to the first trained model M1 acquired by the model acquisition unit 143, thereby generating an object output from the first trained model M1. Obtain identification results. The object identification results obtained by the first control unit 144 are used for various vehicle controls.

The second control unit 145 inputs the image data acquired by the data acquisition unit 140 to the second trained model M2 acquired by the model acquisition unit 143, thereby controlling the camera output from the second trained model M2. Obtain the identification data of the parameter calculation method.

The changing unit 146 changes camera parameters, which are change data for changing the object identification result acquired by the first control unit 144, based on the identification data of the camera parameter calculation method acquired by the second control unit 145. generate.

Specifically, when A, B, and C exist as camera parameter calculation methods, the changing unit 146 determines which camera parameter calculation method identification data output from the second trained model M2 is used. The camera parameter calculation method is selected based on whether the camera parameter calculation method is indicated. For example, if the identification data of the camera parameter calculation method output from the second learned model M2 represents A: 0.8, B: 0.1, C: 0.1, the changing unit 146 selects camera parameter calculation method A with the highest probability. Then, the changing unit 146 calculates the camera parameter _PA using the camera parameter calculation method A.

The third control unit 147 controls the camera 12, which is the object to be controlled, based on the camera parameters generated by the change unit 146.

Specifically, the camera parameter _PA generated by the changing unit 146 is set for the camera 12. Note that the camera parameter _PA may be a parameter of either the sensor device 12A of the camera 12 or the preprocessor 12B. As a result, the characteristics of the image data captured by the camera 12 are changed, so that control can be performed to change the identification result output from the first trained model M1.

<Action of the information processing system 10>

Next, the operation of the information processing system 10 according to this embodiment will be explained. When the information processing system 10 mounted on the vehicle is activated and the camera 12 starts capturing an image in front of the vehicle, the information processing device 14 executes the information processing routine shown in FIG. 5. Note that the information processing routine shown in FIG. 5 is executed every time new image data is captured.

In step S100, the data acquisition unit 140 acquires image data captured by the camera 12.

In step S102, the model acquisition unit 143 acquires the first trained model M1 and the second trained model M2 stored in the trained model storage unit 141.

In step S104, the first control unit 144 inputs the image data obtained in step S100 to the first trained model M1 obtained in step S102, thereby generating an object that is output from the first trained model M1. Obtain identification results.

In step S106, the second control unit 145 inputs the image data obtained in step S100 to the second trained model M2 obtained in step S102, thereby controlling the camera output from the second trained model M2. Obtain the identification data of the parameter calculation method.

In step S108, the changing unit 146 specifies the camera parameter calculation method indicated by the identification data, based on the identification data of the camera parameter calculation method acquired in step S106. The changing unit 146 then generates camera parameters using the specified camera parameter calculation method.

In step S110, the third control unit 147 sets the camera parameters generated in step S108 to the camera 12, and ends the information processing routine.

As is clear from the above description, the information processing device 14 of this embodiment acquires image data that is an example of input data. Further, the information processing device 14 acquires the first trained model M1 and the second trained model M2 from the trained model storage unit 141. The information processing device 14 acquires the object identification result output from the first trained model M1 by inputting the image data to the first trained model M1. The information processing device 14 acquires identification data of the camera parameter calculation method output from the second trained model M2 by inputting the image data to the second trained model M2. The information processing device 14 generates camera parameters, which are change data for changing the object identification result, based on the identification data of the camera parameter calculation method. The information processing device 14 then controls the camera to set camera parameters based on the generated camera parameters. Note that the first trained model M1 is a combination of image data input to the first trained model M1 and first correct data that is the correct answer of the object identification result output from the first trained model M1. This is a learned model that has been trained in advance based on the first learning data. Further, the second trained model M2 is a camera parameter calculation method that is image data input to the first trained model M1 and related data related to the object identification result output from the first trained model M1. Based on the second learning data that is a combination with the second correct data that is the correct answer, the trained model M1 is trained in advance so that the object identification result output by the first trained model M1 approaches the first correct data. It's a model. This makes it possible to support trained models that process data acquired under various environments. Specifically, when the first trained model M1 is an object detector, camera parameters are set for the camera 12 such that the number of detected objects in the object identification results of the first trained model M1 is maximized. Accordingly, the accuracy of the object identification result output from the object detector can be improved.

Note that in order to install a system that outputs object identification results of image data on a vehicle, the system needs to be high-speed. In this regard, in the technique disclosed in Patent Document 1 (for example, FIG. 11 of Patent Document 1), the simple parameter controller 340 sets sensor parameters when the object identifier 332 obtains a detection result. Since the technology disclosed in Patent Document 1 has a configuration in which the sensor is controlled by the output from the simple parameter controller 340, it is necessary to make the simple parameter controller 340, which is a machine learning model, learn all kinds of scenes. . Therefore, with the technology disclosed in Patent Document 1, it is difficult to reduce the weight of the simple parameter controller 340, which is a machine learning model, to shorten the inference time and to speed up the processing. Furthermore, it is practically difficult to collect learning data with different sensor parameters in every possible scene.

For this reason, the second trained model M2 may be subjected to machine learning based only on learning data that improves camera parameters when image data, which the first trained model M1 is weak at, is captured. . In this case, for example, the number of detected objects in the object identification results is already large, and even if such data is included in the training data, the object identification results of the first trained model M1 will not be improved. It is not included in the data for use. According to this, it becomes possible to reduce the weight of the second learned model M2. Furthermore, since only such data is included in the learning data, it is possible to reduce the cost of collecting the learning data.

Furthermore, even if the first trained model M1 is a given one and cannot be changed, by changing the camera parameters for the camera 12, the output from the first trained model M1 can be changed. The accuracy of object identification results can be improved. In particular, trained models used as object detectors are being developed individually at various locations, and it is often difficult for third parties to modify or improve them. Furthermore, since any changes made to the trained model as an object detector will require re-evaluation of the signal processing device itself, there may be circumstances in which such changes are desired to be avoided as much as possible. In contrast, according to the information processing system 10 of the present embodiment, even if the first trained model M1 is a given model and cannot be changed, camera parameters for the camera 12 can be changed. By changing, the accuracy of the object identification result output from the first learned model M1 can be improved.

<Second embodiment>
Next, a second embodiment will be described. Note that, among the configurations of the second embodiment, parts similar to those of the first embodiment are designated by the same reference numerals, and description thereof will be omitted. In the information processing system 10 of the second embodiment, the second output data output from the second trained model M2 is the same type of data as the first output data output from the first trained model M1. Furthermore, the information processing system 10 of the second embodiment corrects the first output data output from the first trained model M1 based on the second output data output from the second trained model M2. This embodiment differs from the first embodiment in the point of control.

An information processing system according to a second embodiment will be described with reference to FIG. 6. FIG. 6 shows a schematic diagram of an information processing system 210 according to the second embodiment.

As shown in FIG. 6, the information processing system 210 includes a visible light camera 212 that is an example of a first sensor, a far-infrared camera 13 that is an example of a second sensor, and an information processing device 214. ing.

The visible light camera 212 has the same configuration as the camera 12 of the first embodiment, and acquires visible image data that is image data of the reflection intensity of the visible light wavelength band. The visible image data is an example of first input data.

The far-infrared camera 13 is a sensor that detects radiation in the far-infrared region emitted by a subject, and acquires far-infrared image data. Far-infrared image data is an example of second input data. The far-infrared camera 13 includes a sensor device 13A that includes an image sensor, and a preprocessor 13B that is a so-called image engine that converts a signal output by the sensor device 13A into an image file.

If the information processing device 214 of the second embodiment is explained in terms of functional blocks divided into function implementation means determined based on hardware and software, as shown in FIG. section 241 , data storage section 142 , model acquisition section 243 , first control section 144 , second control section 245 , change section 246 , and third control section 247 .

The data acquisition unit 240 acquires visible image data acquired by the visible light camera 212 and far-infrared image data acquired by the far-infrared camera 13 as input data.

The trained model storage unit 241 stores a first trained model M1 and a second trained model M22 similar to those in the first embodiment.

7 to 10 are diagrams for explaining an overview of the information processing system 10 of the second embodiment. 7 and 8 are diagrams for explaining an overview of the information processing system 10 of the first embodiment described above. FIG. 7 is a diagram illustrating a learning phase in the information processing system 10 of the first embodiment, and FIG. 8 is a diagram illustrating an operation phase in the information processing system 10 of the first embodiment.

As shown in FIG. 7, in the learning phase in the first embodiment, identification data of the camera parameter calculation method is output from the second trained model M2 that is undergoing machine learning. Then, the difference between the identification data output from the second learned model M2 that is currently being trained and the identification data of the camera parameter calculation method that maximizes the reliability of the object identification result (for example, the number of detected objects) is determined. Machine learning is performed on the second trained model M2 so that the error is small.

In this case, the image data ImA captured when the camera parameters calculated by camera parameter calculation method A were set for the camera 12, and the camera parameters calculated by camera parameter calculation method B are The image data ImB captured when the settings were made for the camera 12 and the image data ImC captured when the camera parameters calculated by the camera parameter calculation method C were set for the camera 12 are the first It is input to the learned model M1. Then, from each object identification result output from the first learned model M1, the identification data of the camera parameter calculation method that maximizes the reliability of the object identification result (for example, the number of detected objects) is used as second correct data. is set as

As shown in FIG. 8, in the operation phase in the first embodiment, identification data of the camera parameter calculation method is output from the second trained model M2. For example, as shown in FIG. 8, when calculation method B is selected, camera parameters calculated by calculation method B are set to the camera 12. Thereby, the image data ImB captured using the parameters is input to the first learned model M1.

On the other hand, as shown in FIG. 9, in the learning phase in the second embodiment, visible image data is input to the first learned model M1, and a first object identification result is output as the object identification result. At this time, as shown in FIG. 9, the first object identification result and the object existing position as the first correct answer data are compared, and the object that was not detected by the first learned model M1 is detected. Weight data with weights added to them is output. Further, the far-infrared image data is input to the second trained model M22, and a second object identification result is output as the object identification result. Next, as shown in FIG. 9, the second object identification result is compared with the object existing position as the second correct answer data, and an error is calculated. Then, weight data is further added to the error. Thereby, the second trained model M22 is trained to detect objects that were not detected by the first trained model M1.

As shown in FIG. 10, in the operation phase in the second embodiment, visible image data is input to the first trained model M1, and the first object identification result is output from the first trained model M1. Ru. Furthermore, the far-infrared image data is input to the second trained model M22, and the second learned model M22 outputs a second object identification result. Then, the first object identification result output from the first trained model M1 and the second object identification result output from the second trained model M22 are integrated to form a final object identification result. In this regard, since the second trained model M22 is configured to preferentially detect objects that the first trained model M1 is not good at, the final object identification result is different from that of the first trained model M1. This may result in overcoming the difficulty in detecting objects. This will be explained in detail below.

The model acquisition unit 243 acquires the first trained model M1 and the second trained model M22 from the trained model storage unit 241.

Similarly to the first embodiment, the first control unit 144 inputs the visible image data acquired by the data acquisition unit 240 to the first trained model M1 acquired by the model acquisition unit 243, thereby controlling the first learned model M1. A first object identification result output from the learned model M1 is obtained. The first object identification result is an example of first output data.

The second control unit 245 inputs the far-infrared image data acquired by the data acquisition unit 240 to the second trained model M22 acquired by the model acquisition unit 243, thereby causing the far-infrared image data to be output from the second trained model M22. A second object identification result is obtained. The second object identification result is an example of second output data.

The changing unit 246 generates change data for changing the first object identification result obtained by the first control unit 144 based on the second object identification result obtained by the second control unit 245. Note that the changed data here is data obtained by changing the second object identification result so that it can be integrated with the first object identification result. For example, if the format of the first object identification result and the format of the second object identification result are different, it is difficult to integrate them, so the format of the second object identification result is changed to facilitate the integration. Generate changed data by changing .

The third control unit 247 corrects the first object identification result output from the first trained model M1, which is the controlled object, based on the second object identification result output from the second trained model M22. control to do so. For example, the third control unit 247 integrates the first object identification result and the second object identification result. Specifically, the third control unit 247 adds up each probability of the object region appearing in the image that is the first object identification result and each probability of the object region appearing in the image that is the second object identification result. By doing so, the final object identification result is generated.

<Operation of information processing system 210>

Next, the operation of the information processing system 210 according to the second embodiment will be explained. When the information processing system 210 mounted on the vehicle is activated and the visible light camera 212 and the far-infrared camera 13 start capturing images in front of the vehicle, the information processing device 214 executes the information processing routine shown in FIG. do. Note that the information processing routine shown in FIG. 11 is executed every time new image data is captured.

In step S200, the data acquisition unit 240 acquires visible image data captured by the visible light camera 212 and far-infrared image data captured by the far-infrared camera 13.

In step S202, the model acquisition unit 243 acquires the first trained model M1 and the second trained model M22 stored in the trained model storage unit 241.

In step S204, the first control unit 144 inputs the visible image data obtained in step S200 to the first trained model M1 obtained in step S202, thereby controlling the visible image data output from the first trained model M1. Obtaining a first object identification result.

In step S206, the second control unit 245 inputs the far-infrared image data obtained in step S200 to the second trained model M22 obtained in step S202, thereby causing the far-infrared image data to be output from the second trained model M22. A second object identification result is obtained.

In step S208, the changing unit 246 generates change data for changing the first object identification result obtained in step S204, based on the second object identification result obtained in step S206.

In step S210, the third control unit 247 selects the first object output from the first trained model M1 based on the second object identification result output from the second trained model obtained in step S206. Correct the identification results. Specifically, the third control unit 247 generates probabilities of object regions appearing in the image that are the first object identification results and probabilities of the object regions appearing in the images that are the second object identification results. By integrating the modified data, a final object identification result is generated.

In step S210, the third control unit 247 outputs the final object identification result generated in step S210 as a result, and ends the information processing routine.

Note that other configurations and operations of the information processing system 210 according to the second embodiment are the same as those in the first embodiment, and therefore, description thereof will be omitted.

As is clear from the above description, in the information processing device 214 of the second embodiment, the data output from the second trained model M22 is the same type of data as the data output from the first trained model M1. It is. The information processing device 214 obtains visible image data obtained by the visible light camera 212 and far-infrared image data obtained by the far-infrared camera 13 as input data. The information processing device 214 acquires the first object identification result output from the first trained model M1 by inputting the visible image data to the first trained model M1. The information processing device 214 acquires the second object identification result output from the second trained model M22 by inputting the far-infrared image data to the second trained model M22. Based on the second object identification result output from the second trained model M22, the information processing device 214 determines the first object identification result output from the first trained model M1, which is the controlled object. Control to fix. Thereby, it is possible to support a trained model that processes data acquired under various environments, and it is possible to improve the accuracy of object identification results using the trained model.

Further, according to the second embodiment, since the second trained model M22 that accurately detects objects that the first trained model M1 would falsely detect is generated, the By integrating the object identification results obtained by the second learned model M22 and the second object identification results obtained by the second trained model M22, the accuracy of the object identification results can be improved.

Furthermore, even if the first trained model M1 is a given one and cannot be changed, the first trained model M1 can accurately detect an object that the first trained model M1 would falsely detect. By generating the second trained model M22, it is possible to improve the accuracy of the object identification result output from the first trained model M1.

In addition, the second trained model M22 is machine learned using learning data related to objects that the first trained model M1 falsely detects, so it is lighter than the first trained model M1 ( In other words, it is a model with a relatively simple network structure.

Note that the present invention is not limited to the embodiments described above, and various modifications and applications can be made without departing from the gist of the present invention.

In the first embodiment, the sensor is the camera 12, but the sensor is not limited to this. Further, in the second embodiment, the first sensor is the visible light camera 212 and the second sensor is the far-infrared camera 13, but the present invention is not limited to this. The sensor may be of any kind; for example, the sensor may be a laser radar.

Further, in the second embodiment, the first sensor is the visible light camera 212 and the second sensor is the far-infrared camera 13, but the present invention is not limited to this. For example, the first sensor and the second sensor may be of the same type.

Further, in this embodiment, the case where the first trained model M1 is an object detector has been described as an example, but the present invention is not limited to this. The first trained model M1 may be any model as long as it performs some processing on sensor data that is the output of a sensor. For example, if the first trained model M1 is an object detector, the input data is image data, and the output data is the object identification result (for example, the number of detected objects, the probability (or likelihood) of object detection, (at least one of the location and the type of object). In this case, as described above, the number of detected objects can be used as the reliability, which is an arbitrary index.

Further, in the case where the first trained model M1 is an object evaluator, when human image data is input, a five-level evaluation result from not beautiful to beautiful may be output data. In this case, a five-level evaluation value can be used as an arbitrary index of reliability. For example, data corresponding to the one with the highest numerical value in the five-level evaluation may be used as the correct answer data.

Furthermore, in the case where the first trained model M1 is a speech recognizer, when speech signal data is input, the transcription result of the speech may be used as output data. In this case, the transcription error rate can be used as an arbitrary index of reliability. For example, data corresponding to transcription with the lowest error rate may be used as correct data.

Note that the program of the present invention can be provided by being stored in a recording medium.

10,210 Information processing system 12 Camera 13 Far-infrared camera 14,214 Information processing device 140,240 Data acquisition section 141,241 Learned model storage section 142 Data storage section 143,243 Model acquisition section 144 First control section 145,245 Second control unit 146, 246 Changing unit 147, 247 Third control unit 212 Visible light camera

Claims (5)

a data acquisition unit that acquires input data;
a model acquisition unit that acquires the first trained model and the second trained model from the storage unit;
By inputting the input data acquired by the data acquisition unit to the first trained model acquired by the model acquisition unit, first output data output from the first trained model is acquired. a first control section;
By inputting the input data acquired by the data acquisition unit to the second trained model acquired by the model acquisition unit, second output data output from the second trained model is acquired. a second control section;
a changing unit that generates change data for changing the first output data acquired by the first control unit based on the second output data acquired by the second control unit;
a third control unit that controls a controlled object based on the change data generated by the change unit,
The first trained model is a combination of the input data input to the first trained model and first correct data of the first output data output from the first trained model. It is a learned model that has been trained in advance based on the first learning data,
The second trained model includes second correct data of related data related to the input data input to the first trained model and the first output data output from the first trained model. is a trained model that has been trained in advance so that the first output data output by the first trained model approaches the first correct data based on second learning data that is a combination of
Information processing device. The controlled object is a camera,
The input data is image data captured by the camera,
The second output data output from the second trained model is the related data representing identification data of a camera parameter or a camera parameter calculation method,
The third control unit sets the camera parameters according to the related data to the camera, which is the object to be controlled, and controls the camera to change characteristics of image data captured by the camera. , controlling to change output data output from the first trained model;
The information processing device according to claim 1. The second output data output from the second trained model is the same type of data as the first output data output from the first trained model,
The third control unit corrects the first output data output from the first trained model, which is the control target, based on the second output data output from the second trained model. to control,
The information processing device according to claim 1. The data acquisition unit acquires first input data acquired by a first sensor and second input data acquired by a second sensor as the input data,
The first control unit inputs the first input data acquired by the data acquisition unit to the first trained model acquired by the model acquisition unit, thereby causing output from the first trained model. obtaining the first output data,
The second control unit inputs the second input data acquired by the data acquisition unit to the second trained model acquired by the model acquisition unit, thereby controlling output from the second trained model. obtain the second output data,
The third control unit corrects the first output data output from the first trained model, which is the control target, based on the second output data output from the second trained model. to control,
The information processing device according to claim 3. computer,
a data acquisition unit that acquires input data;
a model acquisition unit that acquires the first trained model and the second trained model from the storage unit;
By inputting the input data acquired by the data acquisition unit to the first trained model acquired by the model acquisition unit, first output data output from the first trained model is acquired. a first control unit;
By inputting the input data acquired by the data acquisition unit to the second trained model acquired by the model acquisition unit, second output data output from the second trained model is acquired. a second control unit;
a changing unit that generates change data for changing the first output data acquired by the first control unit based on the second output data acquired by the second control unit; A program for functioning as a third control unit that controls a controlled object based on the generated change data, the program comprising:
The first trained model is a combination of the input data input to the first trained model and first correct data of the first output data output from the first trained model. It is a learned model that has been trained in advance based on the first learning data,
The second trained model includes second correct data of related data related to the input data input to the first trained model and the first output data output from the first trained model. is a trained model that has been trained in advance so that the first output data output by the first trained model approaches the first correct data based on second learning data that is a combination of
program.