WO2022201776A1 - Information processing method and information processing device - Google Patents

Abstract

This information processing method verifies the consistency of a first image capturing device with a camera model by comparing the characteristics of the first image capturing device and the characteristics of the camera model by using first image data output from the first image capturing device that captures a specific subject and second image data output from the camera model to which two-dimensional input data, which is based on a measurement result obtained by measuring light from the specific subject, is input.

Description

Information processing method, information processing device

This technology relates to the technical field of information processing methods and information processing devices for verifying the consistency between the real environment and the simulation environment.

In the development of in-vehicle camera systems used for autonomous driving and safety support systems, evaluations using simulation environments are performed to evaluate safety in various driving environments. For example, by reproducing the driving environment with a CG (Computer Graphics) model, inputting data based on the CG model into a camera model imitating an actual camera, and performing recognition processing on the image data output from the camera model. Evaluation is performed using the obtained recognition result.
In Patent Document 1 listed below, input data for a camera model is generated by converting actual image data captured in a real environment into virtual image data.

U.S. Patent Application Publication No. 2019/0171223

By the way, since various virtual models such as driving environment models and optical models are used in the simulation process, the process of verifying consistency with the actual environment is complicated, and time and personnel costs tend to increase. be.
In a simulation using a camera model, there are various types of sensors used by the user, so it is necessary to incorporate a model suitable for each sensor used into the camera model. However, every time the camera model is changed, it is necessary to change the virtual image data to be input and to verify the matching, which leads to an increase in cost.

This technology was created in view of these problems, and aims to propose a method for efficiently verifying the consistency between the real environment and the simulation environment.

An information processing method according to the present technology inputs first image data output from a first imaging device that has captured an image of a specific subject, and two-dimensional input data based on measurement results of measuring light from the specific subject. and second image data output from the camera model that is used to compare the characteristics of the first imaging device and the characteristics of the camera model to compare the characteristics of the first imaging device and the camera model. It verifies consistency.
The comparison of the characteristics of the first imaging device and the camera model may be performed, for example, by comparing the first image data and the second image data, or by applying image recognition processing to the first image data. and the recognition result obtained by applying image recognition processing to the second image data.

In the information processing method according to the present technology, as preprocessing for performing consistency verification regarding a first imaging device as an in-vehicle camera and a camera model as a simulated camera, Calibration is performed by adjusting the yaw direction and pitch direction of the first imaging device so that the center of the first target placed at least two locations at the same height coincides with the center of the angle of view. is performed.
By performing such calibration, it is possible to verify the consistency between the vehicle-mounted camera system equipped with the vehicle-mounted camera and the simulator system equipped with the camera model.

FIG. 4 is a diagram showing a flow performed in an on-vehicle camera system for consistency verification; FIG. 10 is a diagram showing a flow executed in a simulator system for consistency verification; It is a figure which shows the structural example of a camera model. FIG. 10 is a flow chart showing the overall flow of consistency verification; FIG. FIG. 10 is a flowchart showing a specific processing example of camera model consistency verification; FIG. FIG. 4 is a schematic diagram showing a state in which a luminance box is photographed by the first imaging device; FIG. 10 is a diagram showing an example of a verification area set in RAW data output from the first imaging device; FIG. 4 is a diagram showing an example of RAW data output from the first imaging device; FIG. 4 is a diagram showing a state in which a measurement area set in a luminance box is measured using a spectral radiance meter; It is a figure which shows an example of the input data using a spectral radiance value as input data to a camera model. FIG. 4 is a diagram showing an example of RAW data output from a camera model; It is a figure which shows an example of a color chart. It is a figure which shows an example of a gray chart. FIG. 4 is a diagram showing an example of a method of measuring spectral radiance values for color charts and gray charts. FIG. 17 is a flowchart showing a specific example of processing for calibrating the camera posture together with FIG. 16 ; FIG. 16 is a flowchart showing a specific example of processing for calibrating the camera posture together with FIG. 15 ; FIG. 4 is a schematic diagram showing a state in which two indicators are arranged in front of the vehicle; FIG. 4 is a schematic diagram showing a state in which indicators are spaced apart in the width direction of the vehicle; FIG. 10 is an explanatory diagram of calibration in the roll direction of the first imaging device; FIG. 4 is a schematic diagram showing a state in which indices for adjusting the depression angle of the first imaging device are arranged; FIG. 4 is a diagram showing an example of a filter included in the first imaging device, and this diagram is a diagram showing a Bayer array filter. FIG. 4 is a diagram showing an example of a filter provided in the first imaging device, and this diagram is a diagram showing a filter of RCCB arrangement; FIG. 4 is a diagram showing an example of a filter provided in the first imaging device, and this diagram is a diagram showing an RGBIR arrangement filter. FIG. FIG. 4 is a diagram showing an example of a filter included in the first imaging device, and this diagram is a diagram showing a complementary color filter; FIG. 1 is a block diagram of a computer device; FIG.

Hereinafter, embodiments according to the present technology will be described in the following order with reference to the accompanying drawings.
<1. Consistency verification >
<2. Process flow of each system>
<3. Consistency verification flow>
<4. Consistency Verification of Camera Model>
<5. Camera system calibration>
<6. Variation>
<7. Computer device>
<8. Summary>
<9. This technology>

<1. Consistency verification >
This embodiment will be described with reference to the accompanying drawings. In the present embodiment, consistency verification is performed between an in-vehicle camera system S1 equipped with a first imaging device 1 mounted on a vehicle 100 and a simulator system S2 equipped with a camera model 2 simulating an actual imaging device. is.

In order to realize driving support functions such as the collision mitigation brake function and the automatic driving function of the vehicle 100, it is necessary to verify the control algorithm of the vehicle 100. These control algorithms are improved by actually running the vehicle 100 under various driving environments, acquiring data and control results obtained from various sensors, and repeating examinations.

However, in order to prepare the actual environment for running the vehicle 100, a great cost such as securing a place is incurred.
Therefore, algorithms are generally evaluated using a simulation called MBD (Model Based Design).

In MBD, a driving environment model that simulates the actual driving environment, a sensor model that simulates an image sensor such as CMOS (Complementary Metal-Oxide Semiconductor) or CCD (Charge Coupled Device) possessed by the first imaging device 1, and the first An optical model or the like that simulates an optical member such as a lens provided in the imaging device 1 is used.

Therefore, in order to appropriately verify control algorithms using MBD, MBD
It is necessary to match the various models provided by

In this embodiment, a method for efficiently verifying the consistency between these various models and the actual environment will be described. Concretely, consistency verification between the in-vehicle camera system S1 and the simulator system S2 is performed.

<2. Process flow of each system>
FIG. 1 shows the flow of processing executed in the in-vehicle camera system S1 for matching verification, and FIG. 2 shows the flow of processing executed in the simulator system S2.

The in-vehicle camera system S1 uses the first imaging device 1 to capture an actual driving environment scene A1. The first imaging device 1 includes an optical member such as a lens, an image sensor, and the like.
A signal obtained by imaging by the first imaging device 1 is input to the camera signal processing model A2 as an imaging signal (RAW data).

The camera signal processing model A2 executes processing for generating RGB (Red, Green, Blue) images, white balance adjustment, sharpness adjustment, contrast adjustment processing, and the like. The captured image data output from the camera signal processing model A2 is input to the recognition model A3. Captured image data output from the camera signal processing model A2 is referred to as a first image G1.

The recognition model A3 performs a process of identifying and labeling the imaged subject based on the input image data. That is, the recognition model A3 performs processing for detecting the subject captured in the image. Objects to be detected include, for example, pedestrians, traffic lights, signs, and vehicles. The detection result of the recognition model A3 is input to the subsequent integrated control model A4.

The integrated control model A4 controls the vehicle 100 using recognition results based on image data. This realizes a collision mitigation braking function, an ACC (Adaptive Cruise Control) function, and various warning functions.

On the other hand, in the simulator system S2, environment setting B1, scenario generation B2 and rendering B3 are performed in order to simulate the actual driving environment scene A1.

In the environment setting B1, map information corresponding to the actual driving environment scene A1 in which the vehicle 100 actually runs, and subjects such as other vehicles other than the vehicle 100 placed on the driving route, pedestrians, signs, traffic lights, road surfaces, etc. A simulated data set is set.
These data sets include shape information, color information, material information, spectral reflectance information, etc. of polygons and the like used in ray tracing in the subsequent rendering B3.

In the subsequent scenario generation B2, a travel plan that simulates the travel mode of the vehicle 100 is set. The travel plan includes not only the route that the vehicle 100 travels, but also operations such as steering wheel operation and accelerator operation during travel. This can also be rephrased as the behavior of vehicle 100 based on the driving operation.

The driving plan generated in the scenario generation B2 is input to the subsequent rendering B3. In rendering B3, a 3DCG (3-Dimensional CG) model is generated by executing rendering processing based on ray tracing. An output result of rendering processing is input to the camera model 2 .

Here, the output data of the rendering process is the spectral radiance value for each pixel developed in a two-dimensional array corresponding to the two-dimensional pixel array of the image sensor model provided in the camera model 2. Furthermore, the spectral radiance values are output at regular time intervals corresponding to the frame rate of the first imaging device 1 . For example, when the imaging of the first imaging device 1 to be simulated by the camera model 2 is performed at 30 fps (frames per second), the spectral radiance value of each pixel developed in a two-dimensional array is 30 times.

The camera model 2 is a model for simulating the imaging function of the first imaging device 1 of the in-vehicle camera system S1, and as shown in FIG. 3, includes an optical model B7 and a sensor model B8.

The optical model B7 is a model imitating various lens systems, a shutter mechanism, an IR (Infrared) cut filter, etc., which are optical members of the first imaging device 1, and the spectral radiance value input to the camera model 2 is , the effect of the optical member is calculated, and processing is performed to convert it into spectral irradiance for each pixel. Specifically, the optical model B7 performs projection correction processing, aperture correction processing, shading correction processing, IR cut filter correction processing, and the like.

The spectral irradiance for each pixel is input to a sensor model B8 that imitates the image sensor provided in the first imaging device 1. The sensor model B8 includes a filter model simulating various optical filters such as color filters provided in the image sensor, a pixel array model simulating light receiving operation and readout operation in the pixel unit, and the like. The effect of the color filter and the pixel portion on the illuminance is calculated, and output to the subsequent camera signal processing model B4 as RAW data for each pixel. Specifically, the sensor model B8 performs color filter correction processing, photoelectric conversion processing, AD (Analog to Digital) conversion processing, and the like.

Here, the RAW data output from the first imaging device 1 of the in-vehicle camera system S1 and the RAW data output from the camera model 2 of the simulator system S2 have the same data format.
Therefore, the camera signal processing model A2 of the vehicle-mounted camera system S1 to which RAW data is input and the camera signal processing model B4 of the simulator system S2 can be the same. Note that the captured image data output from the camera signal processing model B4 is referred to as a second image G2.

Similarly, the recognition model A3 of the in-vehicle camera system S1 and the recognition model B5 of the simulator system S2 are the same, and the integrated control model A4 of the in-vehicle camera system S1 and the integrated control model B6 of the simulator system S2 are the same.

Therefore, if the detection results input to the integrated control model A4 and integrated control model B6 are the same, the data output from the integrated control model A4 and integrated control model B6 will also match.

Therefore, the flow from the environment setting B1 to the recognition model B5 in the simulator system S2 (broken line portion in FIG. 2) is the flow from the actual driving environment scene A1 to the recognition model A3 in the in-vehicle camera system S1 (broken line portion in FIG. 1). It is necessary to appropriately simulate the Verification of this point is verification of conformity between the in-vehicle camera system S1 and the simulator system S2.

In this embodiment, it is determined whether or not the intermediate data and the like in each system are consistent as the consistency verification.
Specifically, RAW data output from the first imaging device 1 in the in-vehicle camera system S1, RAW data output from the camera model 2 in the simulator system S2, and output from the recognition model A3 in the in-vehicle camera system S1 Consistency verification is performed using the detection result and the detection result output from the recognition model B5 in the simulator system S2.

By performing consistency verification in this way, it is possible to perform the same verification in the simulator system S2 as in the case of verifying the algorithm in the in-vehicle camera system S1 including the first imaging device 1 as the actual imaging device. becomes.

<3. Consistency verification flow>
The flow of consistency verification will be described with reference to FIG. The device that performs each process for verifying consistency may be a computer device included in the simulator system S2, or may be a device separate from the computer device included in the simulator system S2.
The computer device is configured to include an arithmetic processing section including a CPU (Central Processing Unit) or the like in order to execute each process for consistency verification.

In step S101 of FIG. 4, the arithmetic processing unit of the computer device verifies whether or not the camera model 2 appropriately simulates the optical members and image sensor of the first imaging device 1 as an actual device. That is, the arithmetic processing unit verifies the consistency between the camera model 2 and the first imaging device 1 .

The matching verification between the camera model 2 and the first imaging device 1 determines whether or not the RAW data output from the camera model 2 and the RAW data output from the first imaging device 1 match. Note that it is difficult to completely match the RAW data output from the first imaging device 1, which is an actual device, with the RAW data output from the camera model 2 using 3DCG. Consistency is verified using statistical data on the RAW data. Specific details of processing for verifying consistency of RAW data will be described later.

In step S<b>102 , the arithmetic processing unit performs determination processing based on the result of comparison between the RAW data output from the camera model 2 and the RAW data output from the first imaging device 1 . If the difference between the RAW data does not fall within the predetermined range, that is, if it is determined that the matching verification fails, the arithmetic processing unit performs factor analysis and model correction processing in step S103.

In the process of step S103, factor analysis is performed, as well as a process of presenting the result of factor analysis to the operator, a process of presenting a model to be corrected, and the like. Further, a process of changing and overwriting the variables given to the model may be performed in step S103.
Alternatively, the arithmetic processing unit may perform a process of presenting data used for factor analysis to the worker in step S103, and the worker may perform factor analysis based on the presented data.

After finishing the factor analysis and model correction processing, the arithmetic processing unit returns to step S101 and verifies the consistency between the camera model 2 and the first imaging device 1 .

On the other hand, in step S102, when it is determined that the matching verification is passed, that is, when it is determined that the difference between the RAW data is within a predetermined range, the arithmetic processing unit detects the recognition model A3 of the in-vehicle camera system S1 and Consistency verification is performed using each detection result output from B5, which also outputs the recognition of the simulator system S2.

Specifically, in step S<b>104 , the arithmetic processing unit performs calibration for the orientation of the first imaging device 1 and the orientation of the camera model 2 . In the process of step S104, the posture of the first imaging device 1 is calibrated so that the first imaging device 1 captures an image in a predetermined direction. Then, the optical axis direction set in the camera model 2 is changed so as to match the optical axis direction of the first imaging device 1 .
Specific processing contents of step S104 will be described later.

In step S105, the arithmetic processing unit verifies the consistency of the camera system in the running environment. The matching verification of the camera system is to verify the matching between the dashed line portion shown in FIG. 1 in the in-vehicle camera system S1 and the dashed line portion shown in FIG. 3 in the simulator system S2.
Specifically, a process of confirming consistency between the subject detection result output from the recognition model A3 of the in-vehicle camera system S1 and the subject detection result output from the recognition model B5 of the simulator system S2 is executed.

The detection result is, for example, data in which information specifying the pixel area of the first image G1 input to the recognition model A3 and label information that is the result of classifying the subject imaged there are linked. Specifically, it is assumed that a label "car" is associated with a certain predetermined pixel area.

The detection result may include a plurality of pairs of pixel area information and label information for one first image G1.

The same applies to the detection results for the simulator system S2. That is, the detection result is obtained by linking the information specifying the pixel area of the second image G2 input to the recognition model B5 and the label information about the imaged subject.

Note that the detection result may contain information other than the label. For example, it may contain brightness information of pixels or pixel regions. Specifically, information for specifying a certain pixel area, label information for specifying a subject imaged in the area, and brightness information about the subject, for example, the maximum and minimum values of brightness. Information such as the average value and variance may be linked. If the subjects imaged in the specific regions in the first image G1 and the second image G2 are classified into the same category and the brightness of the imaged subjects has a similar tendency, the detection result obtained by the simulator system S2 is used. It is possible to verify the control algorithm in the same way as with the actual equipment. If there is an object detected by only one of the recognition models, if the classification results of the detected object are different, or if the tendency of the object such as brightness is different, the detection result obtained by the simulator system S2 is used. It can be judged that it is impossible to verify the control algorithm by using it in the same way as the actual machine.

In step S106, the arithmetic processing unit performs determination processing based on the result of comparison between the detection result output from the recognition model A3 and the detection result output from the recognition model B5. If both detection results are significantly different, that is, if it is determined that the consistency verification fails, the arithmetic processing unit performs factor analysis and model correction processing in step S107.
However, in the processes of steps S101 and S102, the consistency between the camera model 2 and the first imaging device 1 is ensured, so in step S107, the parts other than the parts indicated by the dashed lines in both FIG. 1 and FIG. factor analysis and model modification.

In the process of step S106, similarly to the process of step S103, factor analysis, process of presenting the factor analysis result to the worker, and process of presenting the model to be corrected may be performed, or the variables given to the model may be changed. A process of overwriting may be performed, or a process of presenting the data used for factor analysis to the worker may be performed to prompt the worker to perform the factor analysis work.

After completing the factor analysis and model correction processing in step S107, the arithmetic processing unit returns to step S105 to verify the consistency of the camera system in the driving environment.

<4. Consistency Verification of Camera Model>
The flow of processing for verification of matching of camera models will be described with reference to FIG. The flow of consistency verification shown in FIG. 5 shows each process of steps S101, S102, and S103 in FIG. 4, and specifically shows the specific process executed in step S101.

An example in which the arithmetic processing unit described above executes each processing shown in FIG. 5 will be described, but the present invention is not limited to this. 5 may be executed, or each process may be executed by a system configured including other devices.

In step S201 of FIG. 5, the arithmetic processing unit determines whether or not an instruction to start shooting has been received. For example, the instruction to start photographing may be input by an operator to a computer device having an arithmetic processing unit, or may be performed by a first imaging device 1 that detects that a subject to be photographed is positioned at a predetermined position within the angle of view of the first imaging device 1 . 1 may be input from the imaging device 1 .

The arithmetic processing unit repeatedly executes the processing of step S201 until an instruction to start shooting is received. The instruction to start photographing is given by the operator, for example, when predetermined conditions are met.
Satisfying a predetermined condition means, for example, completion of installation of a subject to be measured within the angle of view of the first imaging device 1 .

A specific description will be given with reference to FIG. FIG. 6 shows a state in which the luminance box 3 is arranged within the angle of view of the first imaging device 1. In this state, it is possible to image the luminance box 3 as the object to be measured. After installing the brightness box 3 at a predetermined position in front of the first imaging device 1, the operator instructs the arithmetic processing unit to start imaging.

The luminance box 3 has a luminous body disposed therein, and one surface is formed as a diffusion surface 3a for transmitting and diffusing light emitted from the luminous body disposed therein. The brightness box 3 is arranged so that the diffusion surface 3 a faces the first imaging device 1 .

Returning to the description of FIG.
The arithmetic processing unit transmits an imaging operation instruction to the first imaging device 1 in step S202. Thereby, the imaging operation in the first imaging device 1 is executed.

The arithmetic processing unit acquires the RAW data output from the first imaging device 1 in step S203. In the RAW data, a predetermined area is set as the verification area Ar.

Subsequently, in step S204, the arithmetic processing unit performs statistical processing for each verification area Ar set as a predetermined area in the RAW data.
A specific description will be given with reference to FIG. The RAW data is data corresponding to the two-dimensional arrangement of pixels formed in the image sensor of the first imaging device 1 . That is, as shown in FIG. 7, RAW data can be regarded as two-dimensional data. In the RAW data, which is two-dimensional data, a verification area Ar1 is set in the center, a verification area Ar2 is set near the upper left corner, a verification area Ar3 is set near the upper right corner, and a verification area Ar3 is set near the lower left corner. Area Ar4 is set, and verification area Ar5 is set near the lower right corner.

In the statistical processing of step S204, the average value, variance, etc. are calculated for each of the verification areas Ar1 to Ar5. The size of each verification area Ar is, for example, 100 pixels vertically and 100 pixels horizontally.

Here, if the image sensor has a Bayer array color filter or the like, the RAW data includes data Dr indicating the intensity of the red light component and the intensity of the green light component of the light received for each pixel. It contains data Dg and data Db indicating the intensity of the blue light component. Specifically, the RAW data is two-dimensional data in which data Dr, Dg, and Db are developed in a predetermined pattern as shown in FIG. Therefore, the verification areas Ar1 to Ar5 are also two-dimensional data in which the data Dr, Dg, and Db are developed vertically and horizontally.

Taking the verification area Ar1 as an example, in the statistical processing in step S204, the mean value and variance of the data Dr included in the verification area Ar1 are calculated, and the mean value and variance of the data Dg included in the verification area Ar1 are calculated. Then, the average value and variance of the data Db included in the verification area Ar1 are calculated.

That is, when calculating the average value and variance, six statistical data are calculated for one verification area Ar.

Return to the description of Fig. 5. By calculating the statistical data for each verification area Ar and for each color pixel, the arithmetic processing unit completes acquisition of data used for matching verification with respect to the first imaging device 1 as a real device.

Subsequently, the arithmetic processing unit executes the processes from step S205 onward, thereby starting acquisition of data used for consistency verification for the camera model 2 of the simulator system S2. Specifically, in step S205, the arithmetic processing unit determines whether or not preparation for measurement has been completed.

The state in which the measurement preparation is completed is, for example, a state in which the spectral radiance meter 4 is installed at a position facing the diffusion surface 3a of the luminance box 3, as shown in FIG. That is, it is determined that the preparation for measurement is completed when the spectral radiance value of the specific region can be measured using the spectral radiance meter 4 .
The process of step S205 is repeatedly executed until the preparation for measurement is completed.

If it is determined that the preparation for measurement has been completed, or if a user operation indicating that the preparation for measurement has been completed is accepted, the arithmetic processing unit acquires the measured spectral radiance value in step S206. Note that the spectral radiance meter 4 may be instructed to start measurement before step S206.

Note that the spectral radiance meter 4 does not measure the spectral radiance value of the entire brightness box 3, but measures the spectral radiance value of a narrow area with a minute solid angle. Therefore, in order to obtain spectral radiance values for comparison with the five verification areas Ar in FIG. It is necessary to measure the spectral radiance value for

Therefore, as shown in FIG. 9, the arithmetic processing unit performs the processing of step S206 in a state in which the optical axis of the spectral radiance meter 4 is aligned with the center of the measurement area Br1. wait until it's ready. Then, in response to the operator's operation of aligning the optical axis of the spectral radiance meter 4 with the center of the measurement area Br2, the arithmetic processing unit performs the processing of step S206 again. By repeating the processing of steps S205 and S206 in this way, the spectral radiance value for each measurement area Br is obtained.

It should be noted that, as described above, the measurement area Br is provided at a position corresponding to the verification area Ar. That is, the light passing through the central portion of the measurement area Br1 on the diffusion surface 3a affects the RAW data located in the central portion of the verification area Ar1 in the RAW data shown in FIG. It affects the measured spectral radiance values.

In step S207 of FIG. 5, the arithmetic processing unit generates input data to the camera model 2 using the acquired spectral radiance values. The generated data is two-dimensional data corresponding to the pixel array of the sensor model B8 included in the camera model 2. FIG.
An example is shown in FIG. In the input data, the obtained spectral radiance values are arranged in corresponding areas Cr1 to Cr5 corresponding to the respective measurement areas Br. Specifically, the spectral radiance values obtained from the measurement area Br1 are arranged in the corresponding area Cr1, the spectral radiance values obtained from the measurement area Br2 are arranged in the corresponding area Cr2, and the corresponding area Cr3 The spectral radiance values obtained from the measurement area Br3 are arranged, the spectral radiance values obtained from the measurement area Br4 are arranged in the corresponding area Cr4, and the spectral radiance values obtained from the measurement area Br5 are arranged in the corresponding area Cr5. value is placed.

In addition, dummy data is used for areas other than the corresponding area Cr in the input data using the spectral radiance values. The area where dummy data is placed is an area that is not used for consistency verification. The dummy data may be zero values or other values.

The arithmetic processing unit acquires the RAW data output from the camera model 2 in step S208 of FIG. In the RAW data output from the camera model 2, the spectral radiance value input to the camera model 2 is subjected to calculation by the optical model B7 and converted into spectral irradiance, and further calculation is performed by the sensor model B8. It has been applied.

A predetermined area in the RAW data is provided as a verification area Ar'. For example, as shown in FIG. 11, the area of the RAW data output from the pixel area to which the spectral radiance values arranged in the corresponding area Cr1 are input is set as the verification area Ar'1. Similarly, the area of the RAW data output from the pixel area to which the spectral radiance value arranged in the corresponding area Cr2 is input is set as the verification area Ar'2, and the spectral radiance value arranged in the corresponding area Cr3 is input. The area of the RAW data output from the pixel area arranged in the corresponding area Cr4 is designated as the verification area Ar′3, and the area of the RAW data output from the pixel area into which the spectral radiance value arranged in the corresponding area Cr4 is inputted is designated as the verification area Ar '4, and the area of the RAW data output from the pixel area to which the spectral radiance value arranged in the corresponding area Cr5 is input is set as the verification area Ar'5.

In step S209 of FIG. 5, the arithmetic processing unit performs statistical processing for each verification area Ar'. For example, the average value, variance, etc. are calculated for each of the verification areas Ar′1 to Ar′5. The size of the verification area Ar'1 is the same size as the verification area Ar in FIG.
Also, the RAW data output from the sensor model B8 is two-dimensional data in which the data Dr, Dg, and Db are expanded in the same manner as the RAW data output from the first imaging device 1 (see FIG. 8). In the statistical processing of step S209, average values and variances are calculated for each of data Dr, data Dg, and data Db for one verification area Ar'.
That is, when calculating the average value and variance, six statistical data are calculated for one verification area Ar'.

In step S210, the arithmetic processing unit performs comparison processing between the verification area Ar and the verification area Ar'. Specifically, the statistical data calculated in step S204 for the verification area Ar1 and the statistical data calculated in step S209 for the corresponding verification area Ar' are compared.

When the statistical data of all the verification areas Ar and the verification areas Ar' match each other, or when the difference is small, the camera model 2 can appropriately simulate the first imaging device 1. can be determined.

In step S102, the arithmetic processing unit performs branch processing based on the comparison result. Specifically, as a result of the comparison processing in step S210, if it is determined that the difference between all the statistical data is small, it is determined that the consistency between the first imaging device 1 and the camera model 2 is ensured. The series of processing shown in FIG. 5 is terminated assuming that the test has passed.

On the other hand, if it is determined that the difference is greater than the threshold value for at least some of the statistical data, it is determined that the matching between the first imaging device 1 and the camera model 2 is not sufficient, and in step S103 factor analysis and various model correction processes are performed. , the process returns to step S201 again. That is, the series of processing shown in FIG. 5 is repeated until the consistency between the first imaging device 1 and the camera model 2 can be ensured.

Note that the threshold used in the branching process in step S102 may be different for each type of statistical data. Specifically, the threshold used when comparing mean values and the threshold used when comparing variances may be different values.

In addition, in the above-described example of verification of matching between the first imaging device 1 and the camera model 2, the brightness box 3 was used, but other items may be used.
For example, a color chart 5 as shown in FIG. 12 or a gray chart 6 as shown in FIG. 13 may be used instead of using the luminance box 3 as the object to be measured.

When the color chart 5 or the gray chart 6 is used as the object to be measured, a light source 7 for irradiating the color chart 5 or the gray chart 6 with light may be arranged as shown in FIG. That is, the light emitted from the light source 7 may be reflected by the color chart 5 or the gray chart 6 and input to the first imaging device 1 or the spectral radiance meter 4 .

<5. Camera system calibration>
An example of the processing executed by the arithmetic processing unit to calibrate the camera posture in step S104, which is a preliminary preparation for verifying the consistency of the camera system in the driving environment in step S105 of FIG. 4, is shown in FIGS. Description will be made with reference to FIG. 15 and 16 are shown as a connector C1.

15 and 16 may be the same as the arithmetic processing unit for executing each process shown in FIG. 4, or may be a different treatment device. good too.

First, before the arithmetic processing unit executes the process of step S301, the operator installs the index A and the index B at predetermined positions in front of the vehicle 100. FIG. Specifically, as shown in FIG. 17, the first imaging device 1 is mounted on the vehicle 100 so that the horizontal distance from the first imaging device 1 is the distance L1 and the height from the ground is the vehicle 100. An index A is installed so as to match the height Hc of the device 1 .

Next, as shown in FIG. 17, the operator sets the horizontal distance from the first imaging device 1 to the distance L2 longer than the distance L1, and the height from the ground to the height Hc. Set index B so that

After the state shown in FIG. 17 is established, in step S301 of FIG. 15, the arithmetic processing unit adjusts the yaw and pitch directions of the first imaging device 1 so that the center of the index A, the center of the index B, and the center of the angle of view overlap. make adjustments. This adjustment is realized, for example, by the arithmetic processing unit transmitting a control signal for adjusting the imaging direction of the first imaging device 1 to the first imaging device 1 .

Note that the arithmetic processing unit does not transmit a control signal to the first imaging device 1, but holds the first imaging device 1 in a state in which the imaging direction of the first imaging device 1 can be adjusted. A control signal may be transmitted to a device or a control device that controls the imaging direction of the first imaging device 1 . This is the same for other control signals to be described later.

By adjusting the yaw direction and the pitch direction so that the center of the index A and the center of the index B overlap with the center of the angle of view, the optical axis of the first imaging device 1 can be horizontally adjusted.

Then, before the arithmetic processing unit executes the process of step S302, the operator places the index C at a predetermined position. Specifically, as shown in FIG. 18, an index C is installed at a position moved in parallel in the vehicle width direction of the vehicle 100 from the position where the index A was installed. At this time, the height of the center of the index C is made to be the height Hc.

In step S302 of FIG. 15, the arithmetic processing unit performs control for adjusting the roll direction of the first imaging device 1 so that the center of the index C is positioned on the horizontal line passing through the center of the angle of view of the first imaging device 1. Send a signal.
Specifically, as shown in FIG. 19, the center of the index A is imaged at the center of the angle of view of the first imaging device 1, and the center of the index C is imaged at an intermediate position in the vertical direction of the angle of view. Adjust the roll direction as follows.

In step S303 of FIG. 15, the arithmetic processing unit determines whether or not to provide the posture of the first imaging device 1 with a depression angle. Whether or not to provide the depression angle may be set by the operator, or may be automatically set according to subjects such as pedestrians and preceding vehicles imaged at the angle of view of the first imaging device 1. .

When providing the angle of depression, for example, as shown in FIG. 20, the operator sets an index D at a predetermined position. The index D has a distance L3 from the first imaging device 1 and a height Hd at the center position. Any value may be set for the distance L3 and the height Hd.

If the depression angle is provided, the arithmetic processing unit adjusts the pitch direction of the first imaging device 1 in step S304 of FIG. Sends a control signal for adjusting the At this time, no adjustments are made in the yaw or roll direction. Thereby, the imaging direction of the first imaging device 1 can be adjusted without changing the already set yaw direction and roll direction.

The arithmetic processing unit performs the processing from step S305 onward to calibrate the camera model 2 for the simulator system S2.
Specifically, in step S305 of FIG. 15, the arithmetic processing unit sets the index A' and the index B' at predetermined positions on the environment map of the simulator system S2. The set positions of the index A′ and the index B′ are based on the positional relationship of the index A and the index B with respect to the first imaging device 1 . That is, the positional relationship between the camera model 2 and the index A' is the same as the positional relationship between the first imaging device 1 and the index A, and the positional relationship between the camera model 2 and the index B' is the same as that between the first imaging device 1 and the index B. equated with a relationship.
Although the distance L1 and the distance L2 may be changed, the index A' and the index B' are set so that the distances from the position of the vehicle on the environment map are different.

In step S306 of FIG. 15, the arithmetic processing unit adjusts the yaw direction and pitch direction of the camera model 2 so that the center of the index A', the center of the index B', and the center of the angle of view of the camera model 2 overlap.

In step S307 of FIG. 16, the arithmetic processing unit sets an index C' at a predetermined position on the environment map of the simulator system S2. Specifically, the center position of the index C is set so that the positional relationship between the index A and the index C matches the positional relationship between the index A' and the index C'.

In step S308, the arithmetic processing unit adjusts the roll direction of the camera model 2 so that the center of the index C' is positioned on the horizontal line passing through the center of the angle of view of the camera model 2.

In step S309, the arithmetic processing unit determines whether or not to provide the posture of the camera model 2 with a depression angle. When the angle of depression is provided for the posture of the first imaging device 1, the angle of depression is also provided for the posture of the camera model 2 in order to simulate a similar state.

When setting the depression angle, the arithmetic processing unit sets an index D' at a predetermined position on the environment map of the simulator system S2 in step S310. The position and height of the index D' with respect to the camera model 2 are made to match the position and height of the index D with respect to the first imaging device 1 .

Subsequently, in step S<b>311 , the arithmetic processing unit adjusts the pitch direction of the camera model 2 so that the center of the index D′ is positioned on the horizontal line passing through the center of the angle of view of the camera model 2 . At this time, no adjustments are made in the yaw or roll direction. Thereby, the imaging direction of the camera model 2 can be adjusted without changing the already set yaw direction and roll direction.

By performing calibration so that the imaging direction of the camera model 2 matches the imaging direction of the first imaging device 1, the camera model 2 that appropriately simulates the first imaging device 1 can be prepared.

By properly calibrating the first imaging device 1 and the camera model 2 as shown in FIGS. 15 and 16, it is possible to properly perform the matching verification in step S105 of FIG.

<6. Variation>
In the above example, the camera model 2 has the sensor model B8 imitating the color filter by the first imaging device 1 having the Bayer array color filter (see FIG. 21). However, the filter provided in the first image pickup device 1 may be one other than the Bayer array.

Specifically, the first imaging device 1 uses an RCCB array filter (see FIG. 22) in which filters corresponding to R (Red), C (Clear), and B (Blue) are arranged, or R (Red), G ( Green), B (Blue), and IR (Infrared) filters in RGBIR array (see Fig. 23), and Cy (Cyan), Ye (Yellow), G (Green), and Mg (Magenta). A complementary color filter (see FIG. 24) in which corresponding filters are arranged may be provided. In that case, the sensor model B8 of the camera model 2 is also provided with an RCCB arrangement filter model, an RGBIR arrangement filter model, and a complementary color filter model.

In the examples shown in FIGS. 7 and 9 to 11, there are five verification areas Ar, measurement areas Br, corresponding areas Cr, and verification areas Ar′ set at predetermined positions in the angle of view and the image. It had been. However, this is only an example, and other setting examples are also conceivable. For example, if it is not necessary to check the angle of view or the consistency of the peripheral areas in the image, only one predetermined area may be set in the center of each area. Specifically, a verification area Ar1 is set as the verification area Ar, a measurement area Br1 is set as the measurement area Br, a corresponding area Cr1 is set as the corresponding area Cr, and a verification area Ar'1 is set as the verification area Ar'. may be

Alternatively, when confirming the matching only in the peripheral area, each predetermined area may be set in the area around the corner. Also, six or more predetermined areas may be set. For example, the angle of view may be vertically divided into three and horizontally divided into three, for a total of nine divisions, and the verification area Ar may be provided in the central portion of each area.

<7. Computer device>
A configuration of a computer device including an arithmetic processing unit that executes each process shown in FIGS. 4, 5, 15, and 16 will be described with reference to FIG.

The CPU 71 of the computer device functions as an arithmetic processing unit that performs the various processes described above, and programs stored in a non-volatile memory unit 74 such as a ROM 72 or an EEP-ROM (Electrically Erasable Programmable Read-Only Memory), or Various processes are executed according to programs loaded from the storage unit 79 to the RAM 73 . The RAM 73 also appropriately stores data necessary for the CPU 71 to execute various processes.
The CPU 71 , ROM 72 , RAM 73 and nonvolatile memory section 74 are interconnected via a bus 83 . An input/output interface (I/F) 75 is also connected to this bus 83 .

The input/output interface 75 is connected to an input section 76 including operators and operating devices.
For example, as the input unit 76, various operators and operation devices such as a keyboard, mouse, key, dial, touch panel, touch pad, remote controller, etc. are assumed.
A user's operation is detected by the input unit 76 , and a signal corresponding to the input operation is interpreted by the CPU 71 .

The input/output interface 75 is connected integrally or separately with a display unit 77 such as an LCD or an organic EL panel, and an audio output unit 78 such as a speaker.
The display unit 77 is a display unit that performs various displays, and is configured by, for example, a display device provided in the housing of the computer device, a separate display device connected to the computer device, or the like.
The display unit 77 displays images for various types of image processing, moving images to be processed, etc. on the display screen based on instructions from the CPU 71 . Further, the display unit 77 displays various operation menus, icons, messages, etc., ie, as a GUI (Graphical User Interface), based on instructions from the CPU 71 .

The input/output interface 75 may be connected to a storage unit 79 made up of a hard disk, solid-state memory, etc., and a communication unit 80 made up of a modem or the like.

The communication unit 80 performs communication processing via a transmission line such as the Internet, wired/wireless communication with various devices, bus communication, and the like.

A drive 81 is also connected to the input/output interface 75 as required, and a removable storage medium 82 such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory is appropriately mounted.
Data files such as programs used for each process can be read from the removable storage medium 82 by the drive 81 . The read data file is stored in the storage unit 79 , and the image and sound contained in the data file are output by the display unit 77 and the sound output unit 78 . Computer programs and the like read from the removable storage medium 82 are installed in the storage unit 79 as required.

In this computer device, for example, software for the processing of this embodiment can be installed via network communication by the communication unit 80 or via the removable storage medium 82 . Alternatively, the software may be stored in advance in the ROM 72, the storage unit 79, or the like.

The CPU 71 performs processing operations based on various programs, thereby executing necessary information processing and communication processing as an information processing apparatus having the arithmetic processing unit described above.
Note that the information processing apparatus is not limited to being composed of a single computer device as shown in FIG. 2, and may be configured by systematizing a plurality of computer devices. The plurality of computer devices may be systematized by a LAN (Local Area Network) or the like, or may be remotely located by a VPN (Virtual Private Network) or the like using the Internet or the like. The plurality of computing devices may include computing devices as a group of servers (cloud) available through a cloud computing service.

<8. Summary>
As described in each of the examples above, the information processing method of the present technology uses the first imaging device 1 that captures an image of a specific subject (such as the above-described brightness box, charts, subject in an actual driving environment scene, and preceding vehicle). and a second image G2 output from the camera model 2 to which two-dimensional input data (for example, spectral radiance values) based on measurement results of measuring light from a specific subject is input. By comparing the characteristics of the first imaging device 1 and the characteristics of the camera model 2 using and, the matching verification between the first imaging device 1 and the camera model 2 is performed by a computer device (for example, the calculation of the CPU 71, etc.) an information processing apparatus having a processing unit).
The comparison of the characteristics of the first imaging device 1 and the camera model 2 may be performed, for example, by comparing the first image G1 and the second image G2, or applying image recognition processing to the first image G1. may be performed by comparing the recognition result (label information) obtained by applying the image recognition processing to the second image G2.
By verifying the matching between the first imaging device 1 and the camera model 2 through such comparison processing, it is possible to appropriately perform a simulation using the camera model 2 . Note that when the first image G1 and the second image G2 match, the recognition result of the image recognition processing for each image is the same, and the simulation using the camera model 2 appropriately simulates the actual environment. is possible. However, the data input to the camera model 2 is based on the measurement results (spectral radiance value, spectral irradiance, etc.) of measuring the light from the subject. A perfect match is difficult. Therefore, the statistical data obtained from the first image G1 and similar statistical data obtained from the second image G2 may be compared, and if the difference is small, it may be determined that the consistency has been ensured. As a result, consistency verification can be easily performed, and various costs can be reduced.
Alternatively, as described above, the recognition result obtained by applying image recognition processing to the first image G1 may be compared with the recognition result for the second image G2. Even if the first image G1 and the second image G2 are slightly different, if the recognition results are the same, the subsequent control can be performed correctly. In other words, automatic driving control and driving support control can be performed appropriately if the matching between the first imaging device 1 and the camera model 2 can be ensured to the extent that the same recognition result can be output. Ensuring the matching between the first imaging device and the camera model to the extent that the same recognition result can be output eliminates the need for excessive matching between the first imaging device and the camera model. Therefore, the camera model can be verified efficiently.

As described with reference to FIG. 9 and the like, the input data input to the camera model 2 may be spectral radiance values of light from a specific subject.
By inputting a two-dimensional spectral radiance value to the camera model 2, the camera model 2 can output a signal similar to that of the first imaging device 1 that actually exists.
As a result, consistency verification can be performed appropriately.

As described with reference to FIG. 3 and the like, the camera model may have an optical model B7 and an image sensor model (sensor model B8).
For example, the optical model B7 is a model imitating various lenses, projection correction, aperture correction, shading correction, IR cut filter, and the like provided in the first imaging device 1 . The image sensor model is a model imitating a color filter provided in each pixel, photoelectric conversion processing, AD (Analog to Digital) conversion processing, and the like.
In this way, each model is configured including various corrections and processes incorporated in the first imaging device 1, which is an actual imaging device, to construct a camera model 2 that appropriately imitates the first imaging device 1. can do. Further, it is possible to easily change the camera model 2 that is required when some of these elements are changed in the actual first imaging device 1 .

As described with reference to FIGS. 7, 10, etc., the measurement result of measuring the light from the subject is a predetermined area (area corresponding to the verification area Ar) set within the angle of view of the first imaging device 1. ), and the two-dimensional input data may be arranged in an area (corresponding area Cr) corresponding to the verification area Ar.
As a result, it is not necessary to generate input data corresponding to all subjects imaged at the angle of view of the first imaging device 1 .
Therefore, it is possible to reduce the cost required to generate two-dimensional input data.

As described with reference to FIGS. 7 and 11, a pixel region (verification area Ar) corresponding to a predetermined region and a region (verification area Ar) corresponding to the pixel region (verification area Ar) in the second image G2 ') may be performed to verify consistency.
Matching can be simply verified by performing comparison using data (statistical data) output from a part of pixel regions.
Therefore, it is possible to reduce the amount of calculation required for consistency verification.
In addition, it is possible to configure such that the data of the verification area Ar in the RAW data and the data of the corresponding verification area Ar' are compared, and if a certain degree of matching is found, the image recognition processing and the recognition result are compared. is. In this case, when it is determined that the matching between the first imaging device 1 and the camera model 2 cannot be ensured at the time of comparing the data regarding the verification areas Ar and Ar', it is not necessary to perform the image recognition processing. It is possible to reduce the processing load.

As described with reference to FIG. 10 and the like, two-dimensional input data (input data using spectral radiance values) has dummy data arranged in an area other than an area (corresponding area Cr) corresponding to a predetermined area. may be
This eliminates the need to use the measurement result to generate data to be placed in an area other than the area corresponding to the predetermined area.
Therefore, it is possible to reduce the time cost and calculation cost for generating two-dimensional input data.

As described with reference to FIGS. 3 and 7, the image sensor model (sensor model B8) has a filter model, and by performing statistical processing for each pixel group for each filter type in the second image G2 Consistency verification may be performed.
As a result, matching verification is performed in accordance with the actual configuration of the first imaging device 1 having color filters and the like.
Therefore, it is possible to appropriately perform consistency verification.

As described with reference to FIG. 8 and the like, the average value and variance are calculated for each pixel group (that is, each Dr, Dg, and Db) by statistical processing, and the average value and variance are compared with respective thresholds. Consistency verification may be performed by
Consistency can be verified with simple processing by using the average value, variance, or the like. Therefore, it is possible to reduce the amount of calculation for realizing consistency verification.
In addition, by using the average value, variance, etc., it is possible to prevent erroneous determination due to noise or the like.

As described with reference to FIGS. 21 to 24, the filter model may be a model that employs at least one of the Bayer array, RCCB array, RGBIR array, and array using complementary color filters.
By using various models for the filter model, it is possible to verify the conformity of the camera model 2 according to the user's request.
Therefore, it is possible to flexibly perform matching verification according to various demands, and to improve convenience.

As described with reference to the respective drawings in the above examples, the information processing method of the present technology includes pre-processing for verifying consistency between the first imaging device 1 as an in-vehicle camera and the camera model 2 as a simulated camera. , the center of the first targets (indexes A and B) installed at least two locations in front of the first imaging device 1 and at the same height position (height Hc) as the first imaging device 1 is the angle of view. A computer device (for example, an information processing device having an arithmetic processing unit such as a CPU 71) performs calibration by adjusting the yaw direction and the pitch direction of the first imaging device 1 so that the first imaging device 1 is aligned with the center of the .
By performing such calibration, it is possible to verify the consistency between the in-vehicle camera system S1 in which the first imaging device 1 is mounted and the simulator system S2 in which the camera model 2 is mounted.
Therefore, if it can be ensured that the camera model 2 can appropriately simulate the first imaging device 1 in the vehicle-mounted state, the camera model 2 can be used to properly test the vehicle-mounted camera, thereby reducing various costs. It can be performed.

As described with reference to FIGS. 18 and 19, the second target is installed at a position offset in the horizontal direction with respect to the optical axis of the first imaging device 1 and at the same height as the first imaging device 1. Calibration may be performed by adjusting the roll direction of the first imaging device 1 so that the center of the object (marker C) coincides with the middle position in the vertical direction of the angle of view.
This makes it possible to calibrate in the roll direction with a simple method.
Therefore, accuracy of matching verification can be improved.

As described with reference to FIG. 20 and the like, the pitch direction is adjusted so that the center of the third target (index D) installed in front of the first imaging device 1 coincides with the vertical intermediate position. By doing so, the depression angle may be adjusted as calibration.
As a result, it is possible to calibrate not only the first imaging device 1 whose optical axis is substantially horizontal, but also the first imaging device 1 that captures the front of the vehicle 100 from a slightly bird's-eye view.
By setting the optical axis direction of the camera model 2 according to the optical axis direction of the first imaging device 1 calibrated in this way, the first imaging device 1 set in various orientations can be appropriately simulated. A camera model 2 can be provided. Therefore, the simulation for the first imaging device 1 can be properly performed.

As described with reference to FIGS. 15 and 16, etc., the optical axis direction of the camera model 2 may be adjusted based on the optical axis direction of the first imaging device 1 adjusted by calibration.
Thereby, the optical axis direction of the first imaging device 1 and the optical axis direction of the camera model 2 can be matched.
Therefore, it is possible to prepare a camera model 2 that appropriately simulates the first imaging device 1, and it is possible to perform a simulation using the camera model 2 instead of the first imaging device 1, and it is used for driving support control. Algorithm verification work can be performed efficiently.

The information processing apparatus according to the present embodiment uses the first image G1 output from the first imaging device 1 that captures a specific subject (such as the luminance box and charts described above, the subject in the actual driving environment scene, the preceding vehicle, etc.). and a second image G2 output from the camera model 2 to which two-dimensional input data (for example, spectral radiance values) based on measurement results of measuring light from a specific subject is input, It has an arithmetic processing unit (for example, CPU 71) that verifies the consistency between the first imaging device 1 and the camera model 2 by comparing the characteristics of the imaging device 1 and the characteristics of the camera model 2. FIG.

The information processing apparatus according to the present embodiment performs the first imaging in front of the first imaging device 1 as preprocessing for verifying the matching between the first imaging device 1 as an in-vehicle camera and the camera model 2 as a simulated camera. The first imaging device 1 is arranged so that the centers of the first targets (indicators A and B) set at at least two locations at the same height position (height Hc) as the device 1 are aligned with the center of the angle of view. It has an arithmetic processing unit (for example, CPU 71) that performs calibration by adjusting the yaw direction and the pitch direction.

The program to be executed by the information processing apparatus described above can be recorded in advance in a HDD (Hard Disk Drive) as a recording medium built in a device such as a computer device, or in a ROM or the like in a microcomputer having a CPU. . Alternatively, the program may be a flexible disk, a CD-ROM (Compact Disk Read Only Memory), an MO (Magneto Optical) disk, a DVD (Digital Versatile Disc), a Blu-ray Disc (registered trademark), a magnetic disk, a semiconductor It can be temporarily or permanently stored (recorded) in a removable recording medium such as a memory or memory card. Such removable recording media can be provided as so-called package software.
In addition to installing such a program from a removable recording medium to a personal computer or the like, it can also be downloaded from a download site via a network such as a LAN (Local Area Network) or the Internet.

It should be noted that the effects described in this specification are merely examples and are not limited, and other effects may also occur.

Further, the examples described above may be combined in any way, and even when various combinations are used, it is possible to obtain the various effects described above.

<9. This technology>
The present technology can also adopt the following configuration.
(1)
First image data output from a first imaging device that captures an image of a specific subject, and two-dimensional input data based on measurement results of measuring light from the specific subject 2 image data, and comparing the characteristics of the first imaging device and the characteristics of the camera model to verify consistency between the first imaging device and the camera model.
(2)
The information processing method according to (1), wherein the input data is a spectral radiance value of light from the specific subject.
(3)
The information processing method according to any one of (1) to (2) above, wherein the camera model includes an optical model and an image sensor model.
(4)
The measurement result of the light is the measurement result of a predetermined area set within the angle of view of the first imaging device,
The information processing method according to any one of (1) to (3) above, wherein the two-dimensional input data includes the measurement result arranged in an area corresponding to the predetermined area.
(5)
The information processing method according to (4), wherein the matching verification is performed by comparing a pixel area corresponding to the predetermined area and an area corresponding to the pixel area in the second image data.
(6)
The information processing method according to any one of (4) to (5) above, wherein the two-dimensional input data includes dummy data arranged in an area other than an area corresponding to the predetermined area.
(7)
the image sensor model has a filter model;
The information processing method according to (3), wherein the matching verification is performed by performing statistical processing for each pixel group for each filter type in the second image data.
(8)
The information processing method according to (7), wherein an average value and a variance are calculated for each pixel group by the statistical processing, and the matching verification is performed by comparing the average value and the variance with respective threshold values.
(9)
The information processing method according to (7), wherein the filter model employs at least one of a Bayer array, an RCCB array, an RGBIR array, and an array using complementary color filters.
(10)
As a preliminary process for verifying matching between a first imaging device as an in-vehicle camera and a camera model as a simulated camera, at least two cameras positioned in front of the first imaging device and at the same height as the first imaging device. An information processing method, wherein calibration is performed by adjusting the yaw direction and the pitch direction of the first imaging device so that the center of a first target placed at a location coincides with the center of the angle of view.
(11)
The center of a second target placed at a position offset in the horizontal direction with respect to the optical axis of the first imaging device and at the same height as the first imaging device coincides with the middle position in the vertical direction of the angle of view. The information processing method according to (10), wherein the calibration is performed by adjusting the roll direction of the first imaging device such that
(12)
The depression angle adjustment as the calibration is performed by adjusting the pitch direction so that the center of the third target placed in front of the first imaging device coincides with the intermediate position in the vertical direction. 11) The information processing method described in 11).
(13)
The information processing method according to any one of (10) to (12) above, wherein the optical axis direction of the camera model is adjusted based on the optical axis direction of the first imaging device adjusted by the calibration.
(14)
First image data output from a first imaging device that captures an image of a specific subject, and two-dimensional input data based on measurement results of measuring light from the specific subject 2 image data, and comparing the characteristics of the first imaging device and the characteristics of the camera model, thereby verifying the consistency between the first imaging device and the camera model. information processing device.
(15)
As a preliminary process for verifying matching between a first imaging device as an in-vehicle camera and a camera model as a simulated camera, at least two cameras positioned in front of the first imaging device and at the same height as the first imaging device. an arithmetic processing unit that performs calibration by adjusting the yaw direction and the pitch direction of the first imaging device so that the center of the first target placed at the location is aligned with the center of the angle of view. processing equipment.

1 first imaging device 2 camera model B7 optical model B8 sensor model (image sensor model)
G1 First image G2 Second image Ar, Ar1, Ar2, Ar3, Ar4, Ar5 Verification area Cr, Cr1, Cr2, Cr3, Cr4, Cr5 Corresponding area Ar', Ar'1, Ar'2, Ar'3, Ar '4, Ar'5 Verification area A, B index (first target)
C index (second target)
D index (third target)

Claims (15)

1. First image data output from a first imaging device that captures an image of a specific subject, and two-dimensional input data based on measurement results of measuring light from the specific subject 2 image data, and comparing the characteristics of the first imaging device and the characteristics of the camera model to verify consistency between the first imaging device and the camera model.
2. 2. The information processing method according to claim 1, wherein said input data is a spectral radiance value of light from said specific subject.
3. The information processing method according to claim 1, wherein the camera model has an optical model and an image sensor model.
4. The measurement result of the light is the measurement result of a predetermined area set within the angle of view of the first imaging device,
   2. The information processing method according to claim 1, wherein said two-dimensional input data has said measurement result arranged in a region corresponding to said predetermined region.
5. 5. The information processing method according to claim 4, wherein the matching verification is performed by comparing a pixel area corresponding to the predetermined area and an area corresponding to the pixel area in the second image data.
6. 5. The information processing method according to claim 4, wherein the two-dimensional input data has dummy data arranged in an area other than the area corresponding to the predetermined area.
7. the image sensor model has a filter model;
   4. The information processing method according to claim 3, wherein the matching verification is performed by performing statistical processing for each pixel group for each filter type in the second image data.
8. 8. The information processing method according to claim 7, wherein an average value and a variance are calculated for each pixel group by the statistical processing, and the match verification is performed by comparing the average value and the variance with respective threshold values.
9. 8. The information processing method according to claim 7, wherein the filter model adopts at least one of a Bayer array, an RCCB array, an RGBIR array, and an array using complementary color filters.
10. As a preliminary process for verifying matching between a first imaging device as an in-vehicle camera and a camera model as a simulated camera, at least two cameras positioned in front of the first imaging device and at the same height as the first imaging device. An information processing method, wherein calibration is performed by adjusting the yaw direction and the pitch direction of the first imaging device so that the center of a first target placed at a location coincides with the center of the angle of view.
11. The center of a second target placed at a position offset in the horizontal direction with respect to the optical axis of the first imaging device and at the same height as the first imaging device coincides with the middle position in the vertical direction of the angle of view. 11. The information processing method according to claim 10, wherein the calibration is performed by adjusting the roll direction of the first imaging device such that the roll direction is adjusted.
12. The depression angle adjustment as the calibration is performed by adjusting the pitch direction so that the center of a third target placed in front of the first imaging device coincides with the middle position in the vertical direction. 12. The information processing method according to 11.
13. The information processing method according to claim 10, further comprising adjusting the optical axis direction of the camera model based on the optical axis direction of the first imaging device adjusted by the calibration.
14. First image data output from a first imaging device that captures an image of a specific subject, and two-dimensional input data based on measurement results of measuring light from the specific subject 2 image data, and comparing the characteristics of the first imaging device and the characteristics of the camera model, thereby verifying the consistency between the first imaging device and the camera model. information processing device.
15. As a preliminary process for verifying matching between a first imaging device as an in-vehicle camera and a camera model as a simulated camera, at least two cameras positioned in front of the first imaging device and at the same height as the first imaging device. an arithmetic processing unit that performs calibration by adjusting the yaw direction and the pitch direction of the first imaging device so that the center of the first target placed at the location is aligned with the center of the angle of view. processing equipment.

Images