WO2025192569A1 - Information processing apparatus, information processing method, program, imaging apparatus, and image processing method - Google Patents

Abstract

There is provided an information processing apparatus including a model generation unit that generates a learned model by performing learning on a neural network architecture to which constraints on hardware in which the learned model is deployed are given as constraint conditions.

Description

INFORMATION PROCESSING APPARATUS, INFORMATION PROCESSING METHOD, PROGRAM, IMAGING APPARATUS, AND IMAGE PROCESSING METHOD CROSS REFERENCE TO RELATED APPLICATIONS

　　　　This application claims the benefit of Japanese Priority Patent Application JP 2024-038149 filed March 12, 2024, the entire contents of which are incorporated herein by reference.

　　　　The present technology relates to the technical field of an information processing apparatus, an information processing method, a program, an imaging apparatus, and an image processing method that use machine learning to optimize image processing.

　　　　There are increasing cases of solving a problem by inference with use of a learned AI model.　The same is true in the field of image processing.　For example, a case in which predetermined inference processing is performed with use of certain image data as input data is considered.　At this time, an AI model is used to perform predetermined inference processing on a subject included in image data and obtain an inference result.

　　　　For example, the input image data to the AI model is obtained by performing predetermined image processing by an ISP (Image Signal Processor) provided in an imaging apparatus such as a camera.　However, in the image processing of the ISP, parameters for making an image easy for a human to see are applied in many cases.　Therefore, image data obtained by image processing with use of such parameters is not necessarily appropriate as input image data of the AI model.

　　　　That is, in order to perform high-accuracy inference processing with the AI model, it is important to optimize parameters used for image processing in the ISP.

　　　　The AI model can also be used for parameter tuning of the ISP to optimize the input image data of the AI model.

　　　　For example, it is possible to adopt a configuration in which parameters of image processing in an image processing unit such as an ISP are optimized with use of an AI model in a preceding stage, and high-accuracy inference for a subject included in input image data is performed with use of an AI model in a subsequent stage.

　　　　In NPL 1 below, for example, an AI model is used to optimize parameters of various types of image processing executed by an image processing unit and to perform inference with use of image data obtained after image processing.

Haina Qin, six others, "Attention-aware Learning for Hyperparameter Prediction in Image Processing Pipelines" European Conference on Computer Vision (ECCV), 2022, [Searched on January 30, 2024], Internet <URL: https://www.ecva.net/papers/eccv_2022/papers_ECCV/papers/136790265.pdf>

Summary

　　　　NPL 1 proposes an AI model that simultaneously performs parameter optimization and inference processing.
　　　　However, there is still room to further improve the accuracy of inference processing by an AI model.

　　　　The present technology has been developed in view of such problems, and it is desirable to improve the accuracy of inference processing with use of an AI model.

　　　　An information processing apparatus of the present technology includes a model generation unit that generates a learned model by performing learning on a neural network architecture to which constraints on hardware in which the learned model is deployed are given as constraint conditions.
　　　　The learned model is generated by performing learning on the neural network architecture and used for predetermined processing in some cases.　In this case, by using the learned model to which the optimized parameters are applied, high-accuracy processing can be performed.
　　　　Incidentally, the processing using the learned model may be realized by incorporating the parameter calculated in the process of generating the learned model into hardware with relatively low flexibility, instead of using software with relatively high flexibility.
　　　　Accordingly, high-speed processing can be realized at the expense of flexibility.
　　　　However, in many cases, the parameter calculated in the process of generating the learned model may not be applied to hardware without any change, and it may be necessary to change the design of the hardware.　This causes an increase in cost and the like.
　　　　In this configuration, learning is performed in a state where constraints of hardware to which the learned model is to be deployed are incorporated.　The parameter obtained by this learning can be applied to the target hardware without any change.

FIG. 1 is a block diagram for depicting a schematic configuration of an information processing system in the present embodiment. FIG. 2 is a diagram for depicting a configuration example of an imaging apparatus. FIG. 3 is a block diagram for depicting a configuration example of a computer apparatus such as a server apparatus. FIG. 4 is a diagram for schematically depicting a configuration of a neural network architecture. FIG. 5 is a diagram for explaining that parameters obtained in the learning of the neural network architecture are applied to the imaging apparatus. FIG. 6 is a diagram for schematically depicting a process of obtaining a processing order parameter. FIG. 7 is a diagram for schematically depicting a process of obtaining the processing order parameter following FIG. 6. FIG. 8 is a diagram for schematically depicting a process of obtaining the processing order parameter following FIG. 7. FIG. 9 is a diagram for schematically depicting a process of obtaining the processing order parameter following FIG. 8. FIG. 10 is a diagram for depicting an example of a hardware configuration of an image processing unit. FIG. 11 is a diagram for depicting another example of a hardware configuration of the image processing unit. FIG. 12 is a diagram for depicting still another example of a hardware configuration of the image processing unit. FIG. 13 is a diagram for depicting another example of a hardware configuration of the image processing unit. FIG. 14 is a diagram for depicting still another example of a hardware configuration of the image processing unit. FIG. 15 is a diagram for depicting a configuration example of an AWB (Auto White Balance) processing block. FIG. 16 is a diagram for depicting a configuration example of an imaging apparatus in a second embodiment. FIG. 17 is a diagram for depicting an example of a data flow among a prior determination processing unit, an image processing unit, and an inference processing unit. FIG. 18 is a diagram for depicting another example of a data flow among the prior determination processing unit, the image processing unit, and the inference processing unit. FIG. 19 is a diagram for depicting still another example of a data flow among the prior determination processing unit, the image processing unit, and the inference processing unit. FIG. 20 is a flowchart for depicting an example of a flow of processing executed by the imaging apparatus.

　　　　Hereinafter, embodiments of an information processing apparatus according to the present technology will be described in the following order with reference to the accompanying drawings.
　　　　<1. Configuration of information processing system>
　　　　<1-1. Configuration of imaging apparatus>
　　　　<1-2. Configuration of server apparatus>
　　　　<2. Parameter search in server apparatus>
　　　　<3. Hardware configuration of image processing unit>
　　　　<3-1. Other configuration examples of image processing unit>
　　　　<4. Second embodiment>
　　　　<5. Other>
　　　　<6. Summary>
　　　　<7. The present technology>

<1. Configuration of information processing system>
　　　　An information processing system 1A of the present embodiment will be described with reference to the accompanying drawings.

　　　　As depicted in FIG. 1, the information processing system 1A includes an imaging apparatus 2A and a server apparatus 3.　The imaging apparatus 2A and the server apparatus 3 may each be provided in plural number.

　　　　The imaging apparatus 2A and the server apparatus 3 can communicate with each other by being connected to, for example, a communication network NW such as the Internet.

<1-1. Configuration of imaging apparatus>
　　　　An example of a configuration of the imaging apparatus 2A is depicted in FIG. 2.
　　　　The imaging apparatus 2A includes an imaging optical system 21, an image sensor 22, a control unit 23, an image processing unit 24A, and an inference processing unit 25A.　It should be noted that the imaging apparatus 2A further includes necessary units, which are not illustrated, such as a display unit for displaying captured image data, an operation unit used for various types of operations, a power supply unit for supplying driving power to various units, and a storage unit for storing image data.

　　　　The imaging optical system 21 includes, for example, lenses such as a cover lens, a zoom lens, and a focus lens, and a diaphragm (iris) mechanism.　The imaging optical system 21 guides light (incident light) from a subject and condenses the light on a light reception surface of the image sensor 22.

　　　　The image sensor 22 includes a pixel array section 22a in which pixels having photoelectric conversion elements such as photodiodes are two-dimensionally arranged, a readout circuit for reading out charges accumulated in the respective pixels of the pixel array section 22a, and the like.

　　　　The image sensor 22 executes A/D (Analog/Digital) conversion processing and the like on the read-out captured image signal.　It should be noted that in the image sensor 22, CDS (Correlated Double Sampling) processing or AGC (Automatic Gain Control) processing may be executed.

　　　　The image data output from the image sensor 22 is input to the image processing unit 24A in the subsequent stage as RAW image data Gr.

　　　　The control unit 23 includes a microcomputer having, for example, a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), and the like.

　　　　The control unit 23 performs overall control of the imaging apparatus 2A by allowing the CPU to execute various types of processing in accordance with a program stored in the ROM or a program loaded in the RAM.

　　　　The control unit 23 instructs an optical system driving unit, which is not illustrated, to drive the zoom lens, the focus lens, the diaphragm mechanism, and the like.　The optical system driving unit moves the focus lens and the zoom lens, and opens and closes the diaphragm blade of the diaphragm mechanism, that is, drives the imaging optical system 21 in accordance with these driving instructions.

　　　　The control unit 23 controls writing and readout of various types of data to/from a storage unit that is not illustrated.
　　　　The storage unit is, for example, a nonvolatile storage device such as an HDD (Hard Disk Drive) or a flash memory apparatus, and is used for storing and recording the image data output from the image sensor 22.

　　　　The control unit 23 performs various types of data communications with an external apparatus such as the server apparatus 3 via a communication unit that is not illustrated.

　　　　The image processing unit 24A performs predetermined image processing on the RAW image data Gr output from the image sensor 22.　The image data obtained by performing the predetermined image processing by the image processing unit 24A is referred to as “processed image data Gp.”

　　　　The predetermined image processing is realized by executing a plurality of kinds of image processing in order.　The plurality of kinds of image processing includes, for example, AWB processing, DMS (Demosaic) processing, CCM (Color Correction Matrix) processing, GC (Gamma Correction) processing, denoise processing, HSC (Hue Saturation Control) processing, BCC (Brightness Contrast Control) processing, and the like.　It should be noted that these kinds of processing are merely examples.

　　　　In the image processing unit 24A, the processed image data Gp is normally generated by executing various types of image processing as the predetermined image processing in a predetermined order.

　　　　Meanwhile, in the present technology, the processing order of each image processing as the predetermined image processing executed by the image processing unit 24A is variable.　That is, the image processing unit 24A includes a structure in which the processing order of a processing block for performing each image processing is variable.

　　　　Hardware that functions as the image processing unit 24A includes, for example, a CPU, a DSP (Digital Signal Processor), a programmable accelerator, a dedicated hardware accelerator, an ISP (Image Signal Processor), and the like.　Each of these pieces of hardware becomes less flexible and becomes faster in processing in the order of the description.

　　　　The image processing unit 24A is realized by, for example, a programmable accelerator that is more flexible than a dedicated hardware accelerator and may require less sacrifice in high speed.

　　　　The configuration provided in the image processing unit 24A to vary the execution order of each processing block for realizing the predetermined image processing will be described later.

　　　　It should be noted that the control unit 23 may drive the imaging optical system 21 or control the image sensor 22 on the basis of the processing result of the image processing unit 24A.　For example, the control unit 23 may perform adjustment processing of exposure on the basis of the histogram of luminance obtained as a result of the processing by the image processing unit 24A.

　　　　The processed image data Gp output from the image processing unit 24A is input to the inference processing unit 25A.

　　　　The inference processing unit 25A realizes predetermined inference processing by deploying a learned model M1 obtained by performing learning on a predetermined neural network architecture NNA.　The inference processing unit 25A is realized by, for example, a GPU (Graphics Processing Unit), a DSP, and the like.

　　　　For example, the inference processing unit 25A performs processing of detecting a person appearing on the input processed image data Gp, processing of estimating the number of persons, processing of authenticating the face of a person, and the like.

　　　　An inference result Di obtained in the inference processing unit 25A may be presented to the user or transmitted to another apparatus via the control unit 23.　Alternatively, as depicted in FIG. 2, the inference result Di may be directly output from the inference processing unit 25A to the outside of the imaging apparatus 2A.

　　　　The image processing unit 24A and the inference processing unit 25A of the imaging apparatus 2A are used in a production environment or a product environment for obtaining the predetermined inference result Di on the basis of the RAW image data Gr.　That is, the image processing unit 24A and the inference processing unit 25A can perform high-accuracy inference processing on the RAW image data Gr output from the image sensor 22 by applying appropriate parameters to each of them.

　　　　It should be noted that the parameters given to the image processing unit 24A include a processing order parameter PMo for deciding the processing order of the above-described processing blocks and a processing content parameter PMp that is a parameter for each processing block and that changes the processing content of the processing blocks.
　　　　In addition, the parameter that is given to the inference processing unit 25A and that is for the learned model M1 is set as a model parameter PMm.

　　　　The processing order parameter PMo and the processing content parameter PMp of the image processing unit 24A and the inference processing unit 25A are calculated in, for example, a test environment or a learning environment.
　　　　For example, since the learning phase for the neural network architecture NNA may require a large amount of computation, it is preferable to realize it with use of an information processing apparatus with abundant computer resources.

　　　　In the present embodiment, as an example, various types of parameters to be applied to the image processing unit 24A and the inference processing unit 25A of the imaging apparatus 2A are searched for in the server apparatus 3.

<1-2. Configuration of server apparatus>
　　　　A configuration example of the server apparatus 3 is depicted in FIG. 3.
　　　　The server apparatus 3 includes a CPU 71.　The CPU 71 functions as a computation processing unit for performing the various types of processing described above, and executes various types of processing in accordance with a program stored in a nonvolatile memory unit 74 such as a ROM 72 or, for example, an EEP-ROM (Electrically Erasable Programmable Read-Only Memory), or a program loaded into a RAM 73 from a storage unit 79.　The RAM 73 also appropriately stores data and the like necessary for the CPU 71 to execute various types of processing.

　　　　The CPU 71, the ROM 72, the RAM 73, and the nonvolatile memory unit 74 are connected to each other via a bus 83.　An input/output interface (I/F) 75 is also connected to the bus 83.

　　　　An input unit 76 including an operation element and an operation device is connected to the input/output interface 75.
　　　　For example, the input unit 76 is assumed to be various types of operation elements and operation devices such as a keyboard, a mouse, a key, a dial, a touch panel, a touch pad, and a remote controller.
　　　　The operation of the user is detected by the input unit 76, and a signal according to the input operation is interpreted by the CPU 71.

　　　　In addition, a display unit 77 including an LCD, an organic EL panel, or the like and a sound output unit 78 including a speaker or the like are integrally or separately connected to the input/output interface 75.
　　　　The display unit 77 is a display unit for performing various types of displays, and is configured with, for example, a display device provided in the casing of a computer apparatus, a separate display device connected to a computer apparatus, or the like.

　　　　The display unit 77 displays images for various types of image processing and moving images to be processed on the display screen on the basis of an instruction from the CPU 71.　In addition, the display unit 77 displays various types of operation menus, icons, messages, and the like, that is, displays them as a GUI (Graphical User Interface) on the basis of an instruction from the CPU 71.

　　　　The storage unit 79 including a hard disk, a solid state memory, and the like and a communication unit 80 including a modem and the like are connected to the input/output interface 75 in some cases.

　　　　The communication unit 80 performs communication processing via a transmission path such as the Internet, wired/wireless communication with various types of equipment, and communication by bus communication.

　　　　A drive 81 is also connected to the input/output interface 75 as necessary, and a removable storage medium 82 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory is appropriately mounted.

　　　　A data file such as a program used for each processing can be read out from the removable storage medium 82 by the drive 81.　The read-out data file is stored in the storage unit 79, and images and sounds included in the data file are output by the display unit 77 and the sound output unit 78.　In addition, a computer program or the like read out from the removable storage medium 82 is installed in the storage unit 79 as necessary.

　　　　In this computer apparatus, for example, software used for the processing of the present embodiment can be installed via network communication by the communication unit 80 or via the removable storage medium 82.　Alternatively, the software may be preliminarily stored in the ROM 72, the storage unit 79, or the like.
　　　　In addition, this computer apparatus may store the RAW image data Gr received from the imaging apparatus 2A in the removable storage medium 82 via the storage unit 79 or the drive 81.

　　　　When the CPU 71 performs processing operations based on various types of programs, a parameter search function for acquiring various types of parameters in the process of performing learning by inputting the RAW image data Gr to the neural network architecture NNA is realized.

　　　　The communication unit 80 in the server apparatus 3 functions as a reception unit for receiving the RAW image data Gr from the imaging apparatus 2A.　In addition, the communication unit 80 in the server apparatus 3 functions as a transmission unit for transmitting the parameter acquired by the parameter search function to the imaging apparatus 2A.

　　　　It should be noted that the server apparatus 3 is not limited to the configuration of a single computer apparatus as depicted in FIG. 3 and may be configured by systematizing a plurality of computer apparatuses.　The plurality of computer apparatuses may be systemized by a LAN (Local Area Network) or the like, or may be arranged at a remote location by a VPN (Virtual Private Network) or the like with use of the Internet or the like.　The plurality of computer apparatuses may include a computer apparatus as a server group (cloud) that can be used by a cloud computing service.

<2. Parameter search in server apparatus>
　　　　The parameter search function in the server apparatus 3 will be described.　In the parameter search function in the server apparatus 3, the processing order parameter PMo, the processing content parameter PMp, and the model parameter PMm described above are obtained in the process of performing learning on the neural network architecture NNA or after the learning is performed.

　　　　In the learning of the neural network architecture NNA, for example, the RAW image data Gr is input as predetermined image data.　In addition, the neural network architecture NNA outputs the inference result Di obtained as a result of performing some inference processing for the subject included in the RAW image data Gr.

　　　　Here, the neural network architecture NNA used for the learning is one in which constraints on hardware for realizing the image processing unit 24A of the imaging apparatus 2A are set.

　　　　For example, in the case where the image processing unit 24A is realized with use of a programmable accelerator, the learning is performed with use of the neural network architecture NNA incorporating constraints in the programmable accelerator as constraint conditions.

　　　　The constraints on the hardware include, for example, a range of values and the number of bits to be handled for each processing block.

　　　　Specifically, the above-described predetermined image processing includes, for example, the AWB processing.　Although only 24-bit numerical values are handled in the AWB processing realized on the programmable accelerator, if the neural network architecture NNA handled in the server apparatus 3 performs learning in a state where 64-bit numerical values can be handled, there is a possibility that the processing content parameter PMp obtained thereby is not appropriate for the AWB processing block that can handle only 24-bit numerical values.

　　　　That is, the parameters searched for in the learning environment are not necessarily optimized in the product environment.

　　　　In addition, the processing of each processing block such as the AWB processing block is processing inseparable for each processing block, and this may be considered as a constraint condition.

　　　　Specifically, in the case where a certain processing block A includes processing A1 and processing A2 and another processing block B includes processing B1 and processing B2, there are only two processing orders of the processing block A and the processing block B that may be inseparably executed, such as execution in the order of processing A1, A2, B1, and B2, or in the order of processing B1, B2, A1, and A2.

　　　　However, if the parameter search is performed without considering such a constraint in the learning environment in the neural network architecture NNA, the processing order parameter PMo for executing in the processing order such as processing A1, B2, B1, and A2 is obtained as the search result in some cases.　Such a processing order parameter PMo may not be applied without any change to the programmable accelerator in the product environment, and a design change or the like of the programmable accelerator occurs.

　　　　In the present embodiment, constraint conditions that the processing A1 and A2 are inseparably executed and that the order of the processing A1 and A2 may not be switched are given to the neural network architecture NNA.　Similar constraint conditions for the processing B1 and B2 are given to the neural network architecture NNA.

　　　　By performing learning in a state where such constraint conditions are given to the neural network architecture NNA, each parameter obtained as a result of the learning can be preferably applied to hardware used in the product environment.

　　　　A configuration of the neural network architecture NNA used in the parameter search is schematically depicted in FIG. 4.

　　　　The neural network architecture NNA includes an image processing unit 24A part P1 as a first half part and an inference processing part P2 as a second half part.

　　　　The parameters obtained by performing the learning on such a neural network architecture NNA are divided into parameters obtained for the image processing part P1 and parameters obtained for the inference processing part P2.

　　　　The parameters obtained for the image processing part P1 are the processing order parameter PMo and the processing content parameter PMp.　In addition, the parameter obtained for the inference processing part P2 is the model parameter PMm (see FIG. 5).

　　　　The processing order parameter PMo and the processing content parameter PMp are applied to the image processing unit 24A configured with a programmable accelerator and the like.　At this time, since the constraints on the hardware of the image processing unit 24A are given to the neural network architecture NNA as the constraint conditions, the neural network architecture NNA can be implemented in actual hardware in a state where the accuracy obtained at the time of learning is maintained.

　　　　The model parameter PMm is applied to the inference processing unit 25A configured with an ISP or a DSP.　That is, at least a part of the learned model M1 obtained by the learning of the neural network architecture NNA is deployed in the inference processing unit 25A.　It should be noted that the learned model M1 to be deployed in the inference processing unit 25A is a model obtained in the inference processing part P2 in the neural network architecture NNA.
　　　　Therefore, the parameter search function realized by the CPU 71 of the server apparatus 3 can be regarded as a model generation function for generating the learned model M1 to be deployed in the inference processing unit 25A.　That is, the CPU 71 functions as a model generation unit.

　　　　A process of obtaining the processing order parameter PMo indicating the processing order by the neural network architecture NNA is schematically depicted in FIG. 6 and the like.

　　　　Each numerical value described in the table depicted in FIG. 6 is a numerical value indicating the degree of accuracy of the inference result Di output from the inference processing part P2.　It should be noted that the numerical value in the table does not directly indicate the accuracy of the inference result Di and is only a relative evaluation value.　Specifically, as the numerical value in the table is higher, the high-accuracy inference result Di can be obtained.

　　　　The numerical value following “Stage” is the selection order of the processing blocks.　That is, the row of “Stage 1” indicates an evaluation value for deciding a processing block to be selected as the first image processing.　The row of “Stage 1” is an evaluation value for deciding the image processing to be executed first on the RAW image data Gr.

　　　　In addition, “bypass” in the drawing means that the processing of the inference processing part P2 starts without executing additional image processing.　For example, in the case where “bypass” is selected in “Stage 3,” it means that the processed image data Gp obtained by executing the processing selected in Stage 1 and the selection processing in Stage 2 in order is input to the inference processing part P2.

　　　　The inference accuracy in the case where the AWB processing is executed on the RAW image data Gr is higher than that in the case where bypass is selected.　Therefore, the AWB processing is selected in Stage 1 (see the right side of FIG. 6).

　　　　Next, all of the CCM processing, the GC processing, the denoise processing, the HSC processing, and the BCC processing are processing in which processing for the image data obtained after the DMS processing is assumed.

　　　　Therefore, in Stage 2, the DMS processing is automatically selected (see FIG. 7).

　　　　In Stage 3, among the CCM processing, the GC processing, the denoise processing, the HSC processing, the BCC processing, and the bypass, the highest inference accuracy is obtained in the case where the GC processing is selected for the image data obtained after the demosaic processing.　Therefore, in Stage 3, the GC processing is selected (see FIG. 8).

　　　　The selection result of the image processing in each Stage thus obtained is depicted in FIG. 9.　In the illustrated example, each processing block is executed in the order of the AWB processing, the DMS processing, the GC processing, the denoise processing, the HSC processing, the denoise processing, and the BCC processing as the predetermined image processing, so that the inference accuracy in the inference processing part P2 can be enhanced.

　　　　It should be noted that one processing block may be selected a plurality of times.　In the example depicted in FIG. 9, the denoise processing is selected in both Stage 4 and Stage 6.

　　　　It should be noted that the examples depicted in FIG. 6 to FIG. 9 are diagrams for clearly explaining the function of deciding the processing order of the processing blocks, and it is not necessary to decide the processing content in order from the preceding stage side in the image processing part P1 when the learning of the neural network architecture NNA is actually performed.

　　　　In addition, at the time of learning, not only the processing order of the processing blocks but also the processing content parameter PMp for each processing block are simultaneously optimized.

<3. Hardware configuration of image processing unit>
　　　　An example of a hardware configuration when the image processing unit 24A of the imaging apparatus 2A is realized by hardware is depicted in FIG. 10.

　　　　Here, the processing block for performing the AWB processing is referred to as an AWB processing block 24a.　As similar to the above, the processing block for performing the DMS processing is referred to as a DMS processing block 24b, the processing block for performing the CCM processing is referred to as a CCM processing block 24c, the processing block for performing the GC processing is referred to as a GC processing block 24d, the processing block for performing the denoise processing is referred to as a denoise processing block 24e, the processing block for performing the HSC processing is referred to as an HSC processing block 24f, and the processing block for performing the BCC processing is referred to as a BCC processing block 24g.

　　　　The AWB processing block 24a, the DMS processing block 24b, the CCM processing block 24c, the GC processing block 24d, the denoise processing block 24e, the HSC processing block 24f, and the BCC processing block 24g are fabric connected to each other in order to vary the processing order.

　　　　As an example as depicted in FIG. 10, each of the AWB processing block 24a, the DMS processing block 24b, the CCM processing block 24c, the GC processing block 24d, the denoise processing block 24e, the HSC processing block 24f, and the BCC processing block 24g is connected to a bus 26.

　　　　Each processing block can transmit image data as a processing result to another processing block via the bus 26.

　　　　The image sensor 22 and the inference processing unit 25A are also connected to the bus 26.

　　　　That is, in the image processing unit 24A, any processing block receives the RAW image data Gr from the image sensor 22.　Then, the image processing in the processing block and the processing of transmitting the image data as a processing result to another processing block are repeatedly executed.　The processing block that has completed the last image processing outputs the processed image data Gp as input data to the learned model M1 deployed in the inference processing unit 25A via the bus 26.

　　　　As described above, by adopting the configuration in which the respective processing blocks in the image processing unit 24A are fabric connected to each other, the processing order of the processing blocks can be appropriately changed.

　　　　It should be noted that one processing block may be configured to be capable of continuously executing the same image processing.　For example, the predetermined image processing may be performed in a processing order including two consecutive denoise processing.　In this case, the processing block for performing the denoise processing may be configured to be capable of continuously executing the denoise processing as the same image processing a plurality of times without using the bus 26.

<3-1. Other configuration examples of image processing unit>
　　　　Another example of a hardware configuration of the image processing unit 24A is depicted in FIG. 11.
　　　　Among the processing blocks included in the image processing unit 24A, there may be processing blocks that are highly likely to be executed in a predetermined order.　A dedicated data transmission line Tr may be provided between such processing blocks.

　　　　Specifically, as depicted in FIG. 11, the AWB processing block 24a includes a dedicated data transmission line Tr for the DMS processing block 24b.

　　　　As similar to the above, the DMS processing block 24b includes a dedicated data transmission line Tr for the CCM processing block 24c.

　　　　The CCM processing block 24c includes a dedicated data transmission line Tr for the GC processing block 24d.

　　　　The GC processing block 24d includes a dedicated data transmission line Tr for the denoise processing block 24e.

　　　　The denoise processing block 24e includes a dedicated data transmission line Tr for the HSC processing block 24f.

　　　　The HSC processing block 24f includes a dedicated data transmission line Tr for the BCC processing block 24g.

　　　　In the case where a dedicated data transmission line Tr is provided from a processing block as a transmission source of image data to a processing block as a transmission destination, the image data is transmitted with use of the dedicated data transmission line Tr.

　　　　On the other hand, in the case where no dedicated data transmission line Tr is provided from a processing block as a transmission source of image data to a processing block as a transmission destination, the image data is transmitted via the bus 26.

　　　　Accordingly, since the transmission of the image data between the processing blocks having a strong connection is performed with use of the dedicated data transmission line Tr, the transmission speed can be improved.　In addition, in the case where data is sent to the common bus 26, it may be necessary to perform processing to match the common input/output format in the bus 26.　In contrast, in the case where the transmission of the image data is performed with use of the dedicated data transmission line Tr, the processing to match the input/output format becomes unnecessary.

　　　　Therefore, the execution speed of the predetermined image processing in the image processing unit 24A can be increased.

　　　　For example, in the case where the above-described image processing is executed for each imaged frame in the image sensor 22, it is possible to respond even if the imaged frame is raised.

　　　　Still another example of a hardware configuration of the image processing unit 24A is depicted in FIG. 12.
　　　　In the example depicted in FIG. 12, the AWB processing block 24a and the DMS processing block 24b for performing the AWB processing and the DMS processing that are processing for image data that has not yet been subjected to demosaic are connected to a first bus 26a.

　　　　In addition, the CCM processing block 24c, the GC processing block 24d, the denoise processing block 24e, the HSC processing block 24f, and the BCC processing block 24g that perform the CCM processing, the GC processing, the denoise processing, the HSC processing, the denoise processing, and the BCC processing that are processing for the RGB image data as the image data obtained after demosaic is processed are connected to a second bus 26b.

　　　　Data transfer is possible between the first bus 26a and the second bus 26b.

　　　　In the example depicted in FIG. 12, the image processing is performed in the selected order between the AWB processing block 24a and the DMS processing block 24b that are connected to the first bus 26a.　The processed image data is transmitted to the second bus 26b.　The image processing is applied to the image data in the selected order in each of the processing blocks that are connected to the second bus 26b, and the generated processed image data Gp is provided to the learned model M1 of the inference processing unit 25A via the second bus 26b.

　　　　Another example of a hardware configuration of the image processing unit 24A is depicted in FIG. 13.

　　　　It is considered that the AWB processing and the demosaic processing that are processing for the image data that has not yet been subjected to demosaic do not affect the accuracy of the inference result Di even if the processing order is changed.　In this case, the processing of the AWB processing block 24a and the DMS processing block 24b may be fixed, and the processing order of each processing block for the RGB image data obtained after demosaic is processed may be changed.

　　　　For example, as depicted in FIG. 13, only the CCM processing block 24c, the GC processing block 24d, the denoise processing block 24e, the HSC processing block 24f, and the BCC processing block 24g that process the RGB image data obtained after demosaic is processed are connected to the bus 26.

　　　　In this case, as depicted in FIG. 13, it is desirable that the transmission of the image data between the AWB processing block 24a and the DMS processing block 24b is performed via the dedicated data transmission line Tr.
　　　　Accordingly, the speed of data transfer of the image data that has not yet been subjected to demosaic can be increased.

　　　　It should be noted that, in each of the configurations depicted in FIG. 12 and FIG. 13, the dedicated data transmission line Tr and the bus 26 may be used in combination for the transmission of the image data between the processing blocks having a strong connection as similar to FIG. 11.

　　　　As a hardware configuration of the image processing unit 24A, a configuration in which the dedicated data transmission line Tr is more actively used is depicted in FIG. 14.

　　　　In the example depicted in FIG. 14, only the image sensor 22, the AWB processing block 24a, the BCC processing block 24g, and the inference processing unit 25A are connected to the bus 26.

　　　　The DMS processing block 24b, the CCM processing block 24c, the GC processing block 24d, the denoise processing block 24e, and the HSC processing block 24f are connected in series between the AWB processing block 24a and the BCC processing block 24g via the dedicated data transmission lines Tr.

　　　　The RAW image data Gr output from the image sensor 22 is first input to the AWB processing block 24a.　Each of seven processing blocks from the AWB processing block 24a to the BCC processing block 24g is provided with a path for executing predetermined processing and a path for passing predetermined processing.

　　　　Specifically, a configuration of the AWB processing block 24a is schematically depicted in FIG. 15.

　　　　A switch 27 and the like are provided inside the AWB processing block 24a.　The switch 27 is provided with an AWB processing unit 24a1 configured with a circuit for executing the AWB processing and the like and an avoidance path 28 for avoiding the AWB processing.

　　　　That is, the AWB processing block 24a can select the presence or absence of execution of the AWB processing unit 24a1 by operating the switch 27.

　　　　Each of the processing blocks other than the AWB processing block 24a is also provided with a processing unit and an avoidance path 28 as a similar configuration.　For example, in the case of the DMS processing block 24b, a switch 27, a DMS processing unit 24b1 for performing the DMS processing, and an avoidance path 28 are provided.

　　　　The image processing is applied to the image data input to the AWB processing block 24a in any one of the processing blocks until the data is output from the BCC processing block 24g to the bus 26.

　　　　For example, in the case where the image processing in the processing blocks is applied in the processing order as depicted in FIG. 9, in one loop in which the image data is input from the bus 26 to the AWB processing block 24a and the image data is output from the BCC processing block 24g to the bus 26, the switch 27 is controlled on the processing unit side only in the AWB processing block 24a, and the switch 27 is controlled on the avoidance path 28 side in each of other processing blocks.

　　　　In the second loop that follows, the switch 27 is controlled on the processing unit side only in the DMS processing block 24b.

　　　　In the third loop, the switch 27 is controlled on the processing unit side only in the GC processing block 24d.

　　　　The processed image data for which the predetermined image processing has completed is output to the inference processing unit 25A via the bus 26 and is subjected to predetermined inference processing.

　　　　The configuration depicted in FIG. 14 can be realized only by changing a part of the hardware configuration for performing the image processing in which the processing order in the related art is fixed.　That is, the configuration from the AWB processing block 24a to the BCC processing block 24g in the image processing unit 24A can be used without any change.
　　　　Therefore, the configuration for changing the processing order of each image processing in the predetermined image processing can be realized at a low cost.

<4. Second embodiment>
　　　　An information processing system 1B in a second embodiment is an example in which a plurality of sets of processing content parameters PMp and processing order parameters PMo is switched for use.　Specifically, an image processing unit 24B of an imaging apparatus 2B of the information processing system 1B selects a set of parameters from a plurality of prepared sets of processing content parameters PMp and processing order parameters PMo in accordance with the situation, and uses the same for predetermined image processing.

　　　　As depicted in FIG. 1, the information processing system 1B includes an imaging apparatus 2B and a server apparatus 3.　The imaging apparatus 2B and the server apparatus 3 may each be provided in plural number.

　　　　The imaging apparatus 2B and the server apparatus 3 can communicate with each other by being connected to, for example, a communication network NW such as the Internet.

　　　　As depicted in FIG. 16, the imaging apparatus 2B of the information processing system 1B includes an imaging optical system 21, an image sensor 22, a prior determination processing unit 29, a control unit 23, an image processing unit 24B, and an inference processing unit 25B.

　　　　RAW image data Gr is input to the prior determination processing unit 29 from the image sensor 22.　A model that has learned by machine learning is deployed in the prior determination processing unit 29.　The learned model deployed in the prior determination processing unit 29 is referred to as a “predictor M2.”
　　　　It should be noted that the predictor M2 as the learned model is obtained by, for example, learning processing in the server apparatus 3.

　　　　The predictor M2 selects a combination of parameters to be adopted by the image processing unit 24B on the basis of the characteristics of the RAW image data Gr before the RAW image data Gr is input to the image processing unit 24B.　That is, the predictor M2 is a model acquired by learning, and outputs a combination of parameters to be adopted by the image processing unit 24B with use of the RAW image data Gr as input data.

　　　　In the following description, a group of parameters including at least some of the processing order parameter PMo, the processing content parameter PMp, and the model parameter PMm is described as a “parameter set PS.”

　　　　It should be noted that, if the parameter set PS includes the processing order parameter PMo, the image processing unit 24B has the above-described configuration in which the processing order of various types of image processing included in the predetermined image processing can be changed.

　　　　For example, the predictor M2 outputs a parameter set PS1 itself or data specifying the parameter set PS1 in the case where the RAW image data Gr photographed in the daytime is input.　In addition, the predictor M2 outputs a parameter set PS2 itself or data specifying the parameter set PS2 in the case where the RAW image data Gr photographed at night is input.

　　　　The parameter set PS1 is a group of parameters obtained by executing the learning phase of the above-described neural network architecture NNA with use of only the RAW image data Gr photographed in the daytime.

　　　　In addition, the parameter set PS2 is a group of parameters obtained by executing the learning phase of the neural network architecture NNA with use of only the RAW image data Gr photographed at night.

　　　　The parameter set PS1 and the parameter set PS2 are merely examples.　The parameter set PS1 and the parameter set PS2 switch the parameter set PS in accordance with the difference in time, but the parameter set PS in accordance with a season such as summer or winter may be prepared in addition thereto.

　　　　Alternatively, the parameter set PS may be prepared in accordance with a subject.　For example, a parameter set PSa in the case where the subject is a human and a parameter set PSb in the case where the subject is a vehicle may be provided.　In this case, for example, the parameter set PSa in the case where the subject is a human may include the model parameter PMm for performing predetermined inference processing related to a human, such as estimating a posture.　Then, the parameter set PSb in the case where the subject is a vehicle may include the model parameter PMm for performing inference processing different from that in the case where the subject is a human, such as obtaining a vehicle number.

　　　　That is, the parameter set PS selected as a result of the inference for the RAW image data Gr performed by the predictor M2 may not only simply change the mode of the predetermined image processing in the image processing unit 24B, but may also change the inference processing in the inference processing unit 25B to one with a completely different property.

　　　　When the parameter set PS is changed by the selection of the predictor M2, the inference processing to be executed is changed in accordance with the photographed RAW image data Gr, and the predetermined image processing for obtaining the optimum processed image data Gp when realizing the inference processing can be realized.

　　　　An example of a specific data flow among the prior determination processing unit 29, the image processing unit 24B, and the inference processing unit 25B is depicted in FIG. 17, FIG. 18, and FIG. 19.

　　　　The prior determination processing unit 29 is configured with, for example, a DSP and the like.　The image processing unit 24 is configured with, for example, a programmable accelerator and the like.　In addition, the inference processing unit 25B is configured with a DSP and the like.

　　　　The predictor M2 is deployed in the prior determination processing unit 29.　On the basis of the estimation result of the predictor M2, the prior determination processing unit 29 selects a parameter set PS according to the estimation result from a plurality of parameter sets PS and applies it to the image processing unit 24B and the inference processing unit 25B.　At this time, the prior determination processing unit 29 may acquire the parameter set PS itself and transmit it to the image processing unit 24B and the inference processing unit 25B.　Alternatively, the prior determination processing unit 29 may transmit selection information of the parameter set PS to the image processing unit 24B and the inference processing unit 25B, and cause the image processing unit 24B and the inference processing unit 25B to acquire and apply the parameters.

　　　　It should be noted that the prior determination processing unit 29 may apply the parameters to the image processing unit 24B and the inference processing unit 25B via the control unit 23.

　　　　The prior determination processing unit 29 further transmits the RAW image data Gr input from the image sensor 22 in the preceding stage without any change to the image processing unit 24B.　As a result, the RAW image data Gr output from the image sensor 22 is input to the image processing unit 24B.

　　　　It should be noted that the imaging apparatus 2B may adopt a configuration in which the RAW image data Gr is transmitted directly from the image sensor 22 to the image processing unit 24B without using the prior determination processing unit 29.

　　　　The inference processing unit 25B performs high-accuracy inference processing in accordance with the purpose by applying the parameters based on the result of the estimation processing in the prior determination processing unit 29 to the deployed learned model M1.

　　　　It should be noted that, in the case where the content of the inference processing is to be significantly changed, it is considered that the structure itself of the learned model M1 is changed.　In this case, as depicted in FIG. 18, a configuration in which a plurality of learned models M1a and learned models M1b having different network structures, and the like are deployed in the inference processing unit 25B may be adopted.

　　　　The prior determination processing unit 29 selects the learned model M1 to be used from the learned models M1a, M1b, and the like in accordance with the inference result of the predictor M2, and further applies the selected parameter to the selected learned model M1.
　　　　That is, the prior determination processing unit 29 may have a function of selecting the learned model M1.

　　　　It should be noted that the function of the prior determination processing unit 29 can be included in an inference processing unit 25C.

　　　　Specifically, as depicted in FIG. 19, both the learned model M1 and the predictor M2 may be deployed in the inference processing unit 25C.　Further, the learned model M1 to be deployed in the inference processing unit 25C may be a plurality of learned models M1a and learned models M1b, and the like.

　　　　The image sensor 22 inputs the RAW image data Gr not only to an image processing unit 24C but also to the inference processing unit 25C.

　　　　The inference processing unit 25 inputs the RAW image data Gr to the predictor M2, and applies the parameter set PS selected in accordance with the inference result thereof to the image processing unit 24C and the learned model M1.

　　　　The image processing unit 24C performs the predetermined image processing in accordance with the selected parameter set PS on the RAW image data Gr input from the image sensor 22, and supplies the processed image data Gp to the learned model M1 of the inference processing unit 25C.

　　　　That is, the inference processing unit 25C performs the predetermined inference processing while appropriately switching between the learned model M1 and the predictor M2.

　　　　Accordingly, the number of chips and the like configuring the processing unit used for the inference processing can be reduced, and cost reduction and downsizing can be achieved.

　　　　An example of a flow of processing executed in the imaging apparatus 2B of the information processing system 1B in the second embodiment is depicted in FIG. 20.　It should be noted that, as described above, each processing executed by the prior determination processing unit 29 may be realized by the inference processing unit 25C in which the predictor M2 is deployed.

　　　　In Step S101, the imaging apparatus 2B allows the image sensor 22 to execute frame imaging processing.　This processing may be realized by the control unit 23 controlling the image sensor 22.

　　　　In Step S102, the imaging apparatus 2B allows the prior determination processing unit 29 to perform inference processing with use of the predictor M2.　This processing is processing for obtaining the selection information of the parameter set PS based on the character amount of the RAW image data Gr.

　　　　In Step S103, the prior determination processing unit 29 of the imaging apparatus 2B determines whether or not the parameter needs to be changed.　Specifically, the prior determination processing unit 29 determines whether or not the parameter set PS selected in Step S102 is currently being set, and determines as “No” in Step S103 in the case where it is determined that it is being set, and determines as “Yes” in Step S103 in the case where it is determined that it is not being set.

　　　　In the case where it is determined as “Yes” in Step S103, the prior determination processing unit 29 of the imaging apparatus 2B applies the newly selected parameter set PS to the image processing unit 24B and the inference processing unit 25B in Step S104 and changes it.

　　　　Conversely, in the case where it is determined as “No” in Step S103, the imaging apparatus 2B avoids the processing of Step S104.

　　　　The inference processing unit 25B of the imaging apparatus 2B performs inference processing by the learned model M1 in Step S105.

　　　　Subsequently, the inference processing unit 25B of the imaging apparatus 2B outputs the inference result Di in Step S106.

　　　　The imaging apparatus 2B repeatedly executes the processing of Step S101 to Step S106 until a predetermined termination condition is satisfied.

　　　　It should be noted that, in the example depicted in FIG. 20, parameters can be changed for each frame.　The present technology is not limited to this, and a configuration in which the processing of Step S102 to Step S106 is executed for every several frames, or a configuration in which the processing of Step S102 to Step S106 is executed in the case where a predetermined condition is satisfied may be adopted.

　　　　The case where the predetermined condition is satisfied is, for example, a case where the change of the character amount calculated for the RAW image data Gr exceeds a threshold value, and the like, and a moving average of several frames may be used for the character amount.

<5. Other>
　　　　In the following description, in the case where the information processing system 1A, the information processing system 1B, and the like are collectively referred to, they are described as an “information processing system 1” except for the capitalized alphabet added to the end of the reference numeral.　The same applies to other reference numerals.

　　　　In the learning phase of the neural network architecture NNA, an example in which parameters to be applied to the image processing unit 24 and the inference processing unit 25 are searched for has been described.
　　　　In the learning phase of the neural network architecture NNA, setting information of the imaging optical system 21 and the image sensor 22, and setting information in the imaging operation thereof may be further obtained as search results.

　　　　For example, in the learning phase of the neural network architecture NNA, setting information such as the shutter speed in the image sensor 22 and the control positions of various types of lenses of the imaging optical system 21 may be obtained.
　　　　Then, the setting information obtained as search results may include, for example, information set for the control unit 23.

<6. Summary>
　　　　As described by use of the above-described examples, the server apparatus 3 as the information processing apparatus of the present technology includes the model generation unit (CPU 71) for generating the learned model M1 by performing learning on the neural network architecture NNA to which constraints on hardware in which the learned model M1 (M1a and M1b) is deployed are given as constraint conditions.
　　　　The learned model M1 is generated by performing learning on the neural network architecture NNA and used for predetermined processing in some cases.　In this case, by using the learned model M1 to which the optimized parameter (model parameter PMm) is applied, high-accuracy processing can be performed.
　　　　Incidentally, the processing using the learned model M1 may be realized by incorporating the parameter (model parameter PMm) calculated in the process of generating the learned model M1 into hardware with relatively low flexibility, instead of using software with relatively high flexibility.
　　　　Accordingly, fast processing can be realized at the expense of flexibility.
　　　　However, in many cases, the parameter calculated in the process of generating the learned model M1 may not be applied to hardware without any change, and it may be necessary to change the design of the hardware.　This causes an increase in cost and the like.
　　　　In this configuration, learning is performed in a state where the constraints of the hardware to which the learned model M1 is to be deployed are incorporated.　The parameter obtained by the learning is applicable without any change to the target hardware.
　　　　Therefore, the optimum processing can be realized at a high speed by use of the hardware with constraints.

　　　　As described with reference to FIG. 4, FIG. 5, and the like, in the server apparatus 3 as the information processing apparatus, the constraint conditions given to the neural network architecture NNA may be constraints on hardware functioning as the image processing unit 24 for obtaining image data by image processing.
　　　　It is considered that the predetermined processing is, for example, predetermined image processing for image data.　Specifically, there is a case where image data preferable for the inference processing in the subsequent stage is obtained from the RAW image data Gr.
　　　　Such image processing can be performed at a higher speed by realizing with the image processing unit (an ISP or the like) as hardware with relatively low flexibility than by realizing with software with relatively high flexibility.
　　　　Since the parameter obtained at the time of generating the learned model M1 is the parameter obtained in consideration of the constraints of the hardware such as an ISP, it can be applied to the hardware without any change.
　　　　Therefore, the optimum image processing can be performed by the hardware, and the processing speed can be increased while ensuring the inference accuracy.

　　　　As described with reference to FIG. 4, FIG. 5, and the like, in the server apparatus 3 as the information processing apparatus, the neural network architecture NNA may perform the predetermined image processing for obtaining the processed image data Gp from predetermined image data input as input data, and the model generation unit (CPU 71) may search for the parameters (the processing order parameter PMo and the processing content parameter PMp) related to the predetermined image processing in learning.
　　　　For example, it is considered that the predetermined image processing includes the AWB processing, the CCM processing, the denoise processing, and the like.
　　　　In such image processing, the processed image data Gp obtained after the processing differs by adjusting the processing content parameter PMp.
　　　　Then, the optimum processed image data Gp may further differ depending on the content of the inference processing in the subsequent stage.
　　　　The model generation unit (CPU 71) searches for parameters used for the image processing described above at the time of learning for the neural network architecture NNA.
　　　　In the hardware to which such parameters are applied, image processing for obtaining the processed image data Gp suitable for the processing in the subsequent stage is realized.
　　　　Therefore, the search results of parameters can be efficiently utilized.

　　　　As described with reference to FIG. 4 to FIG. 9, and the like, in the server apparatus 3 as the information processing apparatus, the predetermined image processing includes a plurality of kinds of image processing, and the model generation unit (CPU 71) may search for the parameter (the processing order parameter PMo) for deciding the processing order of the plurality of kinds of image processing as a parameter related to the predetermined image processing.
　　　　The plurality of kinds of image processing includes, for example, the AWB processing, the CCM processing, the GC processing, the denoise processing, the HSC processing, the BCC processing, and the like.
　　　　These kinds of processing are usually executed by the image processing unit 24 such as an ISP provided in the image sensor 22 or in the subsequent stage of the image sensor 22.
　　　　When the processing order of these kinds of image processing is changed, the processed image data Gp to be generated is changed.　That is, by changing the processing order, it is possible to obtain the processed image data Gp that is more suitable for the processing in the subsequent stage.
　　　　According to this configuration, when learning is performed on the neural network architecture NNA, the processing order of the plurality of kinds of image processing executed by the image processing unit 24 is searched for as the processing order parameter PMo, so that the processed image data Gp preferable for the processing in the subsequent stage can be generated by hardware such as an ISP.

　　　　As described with reference to FIG. 4, FIG. 5, and the like, in the server apparatus 3 as the information processing apparatus, the parameters related to the predetermined image processing may include the parameter (the processing content parameter PMp) conforming to the processing content for each of the plurality of kinds of image processing.
　　　　The processing unit (the AWB processing block 24a or the like) that performs various types of image processing such as the AWB processing can adjust the processing content by adjusting the parameter.　That is, by adjusting the processing content parameter PMp of each image processing included in the predetermined image processing, the processed image data Gp to be output can be adjusted.
　　　　According to this configuration, when learning is performed on the neural network architecture NNA, the processing content parameter PMp for adjusting the processing content is searched for each of the plurality of kinds of image processing executed by the image processing unit 24.　Therefore, the processed image data Gp more preferable for the processing in the subsequent stage can be generated by hardware such as an ISP.

　　　　As described with reference to FIG. 4, FIG. 5, and the like, in the server apparatus 3 as the information processing apparatus, the neural network architecture NNA may output, as output data, the inference result obtained by performing the predetermined inference processing on the processed image data Gp, and the model generation unit (CPU 71) may search for the parameter (the model parameter PMm) related to the predetermined inference processing in learning.
　　　　For example, the neural network architecture NNA is a model that performs inference processing by use of image data as an input and outputs the inference result.　Then, the neural network architecture NNA includes a structure for realizing image processing that is processing of processing the input image data, and inference processing for the processed image data Gp that has been processed.　The image processing and the inference processing are integrally executed, for example, from the input of the RAW image data Gr to the output of the inference result.
　　　　Further, when learning is performed on the neural network architecture NNA, both of various types of parameters used for the image processing and various types of parameters used for the inference processing are obtained.
　　　　By applying at least some of the parameters (the processing content parameter PMp and the processing order parameter PMo) related to the image processing and the parameter (the model parameter PMm) related to the inference processing obtained here to the hardware, the inference result of the inference processing can be obtained with high accuracy and at a high speed.　In addition, for example, there are a variety of modes in which the image processing is realized by a programmable accelerator that is relatively high-speed and low-flexibility hardware, and the inference processing is realized by software such as a relatively low-speed and high-flexibility CPU or DSP, or by a configuration with high flexibility similar to software.

　　　　As described with reference to FIG. 4, FIG. 5, and the like, in the server apparatus 3 as the information processing apparatus, the predetermined image data may be set as the RAW image data Gr.
　　　　Accordingly, the RAW image data Gr output from the image sensor 22 is subjected to image processing using the hardware to which the optimized parameters are applied.
　　　　Therefore, the optimum and high-speed image processing can be performed, and the processing in the subsequent stage can be preferably performed.

　　　　As described with reference to FIG. 3 to FIG. 5, and the like, the information processing apparatus is the server apparatus 3 including the reception unit (communication unit 80) for receiving the RAW image data Gr output from the image sensor 22, the model generation unit (CPU 71), and the transmission unit (communication unit 80) for transmitting the learned model M1, and the model generation unit (CPU 71) may perform learning on the neural network architecture NNA with use of the received RAW image data Gr.
　　　　Accordingly, the server apparatus 3 optimizes the parameters by learning of the neural network architecture NNA.　That is, the parameter search processing is executed by the information processing apparatus that is generally considered to have higher performance than an edge computer.
　　　　Therefore, the parameter search can be efficiently executed, and the optimized parameters can be acquired in a short period of time.

　　　　The information processing method of the present technology includes a step of generating the learned model M1 by performing learning on the neural network architecture NNA to which constraints on hardware in which the learned model M1 is deployed are given as constraint conditions.

　　　　The program of the present technology causes the information processing apparatus to execute a function of generating the learned model M1 by performing learning on the neural network architecture NNA to which constraints on hardware in which the learned model M1 is deployed are given as constraint conditions.
　　　　By such an information processing method or a program, the various types of working effects described above can be obtained.

　　　　It should be noted that such a program can be preliminarily recorded in an HDD (Hard Disk Drive) as a recording medium incorporated in equipment such as a computer apparatus, a ROM in a microcomputer having a CPU, or the like.　Alternatively, the program can be temporarily or permanently stored (recorded) in a removable recording medium such as a flexible disk, a CD-ROM (Compact Disk Read Only Memory), an MO (Magneto Optical) disk, a DVD (Digital Versatile Disk), a Blu-ray Disc (registered trademark), a magnetic disk, a semiconductor memory, or a memory card.　Such a removable recording medium can be provided as what is generally called a package software.
　　　　In addition, such a program can be installed from a removable recording medium to a personal computer or the like, or downloaded from a download site via a network such as a LAN or the Internet.

　　　　As described using the examples described above, the imaging apparatus 2 of the present technology includes the image processing unit 24 that performs the predetermined image processing on the RAW image data Gr output from the image sensor 22 to obtain the processed image data Gp, and the parameters used for the predetermined image processing may be parameters obtained in the process of generating the learned model M1 by performing learning on the neural network architecture NNA to which constraints on hardware in the image processing unit 24 are given as constraint conditions.
　　　　Accordingly, the parameters optimized by the learning of the neural network architecture NNA can be applied without any change to the image processing unit 24 included in the imaging apparatus 2.
　　　　Therefore, in the case where the image processing unit 24 is realized by relatively high-speed and low-flexibility hardware, it is possible to realize high-speed image processing by applying preferable parameters considering the low-flexibility of the hardware.

　　　　As described with reference to FIG. 2 and the like, the image processing unit 24 in the imaging apparatus 2 may be configured as a programmable accelerator.
　　　　For example, in the case where various types of image processing are executed by the programmable accelerator, the parameter search processing is executed in consideration of the range of values, the number of significant digits, and the like used for each image processing.
　　　　Accordingly, the theoretical processing accuracy when the learned model M1 is used can be matched with the processing accuracy when the parameters are applied to the hardware, and the processing accuracy can be prevented from deteriorating when the model is implemented on the hardware.
　　　　In addition, there may be a constraint condition under which each image processing is inseparably executed in hardware.　For example, there is a case where a configuration in which image processing is executed in the middle of another kind of image processing may not be implemented.
　　　　By performing learning by use of the neural network architecture NNA in which such a constraint condition on hardware is considered, the obtained parameters can be implemented on the hardware without any change, so that desired performance can be appropriately exhibited.

　　　　As described with reference to FIG. 2 and the like, the imaging apparatus 2 may include the inference processing unit 25 that performs the predetermined inference processing on the processed image data Gp by use of the learned model M1.
　　　　Accordingly, in the process of performing learning on the neural network architecture NNA, both the search of parameters used by the image processing unit 24 and the generation of the learned model M1 used for the inference processing can be performed at a time.
　　　　Therefore, setting of the imaging apparatus 2 for realizing desired processing can be efficiently performed.

　　　　As described with reference to FIG. 2 and the like, the image processing unit 24 in the imaging apparatus 2 executes a plurality of kinds of image processing as the predetermined image processing, and the parameters related to the predetermined image processing may include a parameter for deciding the processing order of the plurality of kinds of image processing.
　　　　Accordingly, for example, various types of image processing such as the AWB processing, the CCM processing, and the GC processing can be realized in the optimum processing order as the predetermined image processing.　Then, since these kinds of image processing are executed in consideration of constraints on hardware for realizing the image processing unit 24, they can be efficiently applied to the image processing unit 24 without any change.

　　　　As described with reference to FIG. 10 to FIG. 13, the image processing unit 24 in the imaging apparatus 2 may have a configuration (bus 26) in which the processing blocks (the AWB processing block 24a, the DMS processing block 24b, the CCM processing block 24c, the GC processing block 24d, the denoise processing block 24e, the HSC processing block 24f, the BCC processing block 24g, and the like) for performing each image processing included in a plurality of kinds of image processing are fabric connected to each other.
　　　　Accordingly, the parameter obtained in the learning of the neural network architecture NNA and related to the processing order of various types of image processing can be appropriately reflected on the image processing unit 24.

　　　　As described with reference to FIG. 12 and the like, the image processing unit 24 in the imaging apparatus 2 may have a first fabric connection (first bus 26a) in which processing blocks for image data that has not yet been subjected to demosaic are fabric connected to each other among a plurality of processing blocks, and a second fabric connection (second bus 26b) in which processing blocks for image data obtained after demosaic is processed are fabric connected to each other.
　　　　It is considered that it is not appropriate to switch the processing order between the processing block for image data that has not yet been subjected to demosaic and the processing block for image data obtained after demosaic is processed.
　　　　According to this configuration, the processing order of the processing blocks can be switched between the processing blocks before and after demosaic is processed.
　　　　Accordingly, it is not necessary to give excessive flexibility by fabric connection, and, for example, bus wiring can be simplified.
　　　　In addition, since each of the commonalization of the input/output in the first fabric connection and the commonalization of the input/output in the second fabric connection can be performed, the degree of difficulty of the commonalization of input/output data can be reduced.

　　　　As described with reference to FIG. 9 and the like, the processing order of a plurality of kinds of image processing in the imaging apparatus 2 may include a processing order in which the same image processing is executed a plurality of times.
　　　　Accordingly, the processing blocks can be executed in a more preferable order as the predetermined image processing, and the processed image data Gp suitable for the processing in the subsequent stage can be obtained.

　　　　As described with reference to FIG. 16 to FIG. 20, and the like, the imaging apparatus 2 includes the prior determination processing unit 29 for performing prior determination processing on the RAW image data Gr by use of the inference unit (predictor M2), and the image processing unit 24 may select a parameter used for the predetermined image processing in accordance with the result of the prior determination processing.
　　　　Accordingly, for example, the parameters used for the predetermined image processing can be changed between an image captured in the daytime and an image captured at night.
　　　　These daytime parameters (parameter set PS1) and nighttime parameters (parameter set PS2) are, for example, parameters obtained when learning is performed on the neural network architecture NNA in advance.
　　　　By appropriately changing the parameters on the basis of the imaging conditions or the characteristics of the captured image, it is possible to generate the processed image data Gp more preferable for the processing in the subsequent stage.

　　　　The image processing method of the present technology includes a step of performing the predetermined image processing on the RAW image data Gr output from the image sensor 22 to obtain the processed image data Gp, and a step of performing the predetermined inference processing on the processed image data Gp by use of the learned model M1.　Then, the learned model M1 is obtained by performing learning on the neural network architecture NNA to which constraints on hardware used in the step of obtaining the processed image data Gp are given as constraint conditions.
　　　　By such an image processing method, the various types of working effects described above can be obtained.

　　　　It should be noted that the effects described in the specification are merely exemplary and are not limited, and other effects may be present.

　　　　In addition, the above-described examples may be combined in any manner, and even in the case where various types of combinations are used, the various types of working effects described above can be obtained.

<7. The present technology>
(1) A system comprising: circuitry configured to generate a learned model by performing learning on a neural network architecture (NNA); apply the learned model to determine processing parameters for an inference processing operation and an image processing operation, wherein the processing parameters for the image processing operation include at least a processing order parameter and a processing content parameter; and output the determined processing parameters.

(2) The system of (1), wherein constraints on hardware for performing the image processing operation are set in the NNA used to generate the learned model.
(3) The system of (1), wherein the NNA includes a first portion corresponding to the image processing operation and a second portion corresponding to the inference processing operation.
(4) The system of (1), wherein the image processing operation performs image processing based on the processing order parameter and the processing content parameter.
(5) The system of (4), wherein the image processing operation is performed by processing blocks that each correspond to different image processing functions.
(6)　　　The system of (5), wherein
　　　the processing blocks include at least one of an auto white balancing processing block, a demosaic processing block, a color correction matric processing block, a gamma correction processing block, a denoise processing block, an hue saturation control processing block, and a brightness contrast control processing block.
(7) The system of (5), wherein a processing order of the processing blocks is dynamically adjusted based on the processing order parameter.

(8) The system (5), wherein processing applied within each processing block is adjusted based on the processing content parameter.
(9) The system of (5), wherein the processing blocks are interconnected via a fabric connection allowing for dynamic reconfiguration of processing order.
(10) The system of (5), wherein the learned model sequences the processing blocks based on a learned optimization of inference accuracy.
(11) The system of (1), wherein the inference processing operation applies inference processing using a model parameter determined by the learned model.
(12) The system of (11), wherein the model parameter is optimized during learning of the NNA.
(13) The system of (11), wherein the circuitry includes at least one of an image signal processor (ISP) and a digital signal processor (DSP) configured to perform the inference processing operation.
(14) The system of (11), wherein the inference processing operation processes image data that has been modified according to an image processing order determined by the processing order parameter.
(15) The system of (14), wherein the inference processing operation includes receiving processed image data from a last processing block in a plurality of dynamically ordered processing blocks corresponding to the image processing operation.
(16) The system of (1), wherein the processing order parameter and the processing content parameter are stored for future use in optimizing image and inference processing.
(17) The system of (16), wherein
previously stored processing parameters are used to initialize the learned model.
(18) The system of (2), wherein the constraints on the hardware include at least one of a range of values and a number of bits to be handled for each of a plurality of image processing blocks.
(19) A method for processing image data, the method comprising: generating a learned model by performing learning on a neural network architecture (NNA); applying the learned model to determine a processing order parameter and a processing content parameter for an image processing operation; applying the learned model to determine a model parameter for an inference processing operation; performing the image processing operation on image data based on the processing order parameter and the processing content parameter to generate processed image data; and performing the inference processing on the processed image data based on the model parameter.
(20) A non-transitory computer readable medium including computer program instructions, which, when executed by an information processing system, causes the information processing system to:generate a learned model by performing learning on a neural network architecture (NNA); apply the learned model to determine processing parameters for an image processing operation and inference processing operation; and output the determined processing parameters.

3, 3A: Server apparatus
22: Image sensor
24, 24A, 24B, 24C: Image processing unit
24a: AWB processing block (processing block)
24b: DMS processing block (processing block)
24c: CCM processing block (processing block)
24d: GC processing block (processing block)
24e: Denoise processing block (processing block)
24f: HSC processing block (processing block)
24g: BCC processing block (processing block)
25, 25A, 25B, 25C: Inference processing unit
29: Prior determination processing unit
71: CPU (model generation unit)
80: Communication unit (reception unit and transmission unit)
Gp: Processed image data
Gr: RAW image data
M1, M1a, M1b: Learned model
M2: Predictor (inference unit)
NNA: Neural network architecture
PMm: Model parameter (parameter)
PMo: Processing order parameter (parameter)
PMp: Processing content parameter (parameter)

Claims (20)

1. A system comprising:
   circuitry configured to
   generate a learned model by performing learning on a neural network architecture (NNA);
   apply the learned model to determine processing parameters for an inference processing operation and an image processing operation, wherein the processing parameters for the image processing operation include at least a processing order parameter and a processing content parameter; and
   output the determined processing parameters.
   
2. The system of claim 1, wherein
   constraints on hardware for performing the image processing operation are set in the NNA used to generate the learned model.　　　
   
3. The system of claim 1, wherein
   the NNA includes a first portion corresponding to the image processing operation and a second portion corresponding to the inference processing operation.
   
4. The system of claim 1, wherein
   the image processing operation performs image processing based on the processing order parameter and the processing content parameter.
   
5. The system of claim 4, wherein
   the image processing operation is performed by processing blocks that each correspond to different image processing functions.
   
6. The system of claim 5, wherein
   the processing blocks include at least one of an auto white balancing processing block, a demosaic processing block, a color correction matric processing block, a gamma correction processing block, a denoise processing block, an hue saturation control processing block, and a brightness contrast control processing block.
   
7. The system of claim 5, wherein
   a processing order of the processing blocks is dynamically adjusted based on the processing order parameter.
   
8. The system claim 5, wherein
   processing applied within each processing block is adjusted based on the processing content parameter.
   
9. The system of claim 5, wherein
   the processing blocks are interconnected via a fabric connection allowing for dynamic reconfiguration of processing order.
   
10. The system of claim 5, wherein
    the learned model sequences the processing blocks based on a learned optimization of inference accuracy.
    
11. The system of claim 1, wherein
    the inference processing operation applies inference processing using a model parameter determined by the learned model.
    
12. The system of claim 11, wherein
    the model parameter is optimized during learning of the NNA.
    
13. The system of claim 11, wherein
    the circuitry includes at least one of an image signal processor (ISP) and a digital signal processor (DSP) configured to perform the inference processing operation.
    
14. The system of claim 11, wherein
    the inference processing operation processes image data that has been modified according to an image processing order determined by the processing order parameter.
    
15. The system of claim 14, wherein
    the inference processing operation includes receiving processed image data from a last processing block in a plurality of dynamically ordered processing blocks corresponding to the image processing operation.
    
16. The system of claim 1, wherein
    the processing order parameter and the processing content parameter are stored for future use in optimizing image and inference processing.
    
17. The system of claim 16, wherein
    previously stored processing parameters are used to initialize the learned model.
    
18. The system of claim 2, wherein
    the constraints on the hardware include at least one of a range of values and a number of bits to be handled for each of a plurality of image processing blocks.
    
19. A method for processing image data, the method comprising:
    generating a learned model by performing learning on a neural network architecture (NNA);
    applying the learned model to determine a processing order parameter and a processing content parameter for an image processing operation;
    applying the learned model to determine a model parameter for an inference processing operation;
    performing the image processing operation on image data based on the processing order parameter and the processing content parameter to generate processed image data; and
    performing the inference processing on the processed image data based on the model parameter.
    
20. A non-transitory computer readable medium including computer program instructions, which, when executed by an information processing system, causes the information processing system to:
    generate a learned model by performing learning on a neural network architecture (NNA);
    apply the learned model to determine processing parameters for an image processing operation and inference processing operation; and
    output the determined processing parameters.