JP2021170727A - Device, image processor, imaging device, mobile body, program, and method - Google Patents

Abstract

To provide an image processor which improves image degradations by using a learned neural network.SOLUTION: In an imaging device, a controller has a decryption unit which stores a plurality of learned neural networks for processing a decrypted image in relation to the amount of image degradations by coding, acquires coded data generated by coding an image and the image degradation amount generated by the coding of the image, generates a decrypted image by decrypting the coded data, selects one of the plurality of learned neural networks that is related to the acquired image degradation amount, and processes the decrypted image by using the learned neural networks.SELECTED DRAWING: Figure 7

Description

The present invention relates to an apparatus, an image processing apparatus, an imaging apparatus, a moving body, a program and a method.

Non-Patent Documents 1 and 2 describe a configuration in which a filter by machine learning is inserted in a motion prediction loop in a moving image encoding device. Non-Patent Document 3 describes a technique for evaluating image quality based on structural similarity.
[Prior art literature]
[Patent Document]
[Non-Patent Document 1] Lulu Zhou, Xiaodan Song, Jiabao Yao, Li Wang, Fangdong Chen, "JVET-I0022: Convolutional neural network Video filter (CNNF) for Korea, January 2018 [Non-Patent Document 2] Jiabao Yao, Xiaodan Song, Shuqing Fang, Li Wang, "AHG9: Convolutional Neural Network, Network, Network, 20S July [Non-Patent Document 3] Zhou Wang, Alan C.I. Bobik, Hamid R. Sheikh, Eero P.M. Simoncelli, "Image Quality Assessment: From Error Visibility to Structural Similarity", IEEE Trans on Image Processing, April 200, No. 4, Vol. 13, No. 4, No.

The apparatus according to the first aspect of the present invention includes a circuit configured to store a plurality of learned neural networks for processing the decoded image in correspondence with the amount of image quality deterioration due to coding. The circuit is configured to acquire the coded data generated by the coding of the image and the amount of image quality deterioration caused by the coding of the image. The circuit is configured to generate a decoded image by decoding the coded data. The circuit is configured to select a trained neural network associated with the acquired image quality deterioration amount from a plurality of trained neural networks. The circuit is configured to process the decoded image using the trained neural network of choice.

The amount of image quality deterioration is calculated from the unencoded image and the decoded image (i) Peak signal-to-noise ratio (Peek Signal to Noise Radio), (ii) Structural similarity (SSIM) and (iii) Mean squared error. It may be at least one of the errors (Mean Squared Error).

The image may be a moving image constituting a moving image. The coded data may include quantization difference information obtained by quantization of the predicted difference information of the moving image constituent image obtained by inter-prediction or intra-prediction of the moving image. The circuit generates a difference image based on the predicted difference information obtained by a process including inverse quantization of the quantization difference information acquired from the coded data, and adds an inter-predicted image or an intra-predicted image to the difference image. Is configured to generate a decoded image. The circuit is configured to process the decoded image using the trained neural network of choice.

The image may be a moving image constituting a moving image. The circuit is configured to store a plurality of trained neural networks in association with the picture type of the moving image. The circuit is configured to select a trained neural network associated with the picture type of the moving image constituent image and the acquired image quality deterioration amount from the plurality of trained neural networks.

The neural network may be a convolutional neural network obtained by using the learning image and the coded data of the learning image as training data and performing machine learning according to the amount of image quality deterioration due to coding.

The apparatus according to the second aspect of the present invention comprises a circuit configured to store a plurality of learned neural networks for processing the decoded image data in correspondence with the amount of image quality deterioration due to coding. Be prepared. The circuit is configured to generate encoded data by encoding the image to be encoded by a coding process including inter-prediction or intra-prediction. The circuit is configured to generate a decoded image by decoding the coded data. The circuit is configured to calculate the amount of image quality deterioration caused by the coding of the coded image based on the coded image and the decoded image. The circuit is configured to output the coded data and the calculated image quality deterioration amount. The circuit is configured to select a trained neural network associated with the calculated image quality deterioration amount from the plurality of trained neural networks. The circuit is configured to generate a reference image used for inter-prediction or intra-prediction by processing the generated decoded image using a selected trained neural network.

The image processing apparatus according to the third aspect of the present invention includes an apparatus according to the first aspect and an apparatus according to the second aspect.

The image pickup apparatus according to the fourth aspect of the present invention includes the above-mentioned apparatus and an image sensor for generating an image.

The moving body according to the fifth aspect of the present invention moves with the above-mentioned imaging device.

The moving body may be an unmanned aerial vehicle.

The program according to the sixth aspect of the present invention causes the computer to function as the above-mentioned device. The program may be recorded on a non-temporary recording medium.

The method according to the seventh aspect of the present invention includes a step of storing a plurality of trained neural networks for processing the decoded image in association with the amount of image quality deterioration due to coding. The method includes a step of acquiring the coded data generated by the coding of the image and the amount of image quality deterioration caused by the coding of the image. The method comprises the step of generating a decoded image by decoding the encoded data. The method includes a step of selecting a trained neural network corresponding to the acquired image quality deterioration amount from a plurality of trained neural networks. The method comprises processing the decoded image using the trained neural network of choice.

The method according to the eighth aspect of the present invention includes a step of storing a plurality of trained neural networks for processing the decoded image data in association with the amount of image quality deterioration due to coding. The method comprises a step of generating encoded data by encoding the image to be encoded by a coding process including inter-prediction or intra-prediction. The method comprises the step of generating a decoded image by decoding the encoded data. The method includes a step of calculating the amount of image quality deterioration caused by the coding of the coded image based on the coded image and the decoded image. The method includes a step of outputting the coded data and the calculated image quality deterioration amount. The method includes a step of selecting a trained neural network corresponding to the calculated image quality deterioration amount from a plurality of trained neural networks. The method comprises the step of generating a reference image used for inter-prediction or intra-prediction by processing the generated decoded image using the selected trained neural network.

According to the above aspect of the present invention, the image can be appropriately encoded or decoded.

The outline of the above invention does not list all the necessary features of the present invention. Sub-combinations of these feature groups can also be inventions.

It is a figure which shows an example of the external perspective view of the image pickup apparatus 100 which concerns on this embodiment. It is a figure which shows the functional block of the image pickup apparatus 100 which concerns on this embodiment. The block diagram of the learner is shown. Class information for classifying based on PSNR is shown. The parameter information of the neural network is shown. The block configuration of the encoder included in the control unit 110 is shown. The block configuration of the decoder included in the control unit 110 is shown. The flowchart of the process to be executed when the control unit 110 encodes a picture to be encoded is shown. The flowchart of the process executed when the control unit 110 decodes the decoding target picture is shown. An example of the reference relationship of the picture used in the inter-prediction is shown. An example of an unmanned aerial vehicle (UAV) is shown. An example of a computer 1200 in which a plurality of aspects of the present invention may be embodied in whole or in part is shown.

Hereinafter, the present invention will be described through embodiments of the invention, but the following embodiments do not limit the invention according to the claims. Also, not all combinations of features described in the embodiments are essential to the means of solving the invention. It will be apparent to those skilled in the art that various changes or improvements can be made to the following embodiments. It is clear from the description of the claims that such modified or improved forms may also be included in the technical scope of the present invention.

The claims, description, drawings, and abstracts include matters that are subject to copyright protection. The copyright holder will not object to any person's reproduction of these documents as long as they appear in the Patent Office files or records. However, in other cases, all copyrights are reserved.

Various embodiments of the present invention may be described with reference to flowcharts and block diagrams, wherein the block is (1) a stage of the process in which the operation is performed or (2) a device having a role of performing the operation. May represent the "part" of. Specific steps and "parts" may be implemented by programmable circuits and / or processors. Dedicated circuits may include digital and / or analog hardware circuits. It may include integrated circuits (ICs) and / or discrete circuits. Programmable circuits may include reconfigurable hardware circuits. Reconfigurable hardware circuits include logical AND, logical OR, logical XOR, logical NAND, logical NOR, and other logical operations, flip-flops, registers, field programmable gate arrays (FPGA), programmable logic arrays (PLA), etc. It may include a memory element such as.

The computer-readable medium may include any tangible device capable of storing instructions executed by the appropriate device. As a result, the computer-readable medium having the instructions stored therein will include the product, including instructions that can be executed to create means for performing the operation specified in the flowchart or block diagram. Examples of computer-readable media may include electronic storage media, magnetic storage media, optical storage media, electromagnetic storage media, semiconductor storage media, and the like. More specific examples of computer-readable media include floppy® disks, diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), Electrically erasable programmable read-only memory (EEPROM), static random access memory (SRAM), compact disc read-only memory (CD-ROM), digital versatile disc (DVD), Blu-ray® disc, memory stick, An integrated circuit card or the like may be included.

Computer-readable instructions may include either source code or object code written in any combination of one or more programming languages. Source code or object code includes traditional procedural programming languages. Traditional procedural programming languages are assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcodes, firmware instructions, state-setting data, or Smalltalk®, JAVA®, C ++. It may be an object-oriented programming language such as, and a "C" programming language or a similar programming language. Computer-readable instructions are applied locally or to a processor or programmable circuit of a general purpose computer, special purpose computer, or other programmable data processing device, or a wide area network (WAN) such as a local area network (LAN), the Internet, etc. ) May be provided. The processor or programmable circuit may execute computer-readable instructions to create means for performing the operations specified in the flowchart or block diagram. Examples of processors include computer processors, processing units, microprocessors, digital signal processors, controllers, microcontrollers and the like.

FIG. 1 is a diagram showing an example of an external perspective view of the image pickup apparatus 100 according to the present embodiment. FIG. 2 is a diagram showing a functional block of the image pickup apparatus 100 according to the present embodiment.

The image pickup apparatus 100 includes an image pickup section 102 and a lens section 200. The imaging unit 102 includes an image sensor 120, a control unit 110, a memory 130, an indicator unit 162, and a display unit 160.

The image sensor 120 may be composed of a CCD or CMOS. The image sensor 120 receives light through the lens 210 included in the lens unit 200. The image sensor 120 outputs the image data of the optical image formed through the lens 210 to the control unit 110.

The control unit 110 may be composed of a CPU, a microprocessor such as an MPU, a microcontroller such as an MCU, or the like. The memory 130 may be a computer-readable recording medium and may include at least one of flash memories such as SRAM, DRAM, EPROM, EEPROM, and USB memory. The control unit 110 corresponds to the circuit. The memory 130 stores a program or the like necessary for the control unit 110 to control the image sensor 120 or the like. The memory 130 may be provided inside the housing of the image pickup apparatus 100. The memory 130 may be provided so as to be removable from the housing of the image pickup apparatus 100.

The instruction unit 162 is a user interface that receives an instruction to the image pickup apparatus 100 from the user. The display unit 160 displays an image captured by the image sensor 120 and processed by the control unit 110, various setting information of the image pickup device 100, and the like. The display unit 160 may be composed of a touch panel.

The control unit 110 controls the lens unit 200 and the image sensor 120. For example, the control unit 110 controls the focal position and focal length of the lens 210. The control unit 110 controls the lens unit 200 by outputting a control command to the lens control unit 220 included in the lens unit 200 based on the information indicating the instruction from the user.

The lens unit 200 includes one or more lenses 210, a lens driving unit 212, a lens control unit 220, and a memory 222. In this embodiment, one or more lenses 210 are collectively referred to as "lens 210". The lens 210 may include a focus lens and a zoom lens. At least some or all of the lenses included in the lens 210 are movably arranged along the optical axis of the lens 210. The lens unit 200 may be an interchangeable lens that is detachably provided to the imaging unit 102.

The lens driving unit 212 moves at least a part or all of the lens 210 along the optical axis of the lens 210. The lens control unit 220 drives the lens drive unit 212 in accordance with a lens control command from the image pickup unit 102 to move the entire lens 210 or the zoom lens or focus lens included in the lens 210 along the optical axis direction. Perform at least one of the zoom and focus movements. The lens control command is, for example, a zoom control command, a focus control command, and the like.

The lens driving unit 212 may include a voice coil motor (VCM) that moves at least a part or all of the plurality of lenses 210 in the optical axis direction. The lens driving unit 212 may include an electric motor such as a DC motor, a coreless motor, or an ultrasonic motor. The lens driving unit 212 transmits power from the motor to at least a part or all of the plurality of lenses 210 via mechanical members such as a cam ring and a guide shaft, and makes at least a part or all of the lenses 210 an optical axis. You may move it along.

The memory 222 stores the control values for the focus lens and the zoom lens that move via the lens driving unit 212. The memory 222 may include at least one of flash memories such as SRAM, DRAM, EPROM, EEPROM, and USB memory.

The control unit 110 executes control including control of the imaging operation on the image sensor 120 by outputting a control command to the image sensor 120 based on the information indicating the user's instruction acquired through the instruction unit 162 or the like. The control unit 110 acquires an image captured by the image sensor 120. The control unit 110 performs image processing on the image acquired from the image sensor 120 and stores it in the memory 130.

The coding process and the decoding process executed by the control unit 110 will be described. The control unit 110 stores a plurality of learned neural networks for processing the decoded image in correspondence with the amount of image quality deterioration due to coding. The control unit 110 may store the trained neural network in an external memory 130. The control unit 110 may store the learned neural network in the non-volatile memory in the control unit 110. Further, in the present embodiment, the image to be encoded and the image to be decoded are moving image constituent images constituting the moving image. However, the image may be a still image.

First, the outline of the process in which the control unit 110 encodes the image to be encoded will be described. The control unit 110 generates encoded data by encoding the image to be encoded by a coding process including inter-prediction or intra-prediction. The control unit 110 generates a decoded image by decoding the coded data, and calculates the amount of image quality deterioration caused by the coding of the coded target image based on the coded target image and the decoded image. The control unit 110 outputs the coded data and the calculated image quality deterioration amount. For example, the control unit 110 records in the memory 130 compressed image data including the generated coded data, the amount of image quality deterioration, and compressed information such as a motion vector used for coding.

When generating a reference image used for inter-prediction or intra-prediction, the control unit 110 selects a trained neural network associated with the calculated image quality deterioration amount from the plurality of trained neural networks. The control unit 110 uses the selected trained neural network to process the generated decoded image to generate a reference image used for inter-prediction or intra-prediction.

The amount of image quality deterioration may be the peak signal-to-noise ratio (Peak Signal to Noise Radio) calculated from the unencoded image and the decoded image data. The amount of image quality deterioration may be a structural similarity (SSIM, Structural Similarity) calculated from the image before encoding and the decoded image data. The amount of image quality deterioration may be a mean square error (Mean Square Error) calculated from the image before encoding and the decoded image data. The amount of image quality deterioration may be indexed by the value of an arbitrary loss function based on the unencoded image and the decoded image data.

Next, the outline of the process in which the control unit 110 decodes the image to be decoded will be described. The control unit 110 acquires the coded data generated by the coding of the image and the amount of image quality deterioration caused by the coding of the image. For example, the control unit 110 reads out compressed image data including image coding data, image quality deterioration amount, and compressed information from the memory 130. The control unit 110 generates a decoded image by decoding the coded data. The control unit 110 selects a trained neural network associated with the acquired image quality deterioration amount from the plurality of trained neural networks. Then, the control unit 110 processes the decoded image using the selected trained neural network. The control unit 110 outputs the decoded image processed by using the trained neural network to, for example, the display unit 160.

As an example, the encoded data of the moving image constituent image includes the quantization difference information obtained by quantizing the prediction difference information of the moving image constituent image obtained by the inter-prediction or the intra-prediction of the moving image. For example, the predicted difference information is a DCT coefficient obtained by performing a discrete cosine transform (sometimes called "DCT transform") on the difference image between the coded image and the reference image, and the quantized difference information is It is a quantized DCT coefficient obtained by quantizing the DCT coefficient. The control unit 110 generates a difference image based on the predicted difference information obtained by a process including inverse quantization of the quantization difference information acquired from the coded data, and adds an inter-predicted image or an intra-predicted image to the difference image. By doing so, a decoded image is generated. The control unit 110 processes the decoded image using the selected trained neural network.

The control unit 110 may store a plurality of trained neural networks in association with the picture type of the moving image constituent image. The control unit 110 may select a trained neural network corresponding to the picture type of the moving image constituent image and the acquired image quality deterioration amount from the plurality of trained neural networks.

The "learned neural network" is a convolutional neural network obtained by using a learning image and the coded data of the learning image as training data and performing machine learning according to the amount of image quality deterioration due to coding (a convolutional neural network). CNN) may be. In relation to FIGS. 4 to 10, a mode in which a “CNN filter” that performs image processing using a CNN is adopted will be described.

FIG. 3 shows a block diagram of the learner. FIG. 4 shows class information for classifying based on PSNR. The learner is a device that performs machine learning that generates neural network parameters (CNN parameters) used by the control unit 110. The control unit 110 does not need to include a learning device.

The learner generates CNN parameters by performing machine learning using the input image and the deteriorated image. The degraded image is a decoded image generated by decoding the coded image generated by coding the input image. The decoded image is an image that has undergone image processing by a loop filter included in the decoder. The loop filter will be described later.

The image quality measuring unit 310 measures the amount of image quality deterioration based on the input image and the deteriorated image. In this embodiment, a mode in which PSNR is adopted as the amount of image quality deterioration will be described. The image quality measuring unit 310 calculates the PSNR based on the input image and the deteriorated image. PSNR is defined by the following equation.

MSE is the mean square error. For example, MSE is a value calculated by the mean square error of the difference between the pixel values of each color. MAX _I is the maximum value that the pixel value of the image can take. For example, when the pixel value is represented by 8 bits, MAX _I is 255.

The class determination unit 320 determines the class based on the class information shown in FIG. 4 and the PSNR. As shown in FIG. 4, the class information is information that associates the class identifier with the PSNR range. The class determination unit 320 specifies a range including the calculated PSNR value among a plurality of PSNR ranges in the class information, and specifies a class identifier associated with the specified range.

The CNN learner 330 calculates the CNN parameters constituting the CNN filter described later by performing machine learning using the input image and the deteriorated image for each class determined by the class determination unit 320. Specifically, the CNN learner 330 sets weights and offsets that minimize a predetermined loss function based on the image generated by applying the CNN filter to the degraded image and the input image. calculate.

FIG. 5 shows the parameter information of the neural network. The parameter information is information that associates the CNN parameter calculated by machine learning with the class identifier. CNN parameters include weighted values and offsets that make up the CNN. The control unit 110 stores the class information shown in FIG. 4 and the parameter information shown in FIG. The control unit 110 determines a class with reference to the class information as a part of the image coding process and the decoding process, and executes the image processing using the CNN parameter determined from the determined class and the parameter information.

FIG. 6 shows a block configuration of a encoder included in the control unit 110. A plurality of time-series pictures as moving image constituent images are input to the encoder as input image data to be encoded. The reorder unit 610 determines the order in which the pictures are coded based on the picture type. For example, the reorder unit 610 arranges the order of the pictures to be encoded so as to encode the I picture or the P picture after the B picture before encoding the B picture encoded by the bidirectional prediction. Change.

The orthogonal transform unit 620 calculates the DCT coefficient by performing DCT conversion of the difference image between the picture output from the reorder unit 610 and the reference image. The quantization unit 630 generates a quantization DCT coefficient by quantizing the DCT coefficient output from the orthogonal transform unit 620. The quantization unit 630 adjusts the quantization parameters used for the quantization of the DCT coefficient based on the compression rate output from the rate control unit 660 described later. The entropy coding unit 650 generates a coded picture by applying entropy coding to the quantized DCT coefficient output by the quantization unit 630. The buffer 670 stores the coded picture output by the entropy coding unit 650. The rate control unit 660 determines the compression rate based on the amount of data in the coded picture and outputs the compression rate to the quantization unit 630.

Next, the processing of the loop structure included in the encoder will be described. The inverse quantization unit 641 dequantizes the quantization DCT coefficient output from the quantization unit 630. The inverse orthogonal transform unit 642 generates a difference image by performing an inverse DCT transform on the output of the inverse quantization unit 641. The loop filter 643 applies a filter process to the image information obtained by adding the reference image to the difference image generated by the inverse orthogonal transform unit 642. The loop filter 643 may include, for example, a deblocking filter. The image generated by the loop filter 643 is output to the CNN filter 644 and the image quality measuring unit 646.

The image generated by the loop filter 643 and the input image data are input to the image quality measuring unit 646. The image quality measuring unit 646 calculates the PNSR based on the image generated by the loop filter 643 and the input image data. The class determination unit 647 determines the class identifier using the PSNR calculated by the image quality measurement unit 646 and the class information described in relation to FIG. The class identifier determined by the class determination unit 647 is output to the CNN filter 644 and the entropy coding unit 650.

The CNN filter 644 is a filter composed of CNNs formed by the CNN parameters generated by the learner described above. The CNN filter 644 generates a reference picture by performing a convolution operation on the image output by the loop filter 643 using the CNN. Specifically, the CNN filter 644 refers to the parameter information, and the CNN filter composed of the CNN parameters associated with the class identifier determined by the class determination unit 647 is processed by the loop filter 643. To generate a reference picture. The memory 645 stores a reference picture generated by the CNN filter 644. The intra prediction unit 648 uses the reference picture stored in the memory 645 to perform intra prediction for encoding the coded target picture, and generates an intra prediction image as a reference image. The inter-prediction unit 649 uses the reference picture stored in the memory 645 to perform inter-prediction for encoding another coded target picture, and generates an inter-prediction image as a reference image. The inter-prediction unit 649 may generate an inter-prediction image by, for example, calculating a motion vector and performing motion compensation.

The entropy coding unit 650 entropy-codes the coded picture, the compressed information such as the motion vector, and the PSNR calculated by the image quality measuring unit 646, and stores them in the buffer 670. The control unit 110 records the compressed image data including the information stored in the buffer 670 by the entropy coding unit 650 in the memory 130 or the like.

FIG. 7 shows a block configuration of a decoder included in the control unit 110. The control unit 110 reads the compressed moving image data from the memory 130 and decodes it. The entropy decoding unit 750 acquires the quantized DCT coefficient and PSNR by entropy decoding the compressed image data read from the memory 130. The inverse quantization unit 741 dequantizes the quantization DCT coefficient output from the entropy decoding unit 750. The inverse orthogonal transform unit 742 generates a difference image by performing an inverse DCT transform on the output of the inverse quantization unit 741. The loop filter 743 filters the image information obtained by adding the reference image to the difference image generated by the inverse orthogonal transform unit 742. The loop filter 743 may include, for example, a deblocking filter. The loop filter 743 may be the same filter as the loop filter 643. The image generated by the loop filter 743 is output to the CNN filter 744.

The class determination unit 747 determines the class identifier using the PSNR output from the entropy decoding unit 750 and the class information. Information indicating the class identifier determined by the class determination unit 747 is output to the CNN filter 744.

The CNN filter 744 is a CNN formed by the CNN parameters generated by the learner described above. The CNN filter 744 generates a decoded picture by performing a convolution operation on the image output by the loop filter 743 using a neural network. Specifically, the CNN filter 744 refers to the parameter information, and the CNN filter composed of the CNN parameters associated with the class identifier determined by the class determination unit 747 is processed by the loop filter 743. To generate a decoded picture. The reordering unit 710 performs a process of rearranging the decoded pictures in chronological order based on the picture type, and outputs the decoded image data. The decoded image data is used, for example, for displaying an image on the display unit 160.

The memory 745 stores the decoded picture generated by the CNN filter 744. The intra prediction unit 748 uses the decoded picture stored in the memory 745 as a reference picture to perform intra prediction for encoding the decoding target picture, and generates an intra prediction image as a reference image. The inter-prediction unit 749 uses the decoded picture stored in the memory 745 as a reference picture, performs inter-prediction for encoding another decoding target picture, and obtains an inter-prediction image as a reference image. Generate. The inter-prediction unit 749 may generate an inter-prediction image by, for example, calculating a motion vector and performing motion compensation based on the motion vector.

FIG. 8 shows a flowchart of processing executed when the control unit 110 encodes the coded target picture. In S810, the control unit 110 encodes the picture to be encoded. Specifically, the orthogonal transform unit 620 performs orthogonal transform of the picture to be encoded to calculate the DCT coefficient, and the quantization unit 630 quantizes the calculated DCT coefficient to generate the quantization DCT coefficient. ..

In S820, the coded data generated by the coding in S810 is decoded and a loop filter is applied. Specifically, the inverse quantization unit 641 inversely quantizes the quantization DCT coefficient, and the inverse orthogonal transform unit 642 inversely transforms the DCT coefficient obtained by the inverse orthogonal transform to generate a difference image. Then, the loop filter 643 is applied to the generated difference image.

In S830, the image quality measuring unit 646 calculates the PSNR. In S840, the class determination unit 647 refers to the class information and determines the class identifier based on the PSNR calculated in S830.

In S850, the picture generated by applying the loop filter 643 to the difference image in S820 is image-processed by the CNN filter 644 composed of the CNN parameters associated with the class identifier to generate a reference picture. ..

In S860, compressed image data is generated by entropy coding the coded data including the quantized DCT coefficient generated in S810 and the information including PSNR generated in S830, and the generated compressed image data is output to the memory 130. do.

FIG. 9 shows a flowchart of processing executed when the control unit 110 decodes the decoding target picture. In S910, the entropy decoding unit 750 acquires the coded data and the PSNR of the image by entropy decoding the compressed image data. In S920, the coded data acquired in S910 is decoded. Specifically, the inverse quantization unit 741 inversely quantizes the quantization DCT coefficient included in the coded data, and the inverse orthogonal transform unit 742 inversely transforms the DCT coefficient obtained by the inverse quantization. , Generate a difference image. Further, the prediction image generated by the intra prediction unit 748 or the inter prediction unit is added to the difference image. Subsequently, in S930, the loop filter 743 is applied to the image generated in S920.

In S940, the class determination unit 747 determines the class identifier based on the PSNR acquired in S910 with reference to the class information, and the image to which the loop filter is applied in S930 is associated with the CNN parameter associated with the class identifier. It is processed by the CNN filter 644 configured by the above to generate a decoded picture. As described above, the decoded picture is output as the decoded image data and is used as a reference picture.

In the form described in relation to FIGS. 6 to 9, PSNR is included in the compressed image data and output. However, information other than PSNR may be adopted as the image quality deterioration amount information to be included in the compressed image data. For example, the class identifier determined by the class determination unit 647 may be included in the compressed image data as image quality deterioration amount information.

FIG. 10 shows an example of the reference relationship of the pictures used in the inter-prediction. In FIG. 10, H. The reference relationships of the I picture 1000, the P picture 1004, the Straight-B picture 1002, the Non-Story-B picture 1001, and the Non-Story-B picture 1003 used in an image coding method such as 265 are shown. As shown in FIG. 10, the reference relationship differs depending on the picture type. Therefore, the amount of image quality deterioration may differ depending on the picture type. Therefore, in addition to the above-mentioned classification based on PSNR, classification may be performed according to the picture type. Specifically, the class classification may be performed by setting the class identifier for each combination of the picture type and the PSNR. In machine learning, the learner may perform machine learning for each combination of picture type and PSNR, and calculate a CNN parameter for each combination of picture type and PSNR. The control unit 110 may store the CNN parameter in association with the combination of the picture type and the PSNR, and configures the CNN filter using the CNN parameter associated with the combination of the picture type and the PSNR in encoding and decoding. You can do it.

In the above-described embodiment, the PSNR of the entire image is calculated, the CNN filter corresponding to the calculated PSNR is selected, and the PSNR is applied to the image to which the loop filter is applied. However, a CNN filter corresponding to PSNR may be selected for each subregion of the image, and the CNN filter selected for each subregion may be applied to each subregion of the image to which the loop filter is applied. For example, the PSNR is calculated for each of the plurality of subregions, the class identifier is determined for each of the plurality of subregions, and each of the plurality of subregions in the image to which the loop filter is applied is determined for each subregion. A CNN filter composed of CNN parameters corresponding to the class identifier may be applied. The "partial region" may be an region having an arbitrary shape. For example, slices that are divided along the column direction of the image or tiles that divide the image into a rectangular shape may be applied as a "partial area".

As a modification of the above embodiment, a mode not provided with the loop filter 643 and the loop filter 743 may be adopted.

Non-Patent Document 1 and Non-Patent Document 2 described above describe that the quantization parameter is fixed and the filter parameter is learned. In general, in the compression coding of moving images, the amount of image quality deterioration due to coding differs between the case where the compression rate is low and the case where the compression rate is high. In a encoder that is actually used commercially, the quantization parameter is usually made variable in the picture and the quantization matrix is often applied, so that the amount of image quality deterioration occurs. Therefore, if the filter parameters learned by fixing the quantization parameters as described in Non-Patent Document 1 and Non-Patent Document 2 are applied to a commercial encoder, the coding efficiency may deteriorate. On the other hand, according to the present embodiment, since the CNN parameter is selected by classifying based on the amount of image quality deterioration such as PSNR, an appropriate filter according to the compression ratio is selected in the loop structure of inter-prediction or intra-prediction. Can be applied. As a result, the coding efficiency may be improved.

The image pickup apparatus 100 as described above may be mounted on a moving body. The imaging device 100 may be mounted on an unmanned aerial vehicle (UAV) as shown in FIG. The UAV 10 may include a UAV main body 20, a gimbal 50, a plurality of image pickup devices 60, and an image pickup device 100. The gimbal 50 and the imaging device 100 are examples of an imaging system. The UAV 10 is an example of a moving body propelled by a propulsion unit. The moving body is a concept including a UAV, a flying object such as another aircraft moving in the air, a vehicle moving on the ground, a ship moving on the water, and the like.

The UAV main body 20 includes a plurality of rotor blades. The plurality of rotor blades are an example of a propulsion unit. The UAV main body 20 flies the UAV 10 by controlling the rotation of a plurality of rotor blades. The UAV body 20 flies the UAV 10 using, for example, four rotor blades. The number of rotor blades is not limited to four. Further, the UAV 10 may be a fixed-wing aircraft having no rotor blades.

The imaging device 100 is an imaging camera that captures a subject included in a desired imaging range. The gimbal 50 rotatably supports the imaging device 100. The gimbal 50 is an example of a support mechanism. For example, the gimbal 50 rotatably supports the image pickup device 100 on a pitch axis using an actuator. The gimbal 50 further rotatably supports the image pickup device 100 around each of the roll axis and the yaw axis by using an actuator. The gimbal 50 may change the posture of the image pickup device 100 by rotating the image pickup device 100 around at least one of the yaw axis, the pitch axis, and the roll axis.

The plurality of image pickup devices 60 are sensing cameras that image the surroundings of the UAV 10 in order to control the flight of the UAV 10. Two imaging devices 60 may be provided on the front surface, which is the nose of the UAV 10. Yet two other imaging devices 60 may be provided on the bottom surface of the UAV 10. The two image pickup devices 60 on the front side may form a pair and function as a so-called stereo camera. The two image pickup devices 60 on the bottom surface side may also be paired and function as a stereo camera. Three-dimensional spatial data around the UAV 10 may be generated based on the images captured by the plurality of imaging devices 60. The number of image pickup devices 60 included in the UAV 10 is not limited to four. The UAV 10 may include at least one imaging device 60. The UAV 10 may be provided with at least one imaging device 60 on each of the nose, nose, side surface, bottom surface, and ceiling surface of the UAV 10. The angle of view that can be set by the image pickup device 60 may be wider than the angle of view that can be set by the image pickup device 100. The image pickup apparatus 60 may have a single focus lens or a fisheye lens.

The remote control device 300 communicates with the UAV 10 to remotely control the UAV 10. The remote control device 300 may wirelessly communicate with the UAV 10. The remote control device 300 transmits to the UAV 10 instruction information indicating various commands related to the movement of the UAV 10, such as ascending, descending, accelerating, decelerating, advancing, reversing, and rotating. The instruction information includes, for example, instruction information for raising the altitude of the UAV 10. The instruction information may indicate the altitude at which the UAV 10 should be located. The UAV 10 moves so as to be located at an altitude indicated by the instruction information received from the remote control device 300. The instruction information may include an ascending instruction to ascend the UAV 10. The UAV10 rises while accepting the rise order. Even if the UAV10 accepts the ascending command, the ascending may be restricted if the altitude of the UAV10 has reached the upper limit altitude.

FIG. 12 shows an example of a computer 1200 in which a plurality of aspects of the present invention may be embodied in whole or in part. The program installed on the computer 1200 can cause the computer 1200 to function as an operation associated with the device according to an embodiment of the present invention or as one or more "parts" of the device. For example, a program installed on a computer 1200 can cause the computer 1200 to function as a control unit 110. Alternatively, the program may cause the computer 1200 to perform the operation or the function of the one or more "parts". The program can cause a computer 1200 to perform a process or a step of the process according to an embodiment of the present invention. Such a program may be run by the CPU 1212 to cause the computer 1200 to perform certain operations associated with some or all of the blocks in the flowcharts and block diagrams described herein.

The computer 1200 according to this embodiment includes a CPU 1212 and a RAM 1214, which are connected to each other by a host controller 1210. The computer 1200 also includes a communication interface 1222, an input / output unit, which are connected to the host controller 1210 via an input / output controller 1220. The computer 1200 also includes a ROM 1230. The CPU 1212 operates according to the programs stored in the ROM 1230 and the RAM 1214, thereby controlling each unit.

Communication interface 1222 communicates with other electronic devices via a network. The hard disk drive may store programs and data used by the CPU 1212 in the computer 1200. The ROM 1230 stores in it a boot program or the like executed by the computer 1200 at the time of activation and / or a program depending on the hardware of the computer 1200. The program is provided via a computer-readable recording medium such as a CR-ROM, USB memory or IC card or network. The program is installed in RAM 1214 or ROM 1230, which is also an example of a computer-readable recording medium, and is executed by CPU 1212. The information processing described in these programs is read by the computer 1200 and provides a link between the program and the various types of hardware resources described above. The device or method may be configured to implement the operation or processing of information according to the use of the computer 1200.

For example, when communication is executed between the computer 1200 and an external device, the CPU 1212 executes a communication program loaded in the RAM 1214, and performs communication processing on the communication interface 1222 based on the processing described in the communication program. You may order. Under the control of the CPU 1212, the communication interface 1222 reads the transmission data stored in the transmission buffer area provided in the RAM 1214 or a recording medium such as a USB memory, and transmits the read transmission data to the network, or The received data received from the network is written to the reception buffer area or the like provided on the recording medium.

Further, the CPU 1212 makes the RAM 1214 read all or necessary parts of a file or a database stored in an external recording medium such as a USB memory, and executes various types of processing on the data on the RAM 1214. good. The CPU 1212 may then write back the processed data to an external recording medium.

Various types of information such as various types of programs, data, tables, and databases may be stored in recording media and processed. The CPU 1212 describes various types of operations, information processing, conditional judgment, conditional branching, unconditional branching, and information retrieval described in various parts of the present disclosure with respect to the data read from the RAM 1214. Various types of processing may be performed, including / replacement, etc., and the results are written back to the RAM 1214. Further, the CPU 1212 may search for information in a file, a database, or the like in the recording medium. For example, when a plurality of entries each having an attribute value of the first attribute associated with the attribute value of the second attribute are stored in the recording medium, the CPU 1212 specifies the attribute value of the first attribute. Search for an entry that matches the condition from the plurality of entries, read the attribute value of the second attribute stored in the entry, and associate it with the first attribute that satisfies the predetermined condition. The attribute value of the second attribute obtained may be acquired.

The program or software module described above may be stored on a computer 1200 or in a computer readable storage medium near the computer 1200. Also, a recording medium such as a hard disk or RAM provided within a dedicated communication network or a server system connected to the Internet can be used as a computer readable storage medium, thereby allowing the program to be transferred to the computer 1200 over the network. offer.

Although the present invention has been described above using the embodiments, the technical scope of the present invention is not limited to the scope described in the above embodiments. It will be apparent to those skilled in the art that various changes or improvements can be made to the above embodiments. It is clear from the description of the claims that such modified or improved forms may also be included in the technical scope of the present invention.

The order of execution of operations, procedures, steps, steps, etc. in the devices, systems, programs, and methods shown in the claims, specification, and drawings is particularly "before" and "prior to". It should be noted that it can be realized in any order unless the output of the previous process is used in the subsequent process. Even if the scope of claims, the specification, and the operation flow in the drawings are explained using "first", "next", etc. for convenience, it means that it is essential to carry out in this order. It's not a thing.

10 UAV
20 UAV main unit 50 gimbal 60 imaging device 100 imaging device 102 imaging unit 110 control unit 120 image sensor 130 memory 160 display unit 162 indicator unit 200 lens unit 210 lens 212 lens drive unit 220 lens control unit 222 memory 300 remote control device 310 image quality measurement Part 320 Class determination unit 330 CNN learner 610 Reorder unit 620 Orthogonal conversion unit 630 Quantization unit 641 Inverse quantization unit 642 Inverse orthogonal conversion unit 643 Loop filter 644 CNN filter 645 Memory 646 Image quality measurement unit 647 Class determination unit 648 Intra prediction unit 649 Inter Prediction Unit 650 Entropy Coding Unit 660 Rate Control Unit 670 Buffer 710 Reordering Unit 741 Inverse Quantization Unit 742 Inverse Orthogonal Conversion Unit 743 Loop Filter 744 CNN Filter 745 Memory 747 Class Determination Unit 748 Intra Prediction Unit 748 Inter Prediction Unit 750 Entropy Decoding Unit 1000 I Picture 1001, 1003 Non-Story-B Picture 1002 Straight-B Picture 1004 P Picture 1200 Computer 1210 Host Controller 1212 CPU
1214 RAM
1220 Input / Output Controller 1222 Communication Interface 1230 ROM

Claims (13)

Stores a plurality of trained neural networks for processing the decoded image in correspondence with the amount of image quality deterioration due to coding,
The coded data generated by coding the image and the amount of image quality deterioration caused by the coding of the image are acquired.
By decoding the coded data, a decoded image is generated.
A trained neural network corresponding to the acquired image quality deterioration amount is selected from the plurality of trained neural networks.
A device comprising a circuit configured to process the decoded image using the selected trained neural network. The image quality deterioration amount is calculated from the image before encoding and the decoded image (i) Peak signal-to-noise ratio (Peek Signal to Noise Radio), (ii) Structural similarity (SSIM) and (iii). The apparatus according to claim 1, which is at least one of a mean square error (Mean Squarer). The image is a moving image constituent image constituting a moving image, and is
The coded data includes quantization difference information obtained by quantization of the predicted difference information of the moving image constituent image obtained by inter-prediction or intra-prediction of the moving image.
The circuit
A difference image is generated based on the predicted difference information obtained by a process including dequantization of the quantization difference information acquired from the coded data, and an inter-predicted image or an intra-predicted image is added to the difference image. Generates the decoded image by
The apparatus according to claim 1 or 2, which is configured to process the decoded image using the selected trained neural network. The image is a moving image constituent image constituting a moving image, and is
The circuit
The plurality of trained neural networks are stored in association with the picture type of the moving image, and the trained neural networks are stored.
According to claim 1 or 2, the trained neural network configured to select the trained neural network associated with the picture type of the moving image constituent image and the acquired image quality deterioration amount from the plurality of trained neural networks. The device described. The neural network is a convolutional neural network obtained by using a learning image and coded data of the learning image as training data and performing machine learning according to the amount of image quality deterioration due to the coding. Item 2. The apparatus according to Item 1 or 2. Stores a plurality of trained neural networks for processing the decoded image data in correspondence with the amount of image quality deterioration due to coding.
Encoded data is generated by encoding the image to be encoded by a coding process including inter-prediction or intra-prediction.
A decoded image is generated by decoding the coded data,
Based on the coded image and the decoded image, the amount of image quality deterioration caused by the coding of the coded image is calculated.
The coded data and the calculated image quality deterioration amount are output.
A trained neural network corresponding to the calculated image quality deterioration amount is selected from the plurality of trained neural networks.
An apparatus comprising a circuit configured to generate a reference image used for the inter-prediction or the intra-prediction by processing the generated decoded image using the selected trained neural network. .. The device according to claim 1 and
An image processing device including the device according to claim 6. The device according to claim 1 or 2,
An imaging device including an image sensor that generates an image. A moving body that moves with the imaging device according to claim 8. The mobile body according to claim 9, wherein the mobile body is an unmanned aerial vehicle. A program for operating a computer as the device according to claim 1 or 2. The stage of storing a plurality of trained neural networks for processing the decoded image in correspondence with the amount of image quality deterioration due to coding, and
A step of acquiring the coded data generated by coding the image and the amount of image quality deterioration caused by the coding of the image, and
The stage of generating a decoded image by decoding the coded data, and
A step of selecting a trained neural network corresponding to the acquired image quality deterioration amount from the plurality of trained neural networks, and a step of selecting the trained neural network.
A method comprising a step of processing the decoded image using the selected trained neural network. The stage of storing a plurality of trained neural networks for processing the decoded image data in correspondence with the amount of image quality deterioration due to coding, and
The stage of generating coded data by coding the image to be coded by a coding process including inter-prediction or intra-prediction, and
A step of generating a decoded image by decoding the coded data, and
A step of calculating the amount of image quality deterioration caused by coding of the coded image based on the coded image and the decoded image, and
A step of outputting the coded data and the calculated image quality deterioration amount, and
A step of selecting a trained neural network corresponding to the calculated image quality deterioration amount from the plurality of trained neural networks, and a step of selecting the trained neural network.
A method comprising a step of generating a reference image used for the inter-prediction or the intra-prediction by processing the generated decoded image using the selected trained neural network.

Images