JP2021170728A - Device, image processor, imaging device, mobile body, program, and method - Google Patents

Abstract

To provide an image processor which can improve image degradations by using a learned convolutional neural network (CNN).SOLUTION: In an imaging device, a controller has a decryption unit which stores a plurality of learned CNNs for processing a decrypted image in relation to the amount of ones of spatial frequency components of a coded image that is larger than a predetermined value, acquires coded data generated by coding of the image, acquires a plurality of spatial frequency components of the coded image from the coded data, generates a decrypted image by decrypting the coded data, acquires the amount of ones of a plurality of spatial frequency components acquired from the coded data that are larger than a predetermined value, selects one of a plurality of learned CNNs that is related to the amount of the acquired spatial frequency components, and processes the generated decrypted image by using the selected and learned CNN.SELECTED DRAWING: Figure 7

Description

The present invention relates to an apparatus, an image processing apparatus, an imaging apparatus, a moving body, a program and a method.

Non-Patent Documents 1 and 2 describe a configuration in which a filter by machine learning is inserted in a motion prediction loop in a moving image encoding device. Non-Patent Document 3 describes H. The image coding / decoding technique in 265 (ISO / IEC 23008-2 HEVC) is described.
[Prior art literature]
[Patent Document]
[Non-Patent Document 1] Lulu Zhou, Xiaodan Song, Jiabao Yao, Li Wang, Fangdong Chen, "JVET-I0022: Convolutional neural network Video filter (CNNF) for Korea, January 2018 [Non-Patent Document 2] Jiabao Yao, Xiaodan Song, Shuqing Fang, Li Wang, "AHG9: Convolutional Neural Network, Network, Network, 20S July [Non-Patent Document 3] Gary J. Sullivan, Jens-Rainer Ohm, Woo-Jin Han, and Thomas Wiegand, "Overview of the High Efficiency Video Coding. (HEVC) Standard", IEEE Transactions on Circuits and Systems for Video Technology, 12 May 2012

The apparatus according to the first aspect of the present invention processes a decoded image in association with the amount of spatial frequency components whose size exceeds a predetermined value among the spatial frequency components of the encoded image. It comprises a circuit configured to store a plurality of trained neural networks for the purpose. The circuit is configured to acquire the coded data generated by the coding of the image. The circuit is configured to obtain multiple spatial frequency components of the coded image from the coded data. It is configured to generate a decoded image by decoding the coded data. The circuit acquires the amount of spatial frequency components whose magnitude exceeds a predetermined value among the plurality of spatial frequency components acquired from the coded data. The circuit is configured to select a trained neural network associated with the amount of acquired spatial frequency components from a plurality of trained neural networks. The circuit is configured to process the generated decoded image using the trained neural network of choice.

The circuit may be configured to store a plurality of trained neural networks in association with the proportion of spatial frequency components of the encoded image whose magnitude exceeds a predetermined value. The circuit may be configured to acquire the proportion of spatial frequency components whose magnitude exceeds a predetermined value among the plurality of spatial frequency components acquired from the coded data. The circuit may be configured to select a trained neural network associated with the percentage of acquired spatial frequency components among the plurality of trained neural networks.

The predetermined value may be zero.

The image may be a moving image constituting a moving image. The coded data may include information indicating a plurality of quantization difference values obtained by quantization of the prediction difference values of the plurality of spatial frequency components obtained by inter-prediction or intra-prediction of the moving image constituent image. The circuit may be configured to store a plurality of trained neural networks in association with the amount of quantization difference values whose magnitude exceeds a predetermined value among the plurality of quantization difference values. The circuit may be configured to acquire the amount of the quantization difference value whose magnitude exceeds a predetermined value among the plurality of quantization difference values acquired from the coded data. The circuit may be configured to select a trained neural network associated with the amount of quantization difference values of the acquired spatial frequency components from a plurality of trained neural networks. The circuit generates a difference image based on the predicted difference value of the spatial frequency component obtained by inverse quantization of a plurality of quantization difference values obtained from the coded data, and the generated difference image is used as an inter-predicted image or an intra-predicted image. It may be configured to generate a decoded moving image configuration image by adding the images. The circuit may be configured to process the generated moving image construct using the trained neural network of choice.

The image may be a moving image constituting a moving image. The circuit may be configured to store a plurality of trained neural networks in association with the picture type of the moving image. The circuit may be configured to select a trained neural network associated with a picture type of encoded data and an amount of acquired spatial frequency components from a plurality of trained neural networks.

The trained neural network uses the training image and the coded data of the training image as training data, and the size of a plurality of spatial frequency components acquired from the coded data included in the training data is predetermined. It may be a convolutional neural network obtained by performing machine learning according to the amount of spatial frequency components exceeding the value.

The apparatus according to the second aspect of the present invention processes the decoded image in association with the amount of the spatial frequency component of the encoded image whose size exceeds a predetermined value. It comprises a circuit configured to store a plurality of trained neural networks for the purpose. The circuit is configured to generate coded data including information indicating predicted difference values of a plurality of spatial frequency components by coding the image to be coded by a coding process including inter-prediction or intra-prediction. The circuit is configured to output coded data. The circuit is configured to generate a decoded image by decoding the coded data. The circuit is a trained neural network in which the magnitude of the predicted difference value of a plurality of spatial frequency components in a plurality of trained neural networks exceeds a predetermined value and is associated with the amount of the predicted difference value of the spatial frequency component. Is configured to select. The circuit is configured to generate a reference image used for inter-prediction or intra-prediction by processing the generated decoded image using a selected trained neural network.

The image processing apparatus according to the third aspect of the present invention includes an apparatus according to the first aspect and an apparatus according to the second aspect.

The image pickup apparatus according to the fourth aspect of the present invention includes the above-mentioned apparatus and an image sensor for generating an image.

The moving body according to the fifth aspect of the present invention moves with the above-mentioned imaging device.

The moving body may be an unmanned aerial vehicle.

The program according to the sixth aspect of the present invention causes the computer to function as the above-mentioned device. The program may be recorded on a non-temporary recording medium.

The method according to a seventh aspect of the present invention processes a decoded image in association with the amount of spatial frequency components whose size exceeds a predetermined value among the spatial frequency components of the encoded image. It is provided with a stage of storing a plurality of trained neural networks to be used. The method comprises the step of acquiring the coded data generated by the coding of the image. The method comprises the step of acquiring a plurality of spatial frequency components of a coded image from the coded data. The method comprises the step of generating a decoded image by decoding the encoded data. The method comprises a step of acquiring the amount of spatial frequency components whose magnitude exceeds a predetermined value among the plurality of spatial frequency components acquired from the coded data. The method comprises selecting a trained neural network associated with the amount of acquired spatial frequency components from a plurality of trained neural networks. The method comprises processing the generated decoded image using the trained neural network of choice.

The method according to the eighth aspect of the present invention processes the decoded image in association with the amount of spatial frequency components whose size exceeds a predetermined value among the spatial frequency components of the encoded image. It is provided with a stage of storing a plurality of trained neural networks to be used. The method comprises a step of generating coded data including information indicating predicted difference values of a plurality of spatial frequency components by coding the image to be coded by a coding process including inter-prediction or intra-prediction. The method comprises a step of outputting encoded data. The method comprises the step of generating a decoded image by decoding the encoded data. The method is a trained neural network in which the magnitude of the predicted difference value of a plurality of spatial frequency components in a plurality of trained neural networks exceeds a predetermined value and is associated with the amount of the predicted difference value of the spatial frequency component. Prepare for the stage of selecting. It comprises a step of generating a reference image used for inter-prediction or intra-prediction by processing the generated decoded image using the selected trained neural network.

According to the above aspect of the present invention, the image can be appropriately encoded or decoded.

The outline of the above invention does not list all the necessary features of the present invention. Sub-combinations of these feature groups can also be inventions.

It is a figure which shows an example of the external perspective view of the image pickup apparatus 100 which concerns on this embodiment. It is a figure which shows the functional block of the image pickup apparatus 100 which concerns on this embodiment. The block diagram of the learner is shown. Class information for classifying based on the DCT coefficient is shown. The parameter information of the neural network is shown. The block configuration of the encoder included in the control unit 110 is shown. The block configuration of the decoder included in the control unit 110 is shown. The flowchart of the process to be executed when the control unit 110 encodes a picture to be encoded is shown. The flowchart of the process executed when the control unit 110 decodes the decoding target picture is shown. An example of the reference relationship of the picture used in the inter-prediction is shown. An example of an unmanned aerial vehicle (UAV) is shown. An example of a computer 1200 in which a plurality of aspects of the present invention may be embodied in whole or in part is shown.

Hereinafter, the present invention will be described through embodiments of the invention, but the following embodiments do not limit the invention according to the claims. Also, not all combinations of features described in the embodiments are essential to the means of solving the invention. It will be apparent to those skilled in the art that various changes or improvements can be made to the following embodiments. It is clear from the description of the claims that such modified or improved forms may also be included in the technical scope of the present invention.

The claims, description, drawings, and abstracts include matters that are subject to copyright protection. The copyright holder will not object to any person's reproduction of these documents as long as they appear in the Patent Office files or records. However, in other cases, all copyrights are reserved.

Various embodiments of the present invention may be described with reference to flowcharts and block diagrams, wherein the block is (1) a stage of the process in which the operation is performed or (2) a device having a role of performing the operation. May represent the "part" of. Specific steps and "parts" may be implemented by programmable circuits and / or processors. Dedicated circuits may include digital and / or analog hardware circuits. It may include integrated circuits (ICs) and / or discrete circuits. Programmable circuits may include reconfigurable hardware circuits. Reconfigurable hardware circuits include logical AND, logical OR, logical XOR, logical NAND, logical NOR, and other logical operations, flip-flops, registers, field programmable gate arrays (FPGA), programmable logic arrays (PLA), etc. It may include a memory element such as.

The computer-readable medium may include any tangible device capable of storing instructions executed by the appropriate device. As a result, the computer-readable medium having the instructions stored therein will include the product, including instructions that can be executed to create means for performing the operation specified in the flowchart or block diagram. Examples of computer-readable media may include electronic storage media, magnetic storage media, optical storage media, electromagnetic storage media, semiconductor storage media, and the like. More specific examples of computer-readable media include floppy® disks, diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), Electrically erasable programmable read-only memory (EEPROM), static random access memory (SRAM), compact disc read-only memory (CD-ROM), digital versatile disc (DVD), Blu-ray® disc, memory stick, An integrated circuit card or the like may be included.

Computer-readable instructions may include either source code or object code written in any combination of one or more programming languages. Source code or object code includes traditional procedural programming languages. Traditional procedural programming languages are assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcodes, firmware instructions, state-setting data, or Smalltalk®, JAVA®, C ++. It may be an object-oriented programming language such as, and a "C" programming language or a similar programming language. Computer-readable instructions are applied locally or to a processor or programmable circuit of a general purpose computer, special purpose computer, or other programmable data processing device, or a wide area network (WAN) such as a local area network (LAN), the Internet, etc. ) May be provided. The processor or programmable circuit may execute computer-readable instructions to create means for performing the operations specified in the flowchart or block diagram. Examples of processors include computer processors, processing units, microprocessors, digital signal processors, controllers, microcontrollers and the like.

FIG. 1 is a diagram showing an example of an external perspective view of the image pickup apparatus 100 according to the present embodiment. FIG. 2 is a diagram showing a functional block of the image pickup apparatus 100 according to the present embodiment.

The image pickup apparatus 100 includes an image pickup section 102 and a lens section 200. The imaging unit 102 includes an image sensor 120, a control unit 110, a memory 130, an indicator unit 162, and a display unit 160.

The image sensor 120 may be composed of a CCD or CMOS. The image sensor 120 receives light through the lens 210 included in the lens unit 200. The image sensor 120 outputs the image data of the optical image formed through the lens 210 to the control unit 110.

The control unit 110 may be composed of a CPU, a microprocessor such as an MPU, a microcontroller such as an MCU, or the like. The memory 130 may be a computer-readable recording medium and may include at least one of flash memories such as SRAM, DRAM, EPROM, EEPROM, and USB memory. The control unit 110 corresponds to the circuit. The memory 130 stores a program or the like necessary for the control unit 110 to control the image sensor 120 or the like. The memory 130 may be provided inside the housing of the image pickup apparatus 100. The memory 130 may be provided so as to be removable from the housing of the image pickup apparatus 100.

The instruction unit 162 is a user interface that receives an instruction to the image pickup apparatus 100 from the user. The display unit 160 displays an image captured by the image sensor 120 and processed by the control unit 110, various setting information of the image pickup device 100, and the like. The display unit 160 may be composed of a touch panel.

The control unit 110 controls the lens unit 200 and the image sensor 120. For example, the control unit 110 controls the focal position and focal length of the lens 210. The control unit 110 controls the lens unit 200 by outputting a control command to the lens control unit 220 included in the lens unit 200 based on the information indicating the instruction from the user.

The lens unit 200 includes one or more lenses 210, a lens driving unit 212, a lens control unit 220, and a memory 222. In this embodiment, one or more lenses 210 are collectively referred to as "lens 210". The lens 210 may include a focus lens and a zoom lens. At least some or all of the lenses included in the lens 210 are movably arranged along the optical axis of the lens 210. The lens unit 200 may be an interchangeable lens that is detachably provided to the imaging unit 102.

The lens driving unit 212 moves at least a part or all of the lens 210 along the optical axis of the lens 210. The lens control unit 220 drives the lens drive unit 212 in accordance with a lens control command from the image pickup unit 102 to move the entire lens 210 or the zoom lens or focus lens included in the lens 210 along the optical axis direction. Perform at least one of the zoom and focus movements. The lens control command is, for example, a zoom control command, a focus control command, and the like.

The lens driving unit 212 may include a voice coil motor (VCM) that moves at least a part or all of the plurality of lenses 210 in the optical axis direction. The lens driving unit 212 may include an electric motor such as a DC motor, a coreless motor, or an ultrasonic motor. The lens driving unit 212 transmits power from the motor to at least a part or all of the plurality of lenses 210 via mechanical members such as a cam ring and a guide shaft, and makes at least a part or all of the lenses 210 an optical axis. You may move it along.

The memory 222 stores the control values for the focus lens and the zoom lens that move via the lens driving unit 212. The memory 222 may include at least one of flash memories such as SRAM, DRAM, EPROM, EEPROM, and USB memory.

The control unit 110 executes control including control of the imaging operation on the image sensor 120 by outputting a control command to the image sensor 120 based on the information indicating the user's instruction acquired through the instruction unit 162 or the like. The control unit 110 acquires an image captured by the image sensor 120. The control unit 110 performs image processing on the image acquired from the image sensor 120 and stores it in the memory 130.

The coding process and the decoding process executed by the control unit 110 will be described. The control unit 110 has registered a plurality of learned spatial frequency components for processing the decoded image in association with the amount of the spatial frequency components whose size exceeds a predetermined value among the spatial frequency components of the encoded image. Memorize the neural network. The spatial frequency component is, for example, a spatial frequency component generated by orthogonal transformation. In this embodiment, the spatial frequency component is assumed to be the DCT coefficient generated by the discrete cosine transform (sometimes referred to as "DCT transform") of the image. The control unit 110 may store the learned neural network in the external memory 130. The control unit 110 may store the learned neural network in the non-volatile memory in the control unit 110. Further, in the present embodiment, the image to be encoded and the image to be decoded are moving image constituent images constituting the moving image. However, the image may be a still image.

First, the outline of the process in which the control unit 110 encodes the image to be encoded will be described. The control unit 110 encodes the coded image by a coding process including inter-prediction or intra-prediction to generate coded data including information indicating predicted difference values of a plurality of spatial frequency components. The control unit 110 outputs the generated coded data. For example, the control unit 110 records the generated coded data and the compressed image data including the compressed information such as the motion vector used for the coding in the memory 130.

The control unit 110 generates a decoded image by decoding the generated coded data when generating a reference image used for inter-prediction or intra-prediction. The control unit 110 is learning corresponding to the amount of the predicted difference value of the spatial frequency component whose magnitude exceeds a predetermined value among the predicted difference values of the plurality of spatial frequency components among the plurality of learned neural networks. Select a completed neural network. Then, the control unit 110 generates a reference image to be used for inter-prediction or intra-prediction by processing the generated decoded image using the selected trained neural network.

Next, the outline of the process in which the control unit 110 decodes the image to be decoded will be described. The control unit 110 acquires the coded data generated by coding the image. For example, the control unit 110 reads out compressed image data including image coding data and compressed information from the memory 130. The control unit 110 acquires a plurality of spatial frequency components of the encoded image from the acquired coded data. For example, the control unit 110 acquires a plurality of spatial frequency components by performing entropy decoding on the entropy-coded coded data. The control unit 110 generates a decoded image based on a plurality of spatial frequency components. In this way, the control unit 110 generates a decoded image by decoding the encoded data.

The control unit 110 acquires the amount of the spatial frequency component whose magnitude exceeds a predetermined value among the plurality of spatial frequency components acquired from the coded data. The control unit 110 selects a trained neural network associated with the amount of acquired spatial frequency components from the plurality of trained neural networks. The control unit 110 processes the generated decoded image using the selected trained neural network. The control unit 110 outputs the decoded image processed by using the trained neural network to, for example, the display unit 160.

As an example, the encoded data of the moving image constituent image is information indicating a plurality of quantization difference values obtained by quantization of the predicted difference values of a plurality of spatial frequency components obtained by inter-prediction or intra-prediction of the moving image constituent image. including. The control unit 110 stores a plurality of learned neural networks in advance in association with the amount of the quantization difference value whose magnitude exceeds a predetermined value among the plurality of quantization difference values. The control unit 110 acquires the amount of the quantization difference value whose magnitude exceeds a predetermined value among the plurality of quantization difference values acquired from the coded data, and acquires it from the plurality of trained neural networks. Select a trained neural network that is associated with the amount of quantization difference value of the spatial frequency component. Then, the control unit 110 generates a difference image based on the predicted difference value of the spatial frequency component obtained by inverse quantization of the plurality of quantization difference values acquired from the coded data, and inter-predicts the generated difference image. A decoded moving image constituent image is generated by adding an image or an intra-predicted image. Then, the control unit 110 processes the generated moving image constituent image using the selected learned neural network.

The "amount of spatial frequency components whose size exceeds a predetermined value among the spatial frequency components of the encoded image" is defined as "the size of the spatial frequency components of the encoded image is predetermined". It may be "the ratio of the spatial frequency component exceeding the value". The "predetermined value" may be zero (0). In relation to FIGS. 4 to 10, the DCT coefficient is adopted as the "spatial frequency component", zero (0) is adopted as the "predetermined value", and "of the spatial frequency components of the encoded image". A mode in which "the ratio of DCT coefficients whose magnitude exceeds zero (0) among the DCT coefficients of the encoded image" is adopted as "amount of spatial frequency components whose magnitude exceeds a predetermined value" will be described.

The control unit 110 may store a plurality of trained neural networks in association with the picture type of the moving image constituent image. The control unit 110 may select a trained neural network associated with the picture type of the coded data and the amount of the acquired spatial frequency component from the plurality of trained neural networks.

The "trained neural network" uses the training image and the coded data of the training image as training data, and the size of the plurality of spatial frequency components acquired from the coded data included in the training data is large. It may be a convolutional neural network (CNN) obtained by performing machine learning according to the amount of spatial frequency components exceeding a predetermined value. In relation to FIGS. 4 to 10, a mode in which a “CNN filter” that performs image processing using a convolutional neural network (CNN) is adopted will be described.

FIG. 3 shows a block diagram of the learner. FIG. 4 shows class information for classifying based on the DCT coefficient. The learner is a device that performs machine learning that generates neural network parameters (CNN parameters) used by the control unit 110. The control unit 110 does not need to include a learning device.

The learner generates CNN parameters by performing machine learning using the input image and the deteriorated image. The degraded image is a decoded image generated by decoding the coded image generated by coding the input image. The decoded image is an image that has undergone image processing by a loop filter included in the decoder. The loop filter will be described later. The image compression information used for encoding the input image is further input to the learner. In image coding, intra-prediction or inter-prediction is performed for each predetermined prediction unit in the partial image of each macroblock, and the difference between the predicted image and the partial image obtained by the intra-prediction or inter-prediction. An image is generated, the DCT coefficient is calculated by performing DCT transform for each predetermined conversion unit on the difference image, and the quantized DCT coefficient is calculated by quantizing the DCT coefficient. The image compression information may include the quantized DCT coefficient obtained by quantizing the DCT coefficient.

The DCT coefficient counting unit 310 acquires the quantized DCT coefficient based on the image compression information and counts the DCT coefficient having a magnitude greater than 0. The DCT coefficient counting unit 310 counts, for example, the number M of DCT coefficients whose magnitude exceeds 0 among the N DCT coefficients in all the conversion units of the entire image. The "DCT coefficient having a magnitude exceeding 0" means a DCT coefficient larger than 0 and a coefficient smaller than 0. That is, the "DCT coefficient having a magnitude exceeding 0" means a DCT coefficient other than 0. The class determination unit 320 calculates the ratio of DCT coefficients other than 0 by dividing M by N. In the present embodiment, the ratio of DCT coefficients other than 0 among the plurality of DCT coefficients is referred to as "non-zero DCT ratio".

The class determination unit 320 identifies the class based on the class information shown in FIG. 4 and the non-zero DCT ratio. As shown in FIG. 4, the class information is information that associates the class identifier with the range of the non-0DCT ratio. The class determination unit 320 specifies a range including the acquired non-0DCT ratio among a plurality of ranges in the class information, and specifies a class identifier associated with the specified range.

The CNN learner 330 calculates the CNN parameters constituting the CNN filter described later by performing machine learning using the input image and the deteriorated image for each class determined by the class determination unit 320. Specifically, the CNN learner 330 sets weights and offsets that minimize a predetermined loss function based on the image generated by applying the CNN filter to the degraded image and the input image. calculate.

FIG. 5 shows the parameter information of the neural network. The parameter information is information that associates the CNN parameter calculated by machine learning with the class identifier. CNN parameters include weighted values and offsets in the CNN. The control unit 110 stores the class information and the parameter information shown in FIG. The control unit 110 determines a class with reference to the class information as a part of the image coding process and the decoding process, and executes the image processing using the CNN parameter determined from the determined class and the parameter information.

FIG. 6 shows a block configuration of a encoder included in the control unit 110. A plurality of time-series pictures as moving image constituent images are input to the encoder as input image data to be encoded. The reorder unit 610 determines the order in which the pictures are coded based on the picture type. For example, the reorder unit 610 arranges the order of the pictures to be encoded so as to encode the I picture or the P picture after the B picture before encoding the B picture encoded by the bidirectional prediction. Change.

The orthogonal transform unit 620 calculates the DCT coefficient by performing DCT conversion of the difference image between the picture output from the reorder unit 610 and the reference image. The quantization unit 630 generates a quantization DCT coefficient by quantizing the DCT coefficient output from the orthogonal transform unit 620. The quantization unit 630 adjusts the quantization parameters used for the quantization of the DCT coefficient based on the compression rate output from the rate control unit 660 described later. The entropy coding unit 650 generates a coded picture by applying entropy coding to the quantized DCT coefficient output by the quantization unit 630. The buffer 670 stores the coded picture output by the entropy coding unit 650. The rate control unit 660 determines the compression rate based on the amount of data in the coded picture and outputs the compression rate to the quantization unit 630. The entropy coding unit 650 also entropy-codes the compressed information such as the motion vector and stores it in the buffer 670. The control unit 110 records the compressed image data including the coded picture and the compressed information stored in the buffer 670 in the memory 130 or the like.

Next, the processing of the loop structure included in the encoder will be described. The inverse quantization unit 641 dequantizes the quantization DCT coefficient output from the quantization unit 630. The inverse orthogonal transform unit 642 generates a difference image by performing an inverse DCT transform on the output of the inverse quantization unit 641. The loop filter 643 applies a filter process to the image information obtained by adding the reference image to the difference image generated by the inverse orthogonal transform unit 642. The loop filter 643 may include, for example, a deblocking filter. The image generated by the loop filter 643 is output to the CNN filter 644.

The DCT coefficient counting unit 646 counts the number of DCT coefficients having a magnitude exceeding 0 based on the output of the quantization unit 630, and calculates the non-0DCT ratio. The class determination unit 647 determines the class identifier using the non-0DCT ratio calculated by the DCT coefficient counting unit 646 and the class information described in relation to FIG. The class identifier determined by the class determination unit 647 is output to the CNN filter 644.

The CNN filter 644 is a filter composed of CNNs formed by the CNN parameters generated by the learner described above. The CNN filter 644 generates a reference picture by performing a convolution operation on the image output by the loop filter 643 using the CNN. Specifically, the CNN filter 644 refers to the parameter information, and the CNN filter composed of the CNN parameters associated with the class identifier determined by the class determination unit 647 is processed by the loop filter 643. To generate a reference picture. The memory 645 stores a reference picture generated by the CNN filter 644. The intra prediction unit 648 uses the reference picture stored in the memory 645 to perform intra prediction for encoding the coded target picture, and generates an intra prediction image as a reference image. The inter-prediction unit 649 uses the reference picture stored in the memory 645 to perform inter-prediction for encoding another coded target picture, and generates an inter-prediction image as a reference image. The inter-prediction unit 649 may generate an inter-prediction image by, for example, calculating a motion vector and performing motion compensation.

FIG. 7 shows a block configuration of a decoder included in the control unit 110. The control unit 110 reads the compressed moving image data from the memory 130 and decodes it. The entropy decoding unit 750 generates a quantized DCT coefficient by entropy decoding the compressed image data read from the memory 130. The inverse quantization unit 741 dequantizes the quantization DCT coefficient output from the entropy decoding unit 750. The inverse orthogonal transform unit 742 generates a difference image by performing an inverse DCT transform on the output of the inverse quantization unit 741. The loop filter 743 filters the image information obtained by adding the reference image to the difference image generated by the inverse orthogonal transform unit 742. The loop filter 743 may include, for example, a deblocking filter. The loop filter 743 may be the same filter as the loop filter 643. The image generated by the loop filter 743 is output to the CNN filter 744.

The DCT coefficient counting unit 746 counts the number of DCT coefficients having a magnitude exceeding 0 based on the quantized DCT coefficient output from the entropy decoding unit 750, and calculates the non-0DCT ratio. The class determination unit 747 determines the class using the non-0DCT ratio calculated by the DCT coefficient counting unit 746 and the class information. Information indicating the class determined by the class determination unit 747 is output to the CNN filter 744.

The CNN filter 744 is a CNN formed by the CNN parameters generated by the learner described above. The CNN filter 744 generates a decoded picture by performing a convolution operation on the image output by the loop filter 743 using a neural network. Specifically, the CNN filter 744 refers to the parameter information, and the CNN filter composed of the CNN parameters associated with the class identifier determined by the class determination unit 747 is processed by the loop filter 743. To generate a decoded picture. The reordering unit 710 performs a process of rearranging the decoded pictures in chronological order based on the picture type, and outputs the decoded image data. The decoded image data is used, for example, for displaying an image on the display unit 160.

The memory 745 stores the decoded picture generated by the CNN filter 744. The intra prediction unit 748 uses the decoded picture stored in the memory 745 as a reference picture to perform intra prediction for encoding the decoding target picture, and generates an intra prediction image as a reference image. The inter-prediction unit 749 uses the decoded picture stored in the memory 745 as a reference picture, performs inter-prediction for encoding another decoding target picture, and obtains an inter-prediction image as a reference image. Generate. The inter-prediction unit 749 may generate an inter-prediction image by, for example, calculating a motion vector and performing motion compensation based on the motion vector.

FIG. 8 shows a flowchart of processing executed when the control unit 110 encodes the coded target picture. In S810, the orthogonal transform unit 620 performs the orthogonal transform of the coded picture to calculate the DCT coefficient, and the quantization unit 630 generates the quantization DCT coefficient by quantizing the calculated DCT coefficient.

In S820, the DCT coefficient counting unit 646 calculates the non-0DCT ratio. In S830, the inverse quantization unit 641 inversely quantizes the quantization DCT coefficient, and the inverse orthogonal transform unit 642 generates a difference image by inversely orthogonal transforming the DCT coefficient obtained by the inverse orthogonal transform.

In S840, the intra prediction unit 648 or the inter prediction unit 649 generates a prediction image and adds the difference image to the prediction image to perform compensation. In S850, a loop filter is applied to the image generated by the addition of the predicted image and the difference image in S840.

In S860, the DCT coefficient counting unit 646 calculates the non-0DCT ratio, and the class determination unit 647 determines the class identifier based on the calculated non-0DCT ratio with reference to the class information. In S870, the picture generated in S850 is image-processed by the CNN filter 644 composed of the CNN parameters associated with the class identifier to generate a reference picture.

FIG. 9 shows a flowchart of processing executed when the control unit 110 decodes the decoding target picture. In S910, the entropy decoding unit 750 generates a quantized DCT coefficient by entropy decoding the decoding target picture. In S920, the DCT coefficient counting unit 746 counts the number of DCT coefficients having a magnitude exceeding 0 based on the quantized DCT coefficient generated in S910, and calculates the non-0DCT ratio.

In S930, the inverse quantization unit 741 inversely quantizes the quantization DCT coefficient, and the inverse orthogonal transform unit 742 generates a difference image by inversely orthogonal transforming the DCT coefficient obtained by the inverse orthogonal transform.

In S940, compensation is performed by adding the predicted image generated by the intra prediction unit 748 or the inter prediction unit 749 and the difference image. In S950, a loop filter is applied to the image generated by adding the predicted image and the difference image in S940.

In S960, the class determination unit 747 determines the class identifier based on the non-0DCT ratio calculated in S920 with reference to the class information. In S970, the image to which the loop filter is applied in S950 is processed by the CNN filter 644 configured by the CNN parameter associated with the class identifier to generate a decoded picture. As described above, the decoded picture is output as the decoded image data and is used as a reference picture.

FIG. 10 shows an example of the reference relationship of the pictures used in the inter-prediction. In FIG. 10, H. The reference relationships of the I picture 1000, the P picture 1004, the Straight-B picture 1002, the Non-Story-B picture 1001, and the Non-Story-B picture 1003 used in an image coding method such as 265 are shown. As shown in FIG. 10, the reference relationship differs depending on the picture type. Therefore, the amount of image quality deterioration may differ depending on the picture type. Therefore, in addition to the above-mentioned classification based on the non-zero DCT ratio, the classification may be performed according to the picture type. Specifically, the class classification may be performed by setting the class identifier for each combination of the picture type and the non-0DCT ratio. In machine learning, the learner may perform machine learning for each combination of the picture type and the non-0DCT ratio, and calculate the CNN parameter for each combination of the picture type and the non-0DCT ratio. The control unit 110 may store the CNN parameter in association with the combination of the picture type and the non-0DCT ratio, and uses the CNN parameter associated with the combination of the picture type and the non-0DCT ratio in encoding and decoding. A CNN filter may be configured.

In addition, H. In a coding method such as 265, when performing orthogonal transformation, it is possible to use a plurality of conversion unit sizes in the same picture. For example, H. In 265, it is possible to use a size such as 4x4, 8x8, 16x16, 32x32 as the conversion unit size. In general, for an image area containing fine textures and random movements, the coding efficiency can be improved by selecting and coding a smaller conversion unit size. On the other hand, for an image region containing a flat texture or global motion, the coding efficiency can be improved by selecting and coding a larger conversion unit size. If the size of the conversion unit is changed, the amount of image quality deterioration may also change. Therefore, in the present embodiment, the classification may be performed according to the distribution of the conversion size instead of the non-0DCT ratio or in addition to the non-0DCT ratio. For example, the classification may be performed instead of the non-0DCT ratio, or based on the ratio in which a specific conversion unit size (eg, 4x4 size) is selected in the coding of each picture in addition to the non-0DCT ratio. good.

As described above, the control unit 110 performs coding and decoding processing using the neural network parameters selected based on the non-0DCT ratio. For example, H. In the 265 coding method, orthogonal transformation is performed on conversion units of various sizes such as 4x4, 8x8, 16x16, 32x32 in the picture. Therefore, the number of DCT coefficients whose magnitude exceeds 0 may vary depending on the size of the conversion unit. As described above, according to the present embodiment, the neural network parameters are selected based on the proportion of the DCT coefficients having a magnitude greater than 0. As a result, the influence of the block size can be reduced as compared with the case where the number occupied by the DCT coefficient having a magnitude larger than 0 is used. Further, in the present embodiment, since the classification is performed based on the quantized DCT coefficient obtained by the entropy decoding of the compressed moving image data, it is necessary to transmit the metadata for the classification from the coding side to the decoding side. There is no. Even if orthogonal transform is performed with a plurality of conversion unit sizes different from each other, the non-0DCT ratio is calculated based on the DCT coefficient obtained for each conversion unit size, and the classification is performed based on the combination of the non-0DCT ratios. good. As an example, classification is performed based on the non-0DCT ratio when the conversion unit size of the minimum size (for example, 4 × 4 size) is applied and the non-0DCT ratio when the conversion unit size other than the minimum size is applied. You may.

As a modification of the above embodiment, a mode not provided with the loop filter 643 and the loop filter 743 may be adopted.

Non-Patent Document 1 and Non-Patent Document 2 described above describe that the quantization parameter is fixed and the filter parameter is learned. In general, in the compression coding of moving images, the amount of image quality deterioration due to coding differs between the case where the compression rate is low and the case where the compression rate is high. In a encoder that is actually used commercially, the quantization parameter is usually made variable in the picture and the quantization matrix is often applied, so that the amount of image quality deterioration occurs. Therefore, if the filter parameters learned by fixing the quantization parameters as described in Non-Patent Document 1 and Non-Patent Document 2 are applied to a commercial encoder, the coding efficiency may deteriorate. On the other hand, according to the present embodiment, since the CNN parameter is selected by classifying at least based on the non-zero DCT ratio, an appropriate filter according to the compression ratio can be used in the loop structure of inter-prediction or intra-prediction. Can be applied. As a result, the coding efficiency may be improved.

The image pickup apparatus 100 as described above may be mounted on a moving body. The imaging device 100 may be mounted on an unmanned aerial vehicle (UAV) as shown in FIG. The UAV 10 may include a UAV main body 20, a gimbal 50, a plurality of image pickup devices 60, and an image pickup device 100. The gimbal 50 and the imaging device 100 are examples of an imaging system. The UAV 10 is an example of a moving body propelled by a propulsion unit. The moving body is a concept including a UAV, a flying object such as another aircraft moving in the air, a vehicle moving on the ground, a ship moving on the water, and the like.

The UAV main body 20 includes a plurality of rotor blades. The plurality of rotor blades are an example of a propulsion unit. The UAV main body 20 flies the UAV 10 by controlling the rotation of a plurality of rotor blades. The UAV body 20 flies the UAV 10 using, for example, four rotor blades. The number of rotor blades is not limited to four. Further, the UAV 10 may be a fixed-wing aircraft having no rotor blades.

The imaging device 100 is an imaging camera that captures a subject included in a desired imaging range. The gimbal 50 rotatably supports the imaging device 100. The gimbal 50 is an example of a support mechanism. For example, the gimbal 50 rotatably supports the image pickup device 100 on a pitch axis using an actuator. The gimbal 50 further rotatably supports the image pickup device 100 around each of the roll axis and the yaw axis by using an actuator. The gimbal 50 may change the posture of the image pickup device 100 by rotating the image pickup device 100 around at least one of the yaw axis, the pitch axis, and the roll axis.

The plurality of image pickup devices 60 are sensing cameras that image the surroundings of the UAV 10 in order to control the flight of the UAV 10. Two imaging devices 60 may be provided on the front surface, which is the nose of the UAV 10. Yet two other imaging devices 60 may be provided on the bottom surface of the UAV 10. The two image pickup devices 60 on the front side may form a pair and function as a so-called stereo camera. The two image pickup devices 60 on the bottom surface side may also be paired and function as a stereo camera. Three-dimensional spatial data around the UAV 10 may be generated based on the images captured by the plurality of imaging devices 60. The number of image pickup devices 60 included in the UAV 10 is not limited to four. The UAV 10 may include at least one imaging device 60. The UAV 10 may be provided with at least one imaging device 60 on each of the nose, nose, side surface, bottom surface, and ceiling surface of the UAV 10. The angle of view that can be set by the image pickup device 60 may be wider than the angle of view that can be set by the image pickup device 100. The image pickup apparatus 60 may have a single focus lens or a fisheye lens.

The remote control device 300 communicates with the UAV 10 to remotely control the UAV 10. The remote control device 300 may wirelessly communicate with the UAV 10. The remote control device 300 transmits to the UAV 10 instruction information indicating various commands related to the movement of the UAV 10, such as ascending, descending, accelerating, decelerating, advancing, reversing, and rotating. The instruction information includes, for example, instruction information for raising the altitude of the UAV 10. The instruction information may indicate the altitude at which the UAV 10 should be located. The UAV 10 moves so as to be located at an altitude indicated by the instruction information received from the remote control device 300. The instruction information may include an ascending instruction to ascend the UAV 10. The UAV10 rises while accepting the rise order. Even if the UAV10 accepts the ascending command, the ascending may be restricted if the altitude of the UAV10 has reached the upper limit altitude.

FIG. 12 shows an example of a computer 1200 in which a plurality of aspects of the present invention may be embodied in whole or in part. The program installed on the computer 1200 can cause the computer 1200 to function as an operation associated with the device according to an embodiment of the present invention or as one or more "parts" of the device. For example, a program installed on a computer 1200 can cause the computer 1200 to function as a control unit 110. Alternatively, the program may cause the computer 1200 to perform the operation or the function of the one or more "parts". The program can cause a computer 1200 to perform a process or a step of the process according to an embodiment of the present invention. Such a program may be run by the CPU 1212 to cause the computer 1200 to perform certain operations associated with some or all of the blocks in the flowcharts and block diagrams described herein.

The computer 1200 according to this embodiment includes a CPU 1212 and a RAM 1214, which are connected to each other by a host controller 1210. The computer 1200 also includes a communication interface 1222, an input / output unit, which are connected to the host controller 1210 via an input / output controller 1220. The computer 1200 also includes a ROM 1230. The CPU 1212 operates according to the programs stored in the ROM 1230 and the RAM 1214, thereby controlling each unit.

Communication interface 1222 communicates with other electronic devices via a network. The hard disk drive may store programs and data used by the CPU 1212 in the computer 1200. The ROM 1230 stores in it a boot program or the like executed by the computer 1200 at the time of activation and / or a program depending on the hardware of the computer 1200. The program is provided via a computer-readable recording medium such as a CR-ROM, USB memory or IC card or network. The program is installed in RAM 1214 or ROM 1230, which is also an example of a computer-readable recording medium, and is executed by CPU 1212. The information processing described in these programs is read by the computer 1200 and provides a link between the program and the various types of hardware resources described above. The device or method may be configured to implement the operation or processing of information according to the use of the computer 1200.

For example, when communication is executed between the computer 1200 and an external device, the CPU 1212 executes a communication program loaded in the RAM 1214, and performs communication processing on the communication interface 1222 based on the processing described in the communication program. You may order. Under the control of the CPU 1212, the communication interface 1222 reads the transmission data stored in the transmission buffer area provided in the RAM 1214 or a recording medium such as a USB memory, and transmits the read transmission data to the network, or The received data received from the network is written to the reception buffer area or the like provided on the recording medium.

Further, the CPU 1212 makes the RAM 1214 read all or necessary parts of a file or a database stored in an external recording medium such as a USB memory, and executes various types of processing on the data on the RAM 1214. good. The CPU 1212 may then write back the processed data to an external recording medium.

Various types of information such as various types of programs, data, tables, and databases may be stored in recording media and processed. The CPU 1212 describes various types of operations, information processing, conditional judgment, conditional branching, unconditional branching, and information retrieval described in various parts of the present disclosure with respect to the data read from the RAM 1214. Various types of processing may be performed, including / replacement, etc., and the results are written back to the RAM 1214. Further, the CPU 1212 may search for information in a file, a database, or the like in the recording medium. For example, when a plurality of entries each having an attribute value of the first attribute associated with the attribute value of the second attribute are stored in the recording medium, the CPU 1212 specifies the attribute value of the first attribute. Search for an entry that matches the condition from the plurality of entries, read the attribute value of the second attribute stored in the entry, and associate it with the first attribute that satisfies the predetermined condition. The attribute value of the second attribute obtained may be acquired.

The program or software module described above may be stored on a computer 1200 or in a computer readable storage medium near the computer 1200. Also, a recording medium such as a hard disk or RAM provided within a dedicated communication network or a server system connected to the Internet can be used as a computer readable storage medium, thereby allowing the program to be transferred to the computer 1200 over the network. offer.

Although the present invention has been described above using the embodiments, the technical scope of the present invention is not limited to the scope described in the above embodiments. It will be apparent to those skilled in the art that various changes or improvements can be made to the above embodiments. It is clear from the description of the claims that such modified or improved forms may also be included in the technical scope of the present invention.

The order of execution of operations, procedures, steps, steps, etc. in the devices, systems, programs, and methods shown in the claims, specification, and drawings is particularly "before" and "prior to". It should be noted that it can be realized in any order unless the output of the previous process is used in the subsequent process. Even if the scope of claims, the specification, and the operation flow in the drawings are explained using "first", "next", etc. for convenience, it means that it is essential to carry out in this order. It's not a thing.

10 UAV
20 UAV main unit 50 gimbal 60 image pickup device 100 image pickup device 102 image pickup unit 110 control unit 120 image sensor 130 memory 160 display unit 162 indicator unit 200 lens unit 210 lens 212 lens drive unit 220 lens control unit 222 memory 300 remote control device 310 DCT coefficient Counting unit 320 Class determination unit 330 CNN learner 610 Reorder unit 620 Discrete cosine transform unit 630 Quantization unit 641 Inverse quantization unit 642 Inverse orthogonal transform unit 643 Loop filter 644 CNN filter 645 Memory 646 DCT coefficient Counting unit 647 Class determination unit 648 Intra Prediction unit 649 Inter prediction unit 650 Entropy coding unit 660 Rate control unit 670 Buffer 710 Reordering unit 741 Inverse quantization unit 742 Inverse orthogonal transform unit 743 Loop filter 744 CNN filter 745 Memory 746 DCT coefficient Counting unit 747 Class determination unit 748 Intra Prediction unit 749 Inter-prediction unit 750 Entropy decoding unit 1000 I picture 1001, 1003 Non-Story-B picture 1002 Straight-B picture 1004 P picture 1200 Computer 1210 Host controller 1212 CPU
1214 RAM
1220 Input / Output Controller 1222 Communication Interface 1230 ROM

Claims (14)

Stores a plurality of trained neural networks for processing the decoded image in association with the amount of spatial frequency components of the encoded image whose magnitude exceeds a predetermined value. ,
Get the coded data generated by image coding and
From the coded data, a plurality of spatial frequency components of the coded image are acquired, and a plurality of spatial frequency components are obtained.
By decoding the coded data, a decoded image is generated.
Among the plurality of spatial frequency components acquired from the coded data, the amount of the spatial frequency component whose magnitude exceeds the predetermined value is acquired.
A trained neural network corresponding to the amount of the acquired spatial frequency component is selected from the plurality of trained neural networks, and the trained neural network is selected.
A device comprising a circuit configured to process the generated decoded image using the selected trained neural network. The circuit
The plurality of trained neural networks are stored in association with the ratio of the spatial frequency components whose magnitude exceeds a predetermined value among the spatial frequency components of the encoded image.
Among the plurality of spatial frequency components acquired from the coded data, the proportion of the spatial frequency component whose magnitude exceeds the predetermined value is acquired.
The apparatus according to claim 1, wherein the trained neural network associated with the ratio of the acquired spatial frequency component is selected from the plurality of trained neural networks. The device according to claim 1 or 2, wherein the predetermined value is zero. The image is a moving image constituent image constituting a moving image, and is
The coded data includes information indicating a plurality of quantization difference values obtained by quantization of prediction difference values of a plurality of spatial frequency components obtained by inter-prediction or intra-prediction of a moving image.
The circuit
The plurality of trained neural networks are stored in association with the amount of the quantization difference value whose magnitude exceeds the predetermined value among the plurality of quantization difference values.
Among the plurality of quantization difference values acquired from the coded data, the amount of the quantization difference value whose magnitude exceeds the predetermined value is acquired.
From the plurality of trained neural networks, a trained neural network corresponding to the amount of the quantization difference value of the acquired spatial frequency component is selected.
A difference image is generated based on the predicted difference value of the spatial frequency component obtained by inverse quantization of the plurality of quantization difference values obtained from the coded data, and the inter-predicted image or the intra-predicted image is added to the generated difference image. By adding, a decoded video composition image is generated,
The apparatus according to claim 1 or 2, wherein the trained neural network selected is used to process the generated moving image constituent image. The image is a moving image constituent image constituting a moving image, and is
The circuit
The plurality of trained neural networks are stored in association with the picture type of the moving image, and the trained neural networks are stored.
Claim 1 or 2 configured to select a trained neural network associated with the picture type of the coded data and the amount of the acquired spatial frequency component from the plurality of trained neural networks. The device described. The trained neural network uses the training image and the coded data of the training image as training data, and has a size among a plurality of spatial frequency components acquired from the coded data included in the training data. The apparatus according to claim 1 or 2, wherein is a convolutional neural network obtained by performing machine learning according to the amount of spatial frequency components exceeding the predetermined value. Stores a plurality of trained neural networks for processing the decoded image in association with the amount of spatial frequency components of the encoded image whose magnitude exceeds a predetermined value. ,
By encoding the image to be encoded by a coding process including inter-prediction or intra-prediction, coded data including information indicating predicted difference values of a plurality of spatial frequency components is generated.
Output the coded data and
A decoded image is generated by decoding the coded data,
A trained neural network in which the magnitude of the predicted difference values of the plurality of spatial frequency components exceeds a predetermined value among the plurality of trained neural networks is associated with the amount of the predicted difference values of the spatial frequency components. selection,
An apparatus comprising a circuit configured to generate a reference image used for the inter-prediction or the intra-prediction by processing the generated decoded image using the selected trained neural network. .. The device according to claim 1 and
An image processing device including the device according to claim 7. The device according to claim 1 or 2,
An imaging device including an image sensor that generates an image. A moving body that moves with the imaging device according to claim 9. The mobile body according to claim 10, wherein the mobile body is an unmanned aerial vehicle. A program for operating a computer as the device according to claim 1 or 2. Stores a plurality of trained neural networks for processing the decoded image in association with the amount of spatial frequency components of the encoded image whose magnitude exceeds a predetermined value. Stages and
The stage of acquiring the coded data generated by coding the image, and
A step of acquiring a plurality of spatial frequency components of a coded image from the coded data, and
The stage of generating a decoded image by decoding the coded data, and
A step of acquiring the amount of the spatial frequency component whose magnitude exceeds the predetermined value among the plurality of spatial frequency components acquired from the coded data, and a step of acquiring the amount of the spatial frequency component.
A step of selecting a trained neural network corresponding to the amount of the acquired spatial frequency component from the plurality of trained neural networks, and a step of selecting the trained neural network.
A method comprising a step of processing the generated decoded image using the selected trained neural network. Stores a plurality of trained neural networks for processing the decoded image in association with the amount of spatial frequency components whose magnitude exceeds a predetermined value among the spatial frequency components of the encoded image. Stages and
By encoding the image to be encoded by a coding process including inter-prediction or intra-prediction, coded data including information indicating predicted difference values of a plurality of spatial frequency components is generated.
The stage of outputting the coded data and
A step of generating a decoded image by decoding the coded data, and
A trained neural network in which the magnitude of the predicted difference values of the plurality of spatial frequency components exceeds a predetermined value among the plurality of trained neural networks is associated with the amount of the predicted difference values of the spatial frequency components. The stage of selection and
A method comprising a step of generating a reference image used for the inter-prediction or the intra-prediction by processing the generated decoded image using the selected trained neural network.

Images