JP2021168459A - Apparatus, system, imaging apparatus, mobile, program and method - Google Patents

Abstract

To reduce the data volume of middle layer data generated by multilayer neural network processing for image data that constitutes a moving image.SOLUTION: An imaging apparatus is configured to: generate first and second middle layer data 601-612... by subjecting each image data of first and second moving image component images 400, 401 that constitute a moving image to processing up to a specific middle layer in a multilayer neural network; generate motion data between the first middle layer data and the second middle layer data based on motion information between the first moving image component image and the second moving image component image; generate residual data between the first middle layer data and the second middle layer data by performing motion compensation between the first middle layer data and the second middle layer data based on the motion data; generate middle layer data that is transmitted to other apparatus performing the processing of a latter stage of the specific middle layer in the multilayer neural network based on the residual data; and transmit the middle layer data and the motion data to the other apparatus.SELECTED DRAWING: Figure 6

Description

The present invention relates to devices, systems, imaging devices, mobiles, programs and methods.

Non-Patent Document 1 describes a technique in which a part of a neural network is executed by a mobile edge and the rest of the neural network is executed by a cloud server.
[Prior art literature]
[Patent Document]
[Non-Patent Document 1] Yiping Kang, Johann Hauswald, Cao Gao, Austin Rovinski, Trevor Mudge, Jason Mars, Lingjia Tang, "Neurosergeon: Collaborative intelligence between the cloud and mobile edge", ACM SIGARCH Computer Architecture News, 4 May 2017

The apparatus according to the first embodiment of the present invention performs processing up to a specific intermediate layer in the multi-layer neural network on the image data of the first moving image constituent image constituting the moving image, thereby forming the first intermediate layer. It includes a circuit configured to generate data. The circuit is configured to generate the second intermediate layer data by performing processing up to a specific intermediate layer in the multi-layer neural network on the image data of the second moving image constituent image constituting the moving image. The circuit is configured to generate motion data between the first intermediate layer data and the second intermediate layer data based on the motion information between the first moving image and the second moving image. Will be done. The circuit performs motion compensation between the first and second intermediate layer data based on the motion data, thereby performing a motion compensation between the first intermediate layer data and the second intermediate layer data. It is configured to generate difference data. The circuit is configured to generate intermediate layer data to be transmitted to other devices that perform processing after a specific intermediate layer in a multi-layer neural network based on the residual data. The circuit is configured to transmit intermediate layer data and motion data to other devices.

The circuit is configured to generate quantized data by quantizing the residual data. The circuit may be configured to generate intermediate layer data based on the quantized data.

The circuit may be configured to generate intermediate layer data by entropy coding the quantized data.

The multi-layer neural network is a convolutional neural network, and the first moving image and the second moving image may include one or more intermediate layers for performing a convolution operation.

The circuit generates motion data by scaling the motion information based on the size of the first moving image and the second moving image and the size of the first intermediate layer data and the second intermediate layer data. It may be configured as follows.

The multi-layer neural network may be a neural network for performing super-resolution processing of moving images or image recognition processing from moving images.

In the multi-layer neural network, the neural network parameters from the input layer before the specific intermediate layer of the multi-layer neural network to the output layer after the specific intermediate layer are learned by using the training data. The circuit may be configured to store neural network parameters from the input layer to a particular intermediate layer.

A multi-layer neural network may have a plurality of intermediate layers. The circuit may be configured to select a particular intermediate layer from a plurality of intermediate layers based on at least one of the communication line capacity of the device, the load state of the device, and the power state of the device. ..

The apparatus according to the second aspect of the present invention communicates with another apparatus that processes the first moving image constituent image and the second moving image constituent image constituting the moving image up to a specific intermediate layer in the multi-layer neural network. Then, the processing after the specific intermediate layer in the multi-layer neural network is performed. The device includes (i) first intermediate layer data generated by processing the first moving image constituent image from another device to a specific intermediate layer in the multilayer neural network, and (ii) first. Motion data between the intermediate layer data of 1 and the second intermediate layer data generated by processing up to a specific intermediate layer in the multilayer neural network for the second moving image constituent image, and (iii). Residual data between the first intermediate layer data and the second intermediate layer data, the first intermediate layer data and the second intermediate layer generated by performing motion compensation based on the motion data. It includes a circuit configured to receive residual data with and from the data. The circuit is configured to generate second intermediate layer data based on motion data and residual data. The circuit is configured to perform processing on the first intermediate layer data and the second intermediate layer data after the specific intermediate layer in the multi-layer neural network.

The system according to the third aspect of the present invention includes an apparatus according to the first aspect and an apparatus according to the second aspect.

The image pickup apparatus according to the fourth aspect of the present invention includes the above-mentioned apparatus and an image sensor for generating an image.

The moving body according to the fifth aspect of the present invention moves with the above-mentioned imaging device.

The moving body may be an unmanned aerial vehicle.

The program according to the sixth aspect of the present invention causes the computer to function as the above-mentioned device. The program may be recorded on a non-temporary recording medium.

In the method according to the seventh aspect of the present invention, the image data of the first moving image constituent image constituting the moving image is processed up to a specific intermediate layer in the multi-layer neural network, so that the first intermediate layer is processed. It has a stage to generate data. The method includes a step of generating a second intermediate layer data by performing processing up to a specific intermediate layer in the multi-layer neural network on the image data of the second moving image constituting the moving image. The method is a step of generating motion data between the first intermediate layer data and the second intermediate layer data based on the motion information between the first moving image constituent image and the second moving image constituent image. Be prepared. The method is to perform motion compensation between the first intermediate layer data and the second intermediate layer data based on the motion data, thereby performing a residual between the first intermediate layer data and the second intermediate layer data. It has a stage to generate difference data. The method comprises a step of generating intermediate layer data to be transmitted to another device that performs processing after a specific intermediate layer in a multi-layer neural network based on the residual data. The method comprises transmitting intermediate layer data and motion data to another device.

The method according to the eighth aspect of the present invention communicates with another device that processes the first moving image constituent image and the second moving image constituent image constituting the moving image up to a specific intermediate layer in the multi-layer neural network. Then, the processing after the specific intermediate layer in the multi-layer neural network is performed. The method comprises (i) first intermediate layer data generated by processing the first moving image constituent image from another device to a specific intermediate layer in the multilayer neural network, and (ii) first. Motion data between the intermediate layer data of 1 and the second intermediate layer data generated by processing up to a specific intermediate layer in the multilayer neural network for the second moving image constituent image, and (iii). Residual data between the first intermediate layer data and the second intermediate layer data, the first intermediate layer data and the second intermediate layer generated by performing motion compensation based on the motion data. It is provided with a step of receiving residual data from the data. The method comprises generating a second intermediate layer data based on the motion data and the residual data. The method includes a step of processing the first intermediate layer data and the second intermediate layer data after the specific intermediate layer in the multi-layer neural network.

According to the above aspect of the present invention, it is possible to reduce the amount of intermediate layer data of the neural network transmitted to other devices.

The outline of the above invention does not list all the necessary features of the present invention. Sub-combinations of these feature groups can also be inventions.

It is a figure which shows an example of the external perspective view of the image pickup apparatus 100 which concerns on this embodiment. It is a figure which shows the functional block of the image pickup apparatus 100 which concerns on this embodiment. The overall view of the system 310 including the image pickup apparatus 100 and the server 180 is schematically shown. The moving image composition image constituting the moving image captured by the image pickup apparatus 100 is schematically shown. The state in which the moving image is processed by the multi-layer neural network 580 is schematically shown. The outline of the compression process of the intermediate layer data in the image pickup apparatus 100 is schematically shown. The block configuration of the control unit 110 is schematically shown. The block configuration of the server 180 is schematically shown. The flowchart which shows the flow of the process which the image pickup apparatus 100 generates the intermediate layer data is shown. A flowchart showing a flow of processing executed by the server 180 is shown. An example of an unmanned aerial vehicle (UAV) is shown. An example of a computer 1200 in which a plurality of aspects of the present invention may be embodied in whole or in part is shown.

Hereinafter, the present invention will be described through embodiments of the invention, but the following embodiments do not limit the invention according to the claims. Also, not all combinations of features described in the embodiments are essential to the means of solving the invention. It will be apparent to those skilled in the art that various changes or improvements can be made to the following embodiments. It is clear from the description of the claims that such modified or improved forms may also be included in the technical scope of the present invention.

The claims, description, drawings, and abstracts include matters that are subject to copyright protection. The copyright holder will not object to any person's reproduction of these documents as long as they appear in the Patent Office files or records. However, in other cases, all copyrights are reserved.

Various embodiments of the present invention may be described with reference to flowcharts and block diagrams, wherein the block is (1) a stage of the process in which the operation is performed or (2) a device having a role of performing the operation. May represent the "part" of. Specific steps and "parts" may be implemented by programmable circuits and / or processors. Dedicated circuits may include digital and / or analog hardware circuits. It may include integrated circuits (ICs) and / or discrete circuits. Programmable circuits may include reconfigurable hardware circuits. Reconfigurable hardware circuits include logical AND, logical OR, logical XOR, logical NAND, logical NOR, and other logical operations, flip-flops, registers, field programmable gate arrays (FPGA), programmable logic arrays (PLA), etc. It may include a memory element such as.

The computer-readable medium may include any tangible device capable of storing instructions executed by the appropriate device. As a result, the computer-readable medium having the instructions stored therein will include the product, including instructions that can be executed to create means for performing the operation specified in the flowchart or block diagram. Examples of computer-readable media may include electronic storage media, magnetic storage media, optical storage media, electromagnetic storage media, semiconductor storage media, and the like. More specific examples of computer-readable media include floppy® disks, diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), Electrically erasable programmable read-only memory (EEPROM), static random access memory (SRAM), compact disc read-only memory (CD-ROM), digital versatile disc (DVD), Blu-ray® disc, memory stick, An integrated circuit card or the like may be included.

Computer-readable instructions may include either source code or object code written in any combination of one or more programming languages. Source code or object code includes traditional procedural programming languages. Traditional procedural programming languages are assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcodes, firmware instructions, state-setting data, or Smalltalk®, JAVA®, C ++. It may be an object-oriented programming language such as, and a "C" programming language or a similar programming language. Computer-readable instructions are applied locally or to a processor or programmable circuit of a general purpose computer, special purpose computer, or other programmable data processing device, or a wide area network (WAN) such as a local area network (LAN), the Internet, etc. ) May be provided. The processor or programmable circuit may execute computer-readable instructions to create means for performing the operations specified in the flowchart or block diagram. Examples of processors include computer processors, processing units, microprocessors, digital signal processors, controllers, microcontrollers and the like.

FIG. 1 is a diagram showing an example of an external perspective view of the image pickup apparatus 100 according to the present embodiment. FIG. 2 is a diagram showing a functional block of the image pickup apparatus 100 according to the present embodiment.

The image pickup apparatus 100 includes an image pickup section 102 and a lens section 200. The image pickup unit 102 includes an image sensor 120, a control unit 110, a memory 130, an instruction unit 162, a display unit 160, and a communication unit 170.

The image sensor 120 may be composed of a CCD or CMOS. The image sensor 120 receives light through the lens 210 included in the lens unit 200. The image sensor 120 outputs the image data of the optical image formed through the lens 210 to the control unit 110.

The control unit 110 may be composed of a CPU, a microprocessor such as an MPU, a microcontroller such as an MCU, or the like. The memory 130 may be a computer-readable recording medium and may include at least one of flash memories such as SRAM, DRAM, EPROM, EEPROM, and USB memory. The control unit 110 corresponds to the circuit. The memory 130 stores a program or the like necessary for the control unit 110 to control the image sensor 120 or the like. The memory 130 may be provided inside the housing of the image pickup apparatus 100. The memory 130 may be provided so as to be removable from the housing of the image pickup apparatus 100.

The instruction unit 162 is a user interface that receives an instruction to the image pickup apparatus 100 from the user. The display unit 160 displays an image captured by the image sensor 120 and processed by the control unit 110, various setting information of the image pickup device 100, and the like. The display unit 160 may be composed of a touch panel.

The control unit 110 controls the lens unit 200 and the image sensor 120. For example, the control unit 110 controls the focal position and focal length of the lens 210. The control unit 110 controls the lens unit 200 by outputting a control command to the lens control unit 220 included in the lens unit 200 based on the information indicating the instruction from the user.

The lens unit 200 includes one or more lenses 210, a lens driving unit 212, a lens control unit 220, and a memory 222. In this embodiment, one or more lenses 210 are collectively referred to as "lens 210". The lens 210 may include a focus lens and a zoom lens. At least some or all of the lenses included in the lens 210 are movably arranged along the optical axis of the lens 210. The lens unit 200 may be an interchangeable lens that is detachably provided to the imaging unit 102.

The lens driving unit 212 moves at least a part or all of the lens 210 along the optical axis of the lens 210. The lens control unit 220 drives the lens drive unit 212 in accordance with a lens control command from the image pickup unit 102 to move the entire lens 210 or the zoom lens or focus lens included in the lens 210 along the optical axis direction. Perform at least one of the zoom and focus movements. The lens control command is, for example, a zoom control command, a focus control command, and the like.

The lens driving unit 212 may include a voice coil motor (VCM) that moves at least a part or all of the plurality of lenses 210 in the optical axis direction. The lens driving unit 212 may include an electric motor such as a DC motor, a coreless motor, or an ultrasonic motor. The lens driving unit 212 transmits power from the motor to at least a part or all of the plurality of lenses 210 via mechanical members such as a cam ring and a guide shaft, and makes at least a part or all of the lenses 210 an optical axis. You may move it along.

The memory 222 stores the control values for the focus lens and the zoom lens that move via the lens driving unit 212. The memory 222 may include at least one of flash memories such as SRAM, DRAM, EPROM, EEPROM, and USB memory.

The control unit 110 executes control including control of the imaging operation on the image sensor 120 by outputting a control command to the image sensor 120 based on the information indicating the user's instruction acquired through the instruction unit 162 or the like. The control unit 110 acquires an image captured by the image sensor 120. The control unit 110 performs image processing on the image acquired from the image sensor 120 and stores it in the memory 130.

The communication unit 170 is responsible for communication with the outside. The communication unit 170 transmits the information generated by the control unit 110 to the outside through the communication network. The communication unit 170 provides the control unit 110 with information received from the outside through the communication network.

FIG. 3 schematically shows an overall view of the system 310 including the image pickup apparatus 100 and the server 180. The image pickup apparatus 100 processes the moving image data captured by the image pickup apparatus 100 by using the multilayer neural network. Specifically, the image pickup apparatus 100 performs processing up to a specific intermediate layer in the multi-layer neural network and generates data of the specific intermediate layer. The image pickup apparatus 100 compresses the data of the intermediate layer using the motion information of the moving image to generate the intermediate layer data. The communication unit 170 of the image pickup apparatus 100 transmits the intermediate layer data and the motion data to the server 180 through the communication network 190.

When the server 180 receives the intermediate layer data and the motion information, the server 180 acquires the intermediate layer data by extending the intermediate layer data using the motion information. Then, the server 180 reflects the data of the intermediate layer on the specific intermediate layer in the multi-layer neural network, and performs the processing after the specific intermediate layer in the multi-layer neural network. The server 180 transmits the processing result based on the information of the output layer of the multiple neural network to the image pickup apparatus 100 through the communication network 190. For example, when the multi-layer neural network is a neural network for image recognition, the image pickup device 100 transmits the image recognition result to the image pickup device 100.

The outline of the processing related to the neural network executed by the image pickup apparatus 100 will be described. The control unit 110 generates the first intermediate layer data by performing processing up to a specific intermediate layer in the multi-layer neural network on the image data of the first moving image constituting the moving image. The control unit 110 generates the second intermediate layer data by performing processing up to a specific intermediate layer in the multi-layer neural network on the image data of the second moving image constituent image constituting the moving image. The control unit 110 generates motion data between the first intermediate layer data and the second intermediate layer data based on the motion information between the first moving image constituent image and the second moving image constituent image. .. The control unit 110 obtains the first intermediate layer data and the second intermediate layer data by performing motion compensation between the first intermediate layer data and the second intermediate layer data based on the motion data. Generate residual data between. Based on the residual data, the control unit 110 generates intermediate layer data to be transmitted to another device that performs processing after a specific intermediate layer in the multi-layer neural network. The communication unit 170 transmits the intermediate layer data and the motion data to another device. For example, the communication unit 170 transmits the intermediate layer data and the motion data to the server 180 through the communication network 190.

The control unit 110 generates the quantized data by quantizing the residual data. The control unit 110 generates intermediate layer data based on the quantization data. For example, the control unit 110 generates intermediate layer data by entropy coding the quantization data.

The control unit 110 scales the motion information based on the sizes of the first moving image and the second moving image and the sizes of the first intermediate layer data and the second intermediate layer data to obtain the motion data. May be generated.

The multi-layer neural network may be a convolutional neural network. The multi-layer neural network may include one or more intermediate layers for performing a convolution operation on the first moving image and the second moving image. The multi-layer neural network may be a neural network for performing super-resolution processing of moving images. The multi-layer neural network may be a neural network for performing image recognition processing from a moving image. Applications of multi-layer neural networks are not limited to super-resolution processing and image recognition processing for results. The multi-layer neural network may be any neural network that inputs an image.

In the multi-layer neural network, the neural network parameters from the input layer before the specific intermediate layer of the multi-layer neural network to the output layer after the specific intermediate layer may be learned by using the training data. The control unit 110 stores neural network parameters from the input layer to a specific intermediate layer. For example, the control unit 110 stores in advance the neural network parameters from the input layer to the specific intermediate layer in the memory 130. The control unit 110 does not have to store the neural network parameters in the stage after the intermediate layer.

A multi-layer neural network may have a plurality of intermediate layers. The control unit 110 has a plurality of intermediate layers included in the multilayer neural network based on at least one of the communication line capacity available in the image pickup apparatus 100, the load state of the image pickup apparatus 100, and the power supply state of the image pickup apparatus 100. Select a specific intermediate layer from among. For example, the control unit 110 may select the earlier layer as a specific intermediate layer as the communication line capacity available in the image pickup apparatus 100 becomes smaller. The control unit 110 may select a layer in the front stage as a specific intermediate layer as the load state of the image pickup apparatus 100 increases. The control unit 110 may select a layer in the front stage as a specific intermediate layer as the remaining power supply capacity of the image pickup apparatus 100 is lower.

Next, an outline of the processing executed by the server 180 will be described. The server 180 is a device that performs processing after the specific intermediate layer in the multi-layer neural network. The server 180 includes (i) first intermediate layer data generated by processing the first moving image constituent image from the image pickup device 100 to a specific intermediate layer in the multilayer neural network, and (ii). Motion data between the first intermediate layer data and the second intermediate layer data generated by processing up to a specific intermediate layer in the multilayer neural network for the second moving image constituent image, and (iii). ) Residual data between the first intermediate layer data and the second intermediate layer data, which is between the first intermediate layer data and the second intermediate layer data generated by performing motion compensation based on the motion data. Receives residual data with and from layer data. The server 180 generates the second intermediate layer data based on the motion data and the residual data. The server 180 processes the first intermediate layer data and the second intermediate layer data after the specific intermediate layer in the multi-layer neural network.

FIG. 4 schematically shows a moving image constituent image constituting a moving image captured by the imaging device 100. The moving image is composed of a plurality of time-series moving image constituent images such as the moving image constituent image 400, the moving image constituent image 401, the moving image constituent image 402, the moving image constituent image 403, and so on.

FIG. 5 schematically shows how the multilayer neural network 580 processes the moving image. The multi-layer neural network 580 includes an input layer 500, an intermediate layer 501, an intermediate layer 502, an intermediate layer 503, an intermediate layer 508, and an output layer 509. The multi-layer neural network 580 is, for example, a convolutional neural network (CNN) for performing image recognition processing from a moving image constituent image and super-resolution processing of a moving image constituent image. The moving image constituent image 400, the moving image constituent image 401, the moving image constituent image 402 ... Are sequentially input to the input layer 500 in a predetermined order. The moving image constituent image input to the input layer 500 is sequentially processed among a plurality of intermediate layers including the intermediate layer 501, the intermediate layer 502, the intermediate layer 503, and the intermediate layer 508, and is output from the output layer 509. The processing result is generated by the multi-layer neural network 580 based on the data of the output layer 509.

In the present embodiment, the image pickup apparatus 100 is in charge of the processing from the input layer 500 to the intermediate layer 502 in the multilayer neural network 580, and the server 180 is in charge of the processing from the intermediate layer 502 to the output layer 509 in the subsequent stage.

FIG. 6 schematically shows an outline of the intermediate layer data compression process in the image pickup apparatus 100. The control unit 110 detects the movement between the plurality of moving image constituent images. For example, the control unit 110 may detect the movement by a method such as block matching. FIG. 6 shows a state in which motion information M0 is detected as a motion related to the moving image constituent image 401 with the moving image constituent image 400 as a reference image.

The intermediate layer data 601 shows the state of the intermediate layer 501 obtained by inputting the moving image constituent image 400 into the input layer 500, and the intermediate layer data 602 is obtained by inputting the moving image constituent image 400 into the input layer 500. The state of the intermediate layer 502 is shown. The intermediate layer data 611 shows the state of the intermediate layer 501 obtained by inputting the moving image constituent image 401 to the input layer 500, and the intermediate layer data 612 is obtained by inputting the moving image constituent image 401 to the input layer 500. The state of the intermediate layer 502 is shown.

First, the control unit 110 performs quantization 630 and coding 640 on the intermediate layer data 602. The communication unit 170 transmits the data obtained by the quantization 630 and the coding 640 of the intermediate layer data 602 to the server 180.

Next, the control unit 110 acquires the motion information M1 based on the motion information M0 acquired from the moving image constituent image 400 and the moving image constituent image 401. The control unit 110 acquires the motion information M1 by scaling the motion information M0 based on the sizes of the moving image constituent image 400 and the moving image constituent image 401 and the size of the intermediate layer 502. When the size of the moving image constituent image 400 and the moving image constituent image 401 and the size of the intermediate layer 502 are the same, the motion information M0 may be applied as it is as the motion information M1. When the size of the intermediate layer 502 is smaller than that of the moving image constituent image 400 and the moving image constituent image 401 due to the convolution processing and the pooling processing existing up to the intermediate layer 502 in the multi-layer neural network 580, the control unit 110 causes motion information. Motion information M1 is generated by scaling M0 according to the reduction ratio.

The control unit 110 generates the prediction data of the middle layer data 612 from the middle layer data 602 using the motion information M1, and generates the residual information 620 between the middle layer data 612 and the prediction data. The control unit 110 quantizes the residual information 620 650, and encodes the information obtained by the quantization of the residual information 620 660. For example, the control unit 110 encodes the information obtained by the quantization of the residual information 620 by the entropy coding. The communication unit 170 transmits the entropy-encoded information to the server 180.

FIG. 7 schematically shows the block configuration of the control unit 110. The first CNN part 710 shows the part from the input layer 500 to the intermediate layer 502 in the multi-layer neural network 580. The moving image of the input moving image is processed by the first CNN unit 710 and stored in the intermediate data buffer 720.

The motion detection unit 730 detects motion information between a plurality of moving image constituent images. The motion mapping unit 750 acquires the motion information in the intermediate layer 502 by scaling the motion information detected by the motion detection unit 730 to the size of the intermediate layer 502. The intermediate data prediction unit 740 generates prediction data of the intermediate layer data based on the motion information in the intermediate layer 502 acquired by the motion mapping unit 750 and the intermediate layer data stored in the intermediate data buffer 720. Generate residual information between the middle layer data and the forecast data. The quantization / entropy coding unit 760 quantizes the residual information generated by the intermediate data prediction unit 740, and performs entropy coding on the quantized residual information. The communication unit 170 generates transmission data 780 including the coded data of the residual information obtained by the quantization / entropy coding unit 760 and the motion information detected from the moving image constituent image by the motion detection unit 730. , Send to server 180.

It can be expected that the residual information of the intermediate layer data will be a small value in the area where the still subject is shown or the area where there is global movement. Therefore, it can be expected that the value of the residual information becomes 0 in many cases due to the quantization process. Therefore, it can be expected that the compression rate of the intermediate layer data is increased by performing a coding process such as run-length coding. Arithmetic coding may be applied as the coding. Since it is expected that the probability of occurrence of 0 and 1 values after quantization will be biased, it can be expected that the compression rate can be increased by using arithmetic coding.

FIG. 8 schematically shows a block configuration of the server 180. The second CNN part 810 shows a part from the intermediate layer 502 to the output layer 509 in the multi-layer neural network 580. The entropy decoding / quantization unit 830 generates residual information of the intermediate layer data by performing entropy decoding and dequantization of the intermediate layer data received from the image pickup apparatus 100. Further, the motion mapping unit 850 maps the motion information received from the image pickup apparatus 100 to the motion information in the intermediate layer 502, and acquires the motion information in the intermediate layer 502. The intermediate data reconstruction unit 840 generates prediction data of the intermediate layer data based on the intermediate layer data stored in the intermediate data buffer 820 and the motion information acquired by the motion mapping unit 850. The intermediate data reconstruction unit 840 reconstructs the state of the intermediate layer 502 by adding the difference information of the intermediate layer data to the prediction data. The second CNN unit 810 performs the processing after the intermediate layer 502, and outputs the state of the output layer 509 as the processing result.

FIG. 9 shows a flowchart showing a flow of processing in which the image pickup apparatus 100 generates intermediate layer data. In S910, the control unit 110 inputs the target moving image configuration image to the first CNN unit 710. In S920, the intermediate layer data of the intermediate layer 502 is generated. In S930, the motion detection unit 730 detects motion information between the target moving image constituent image and the referenced moving image constituent image. In S940, the motion information detected in S930 is scaled to the motion information of the intermediate layer 502. In S950, the intermediate data prediction unit 740 generates prediction data in the intermediate layer 502 using the motion information acquired in S940, and generates residual information between the intermediate layer data and the prediction data in the intermediate layer 502.

In S960, the quantization / entropy coding unit 760 generates intermediate layer data for transmission by performing quantization and entropy coding on the residual information generated in S950. In S970, the communication unit 170 transmits the transmission data including the intermediate layer data generated in S960 and the motion information detected in S930 to the server 180.

FIG. 10 shows a flowchart showing the flow of processing executed by the server 180. In S1010, the intermediate layer data transmitted from the image pickup apparatus 100 is acquired. In this flowchart, the case where the residual information of the intermediate layer data is acquired will be described.

In S1020, entropy decoding and dequantization are performed on the intermediate layer data. In S1030, the server 180 acquires motion information between the moving image constituent images. In S1040, the motion information acquired in S1030 is scaled to the motion information of the intermediate layer 502. In S1050, the prediction data in the intermediate layer 502 is generated using the motion information acquired in S1040, and the intermediate layer data acquired in S1010 is added to the prediction data to reconstruct the intermediate layer data in the intermediate layer 502.

In S1060, the process by the second CNN unit 810 is performed. In S1070, the state of the output layer 509 of the second CNN unit 810 is acquired and transmitted to the image pickup apparatus 100 as the processing result of the multilayer neural network 580.

In the embodiment described above, the communication unit 170 transmits the motion information detected from the moving image configuration image to the server 180. The communication unit 170 may transmit a moving image configuration image as another example of motion information to the server 180. The server 180 may acquire motion information between the moving image constituent images by performing motion detection by the same algorithm as the motion information detection algorithm in the control unit 110 using the moving image constituent image received from the imaging device 100. .. In this case, the communication unit 170 does not have to transmit the motion information detected from the moving image configuration image to the server 180.

In the embodiment described above, the mode in which the communication unit 170 transmits the residual information of the intermediate layer data of the intermediate layer 502 to the server 180 has been described. However, the communication unit 170 may also transmit the residual information of the intermediate layer data of the intermediate layer 501. When the information of the intermediate layer or the input layer before the intermediate layer 502 is required in the processing after the intermediate layer 502 in the multi-layer neural network 580, the residual information of all the layers required in the subsequent processing is stored in the server 180. May be sent to.

In the embodiment described above, the image pickup apparatus 100 processes the layers from the input layer 500 to the intermediate layer 502 in the multilayer neural network 580. The layer to be processed by the image pickup device 100 may be selected according to the communication capacity available to the image pickup device 100, the remaining capacity of the power supply of the image pickup device 100, the processing capacity of the image pickup device 100, and the current load of the image pickup device 100. .. For example, when the remaining capacity of the power supply of the image pickup apparatus 100 is smaller than a predetermined value, the image pickup apparatus 100 may be in charge of the layers up to the intermediate layer 501. In this case, the communication unit 170 may transmit information identifying the selected intermediate layer to the server 180.

In the embodiment described above, the motion detection unit 730 acquires motion information by detecting motion from the moving image. Even if the motion detection unit 730 acquires motion information calculated for other purposes such as noise removal and image compression. Further, the motion detection unit 730 may use the information obtained from the gyro sensor as the motion information. When a motion vector or the like calculated by image compression is used as motion information, based on the GOP structure of the image compression information, the prediction data of the intermediate layer data is used not only in the past but also in the future video composition image in the video composition image. May be generated to generate residual information.

According to the embodiment described above, the image pickup apparatus 100 and the server 180 can share the processing of one multilayer neural network 580. As described above, the control unit 110 compensates for the motion of the intermediate layer data by using the motion information between the moving image constituent images. Therefore, the amount of intermediate layer data transmitted from the image pickup apparatus 100 to the server 180 can be reduced. As a result, the network load can be reduced. Further, the image pickup apparatus 100 and the server 180 can share the processing of the multilayer neural network 580. This makes it possible to reduce the possibility that, for example, the communication band or the overload on the server 180 becomes a bottleneck in the processing speed. Therefore, it can be expected that the image pickup apparatus 100 can quickly acquire the processing result of the multilayer neural network 580.

Some or all the functions of the image pickup apparatus 100 may be incorporated into a mobile terminal such as a mobile phone. The image pickup apparatus 100 may be a surveillance camera. The image pickup device 100 may be a video camera or the like. Some or all of the functions of the imaging device 100 may be incorporated into any device capable of capturing moving images.

The image pickup apparatus 100 as described above may be mounted on a moving body. The imaging device 100 may be mounted on an unmanned aerial vehicle (UAV) as shown in FIG. The UAV 10 may include a UAV main body 20, a gimbal 50, a plurality of image pickup devices 60, and an image pickup device 100. The gimbal 50 and the imaging device 100 are examples of an imaging system. The UAV 10 is an example of a moving body propelled by a propulsion unit. The moving body is a concept including a UAV, a flying object such as another aircraft moving in the air, a vehicle moving on the ground, a ship moving on the water, and the like.

The UAV main body 20 includes a plurality of rotor blades. The plurality of rotor blades are an example of a propulsion unit. The UAV main body 20 flies the UAV 10 by controlling the rotation of a plurality of rotor blades. The UAV body 20 flies the UAV 10 using, for example, four rotor blades. The number of rotor blades is not limited to four. Further, the UAV 10 may be a fixed-wing aircraft having no rotor blades.

The imaging device 100 is an imaging camera that captures a subject included in a desired imaging range. The gimbal 50 rotatably supports the imaging device 100. The gimbal 50 is an example of a support mechanism. For example, the gimbal 50 rotatably supports the image pickup device 100 on a pitch axis using an actuator. The gimbal 50 further rotatably supports the image pickup device 100 around each of the roll axis and the yaw axis by using an actuator. The gimbal 50 may change the posture of the image pickup device 100 by rotating the image pickup device 100 around at least one of the yaw axis, the pitch axis, and the roll axis.

The plurality of image pickup devices 60 are sensing cameras that image the surroundings of the UAV 10 in order to control the flight of the UAV 10. Two imaging devices 60 may be provided on the front surface, which is the nose of the UAV 10. Yet two other imaging devices 60 may be provided on the bottom surface of the UAV 10. The two image pickup devices 60 on the front side may form a pair and function as a so-called stereo camera. The two image pickup devices 60 on the bottom surface side may also be paired and function as a stereo camera. Three-dimensional spatial data around the UAV 10 may be generated based on the images captured by the plurality of imaging devices 60. The number of image pickup devices 60 included in the UAV 10 is not limited to four. The UAV 10 may include at least one imaging device 60. The UAV 10 may be provided with at least one imaging device 60 on each of the nose, nose, side surface, bottom surface, and ceiling surface of the UAV 10. The angle of view that can be set by the image pickup device 60 may be wider than the angle of view that can be set by the image pickup device 100. The image pickup apparatus 60 may have a single focus lens or a fisheye lens.

The remote control device 300 communicates with the UAV 10 to remotely control the UAV 10. The remote control device 300 may wirelessly communicate with the UAV 10. The remote control device 300 transmits to the UAV 10 instruction information indicating various commands related to the movement of the UAV 10, such as ascending, descending, accelerating, decelerating, advancing, reversing, and rotating. The instruction information includes, for example, instruction information for raising the altitude of the UAV 10. The instruction information may indicate the altitude at which the UAV 10 should be located. The UAV 10 moves so as to be located at an altitude indicated by the instruction information received from the remote control device 300. The instruction information may include an ascending instruction to ascend the UAV 10. The UAV10 rises while accepting the rise order. Even if the UAV10 accepts the ascending command, the ascending may be restricted if the altitude of the UAV10 has reached the upper limit altitude.

FIG. 12 shows an example of a computer 1200 in which a plurality of aspects of the present invention may be embodied in whole or in part. The program installed on the computer 1200 can cause the computer 1200 to function as an operation associated with the device according to an embodiment of the present invention or as one or more "parts" of the device. For example, a program installed on a computer 1200 can cause the computer 1200 to function as a control unit 110. Alternatively, the program may cause the computer 1200 to perform the operation or the function of the one or more "parts". The program can cause a computer 1200 to perform a process or a step of the process according to an embodiment of the present invention. Such a program may be run by the CPU 1212 to cause the computer 1200 to perform certain operations associated with some or all of the blocks in the flowcharts and block diagrams described herein.

The computer 1200 according to this embodiment includes a CPU 1212 and a RAM 1214, which are connected to each other by a host controller 1210. The computer 1200 also includes a communication interface 1222, an input / output unit, which are connected to the host controller 1210 via an input / output controller 1220. The computer 1200 also includes a ROM 1230. The CPU 1212 operates according to the programs stored in the ROM 1230 and the RAM 1214, thereby controlling each unit.

Communication interface 1222 communicates with other electronic devices via a network. The hard disk drive may store programs and data used by the CPU 1212 in the computer 1200. The ROM 1230 stores in it a boot program or the like executed by the computer 1200 at the time of activation and / or a program depending on the hardware of the computer 1200. The program is provided via a computer-readable recording medium such as a CR-ROM, USB memory or IC card or network. The program is installed in RAM 1214 or ROM 1230, which is also an example of a computer-readable recording medium, and is executed by CPU 1212. The information processing described in these programs is read by the computer 1200 and provides a link between the program and the various types of hardware resources described above. The device or method may be configured to implement the operation or processing of information according to the use of the computer 1200.

For example, when communication is executed between the computer 1200 and an external device, the CPU 1212 executes a communication program loaded in the RAM 1214, and performs communication processing on the communication interface 1222 based on the processing described in the communication program. You may order. Under the control of the CPU 1212, the communication interface 1222 reads the transmission data stored in the transmission buffer area provided in the RAM 1214 or a recording medium such as a USB memory, and transmits the read transmission data to the network, or The received data received from the network is written to the reception buffer area or the like provided on the recording medium.

Further, the CPU 1212 makes the RAM 1214 read all or necessary parts of a file or a database stored in an external recording medium such as a USB memory, and executes various types of processing on the data on the RAM 1214. good. The CPU 1212 may then write back the processed data to an external recording medium.

Various types of information such as various types of programs, data, tables, and databases may be stored in recording media and processed. The CPU 1212 describes various types of operations, information processing, conditional judgment, conditional branching, unconditional branching, and information retrieval described in various parts of the present disclosure with respect to the data read from the RAM 1214. Various types of processing may be performed, including / replacement, etc., and the results are written back to the RAM 1214. Further, the CPU 1212 may search for information in a file, a database, or the like in the recording medium. For example, when a plurality of entries each having an attribute value of the first attribute associated with the attribute value of the second attribute are stored in the recording medium, the CPU 1212 specifies the attribute value of the first attribute. Search for an entry that matches the condition from the plurality of entries, read the attribute value of the second attribute stored in the entry, and associate it with the first attribute that satisfies the predetermined condition. The attribute value of the second attribute obtained may be acquired.

The program or software module described above may be stored on a computer 1200 or in a computer readable storage medium near the computer 1200. Also, a recording medium such as a hard disk or RAM provided within a dedicated communication network or a server system connected to the Internet can be used as a computer readable storage medium, thereby allowing the program to be transferred to the computer 1200 over the network. offer.

Although the present invention has been described above using the embodiments, the technical scope of the present invention is not limited to the scope described in the above embodiments. It will be apparent to those skilled in the art that various changes or improvements can be made to the above embodiments. It is clear from the description of the claims that such modified or improved forms may also be included in the technical scope of the present invention.

The order of execution of operations, procedures, steps, steps, etc. in the devices, systems, programs, and methods shown in the claims, specification, and drawings is particularly "before" and "prior to". It should be noted that it can be realized in any order unless the output of the previous process is used in the subsequent process. Even if the scope of claims, the specification, and the operation flow in the drawings are explained using "first", "next", etc. for convenience, it means that it is essential to carry out in this order. It's not a thing.

10 UAV
20 UAV main unit 50 gimbal 60 imaging device 100 imaging device 102 imaging unit 110 control unit 120 image sensor 130 memory 160 display unit 162 indicator unit 170 communication unit 180 server 190 communication network 200 lens unit 210 lens 212 lens drive unit 220 lens control unit 222 Memory 300 Remote control device 310 System 400, 401, 402, 403 Video configuration image 500 Input layer 501, 502, 503, 508 Intermediate layer 509 Output layer 580 Multilayer neural network 601 Intermediate layer data 602 Intermediate layer data 611 Intermediate layer data 612 Intermediate Layer data 620 Residual information 630 Quantization 640 Coding 650 Quantization 660 Coding 710 1st CNN part 720 Intermediate data buffer 730 Motion detection part 740 Intermediate data prediction part 750 Mapping part 760 Quantization / entropy coding part 780 Transmission data 810 2nd CNN part 820 Intermediate data buffer 830 Entropy decoding / quantization unit 840 Intermediate data reconstruction unit 850 Motion mapping unit 1200 Computer 1210 Host controller 1212 CPU
1214 RAM
1220 Input / Output Controller 1222 Communication Interface 1230 ROM

Claims (16)

The first intermediate layer data is generated by processing the image data of the first moving image constituting the moving image up to a specific intermediate layer in the multi-layer neural network.
The second intermediate layer data is generated by performing processing up to the specific intermediate layer in the multi-layer neural network on the image data of the second moving image constituent image constituting the moving image.
Based on the motion information between the first moving image and the second moving image, motion data between the first intermediate layer data and the second intermediate layer data is generated.
By performing motion compensation between the first intermediate layer data and the second intermediate layer data based on the motion data, between the first intermediate layer data and the second intermediate layer data. Generate residual data for
Based on the residual data, intermediate layer data to be transmitted to another device that performs processing after the specific intermediate layer in the multi-layer neural network is generated.
A device including a circuit configured to transmit the intermediate layer data and the motion data to the other device. The circuit
Quantized data is generated by quantizing the residual data,
The apparatus according to claim 1, wherein the intermediate layer data is generated based on the quantization data. The apparatus according to claim 2, wherein the circuit is configured to generate the intermediate layer data by entropy coding the quantization data. The apparatus according to claim 1 or 2, wherein the multi-layer neural network is a convolutional neural network, and includes one or more intermediate layers for performing a convolution operation on the first moving image and the second moving image. .. The circuit scales the motion information based on the sizes of the first moving image and the second moving image and the sizes of the first intermediate layer data and the second intermediate layer data. The apparatus according to claim 4, wherein the motion data is generated. The apparatus according to claim 1 or 2, wherein the multi-layer neural network is a neural network for performing super-resolution processing of the moving image or image recognition processing from the moving image. In the multi-layer neural network, the neural network parameters from the input layer before the specific intermediate layer to the output layer after the specific intermediate layer of the multi-layer neural network are learned by using the training data.
The device according to claim 1 or 2, wherein the circuit is configured to store neural network parameters from the input layer to the specific intermediate layer. The multi-layer neural network has a plurality of intermediate layers and has a plurality of intermediate layers.
The circuit selects the specific intermediate layer from the plurality of intermediate layers based on at least one of a communication line capacity in the device, a load state of the device, and a power state of the device. The device according to claim 1 or 2, which is configured as described above. The first moving image constituent image and the second moving image constituent image constituting the moving image are communicated with another device that processes up to a specific intermediate layer in the multi-layer neural network, and the specific intermediate layer in the multi-layer neural network is communicated with. It is a device that performs the subsequent processing,
From the other device
(I) The first intermediate layer data generated by performing the processing up to the specific intermediate layer in the multilayer neural network on the first moving image constituent image, and
(Ii) Between the first intermediate layer data and the second intermediate layer data generated by performing processing up to the specific intermediate layer in the multilayer neural network on the second moving image constituent image. Motion data and
(Iii) Residual data between the first intermediate layer data and the second intermediate layer data, and the first intermediate layer generated by performing motion compensation based on the motion data. Residual data between the data and the second intermediate layer data and
Received
Based on the motion data and the residual data, the second intermediate layer data is generated.
A device including a circuit configured to perform processing on the first intermediate layer data and the second intermediate layer data after the specific intermediate layer in the multilayer neural network. The device according to claim 1 and
A system including the device according to claim 9. The device according to claim 1 or 2,
An imaging device including an image sensor that generates the moving image. A moving body that moves with the imaging device according to claim 11. The mobile body according to claim 12, wherein the mobile body is an unmanned aerial vehicle. A program for operating a computer as the device according to claim 1 or 2. The stage of generating the first intermediate layer data by processing the image data of the first moving image constituting the moving image up to a specific intermediate layer in the multi-layer neural network, and the stage of generating the first intermediate layer data.
A step of generating a second intermediate layer data by performing processing up to the specific intermediate layer in the multi-layer neural network on the image data of the second moving image constituent image constituting the moving image.
A step of generating motion data between the first intermediate layer data and the second intermediate layer data based on motion information between the first moving image and the second moving image. When,
By performing motion compensation between the first intermediate layer data and the second intermediate layer data based on the motion data, between the first intermediate layer data and the second intermediate layer data. At the stage of generating the residual data of
Based on the residual data, a step of generating intermediate layer data to be transmitted to another device that performs processing after the specific intermediate layer in the multilayer neural network, and a step of generating the intermediate layer data.
A method including a step of transmitting the intermediate layer data and the motion data to the other device. The first moving image constituent image and the second moving image constituent image constituting the moving image are communicated with another device that processes up to a specific intermediate layer in the multi-layer neural network, and the specific intermediate layer in the multi-layer neural network is communicated with. It is a method to perform the later processing,
From the other device
(I) The first intermediate layer data generated by performing the processing up to the specific intermediate layer in the multilayer neural network on the first moving image constituent image, and
(Ii) Between the first intermediate layer data and the second intermediate layer data generated by performing processing up to the specific intermediate layer in the multilayer neural network on the second moving image constituent image. Motion data and
(Iii) Residual data between the first intermediate layer data and the second intermediate layer data, and the first intermediate layer generated by performing motion compensation based on the motion data. Residual data between the data and the second intermediate layer data and
At the stage of receiving
A step of generating the second intermediate layer data based on the motion data and the residual data, and
A method including a step of performing processing on the first intermediate layer data and the second intermediate layer data after the specific intermediate layer in the multilayer neural network.

Images