JP2020113829A - Moving image processing method and moving image processing apparatus - Google Patents

Abstract

To provide a moving image processing method that can determine an appropriate exposure pattern according to the type of an image sensor.SOLUTION: A moving image processing method includes a compression step of generating a compressed moving image by performing imaging with repeated exposure that is thinned out temporally and spatially by using an image sensor in which pixels are arranged two-dimensionally, and a first machine learning step S1 of optimizing an exposure pattern for identifying an exposure mode by machine learning prior to the compression step. In the compression step, a compressed moving image is generated using the exposure pattern obtained by the optimization in the first machine learning step S1.SELECTED DRAWING: Figure 12

Description

The present disclosure relates to a moving image processing method and an apparatus that executes the method.

2. Description of the Related Art In recent years, video images captured by IoT (Internet of Things) devices such as surveillance cameras and vehicle-mounted cameras have been actively analyzed. Video images (that is, moving images) captured by these cameras are collected in a data center and used for analysis or the like. At this time, in order to reduce the capacity of the communication path, it is necessary to perform compression processing such as lowering the spatial resolution of video and temporal resolution (hereinafter, also referred to as frame rate). However, if the spatial resolution is lowered, the image becomes unclear, and if the frame rate is lowered, the motion information in the image is lost. As a means for solving the trade-off between the spatial resolution and the temporal resolution, a compressed video sensing method using a coded exposure image has been proposed.

For example, U.S. Pat. No. 6,096,839, modulates the light field acquired at an individual pixel of a camera sensor according to a corresponding modulation function to generate an integrated frame during each exposure time, and convex optimizes the generated frame. A method of reconstructing by a method is disclosed.

Japanese Patent No. 572657

T. Sonoda, H. Nagahara, K. Endo, Y. Sugiyama, R. Taniguchi, “High-speed imaging using CMOS image sensor with quasi pixel-wise exposure”, International Conference on Computational Photography (ICCP), pp.1- 11, 2016. M. Iliadis, L. Spinoulas, A. K. Katsaggelos, “Deep fully-connected networks for video compressive sensing”, Digital Signal Proessing 72: 9-18, 2018. Y. Hitomi, J. Gu, M. Gupta, T. Mitsuniga, SK Nayar, “Video from a single coded exposure photograph using a learned over-complete dictionary”, International Conference on Computer Vision (ICCV), pp.287-294. , 2011. J. Yang, X. Yuan, X. Liao, P. Llull, DJ Brady, G. Sapiro, L. Carin, “Video compressive sensing using Gaussian mixture models”, IEEE Transactions on Image Processing, pp.4863-4878, 2014 . M. Iliadis, L. Spinoulas, A. K. Katsaggelos, “DeepBinaryMask: Learning a Binary Mask for Video Compressive Sensing”, arXiv preprint arXiv: 1607.03343 2016. M. Courbariaux, I. Hubara, D. Soudry, R. El-Yaniv, Y. Bengio, “Binarized neural networks: Training neural networks with weights and activations constrained to +1 or -1”, arXiv preprint arXiv: 1602.02830 2016. M. Gygli, H. Grabner, H. Riemenschneider, L. V. Gool, “Creating Summaries from User Videos”, ECCV,2014, https://people.ee.ethz.ch/gyglim/vsum/ Rty. T. D.: Survey on Contemporary Remote Surveillance Systems for Public Safety, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), Vol. 40, No. 5, pp. 93-515, 2010. Li, Y., Ai, H., Yamashita, T., Lao, S. and Kawade,, M.: Tracking in Low Frame Rate Video: A Cascade Particle Filter with Discriminative Ovservers of Different Life Spans, Vol. 30, No . 10, pp. 1728-1740, 2008. Yoshida, M., Torii, A., Okutomi, M., Endo, K., Sugiyama, Y., Tanigushi, R.-i. and Nagahara, H.: Joint optimization for compressive video sensing and reconstruction under hardware constraints, Proceedings of European Conference on Conmputer Vision (ECCV), 2018. Bobick, A. F. and Davis, J. W.: The recognition of human movement using temporal templates, IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 23, No. 3, pp. 257-267, 2001. Blank, M., Gorelick, L., Shechtman, E., Irani, M. and Basri, R.: Actions as Space-Time Shapes, Proceedings of International Conference on Computer Vision (ICCV), pp. 1395-1402, 2005 . Laptev, I.: On Space-Time Interest Points, International Journal of Comnputer Vision, Vol. 64, No. 2, pp. 107-123, 2005. Dalal, N. and Triggs, B.: Histograms of oriented gradients for human detection, Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Vol. 1, pp. 886-893, 2005. Klaser, A., Marszalek, M. and Schmid, C.: A Spatio-Temporal Descriptor Based on 3D-Gradients, Proceedings of British Machine Vision Conference (BMVC) (Everningham, M., Needham, C. and Fraile, R. , eds.), Leeds, United Kingdom, British Machine Vision Association, pp. 275:1-10, 2008. Csurka, G., Dance, C. R., Fan, L., Willamowski, J. and Bray, C.: Visual categorization with bags of keypoints, Proceedings of European Conference on Computer Vision (ECCV), pp. 1-22, 2004. Lptev, I., Marszalek, M., Schmid, C. and Rozenfeld, B.: Learning realistic human actions from movies, Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1-8, 2008. Simonyan, K. and Zisserman, A.: Two-Stream Convolutional Networks for Action Recognition in Videos, Advances in Neural Information Processing System (NIPS) (Ghahramani, Z., Welling, M., Cortes, C., Lawrence, ND and Weinberger, KQ, eds.), Curran Associates, Inc., pp. 568-576, 2014. Tran, D., Bourdev, L., Fergus, R., Torresani, L. and Paluri, M.: Learning SpatiotemporalFeatures with 3D Convolutional Networks, Proceedings of International Conference on Computer Vision (ICCV), pp. 4489-4497, 2015 . Kay, W., Carreira, J., Simonyan, K., Zhang, B., Hillier, C., Vijayanarasimhan, S., Viola, F., Green, T., Back, T., Natsev, P., Suleyman, M. and Zisserman, A.: The Kinetics Human Action Video Dataset, CoRR, Vol. abs/1705.06950, 2017. Carreira, J. and Zisserman, A.: Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset, Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4724-4733, 2017. Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V. and Rabinovich, A.: Going deeper with convolutions, Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1-9, 2015. Schldt, C., Laptev, I. and Caputo, B.: Recognizing Human Actions: A Local SVM Approach, Proceedings of International Conference on Pattern Recognition (ICPR), Washington, DC, USA, IEEE Computer Society, pp. 32-36 , 2004.

In the conventional technique described in Patent Document 1, the exposure state of each pixel is modulated based on the modulation function, but the optimum exposure pattern in each frame of the image captured by the camera is determined according to the type of image sensor. It is hard to say that we have made an appropriate decision.

Therefore, the present disclosure provides a moving image processing method and a moving image processing apparatus capable of determining an appropriate exposure pattern according to the type of image sensor.

A moving image processing method according to an aspect of the present disclosure generates a compressed moving image by performing imaging by repeated exposure that is thinned out temporally and spatially using an image sensor in which pixels are arranged two-dimensionally. A compression step, and a first machine learning step of optimizing an exposure pattern for specifying the aspect of the exposure by machine learning prior to the compression step, wherein the compression step is optimized by the first machine learning step. The compressed moving image is generated using the exposure pattern obtained by the conversion.

Further, the moving image processing apparatus according to an aspect of the present disclosure captures a compressed moving image by performing imaging by repeated exposure that is thinned temporally and spatially using an image sensor in which pixels are two-dimensionally arranged. A moving image processing apparatus used for a camera to generate, which is obtained by a first machine learning unit for optimizing an exposure pattern for specifying an aspect of the exposure by machine learning, and an optimization by the first machine learning unit. And an output unit that outputs the exposed exposure pattern.

Note that these comprehensive or specific aspects may be realized by a system, an apparatus, a method, an integrated circuit, a computer program, or a non-transitory recording medium such as a computer-readable CD-ROM, It may be realized by any combination of a system, an apparatus, a method, an integrated circuit, a computer program, and a recording medium.

According to the moving image processing method and the moving image processing apparatus according to an aspect of the present disclosure, it is possible to determine an appropriate exposure pattern according to the type of image sensor.

FIG. 1 is a diagram showing an example of a flow of compression sensing of a general moving image. FIG. 2 is a diagram showing an example of an exposure pattern that satisfies the restrictions on mounting on hardware. FIG. 3 is a diagram showing an example of the structure of an SBE (Single Bump Exposure) sensor. FIG. 4 is a diagram showing the number of exposures during one frame in the SBE sensor. FIG. 5: is a figure which shows an example of the structure of a RCE(Row Column width Exposure) sensor. FIG. 6 is a diagram showing the number of exposures during one frame in the RCE sensor. FIG. 7 is a diagram showing an example of a motion expression considering the whole. FIG. 8 is a diagram showing another example of a motion expression considering the whole. FIG. 9 is a diagram for explaining the outline of a method of recognizing human actions. FIG. 10 is a block diagram showing an example of the functional configuration of the moving image processing system according to the embodiment. FIG. 11 is a diagram illustrating an example of the configuration of the machine learning unit according to the embodiment. FIG. 12 is a flowchart showing an example of the moving image processing method according to the embodiment. FIG. 13 is a diagram showing an example of the artificial intelligence used in the embodiment. FIG. 14 is a diagram showing an example of the configuration of the machine learning step in the embodiment. FIG. 15 is a diagram showing an example of updating a binarized exposure pattern. FIG. 16 is a diagram showing the results of Experimental Example 2. FIG. 17 is a diagram showing the results of Experimental Example 3. FIG. 18 is a diagram illustrating an example of a flow of compressed sensing of a color moving image. FIG. 19 is a diagram showing an example of a color filter pattern. FIG. 20 is a diagram showing an example of the exposure pattern and the color filter pattern used in the experimental example. FIG. 21 is a diagram showing the results of Experimental Example 4. FIG. 22 is a flowchart showing an example of the moving image processing method according to Modification 2. FIG. 23 is a diagram illustrating an example of elderly in the machine learning step according to the second modification. FIG. 24 is a diagram showing one scene of each action class in the KTH Action data set. FIG. 25 is a diagram illustrating an example of the comparison method in Experimental Example 5. FIG. 26 is a diagram showing an example of exposure in a pixel of an image input to the neural network. FIG. 27 is a diagram showing a confusion matrix of each comparison method. 28: is a figure which shows the result of Experimental example 6. FIG.

(Findings that form the basis of this disclosure)
Video with high spatial resolution and high frame rate is useful for analyzing what is really happening. Usually, such a moving image is captured by a high speed camera. The high-speed camera is provided with a buffer for each pixel in order to perform high-speed reading from the sensor, and is also equipped with a parallel AD converter in order to shorten the time of analog-digital (AD) conversion. Such a special sensor is very expensive, and the circuit becomes complicated, so that the area of the phototransistor is reduced and the sensitivity is deteriorated. Therefore, a method using compressed sensing has been proposed as one of means for acquiring a moving image with high spatial resolution and high frame rate (Non-Patent Documents 1 to 4).

Usually, shooting of a moving image is realized by continuously shooting a plurality of still images using a sensor having a global shutter in which all pixels are simultaneously exposed. On the other hand, the compressed video sensing has a compression step and a reconstruction step, compresses the moving image while shooting the moving image, and reconstructs the compressed moving image into the original moving image. More specifically, in the compression step, the image sensor randomly shifts the exposure timing for each adjacent pixel to capture a single image. This makes it possible to obtain a coded exposure image obtained by sampling the time information into a single image. Next, in the reconstruction step, moving images of a plurality of frames are reconstructed from the single image using different time information included in the coded exposure image obtained in the compression step. FIG. 1 is a diagram showing an example of a flow of compression sensing of a general moving image. As shown in FIG. 1, in the compressed video sensing, the sensing unit shoots a moving image including a series of scenes by shifting the exposure timing for each pixel using an exposure pattern, and thus the time information is converted into a single image. Create an aggregated coded exposure image. Then, the reconstruction unit reconstructs a moving image including a series of scenes by using different time information included in the coded exposure image. This compressed sensing model is expressed by the following equation (1).

In the formula, x is an unknown dynamic scene (unknown moving image), y is a coded exposure image, and φ is a coded exposure pattern.

Generally, in compressed sensing, an unknown moving image x is reconstructed from a coded exposure image y using a coded exposure pattern φ. From equation (1), it can be seen that the quality of the unknown moving image x reconstructed from the coded exposure image y using the coded exposure pattern φ depends on the compression performance of the coded exposure pattern φ.

In compressed video sensing, it is necessary to capture an image exposed at random timing at each pixel. Therefore, various coded exposure patterns have been proposed. However, a general CCD (Charge Coupled Device) or CMOS (Complementary Metal Oxide Semiconductor) sensor is generally a global shutter in which all pixels are exposed simultaneously or a rolling shutter in which exposure is performed in the order of reading out pixels, and compressed video sensing. There is generally no ideal sensor in. Therefore, a coded exposure pattern that assumes ideal random exposure, or a coded exposure pattern that considers restrictions on hardware implementation is used.

For example, Non-Patent Document 5 discloses a method of optimizing an exposure pattern by assuming an ideal sensor (hereinafter, a completely random sensor) capable of controlling random exposure for each pixel. Specifically, in Non-Patent Document 5, the exposure time of each pixel is divided into 16 and a random exposure pattern of 4 pixels×4 pixels×16 is repeated to obtain a coded exposure pattern of 8 pixels×8 pixels×16. I am conducting an experiment. In Non-Patent Document 1, pseudo-random exposure coded exposure is realized by using a prototype CMOS sensor capable of controlling exposure for each pixel. A demonstration experiment was performed using an 8×8 coded exposure pattern in which exposure was performed simultaneously in columns and rows due to hardware restrictions.

Hereinafter, as an example of restrictions on mounting on hardware, a CMOS sensor (see Non-Patent Document 3), which is assumed to be a realistic sensor that can control exposure for each pixel, and a prototype that can control exposure for each pixel A CMOS sensor (see Non-Patent Document 1) will be described with reference to the drawings. The CMOS sensor assumed in Non-Patent Document 3 is referred to as a SBE (Single Pump Exposure) sensor, and the prototype CMOS sensor assumed in Non-Patent Document 1 is referred to as an RCE (Row Columnwise Exposure) sensor.

FIG. 2 is a diagram showing an example of an exposure pattern that satisfies the restrictions on mounting on hardware. 2A shows an exposure pattern that can be mounted on the above-described completely random sensor, FIG. 2B shows an exposure pattern that can be mounted on an SBE sensor, and FIG. 2C shows an RCE. The exposure pattern that can be mounted on the sensor is shown.

FIG. 3 is a diagram showing an example of the structure of the SBE sensor. As shown in FIG. 3, the SBE sensor is a feasible sensor in which an address line is added to a normal CMOS sensor in order to control the exposure for each pixel. An ordinary CMOS sensor is often equipped with a rolling shutter that reads out one row by controlling an address for each row. Moreover, since a normal CMOS sensor does not have a buffer for each pixel, nondestructive reading is impossible. On the other hand, in the SBE sensor, by incorporating a circuit for determining an address for each column in a normal CMOS sensor, it is possible to read out for each pixel. FIG. 4 is a diagram showing an example of the number of exposures in one frame in the SBE sensor. As shown in FIG. 4, in the SBE sensor, each pixel is exposed once during one frame. The timing of starting and ending the exposure is an example, and is random in each frame. As shown in FIG. 2B, in Non-Patent Document 3, the exposure pattern that can be mounted on the SBE sensor (hereinafter, also referred to as a coded exposure pattern) has an arbitrary start and end with one exposure. A single exposure coded exposure pattern is disclosed. In Non-Patent Document 3, a simulation experiment and a pseudo mounting experiment using a reflective optical system and Liquid Crystal on Silicon (LCoS) are performed using a 7×7 coded exposure pattern.

FIG. 5 is a diagram showing an example of the structure of the RCE sensor. As shown in FIG. 5, the RCE sensor is a prototype CMOS sensor in which a signal line is added to control exposure. FIG. 5 shows the upper left of the RCE sensor. The RCE sensor has an 8×8 block structure. The RCE sensor includes eight Reset signal lines and eight Transfer signal lines as additional signal lines for controlling the exposure, and each Reset signal line is shared every eight columns, and each Transfer signal line is shared. The line is shared every 8 lines. Therefore, the coded exposure pattern is the same for each block. Further, the RCE sensor is capable of nondestructive readout. FIG. 6 is a diagram showing the number of exposures during one frame in the RCE sensor. As shown in FIG. 6, in the RCE sensor, each pixel can be exposed multiple times during one frame. However, the RCE sensor includes only eight Reset signal lines and only eight Transfer signal lines, and one Reset signal line and one Transfer signal line are each a column and a column in one block. Is shared between pixels. Therefore, in Non-Patent Document 1, as a coded exposure pattern that can be mounted on the RCE sensor, a demonstration experiment is performed using an 8x8 coded exposure pattern that is simultaneously exposed in columns and rows.

As described above, since there are various restrictions on the camera used for actual compressed sensing, it is necessary to optimize the coded exposure pattern while considering the restrictions on the hardware implementation.

Therefore, the inventors of the present application determine the optimum coded exposure pattern that satisfies the restrictions on mounting on hardware by using DNN (Deep Neural Network), and thus the coded exposure pattern determined by the conventional method. It was found that it is possible to reconstruct an image having a better image quality than an image reconstructed from an image compressed by using (hereinafter, compressed image). Further, the inventors of the present application optimize the coded exposure pattern and, at the same time, optimize the decoder for reconstructing the video (moving image) from the compressed image, thereby further improving the reconstruction quality as compared with the conventional method. I found that I can do it.

Next, a conventional technique regarding action recognition will be described. In the past, 3D models were used for action recognition. However, since it is difficult to construct an accurate 3D model from an image, in many cases, a method of using a global or local motion expression is used instead. The motion expression that considers the whole uses a global expression of the structure or shape of the human body or motion. FIG. 7 is a diagram showing an example of a motion expression considering the whole, and FIG. 8 is a diagram showing another example of a motion expression considering the whole. For example, as illustrated in FIG. 7, in Non-Patent Document 11, a Motion Energy Image (MEI) in which a binary image that encodes information about motion into a single image is accumulated, or a Motion History Image (MEI) that represents time by luminance. MHI) is disclosed. Further, as shown in FIG. 8, Non-Patent Document 12 discloses a Space-Time Volume (STV) in which contours of objects are stacked along a time axis. These all-inclusive approaches have the problem that it is difficult to capture changes in viewpoint and appearance, and STV cannot capture details. On the other hand, in the motion representation considering the local area, the local features for the action recognition are detected according to the procedures of the interest point detection, the local descriptor extraction, and the local descriptor aggregation simultaneously with the general image recognition. create. Non-Patent Document 13 discloses Space-Time Interest Points (STIP) as a three-dimensional extension of a two-dimensional Harris corner detector as a detection of a point of interest in a spatiotemporal region. Non-Patent Document 15 discloses the use of Histograms of Orienter Gradients (HOG) described in Non-Patent Document 14 as a motion descriptor at the pixel level in a video clip as a local spatiotemporal descriptor. A Histograms of Optical Flow (HOF) for encoding is disclosed. For aggregation of descriptors, Bag-of-Features (BoF) (Non-Patent Document 16) was used as in image recognition. In particular, in the category classification, the Support Vector Machine (STM), which has been highly evaluated in the text classification, is also used for the BoF vector (Non-Patent Document 17).

When convolutional neural networks (CNN) have come to the attention in the field of image recognition, CNN has come to be used also in the field of image recognition. CNN can be used at any stage of interest point detection, local descriptor extraction, and local descriptor aggregation, and is used not only to characterize image frames but also in combination with optical flow or HOG. .. Non-Patent Document 18 discloses that the accumulation of RGB image frames and optical flows is used as the appearance and motion information, respectively, and further improves the accuracy by combining two streams. .. There have been many studies based on two-stream networks, which greatly improve the recognition accuracy that was once without Deep Learning in datasets such as UCF101 or HMDB51. On the other hand, Non-Patent Document 19 discloses a network (C3D: Convolution 3D) that simultaneously models appearance and motion by convolving in three dimensions. This achieves good accuracy using Sports-1M, which is a large-scale moving image data set, which is inferior to 2-stream 2D CNN. Non-Patent Document 20 discloses Kinetics, which is a large-scaled and calibrated data set of action recognition. This shows that, in a relatively small-scale 3D CNN, a model without pre-training can achieve accuracy close to that of the 2D CNN pre-trained by ImageNet by training with the constructed data. Non-Patent Document 21 discloses I3D, which is an extension of 22 layers of 2D CNN, GoogleLeNet (Inception v1) (Non-Patent Document 22) to 3D, and achieves the most advanced accuracy by learning using Kinetics data set. There is.

As described above, various techniques regarding action recognition have been disclosed, but a solution by compressed sensing is considered for human action recognition in a video surveillance system, that is, a trade-off problem of data compression in video analysis. When simply considering the application of compressed video sensing, it is possible to perform video analysis in the same manner as normal moving images by reconstructing moving images from coded exposure images.

The information amount of the coded exposure image is W×H, which is the size of the coded exposure image. When the exposure time is T, the information amount of the unknown moving image is W×H×T. This cannot be uniquely defined because it restores more information than the observed information. Therefore, in Non-Patent Document 3 and Non-Patent Document 1, it is more sufficient than the observed information by using the reconstruction method by sparse optimization that assumes that the moving image can be represented by the moving image serving as the base and its sparse coefficient. The moving image is reconstructed by obtaining a small number of coefficients. In Non-Patent Document 3, an Orthogonal Matching Pursuit (OMP) algorithm that performs L ₀ Nomul regularization is used as a sparse optimization method. In general, sparse optimization is known to be an NP-hard problem. Therefore, the reconstruction method using the sparse optimization requires a huge amount of time and cannot be said to be a practical method. Non-Patent Document 4 assumes that the moving image can be represented by the Gaussian Mixture Model (GMM), and reconstructs the moving image from the expected value of the posterior probability given the coded exposure image, which is a faster means. Is disclosed. Further, Non-Patent Document 2 uses Deep Learning to learn an AutoEncoder that uses coded exposure as an encoder, thereby creating a decoder that reconstructs a moving image from a coded exposed image, and a faster reconstruction means. Is disclosed.

In addition, in the automatic monitoring system, it is necessary to detect or predict a suspicious action of a person within the field of view of the camera and warn the operator. Therefore, the inventors of the present application focus on human behavior recognition as image analysis. FIG. 9 is a diagram for explaining the outline of a method of recognizing human actions. For example, as shown in (b) of FIG. 9, when application of compressed video sensing to human behavior recognition is considered, after performing high-dimensional reconstruction of a moving image from a coded exposure image, the moving image is converted from the moving image. It is inefficient because it reduces the dimension of action label estimation. Since the encoded exposure image includes time information, it is considered that the action recognition can be directly performed without the reconstruction of the moving image as shown in (a) of FIG. Therefore, the inventors of the present application have found a method of directly recognizing human behavior from a single coded exposure image captured by a coded exposure camera by using Deep Learning.

The outline of one aspect of the present disclosure is as follows.

A moving image processing method according to an aspect of the present disclosure generates a compressed moving image by performing imaging by repeated exposure that is thinned out temporally and spatially using an image sensor in which pixels are arranged two-dimensionally. A compression step, and a first machine learning step of optimizing an exposure pattern for specifying the aspect of the exposure by machine learning prior to the compression step, wherein the compression step is optimized by the first machine learning step. The compressed moving image is generated using the exposure pattern obtained by the conversion.

As a result, since the exposure pattern is optimized by machine learning, an appropriate exposure pattern can be determined according to the type of image sensor.

For example, in the moving image processing method according to an aspect of the present disclosure, the exposure pattern is information specifying pixels used for exposure among pixels forming the image sensor for each frame forming the compressed moving image. Good.

Thus, the compressed moving image captured using the exposure pattern has temporal information indicating in which frame of each frame each pixel was exposed, and a space indicating the position of each pixel in the compressed moving image. Information. Therefore, higher compression efficiency can be obtained as compared with the case where a compressed moving image is generated at the expense of only temporal information or only spatial information as in the conventional method.

For example, in the moving image processing method according to an aspect of the present disclosure, further, in the compressed moving image generated in the compressing step, all pixels forming the image sensor are exposed in all frames. It may include a reconstruction step of generating an output moving image by reconstructing the unknown moving image obtained in 1. as a target.

As a result, an output image close to an unknown moving image obtained by shooting with an exposure pattern that is not thinned temporally and spatially is reconstructed from the compressed moving image.

For example, in the moving image processing method according to an aspect of the present disclosure, prior to the reconstructing step, machine learning of artificial intelligence for inputting the compressed moving image and outputting the output moving image is performed. In the reconstructing step, the output moving image may be generated using the artificial intelligence machine-learned in the second machine learning step.

Thereby, by using machine learning, an output image reconstructed with high quality from the compressed moving image can be obtained.

For example, in the moving image processing method according to an aspect of the present disclosure, the artificial intelligence is a neural network, and the compressed moving image is generated from the unknown moving image by calculation using a weighting factor corresponding to the exposure pattern. The first machine learning step and the second machine learning step include a sensing layer and a reconstruction layer that reconstructs the compressed moving image generated by the sensing layer to generate the output moving image. , Supervised learning for the artificial intelligence including the sensing layer and the reconstruction layer.

Accordingly, the process of compressing the unknown moving image and the process of reconstructing the unknown moving image from the compressed moving image can be performed using one artificial intelligence. Furthermore, the artificial intelligence performs optimization of an exposure pattern for compression of an unknown moving image and optimization of a reconstruction algorithm for reconstructing an unknown moving image from a compressed moving image by supervised learning. , Can be learned efficiently based on the input and the correct answer data.

For example, the moving image processing method according to an aspect of the present disclosure may be further obtained when all the pixels configuring the image sensor are exposed in all the frames from the compressed moving image generated in the compression step. The motion detection step of specifying the type of motion indicated by the unknown moving image and generating the motion information indicating the specified type of motion may be included.

This makes it possible to directly generate motion information indicating the type of motion indicated by a moving image without reconstructing the moving image from the temporal and spatial information of the compressed moving image. Therefore, the amount of data is reduced as compared with the related art, and thus it is possible to quickly and accurately identify the type of motion indicated by the moving image.

For example, in the moving image processing method according to an aspect of the present disclosure, prior to the motion detection step, machine learning of artificial intelligence for inputting the compressed moving image and outputting the motion information is performed. A machine learning step may be included, and in the motion detecting step, the motion information may be generated using the artificial intelligence machine-learned in the third machine learning step.

As a result, by using machine learning, motion is detected with high quality from the compressed moving image.

For example, in the moving image processing method according to an aspect of the present disclosure, the artificial intelligence is a neural network, and the compressed moving image is generated from the unknown moving image by calculation using a weighting factor corresponding to the exposure pattern. A sensing layer and a motion detection layer that generates the motion information from the compressed moving image generated by the sensing layer, wherein the first machine learning step and the third machine learning step include the sensing layer and the motion. It may be performed by supervised learning for the artificial intelligence including a detection layer.

Thereby, the process of compressing the unknown moving image and the process of generating the motion information indicating the type of motion of the unknown moving image from the compressed moving image can be performed using one artificial intelligence. Furthermore, the artificial intelligence performs optimization of an exposure pattern for compression of an unknown moving image and optimization of a motion information generation algorithm for generating motion information indicating a type of unknown moving image motion from a compressed moving image. Since learning is performed by supervised learning, learning can be efficiently performed based on input and correct answer data.

For example, in the moving image processing method according to an aspect of the present disclosure, the image sensor includes a color filter that selectively passes light of a specific color corresponding to each of the pixels, and in the compression step, the The compressed moving image is generated by performing imaging by repeated exposure in which the pattern of the color filter is temporally and spatially changed, and in the first machine learning step, the color filter is further preceded by the compression step. The color filter pattern that specifies the temporal and spatial variation of the pattern is optimized by machine learning, and the compression step uses the color filter pattern obtained by the optimization by the first machine learning step. The compressed moving image may be generated.

As a result, it is possible to select and apply the optimum color filter pattern to each frame that makes up an unknown color moving image, while leaving sufficient information for reconstruction of the moving image while compressing the data of the compressed moving image. The amount can be reduced. Therefore, the compression performance of an unknown color moving image is improved. Since not only the exposure pattern but also the color filter pattern is optimized by machine learning, it is possible to determine an appropriate exposure pattern and color filter pattern according to the type of image sensor compatible with color imaging.

Further, the moving image processing apparatus according to an aspect of the present disclosure captures a compressed moving image by performing imaging by repeated exposure that is thinned temporally and spatially using an image sensor in which pixels are two-dimensionally arranged. A moving image processing apparatus used for a camera to generate, which is obtained by a first machine learning unit for optimizing an exposure pattern for specifying an aspect of the exposure by machine learning, and an optimization by the first machine learning unit. And an output unit that outputs the exposed exposure pattern.

As a result, since the exposure pattern is optimized by machine learning, an appropriate exposure pattern can be determined according to the type of image sensor.

Further, these comprehensive or specific aspects may be realized by a system, a device, a method, an integrated circuit, a computer program, or a non-transitory recording medium such as a computer-readable CD-ROM, It may be realized by any combination of a system, an apparatus, a method, an integrated circuit, a computer program, and a recording medium.

Hereinafter, embodiments will be specifically described with reference to the drawings. It should be noted that each of the embodiments described below shows a comprehensive or specific example. Numerical values, shapes, materials, constituent elements, arrangement positions and connection forms of constituent elements, steps, order of steps, and the like shown in the following embodiments are examples, and are not intended to limit the scope of the claims. Further, among the constituent elements in the following embodiments, constituent elements not described in the independent claim showing the highest concept are described as arbitrary constituent elements.

Further, in the following description, the ordinal numbers such as the first, second, and third cases may be attached to the elements. These ordinal numbers are attached to the elements to identify the elements and do not necessarily correspond to a meaningful order. These ordinal numbers may be replaced appropriately, may be newly added, or may be removed.

(Embodiment)
First, the moving image processing system according to the present embodiment will be described with reference to FIG. FIG. 10 is a block diagram showing an example of the functional configuration of the moving image processing system 300 according to the embodiment.

As shown in FIG. 10, the moving image processing system 300 includes a moving image processing device 100 and a camera 200. The camera 200 includes an image sensor in which pixels are two-dimensionally arranged, and uses an exposure pattern output from the moving image processing apparatus 100 to perform imaging by repeated exposure that is thinned temporally and spatially. Generate a compressed moving image. The camera 200 includes an exposure pattern holding unit 90 that acquires and holds the exposure pattern optimized by the moving image processing apparatus 100, and a plurality of exposure patterns held by the exposure pattern holding unit 90 according to the type of the image sensor. A compressed moving image generation unit 80 that generates a compressed moving image by selecting an appropriate exposure pattern and applying it to an image sensor.

The moving image processing apparatus 100 includes a communication unit 10, a control unit 20, a display unit 60, and an input unit 70. The control unit 20 includes a machine learning unit 30, a reconstruction unit 40, and a motion information generation unit 50.

The machine learning unit 30 causes artificial intelligence (not shown) such as a neural network to perform learning. The machine learning unit 30 may be divided into a plurality of functional units such as a first functional unit, a second functional unit, and a third functional unit, depending on the learning content to be learned by the artificial intelligence. For example, a first machine learning unit (not shown) trains artificial intelligence for optimizing an exposure pattern that specifies an exposure mode. The second machine learning unit (not shown) receives the compressed moving image as an input and causes the artificial intelligence for outputting the output moving image to learn. The third machine learning unit (not shown) receives the compressed moving image as an input and causes the artificial intelligence for outputting the motion information to learn. The machine learning unit 30 causes the artificial intelligence to learn using, for example, teacher data. The exposure pattern is information designating pixels to be used for exposure among the pixels forming the image sensor for each frame forming the compressed moving image. Further, optimizing the exposure pattern means selecting, from a plurality of exposure patterns, an exposure pattern that satisfies the mounting restrictions on the hardware and that is optimum for each frame forming a moving image.

The reconstructing unit 40 reconstructs the compressed moving image generated by the camera 200 by targeting an unknown moving image obtained when all pixels forming the image sensor are exposed in all frames, Generate an output moving image.

The re-motion information generation unit 50 specifies, from the compressed moving image generated by the camera 200, the type of movement indicated by the unknown moving image obtained when all the pixels forming the image sensor are exposed in all frames, Motion information indicating the identified motion type is generated.

The communication unit 10 includes an output unit (not shown) that outputs the exposure pattern obtained by the optimization by the first machine learning unit (not shown) to the camera 200, and an acquisition unit that acquires the compressed moving image generated by the camera 200. (Not shown). The communication unit 10 may be a wireless communication such as Wi-Fi (registered trademark) or a communication using wired communication such as Ethernet (registered trademark). The communication may be electric power wireless communication or visible light communication.

The display unit 60 is, for example, a display, and displays the moving image reconstructed by the reconstructing unit 40, for example, based on the user's instruction input to the input unit 70. The input unit 70 is, for example, a keyboard, a mouse, a touch panel, a microphone, or the like, and receives an input of a user's instruction. The moving image processing apparatus 100 may not include the display unit 60 and the input unit 70. The display unit 60 and the input unit 70 may be included in a device other than the moving image processing device 100, for example. The moving image processing apparatus 100 may be mounted on the camera 200, a computer, or a server connected via a communication network such as the Internet.

Next, a moving image processing method according to the embodiment will be described. FIG. 11 is a flowchart showing an example of the moving image processing method according to the embodiment.

As shown in FIG. 11, the camera 200 generates a compressed moving image (compression step S10). More specifically, the camera 200 uses a image sensor in which pixels are arranged two-dimensionally to perform photographing by repeated exposure with temporal and spatial thinning to generate a compressed moving image. In the compression step, the compressed moving image is generated using the exposure pattern obtained by the optimization in the first machine learning step described later. Here, imaging by repeated exposure that is thinned out temporally and spatially means that for each of a plurality of frames that form a moving image, from among a plurality of exposure patterns that specify the exposure mode for each pixel of the image sensor. The optimum exposure pattern is selected and applied to each frame.

Next, the moving image processing apparatus 100 reconstructs the compressed moving image generated by the camera 200 into a moving image (reconstructing step S20). More specifically, the moving image processing apparatus 100 obtains an unknown value obtained when all the pixels forming the image sensor are exposed in all the frames of the compressed moving image generated by the camera 200 in the compression step S10. An output moving image is generated by reconstructing the moving image as a target. In the reconstruction step S20, the output moving image is generated using the artificial intelligence machine-learned in the second machine learning step described later.

Note that, prior to each of these two steps, the machine learning unit 30 may cause the artificial intelligence used in each step to learn. Hereinafter, each of the learning steps that the machine learning unit 30 causes the artificial intelligence to learn and the artificial intelligence will be described.

FIG. 12 is a diagram showing an example of an artificial intelligence learning step used in the compression and reconstruction steps. As shown in FIG. 12, in the machine learning step, prior to the compression step S10, prior to the first machine learning step S1 in which artificial intelligence for optimizing the exposure pattern is learned and the reconstruction step S20, A second machine learning step S2 of inputting the compressed moving image and learning the artificial intelligence for outputting the output moving image to optimize the reconstruction algorithm. Note that these steps may be performed simultaneously or individually. Also, these steps may be performed in any order. Also, both of these steps may be performed, or only one of them may be performed. That is, it may be implemented as needed.

Next, an example of artificial intelligence used for compression and reconstruction of moving images will be described with reference to FIG. FIG. 13 is a diagram showing an example of artificial intelligence used for compression and reconstruction of moving images in the embodiment.

Artificial intelligence consists of neural networks (NN). The neural network is, for example, Deep Neural Network (DNN). The artificial intelligence includes a sensing layer (hereinafter, also referred to as a compressed sensing layer) that generates a compressed moving image from an unknown moving image by a calculation using a weighting coefficient corresponding to an exposure pattern, and a compressed moving image generated by the sensing layer. And a reconstruction layer that generates an output moving image by reconstructing.

As shown in FIG. 13, the sensing layer selects an optimum exposure pattern for each frame constituting a moving image (Wp×Hp×T) captured by the camera 200 from among a plurality of binarized exposure patterns. A compressed moving image, that is, a coded moving image (Wp×Hp) is generated by selecting and applying it to each frame.

Here, the plurality of binarized exposure patterns are, for example, as shown in FIG. 2A, a plurality of exposure patterns that can be mounted on a sensor capable of completely random exposure in all pixels. , (B) and (c) of FIG. 2, a plurality of exposure patterns prepared in consideration of restrictions on mounting on hardware. It should be noted that the completely random exposure of all pixels means that randomly selected pixels of all pixels are exposed for each frame forming a moving image. For example, with respect to any hardware that can be considered for mounting, all kinds of exposure patterns satisfying the restrictions on mounting on these hardware are prepared in advance, and the plurality of exposure patterns are stored in a memory (not shown). The artificial intelligence selects the optimum exposure pattern for each frame of the moving image captured by the camera 200 from a plurality of exposure patterns stored in a memory (not shown), and outputs it from the moving image processing apparatus 100 to the camera 200. By doing so, optimal encoding of the moving image, that is, compression is performed.

An exposure pattern that has restrictions in terms of mounting on hardware can be easily derived from the hardware structure. For example, when the hardware is an SBE sensor (see FIG. 3), it is desirable that the exposure time be the same for all pixels, considering the dynamic range of the SBE sensor. Therefore, what can be controlled by the SBE sensor in order to enhance the compression performance is to control the exposure start timing. Therefore, in the SBE sensor, all possible exposure patterns are derived by determining all possible exposure start timings (start time (second) t=0, 1, 2,..., Td). (See (b) of FIG. 2). Here, d is the exposure time.

Further, for example, when the hardware is an RCE sensor (see FIG. 5), the RCE sensor first generates a set of all Reset signals (8 bits) and Transfer signals (8 bits). Next, exposure patterns of all types are derived by simulating the exposure patterns generated from all the generated signal sets (see (c) of FIG. 2 ).

As shown in FIG. 13, the reconstruction layer inputs the compressed moving image created in the sensing layer into the input layer, and outputs the output moving image from the output layer. More specifically, the reconstruction layer is composed of a plurality of frames from a single image (compressed moving image) compressed using an exposure pattern that is optimal for each frame that constitutes the moving image in the compressed sensing layer. Reconstruct a moving image. The reconstruction layer learns a non-linear mapping from a single input image to a moving image composed of a plurality of frames by using DNN. As shown in FIG. 13, this DNN has four hidden layers and uses ReLU (Rectified Linear Unit) as a transfer coefficient. The DNN learns to reduce the error between the training moving image and the reconstructed moving image. The peak signal-to-noise ratio (PSNR) is used to evaluate the reconstructed video. Therefore, the loss function uses a mean square error (MSE) that is closely related to PSNR.

As described above, the artificial intelligence (here, DNN) that compresses and reconstructs a moving image includes a sensing layer and a reconstruction layer, and includes a first machine learning step and a second machine learning step that are machine learning for the artificial intelligence. In the machine learning step, supervised learning using the training moving image is performed. Thereby, the artificial intelligence in the present embodiment can simultaneously learn the optimization of the exposure pattern for compressed sensing and the optimization of the reconstruction algorithm of the decoder.

Next, the procedure of machine learning of artificial intelligence (here, DNN) used for compressing and reconstructing a moving image will be described more specifically. FIG. 14 is a diagram showing an example of the configuration of the machine learning step in the embodiment.

As described above, the DNN includes a sensing layer that optimizes the exposure pattern while satisfying the restrictions on mounting on hardware, and a reconstruction layer that reconstructs a moving image from an observed image that is a compressed moving image. It consists of two layers. As shown in FIG. 14, DNN training (that is, machine learning) is performed, for example, in the following procedure. Here, an example of machine learning for simultaneously performing the first learning step and the second learning step will be described.
(1) During Forwarding when processing from the sensing layer to the reconstruction layer, a binary exposure pattern that is a binarized weight is used in the sensing layer, and a continuous value weight is used in the reconstruction layer.
(2) Gradient is obtained by error back propagation.
(3) The continuous value weight of the entire network is updated using the obtained gradient.
(4) The updated continuous value weights are binarized in consideration of the restrictions on mounting on hardware. As a result, the binarization weight used in the sensing layer is updated.

Since binary exposure patterns are used in actual compressed sensing, binarization weights are used during Forward in neural network training, but they are relaxed to continuous values because they are differentiable during Backward (Non-Patent Document 6). .. FIG. 15 is a diagram showing an example of updating a binarized exposure pattern. The weight to be used at the next Forward is selected from a plurality of binarized exposure patterns generated in advance, and the one closest to the continuous value weight derived at the Backward is selected using the inner product, and the binarized exposure pattern is selected. To update.

[Experimental example]
[Experimental Example 1] Machine learning of DNN Machine learning of DNN was performed by the following procedure. The size of the network (DNN) was determined based on the size of the patch to be reconstructed. In this experimental example, the prototype sensor described in Non-Patent Document 1 that can control exposure was used. Therefore, the size of the patch is set to Wp=Hp=8 and T=16 (Wp×Hp×T in FIG. 13). The hidden layer of the reconstructed layer was 4 layers. As the training data (training moving image), the same data was used in all methods in the following experimental examples (Non-Patent Document 7). This training data is a data set for benchmarking video summarization, and 16 frames are randomly extracted from each of 20 moving images in 4 scenes, and rotated (90°, 180°, 270°) and inverted respectively. What was done was used. Using the 829 and 440 patches prepared in this manner, machine learning of the network (DNN) for simultaneously optimizing the exposure pattern and the decoder for reconstruction was performed end-to-end. The machine learning was performed at a minipatch size of 200 and 250 epochs.

[Experimental Example 2] Simulation experiment A simulation experiment of compression and reconstruction of a moving image was performed assuming that the SBE sensor and the RCE sensor are mounting targets. The moving images used in the experiment were 14 moving images composed of 16 frames with a spatial resolution of 256×256. The reconstruction quality of the reconstructed moving image was evaluated by the peak signal-to-noise ratio (PSNR). FIG. 16 is a diagram showing the results of Experimental Example 2.

In a simulation experiment in which the SBE sensor is mounted, an exposure pattern randomly selected from a plurality of exposure patterns that can be mounted on the SBE sensor illustrated in FIG. Each was used to simulate a compressed video image taken. Next, the compressed moving image obtained by simulation was input to the reconstruction network, and the moving image composed of 16 frames was reconstructed (Handcraft SBE in FIG. 16). At this time, the machine learning of the DNN was performed only for the decoder, that is, only the learning for optimizing the reconstruction algorithm in the reconstruction layer.

On the other hand, as disclosed in the above embodiment, the machine learning of the DNN is performed by the first learning step and the second learning step, and the optimum exposure pattern for each frame forming the moving image is selected and compressed. The moving image was simulated and the moving image was reconstructed (Optimized SBE in FIG. 16).

The simulation experiment with the RCE sensor as the mounting target was performed in the same manner as the simulation experiment with the SBE sensor as the mounting target, except that a plurality of exposure patterns mountable on the RCE sensor illustrated in FIG. 2C were used. (Handcraft RCE and Optimized RCE in FIG. 16).

FIG. 16 shows only the evaluation results of 3 cases out of the results using 14 moving images. The leftmost column in FIG. 16 shows one scene of the moving image used in the test. The upper part of FIG. 16 is a moving image of a mail delivery car, and a letter mark is written on the side of the mail delivery car. The middle part of FIG. 16 is a moving image in which a plurality of vehicles are moving. The lower part of FIG. 16 is a moving image in which the performer is accompanied by a cello.

As shown in FIG. 16, in the simulation experiment in which the SBE sensor and the RCE sensor were mounted, the following evaluation results were obtained for each sensor.

Comparing the moving images in the upper part (Car) of FIG. 16, the moving images (Optimized SBE and Optimized RCE) obtained by compressing and reconstructing the moving image using the DNN disclosed in the present embodiment are randomly exposed. The letter marks were reproduced more clearly than when the patterns were selected (Handcraft SBE and Handcraft RCE). Even when comparing the PSNR values of Handcrafted and Optimized, it was confirmed that the reconstructed moving image has less noise and high quality because Optimized is higher than Handcrafted.

Further, also in the middle (Traffic) moving image and the lower (Cello) moving image in FIG. 16, a moving image (Optimized SBE In the Optimized RCE), the outline of the subject was clearer, the difference in pixel value between the patches was small, and the change in pixel value at the patch boundary was a smooth and continuous value.

Although not shown, the reconstruction quality of each of Handcraft and Optimized was evaluated by the average value of PSNR in 14 reconstructed moving images. The average value of SPNR in the reconstructed moving image of 14 moving images is 28.37 dB for Handcrafted SBE, 29.32 dB for Optimized SBE, 27.58 dB for Handcrafted RCE, and 27.82 dB for Optimized RCE. Met. Also from this result, the reconstructed moving image obtained by using the optimal exposure pattern when compressing the moving image using the DNN disclosed in the present embodiment and optimizing the reconstruction algorithm has better reconstruction quality. Was confirmed to be good.

From the above results, the quality of the moving image obtained by compressing and reconstructing the moving image (hereinafter, reconstruction quality) is the same in all scenes regardless of which of the SBE sensor and the RCE sensor is mounted. The reconstruction quality was better when the compressed exposure image was generated by applying the optimized exposure pattern. Therefore, the DNN disclosed in the present embodiment can find an exposure pattern that is more suitable for each frame of a moving image even under the restrictions on hardware implementation, and at the same time, reconstructs in a reconstruction layer. Since the algorithm can be optimized, it was confirmed that not only compression performance but also reconstruction performance was improved.

Experimental Example 3 Actual Experiment Subsequently, the DNN disclosed in the present embodiment was applied to an actual machine to compress and reconstruct a moving image. The sensor used is a sensor that can actually control the exposure for each pixel (Patent Document 2: JP-A-2015-216594). The moving image is captured at 15 frames per second (fps: Frames per second), and 16 subframes are reconstructed, so the reconstructed moving image corresponds to 240 fps. A moving image was shot using the optimized exposure pattern in the compressed sensing layer, and the moving image was reconstructed from the compressed moving image using the reconstruction layer with the optimized reconstruction algorithm. Since the sensor is a rolling shutter system, the order in which the exposure pattern is applied differs depending on the position of the pixel on the sensor (here, in each row). The sensor size of the sensor is 672×512 pixels. Since the moving image captured by the sensor is divided into 16 subframes, the order of the exposure patterns applied to each frame is different every 512/16=32 rows. Therefore, it is necessary to use different decoders for each row. FIG. 17 is a diagram showing the results of Experimental Example 3. The image at the left end of FIG. 17 is a moving image that was actually taken, the second image from the left is a compressed moving image, and the two images on the right side are 16 subframes that make up the reconstructed moving image. It is an image of three sub-frames. I took a picture of the metronome weight swaying left and right with a camera equipped with the sensor. As shown in FIG. 17, it was confirmed from the images of three subframes of the 16 subframes forming the reconstructed moving image that the metronome weight swayed from side to side.

(Summary)
In the present embodiment, in consideration of the restrictions on the mounting of the exposure pattern in the compressed video sensing on the hardware, the selection of the exposure pattern suitable for each frame forming the moving image using DNN as an example of artificial intelligence, Disclosed is a moving image processing method for simultaneously optimizing a reconstruction algorithm of a decoder for reconstructing a moving image from a compressed moving image. In the above experimental example, the moving image is compressed and reconstructed by using the moving image processing method of the present disclosure with respect to the restrictions on mounting the two sensors of the SBE sensor and the RCE sensor. From the results of the above experimental example, by using the moving image processing method of the present disclosure, it is possible to select an optimal exposure pattern for each frame of a moving image to be shot, regardless of restrictions of hardware to be mounted, Furthermore, it was confirmed that the quality of the unknown moving image reconstructed from the compressed moving image is further improved.

(Modification 1)
In the embodiment, the method of compressing and reconstructing a monochrome moving image has been described, but the moving image processing method according to the first modification can be applied to a color moving image. Hereinafter, the points different from the embodiment will be mainly described.

The moving image processing system according to Modification 1 differs from the embodiment in that the camera includes an image sensor including a color filter that selectively passes light of a specific color corresponding to each pixel.

Further, in the moving image processing method according to the first modification, in the compression step, a compressed moving image is generated by performing shooting by exposure in which the pattern of the color filter is temporally and spatially changed, and the first machine learning is performed. In the step, prior to the compression step, the color filter pattern that specifies the temporal and spatial variation of the color filter pattern is optimized by machine learning, and in the compression step, the optimization by the first machine learning step is performed. A compressed moving image is generated using the color filter pattern obtained by the conversion. That is, the moving image processing method according to Modification 1 includes a step of optimizing the color filter pattern in addition to the optimization of the exposure pattern in the first learning step, and a moving image is configured in the compression step. The present embodiment differs from the embodiment in that the exposure pattern is selected according to the optimum color filter pattern applied to each frame.

FIG. 18 is a diagram illustrating an example of a flow of compressed sensing of a color moving image. As shown in FIG. 18, in the moving image processing method according to the modified example 1, except that a color filter pattern suitable for each frame forming a moving image is selected from the plurality of optimized color filter patterns. The flow is the same as the flow of the moving image processing method according to the embodiment.

FIG. 19 is a diagram showing an example of a color filter pattern. 19A shows an example of a Bayer pattern color filter, and FIG. 19B shows an example of a color filter optimized by the moving image processing method in the first modification.

FIG. 20 is a diagram showing an example of an exposure pattern applied according to the difference in color filters. The diagram on the left end of FIG. 20 is an example of an exposure pattern used when capturing a monochrome moving image. The center diagram of FIG. 20 is an example of an exposure pattern used when a color moving image is captured by a Bayer pattern color filter. The diagram on the right end of FIG. 20 is an example of an exposure pattern used when a color moving image is captured with an optimized color filter pattern. In the first modification, the performance of compression and reconstruction of the color moving image is improved by selecting a color filter pattern suitable for each frame forming the moving image to be captured.

[Experimental Example 4]
In this experimental example, a simulation experiment of compression and reconstruction of a color moving image was performed assuming that the RCE sensor is a mounting target. The moving images used in the experiment were 25 moving images composed of 16 frames with a spatial resolution of 256×256. The reconstruction quality of the reconstructed moving image was evaluated by the peak signal-to-noise ratio (PSNR). 21: is a figure which shows an example of the result of Experimental example 4. FIG. The leftmost diagram (Original Video) in FIG. 21 is an example of the color moving image used in the experiment. Here, the contents and results of the simulation experiment will be described by taking the result of subjecting the Original Video shown in the left end of FIG. 21 to the simulation experiment as an example.

First, in a simulation experiment, an exposure randomly selected from a plurality of exposure patterns that can be mounted on the color filter of the Bayer pattern and the RCE sensor illustrated in FIG. A compressed moving image captured using each of the pattern and was simulated. Next, the compressed moving image obtained by simulation was input to the reconstruction network to reconstruct a moving image composed of 16 frames (second diagram from the left in FIG. 21). At this time, the machine learning of the DNN was performed only for the decoder, that is, only the learning for optimizing the reconstruction algorithm in the reconstruction layer. In this way, the reconstructed moving image obtained by using the DNN optimized only for the decoder is referred to as “decoder only”.

Next, a Bayer pattern color filter is selected for each frame that composes the color moving image, and an optimal exposure pattern is selected for each frame that composes the color moving image. It was configured (third drawing from the left in FIG. 21). At this time, in the machine learning of the DNN, learning for optimizing the exposure pattern and the reconstruction algorithm in the reconstruction layer was performed by the first machine learning step and the second machine learning step. The reconstructed moving image obtained by using the DNN in which the exposure pattern and the decoder are optimized as described above is referred to as “exposure pattern+decoder”.

Then, a color filter pattern most suitable for each frame forming the color moving image and an exposure pattern most suitable for each frame forming the color moving image are selected to simulate a compressed moving image captured, Was reconstructed (the rightmost diagram in FIG. 21). At this time, in the machine learning of the DNN, the learning for optimizing the color filter pattern and the exposure pattern is performed in the first machine learning step, and the reconstruction algorithm in the reconstruction layer is optimized in the second machine learning step. I learned to do it. The reconstructed moving image obtained by using the DNN in which the color filter pattern, the exposure pattern, and the decoder are optimized as described above is referred to as "color filter+exposure pattern+decoder".

As shown in FIG. 21, when the reconstructed moving images of the decoder only, the exposure pattern+decoder, and the color filter+exposure pattern+decoder are compared, the PSNR value of the reconstructed moving image of only the decoder is 24.18 dB. Yes, the PSNR value of the reconstructed moving image of the exposure pattern+decoder was 23.92 dB, and the PSNR value of the reconstructed moving image of the color filter+exposure pattern+decoder was 23.34 dB. Therefore, among these reconstructed moving images, it was confirmed that the reconstructed moving image of the color filter+exposure pattern+decoder has the least noise and high reconstruction quality.

Further, among these reconstructed moving images, the reconstructed moving image obtained by using the DNN that has undergone learning for optimizing all of the color filter+exposure pattern+decoder as disclosed in Modification 1 is The color and outline of the subject were clear, and the change in pixel value at the boundary of the patch was a smoother and continuous value.

Although not shown, when only the decoder is optimized (hereinafter, only the decoder), when the exposure pattern and the decoder are optimized (hereinafter, the exposure pattern), the color filter pattern, the exposure pattern and the decoder are optimized. In each case (hereinafter, color filter+exposure pattern), the reconstruction quality was evaluated by the average value of PSNR in 25 reconstructed moving images. The average value of SPNR in the reconstructed moving image of the 25 moving images was 26.56 dB only in the decoder, 26.43 dB in the exposure pattern, and 26.76 dB in the color filter+exposure pattern. Also from this result, the reconstructed moving image obtained by using the optimal color filter pattern and exposure pattern when compressing the color moving image by the moving image processing method according to the modified example 1 and optimizing the reconstruction algorithm is: It was confirmed that the reconstruction quality was good.

(Modification 2)
In the embodiment and the modified example 1, the method of compressing and reconstructing the moving image has been described, but in the modified example 2, the method of detecting the movement of the subject from the compressed moving image will be described. Hereinafter, differences from the embodiment and the first modification will be mainly described. FIG. 22 is a flowchart showing an example of the moving image processing method according to Modification 2. FIG. 23 is a diagram showing an example of the configuration of the machine learning step in Modification 2.

As shown in FIG. 22, in the moving image processing method according to the second modification, the camera 200 (see FIG. 10) generates a compressed moving image (compression step S10). Next, the motion information generation unit 50 (see FIG. 10) generates an unknown moving image obtained by exposing all the pixels forming the image sensor in all the frames from the compressed moving image generated in the compression step S10. The type of motion shown is specified, and motion detection is performed to generate motion information indicating the specified type of motion (detection step S30).

As shown in FIG. 23, further, in the moving image processing method according to the second modification, prior to the motion detection step S30, a compressed moving image is input and machine learning of artificial intelligence for outputting motion information is performed. In the motion detection step S30, which includes three machine learning steps S3, motion information is generated using the artificial intelligence machine-learned in the third machine learning step S3.

Further, although not shown, artificial intelligence is a neural network, and a sensing layer that generates a compressed moving image from an unknown moving image by an operation using a weighting factor corresponding to an exposure pattern, and a compressed moving image generated by the sensing layer. The first machine learning step S1 and the third machine learning step S3 include a motion detection layer that generates motion information from an image, and are performed by supervised learning for artificial intelligence including a sensing layer and a motion detection layer.

In the moving image processing method according to the modified example, the following effects are expected as in the moving image processing method according to the embodiment and the modified example 1.

Also in the moving image processing method disclosed in the second modification, as described above in the embodiment and the first modification, a coded exposure image (so-called compressed moving image) is formed using a sensor capable of randomly exposing each pixel of the image sensor. ) Is taken. It is possible to compress the data amount by the length of the coded exposure, that is, by the number of frames captured by applying the exposure pattern. For example, when a compressed moving image is generated by applying an optimum exposure pattern for all frames of a moving image composed of 16 frames, the data amount of the compressed moving image is 1 of the data amount of the original moving image. /16 times. Therefore, it is expected that the amount of communication and the power consumption for transmission will be reduced.

In a normal compression method, a moving image is captured by a camera and then the moving image is compressed. On the other hand, also in the moving image processing method disclosed in the second modification, as described above in the embodiment and the first modification, each pixel of the image sensor is randomly exposed and a coded exposure image is captured to obtain a moving image. This is very efficient because it is possible to compress and acquire sufficient information for the reconstruction of the. Therefore, compared to the conventional method, cost reduction such as power consumption for moving image compression processing is expected.

Further, also in the moving image processing method disclosed in the second modification, as described above, information sufficient for reconstructing a moving image can be compressed and acquired in a single frame, and therefore, compared to the conventional method, The amount can be significantly reduced. For example, as a conventional method, there is a method of recognizing a motion of a subject in a moving image in recent years. In the recognition method, the recognition accuracy is improved by characterizing the spatiotemporal information by three-dimensional convolution of the temporal and spatial information of the moving image captured by the camera. However, the spatiotemporal information of a moving image obtained by three-dimensional convolution has a large number of parameters and a large amount of data. Therefore, in order to recognize (identify) the movement of a subject in a moving image from these spatio-temporal information, the neural network becomes a large-scale network having more layers than a normal neural network, and the parameters of the network. The number also increases. In addition, a large-scale data set is required because the number of data required to sufficiently train the network increases. Therefore, in this network, computational resources such as a large-scale GPU cluster are required for learning, and the time required for learning becomes enormous. On the other hand, in the moving image processing method disclosed in the second modification, the coded exposure image generated in the same manner as in the embodiment and the first modification is input to the neural network for identifying the motion of the subject in the moving image. By doing so, it is possible to characterize the spatiotemporal information by the two-dimensional convolution without requiring the three-dimensional convolution as in the conventional method. Therefore, the number of parameters required for machine learning is smaller and the number of data is smaller than that by the conventional method of three-dimensional convolution, so that the efficiency of machine learning can be expected and the accuracy can be improved even with a small amount of learning data. To be done.

A two-dimensional convolutional neural network (CNN) shown in Table 1 is considered as a network architecture in the moving image processing method according to the second modification.

As shown in Table 1, the two-dimensional CNN is composed of, for example, two-dimensional convolution of eight layers of 3×3 stride 1, maximum pooling of five layers of 2×2, and two fully connected layers. .. For simplification of calculation, the parameter number P of a certain convolutional layer when the bias term is ignored, using the input channel number C _in , the output channel number C _out, and the kernel size K of the layer, the following equation (2) is obtained. It is represented by.

Therefore, when the 3D convolution kernel 3×3×3, which is the best in Non-Patent Document 19, is used, the method disclosed in this modification is a 2D convolution kernel 3×3, and the number of parameters of the convolution layer is Is about 1/3 of the method of Non-Patent Document 19.

The neural network of the method disclosed in this modification performs learning and evaluation as follows. It is assumed that K kinds of actions C={C ₁ , C ₂ ,..., C _K } are classified. A moving image of length N in a certain action aεC is I={I ₁ , I ₂ ,..., I _N }. When the length of the coded exposure pattern is L, the length of the video clip is L, and the video clip is represented by the following formula (3).

Apply the coded exposure pattern to the video clip,
And For I, train the network using {(X _i ,a)} pairs. The output Yi for each input Xi is averaged over the entire moving image, and the maximum value is evaluated as the action label in the moving image. That is, the probability p(C _j |X _i ) that the input Xi at a certain time belongs to the behavior Cj is represented by the following formula (5).

At this time, the action label a ^* estimated for I is represented by the following equation (6).

The recognition accuracy (Accuracy) S is calculated using the following equation (7), where M is the total number of moving images in the data set.

[Experimental Example 5] Evaluation experiment A simulation experiment was performed in which the behavior was directly recognized from the encoded exposure image.

[1] Data Set The KTH Action data set (Non-Patent Document 23) was used for the simulation experiment. FIG. 24 is a diagram showing one scene of each action class in the KTH Action data set. As shown in FIG. 24, the data set is classified into six types of action classes, “walking”, “jogging”, “running”, “boxing”, “hand waving”, and “hand clipping”. In each action class, the position of the camera used for shooting is fixed, and 25 subjects are shooting six types of actions in four scenarios. The moving image of each action class is an average of 4 seconds, and is a grayscale video (hereinafter, moving image) having an image resolution of 600 dpi. These moving images are captured at 25 fps and downsampled to a spatial resolution of 160×120. According to the division method of Non-Patent Document 23, 25 subjects were divided into 8 for neural network training, 8 for verification, and 9 for experiment.

[2] Comparison method Each moving image of the above data set is divided into 16-frame video clips selected so that there is no overlapping data between preceding and following frames at the time of learning, and randomly divided into 112 × 112 spatial resolution I cut it out. The method shown in (d) below is a method in which the behavior class is identified using a neural network (NN) that has been machine-learned using this video clip as an input. The methods shown in (a) to (c) below use the above-mentioned action class by using a neural network that has learned using compressed moving images obtained by performing different compression processes on this video clip as input. This is the method of identifying. In the following (a) to (d), the video clip compression processing is performed so that the video clip is compressed to 1/16 times the information amount of the video clip.

FIG. 25 is a diagram illustrating an example of the comparison method in Experimental Example 5. FIG. 26 is a diagram showing an example of exposure in a pixel of an image input to the NN. Hereinafter, the methods shown in (a) to (d) will be described more specifically with reference to FIGS. 25 and 26.

(A) Coded exposure image A video clip was compressed by the compression method disclosed in the present disclosure. More specifically, the optimum coded exposure pattern was applied to each frame constituting the video clip to generate a coded exposure image. This coded exposure image was used as the input of CNN. The coded exposure pattern was 8x8 in size and used a random pattern that exposed 1/16 of the pixels in each video clip. The exposure time of each pixel changes depending on the coded exposure pattern at a frame rate of 1/16 for a moving image. For example, as shown in (a) of FIG. 26, the coded exposure image is an image of a single frame, and exposure at a pixel of the single frame is performed several times in one frame, for example. There is. In the coded exposure pattern used in this experiment, the exposure time is equal to the exposure time for capturing one frame of a moving image.

(B) Averaged image As a simple method of compressing time information into one image, an averaged image obtained by averaging video clips in the time direction is used. This averaged image was used as the input of CNN. As shown in FIG. 26B, a pixel having an averaged image is exposed for one frame. Therefore, the averaged image is equal to one frame of a moving image having a frame rate of 1/16 and an exposure time of 16 times.

(C) One-frame image In order to compare with an image without time information, it was compared with one-frame image. One frame was selected from the 16 frames forming the video clip, and this was used as the input of CNN. As shown in (c) of FIG. 26, one frame image is equal to one frame of a moving image having the same exposure time at a frame rate of 1/16.

(D) Moving image As a method corresponding to a conventional three-dimensional convolutional network (C3D: Convolution 3D), a video clip was input and learning was performed in C3D (Non-Patent Document 19). C3D originally has 3 channels of RGB, but was changed to 1 channel of gray scale and learned without prior learning. The video clip has all pixels exposed in every frame. Therefore, as shown in FIG. 26D, a pixel in every frame is exposed during each frame.

[3] Experimental Results Table 2 shows the results of a simulation experiment in which all the behavior classes of the data set were identified using the comparison methods (a) to (d) above. The identification accuracy in Table 2 indicates the average of the identification accuracy of each behavior class of the data set.

As shown in Table 2, when (a) the coded exposure image is used as the input of CNN and machine learning is performed to identify the motion of the subject in the moving image, (d) the accuracy of identifying the moving image by the conventional method is very close. The identification accuracy was obtained. However, (b) the method of identifying a moving image that has been learned by using the averaged image as the input of CNN and (c) the method of identifying the moving image that is learned by using the one-frame image as the input of CNN are Since the spatiotemporal information of the moving image has to be discriminated from the spatial information or the temporal information, (a) the discrimination accuracy of the moving image is remarkably lowered as compared with the case where the coded exposure image is input to the CNN. ..

FIG. 27 is a diagram showing a confusion matrix of each comparison method of (a) to (d). From the confusion matrix in FIG. 27, the method (b) in which the averaged image is input to CNN is the same as the method (c) in which one frame image is input to CNN, and the method in which the coded exposure image is input to CNN is used. As compared with (a), the recognition accuracy of “hand saving” was lower. Furthermore, the methods (b) and (c) have significantly lower identification accuracy of “walking”, “jogging”, and “running” than the method (a). I found it difficult to distinguish.

On the other hand, the method (a) in which the coded exposure image is input to CNN is a method (d) corresponding to the conventional method in which a moving image (here, a video clip) is input in C3D in the identification of each action class. It showed a similar tendency to. Furthermore, the method (a) achieved a high accuracy in recognition of each of the above-described action classes, which is close to the recognition accuracy of the method (d).

[Experimental Example 6]
In Experimental Example 5, a 16-frame video clip was used as an image to be identified, but in Experimental Example 6, the length L (hereinafter, the number of frames) of the video clip was changed, and the number of frames forming a moving image was increased. In this case, we conducted a simulation experiment to confirm how the recognition accuracy of the action class changes. 28: is a figure which shows the result of Experimental example 6. FIG.

Regarding method (d), since C3D inputs a video clip of 16 frames, when using a video clip of less than 16 frames, the same frame is repeated 16/L times to adjust to a moving image of 16 frames, and the adjustment is performed. The video clip was input to C3D. When using more than 16 video clips, the number of C3D input frames is changed to L. Therefore, it should be noted that the one using C3D cannot make a fair comparison due to the improved expressiveness of the network and the lack of data sets.

Regarding the method (b) that uses an averaged image as input, there was a slight improvement in the identification accuracy when the number of frames in the video clip was 4 to 8, but the number of frames in the video clip was better than 8 frames. The recognition accuracy declined as the number increased. The same tendency as in the method (b) was observed in the method (c) in which one frame image was input. Therefore, when the moving image to be identified is averaged in the time direction like the averaged image of the method (b), the time information of the moving image is lost. It is thought that the necessary time information will not be obtained.

On the other hand, in the method (a) in which the coded exposure image is input, the recognition accuracy is improved up to the frame number L of the video clip being 16 frames. Since this shows the same tendency as the method (d) in which a moving image (video clip) is input, it is considered that the coded exposure image has sufficient time information. However, in the method (a) in which the coded exposure image is input, the recognition accuracy is deteriorated when the number of frames of the video clip is longer than 16 frames. It is considered that this is because the time information that must be characterized increases, and the coded exposure pattern used in the method (a) cannot express the time information.

(Summary)
In the second modification, compressed sensing is applied to the trade-off problem of action recognition in a video surveillance system, and a two-dimensional CNN is used from a single image (so-called compressed moving image) captured by a coded exposure camera. Thus, an example of the moving image processing method for directly recognizing the action of a person without generating a reconstructed moving image from a compressed moving image has been described. In order to evaluate the effectiveness of the moving image processing method according to Modified Example 2, a simulation experiment using the KTH Action data set was performed in Experimental Example 5. From the results of Experimental Example 5, the moving image processing method according to the modified example 2 uses the moving image as the input in the three-dimensional method even though the input data amount to the neural network is compressed to 1/16. A high identification accuracy approaching the identification accuracy when the behavior of a person is identified using CNN (for example, C3D) is achieved.

(Other embodiments)
Although the moving image processing method and the moving image processing device according to one or more aspects of the present disclosure have been described above based on the embodiment, the present disclosure is not limited to this embodiment. As long as it does not depart from the gist of the present disclosure, various modifications made by those skilled in the art may be applied to the embodiment, and configurations configured by combining components in different embodiments are also one or more aspects of the present disclosure. It may be included in the range of.

For example, the moving image processing system according to the above-described embodiment has been described with the case of having one camera, but it may have two or more cameras. With this, a plurality of captured moving images can be acquired, and thus abnormal behavior can be detected more quickly and accurately from the obtained plurality of moving images.

Further, for example, some or all of the components included in the moving image processing apparatus according to the above-described embodiments may be configured by one system LSI (Large Scale Integration). For example, the moving image processing apparatus may be composed of a system LSI having a communication unit and a control unit.

The system LSI is a super-multifunctional LSI manufactured by integrating a plurality of components on one chip, and specifically, a microprocessor, a ROM (Read Only Memory), a RAM (Random Access Memory), and the like. It is a computer system configured to include. A computer program is stored in the ROM. The system LSI achieves its function by the microprocessor operating according to the computer program.

The system LSI is used here, but it may also be called IC, LSI, super LSI, or ultra LSI depending on the degree of integration. Also, the method of circuit integration is not limited to LSI, and may be realized by a dedicated circuit or a general-purpose processor. A programmable programmable gate array (FPGA) that can be programmed after the LSI is manufactured, or a reconfigurable processor that can reconfigure connection and setting of circuit cells inside the LSI may be used.

Furthermore, if integrated circuit technology comes out to replace LSI's as a result of the advancement of semiconductor technology or a derivative other technology, it is naturally also possible to carry out function block integration using this technology. The application of biotechnology is possible.

In addition, one aspect of the present disclosure may be not only such a moving image processing apparatus but also a moving image processing method having a characteristic configuration unit included in the moving image processing apparatus as a step. Further, one aspect of the present disclosure may be a computer program that causes a computer to execute the characteristic steps included in the moving image processing method. Further, one aspect of the present disclosure may be a computer-readable non-transitory recording medium in which such a computer program is recorded.

In addition, in each of the above-described embodiments, each component may be configured by dedicated hardware, or may be realized by executing a software program suitable for each component. Each component may be realized by a program execution unit such as a CPU or a processor reading and executing a software program recorded in a recording medium such as a hard disk or a semiconductor memory. Here, the software that realizes the moving image processing apparatus and the like of the above-described embodiment is the following program.

That is, the program includes a compression step of generating a compressed moving image by performing imaging with repeated exposure in a computer by using an image sensor in which pixels are two-dimensionally arranged, and which is thinned out temporally and spatially. Prior to the compression step, a first machine learning step of optimizing an exposure pattern for specifying an exposure mode by machine learning is performed. In the compression step, the exposure pattern obtained by the optimization by the first machine learning step is A moving image processing method for generating a compressed moving image is executed.

The present disclosure can shoot a moving image by applying an appropriate exposure pattern depending on the type of hardware to each frame that constitutes the moving image, regardless of the constraints of the hardware to be mounted. It is possible to generate a compressed moving image with high compression quality while capturing the moving image. Therefore, the moving image processing device of the present disclosure can be widely used for, for example, an observation camera, a surveillance camera, and the like.

10 communication unit 20 control unit 30 machine learning unit 40 reconstruction unit 50 motion information generation unit 60 display unit 70 input unit 80 compressed moving image generation unit 90 exposure pattern holding unit 100 moving image processing device 200 camera 300 moving image processing system

Claims (10)

A compression step of generating a compressed moving image by performing imaging by repeated exposure with temporal and spatial thinning using an image sensor in which pixels are arranged two-dimensionally;
A first machine learning step of optimizing an exposure pattern for specifying the exposure mode by machine learning prior to the compression step;
Including
In the compression step, the compressed moving image is generated using the exposure pattern obtained by the optimization in the first machine learning step,
Video processing method. The exposure pattern is information designating pixels used for exposure among the pixels forming the image sensor for each frame forming the compressed moving image.
The moving image processing method according to claim 1. Furthermore, with respect to the compressed moving image generated in the compression step, by reconstructing an unknown moving image obtained when all the pixels forming the image sensor are exposed in all frames, Including a reconstruction step to generate an output moving image,
The moving image processing method according to claim 1. Further, prior to the reconstructing step, a second machine learning step of machine learning the artificial intelligence for outputting the output moving image using the compressed moving image as an input is included.
In the reconstruction step, the output moving image is generated by using the artificial intelligence machine-learned in the second machine learning step.
The moving image processing method according to claim 3. The artificial intelligence is a neural network, and a sensing layer that generates the compressed moving image from the unknown moving image by a calculation using a weighting factor corresponding to the exposure pattern, and the compressed moving image generated by the sensing layer. A reconstruction layer for generating the output moving image by reconstructing,
The first machine learning step and the second machine learning step are performed by supervised learning for the artificial intelligence including the sensing layer and the reconstruction layer.
The moving image processing method according to claim 4. Further, from the compressed moving image generated in the compression step, the type of motion indicated by the unknown moving image obtained when all the pixels forming the image sensor are exposed in all the frames is specified and specified. A motion detection step of generating motion information indicating the type of motion,
The moving image processing method according to claim 1. Further, prior to the motion detection step, a third machine learning step of machine learning the artificial intelligence for inputting the compressed moving image and outputting the motion information is included,
In the motion detecting step, the motion information is generated using the artificial intelligence machine-learned in the third machine learning step.
The moving image processing method according to claim 6. The artificial intelligence is a neural network, and a sensing layer that generates the compressed moving image from the unknown moving image by a calculation using a weighting factor corresponding to the exposure pattern, and the compressed moving image generated by the sensing layer. A motion detection layer for generating the motion information,
The first machine learning step and the third machine learning step are performed by supervised learning for the artificial intelligence including the sensing layer and the motion detection layer,
The moving image processing method according to claim 7. The image sensor includes a color filter that selectively passes light of a specific color corresponding to each of the pixels,
In the compression step, the compressed moving image is generated by performing photography by exposure in which the pattern of the color filter is temporally and spatially changed,
In the first machine learning step, prior to the compressing step, a color filter pattern that specifies a temporal and spatial variation of the color filter pattern is optimized by machine learning.
In the compression step, the compressed moving image is generated using the color filter pattern obtained by the optimization in the first machine learning step.
The moving image processing method according to claim 1. A moving image processing apparatus used in a camera that generates a compressed moving image by performing imaging by repeated exposure with temporal and spatial thinning using an image sensor in which pixels are arranged two-dimensionally,
A first machine learning unit that optimizes an exposure pattern that specifies the mode of exposure by machine learning;
An output unit for outputting the exposure pattern obtained by the optimization by the first machine learning unit;
With
Video processing device.