JP2022112364A - Imaging device and measurement coding device - Google Patents

Abstract

To enable efficiently compressing an image while suppressing deterioration of image quality in use of compressed sensing.SOLUTION: A compressed sensing unit uses a CoWH (Continuously ordered Walsh-Hadamard) matrix based on a SoWH (Sequence ordered Walsh-Hadamard) matrix in which respective rows of a NoH (Naturally ordered Hadamard) matrix are arranged in sequence as a measurement matrix to perform compressed sensing processing. The respective rows of the CoWH matrix have mutually different projection patterns when projected onto an image block. The CoWH matrix is configured such that the number of consecutive elements of the same value in the projection pattern decreases from the upper row to the lower row in the CoWH matrix.SELECTED DRAWING: Figure 6

Description

The present invention relates to imaging devices and measurement coding devices.

Although the Nyquist-Shannon sampling theorem states that the sampling rate (SR) must be at least twice the frequency of the signal, conventional imaging techniques face implementation limits. Recently developed CMOS image sensors (CIS) have resolutions of 4K, 8K, and 16K, and are expensive and consume a lot of power. Furthermore, it requires a huge storage space to store a large amount of image data.

Multimedia compression tools such as JPEG/JPEG2000, AVC (Advanced Video Coding), HEVC (High-Efficiency Video Coding), and VVC (Versatile Video Coding) are widely used to reduce image data storage and transmission costs. ing. Software and hardware implementations can reach up to ten times complexity per generation, which is a result of CIS design impediments.

All data compression algorithms reduce spatial and temporal redundancy. The original signal is obtained by efficiently compressing the image data from the viewpoint of a sparse vector of coefficients using a general transform basis such as DCT (Discrete Cosine Transform), Fourier transform, DWT (Discrete Wavelet Transform), This is irreversible.

When performing lossy compression to meet low bitrates, the question arises why we need to collect high-dimensional and redundant measurements in the first place. Now, instead of collecting and compressing high-dimensional measurements, we can take the compressed measurements and derive the sparsest high-dimensional signal that matches the measurements. This is called "compressed sensing" or "compressed sampling" and will be hereinafter appropriately denoted as "CS".

Such compressed sensing is a new perspective that has the potential to revolutionize data acquisition and processing: signal processing, data compression, inverse problem solving, radiation system design, radar, imaging through walls, SAR. (Synthetic Aperture Radar) antennas allow for some enhanced approaches.

Advances in Block-based Computational Imaging (BCI) using compressed sensing have attracted attention in recent years. This paradigm will enable next-generation image sensors to deliver superior performance in reduced readout delays and reduced power consumption.

Nonetheless, there is a growing need for new coding algorithms for BCI measurements with minimal processing. Compressed sensing is generally built on two main basic conditions consisting of sparsity and incoherence.

An important condition for applying CS in a particular application is that the signal properties should be sparse when represented in a particular orthogonal transform basis. This property means that only a few coefficients contain most of the signal information and can be expressed as:

ここで、

here,

はベクトル化された信号である。

is the vectorized signal.

は、２次元信号である基底

is a two-dimensional signal basis

におけるＸの射影を含むスパースベクトル、すなわち、画像又はビデオがラスタースキャンによって列ベクトルに変形される。

A sparse vector, i.e. an image or video, containing the projection of X at is transformed into a column vector by raster scanning.

Compressed sensing theory claims that incoherence is key to the quality of the measurement and how the signal is sampled. Therefore, the measurement matrix used for sampling X

はコヒーレンスが低くなければならない。

must have low coherence.

by various distributions such as Bernoulli, Gaussian, and binary

が得られる。圧縮された測定値

is obtained. Compressed measurements

を劣決定系の射影を用いて得ることができる。これは数学的に

can be obtained using the projection of the underdetermined system. this is mathematically

と記述できる。ここで、ｍの次元はｎ未満でなければならない。

can be described as where the dimension of m must be less than n.

A matrix property called RIP (Restricted Isometry Property) is important to ensure good signal restoration. The following formula can be used to determine the lower bound dimension of m for a heterogeneous variance measure matrix.

ここで、

here,

である。したがって、不均一な分散測定行列に対するｍの下限の次元は、凸最適化を介して非適切（ill-posed）な線形逆問題を解くことによって、次のように記述できる。

is. Therefore, the lower bound dimension of m for a heterogeneous variance measure matrix can be written by solving the ill-posed linear inverse problem via convex optimization:

信号を復元するために、圧縮センシングは、信号Ｘが疎変換

Compressive sensing is used to restore the signal, where the signal X is sparsely transformed

により圧縮可能であり、

is compressible by

は、

teeth,

と非常にインコヒーレントである必要がある。したがって、係数ドメイン（測定ドメイン）の不完全な測定値の次元ｍから以下のように高精度に復元できる。

and must be very incoherent. Therefore, the dimension m of imperfect measurements in the coefficient domain (measurement domain) can be restored with high accuracy as follows.

ここで、

here,

は逆変換である。

is the inverse transform.

Thus, convex optimization, called l ₁ minimization, can be used to recover the measurements by the classical method, as follows:

これは次のように単純化できる。

This can be simplified as follows.

As greedy-based restoration algorithms, OMP (Orthogonal Matching Pursuit), its extensions Stepwise OMP, A*Omp, CoSaMP, and TwIST2 have been proposed for compressed sensing.

Compared to l ₁ minimization, greedy-based restoration algorithms are generally faster by minimizing a series of subspace problems. However, it requires higher prior knowledge and better measurements, and for some applications it is impractical to obtain such information when the sensing device needs to reduce the SR.

Scalar quantization (SQ) provided a simple measurement coding approach [1, 2]. It was established that SQ coding is very inefficient in terms of rate-distortion (RD) performance according to information theory. Compared to conventional encoding, SQ encoding yields low bitrate utilization, but results in high reconstruction error. Therefore, complex techniques such as QIHT (Quantized Iterative Hard Thresholding), QCoSaMP (Quantized Compressed Sampling Matching Pursuit), AOP-QIHT (Adaptive Outlier Pursuit For Quantized Iterative Hard Thresholding) are used to predict corrupted coded measurements. A decompression algorithm is required. The quality degradation will show up in the final reconstructed image/video.

DPCM (Differential Pulse-Code Modulation) is also a simple approach applied to reduce the bit rate on the transmission path (see Non-Patent Document 3). DPCM coding uses one previously coded block to predict the current block. However, previously encoded blocks may contain extraneous information unrelated to the current block, resulting in an unnecessary increase in bitrate. Nevertheless, DPCM encoding provides a practical encoding method.

The encoding method in Non-Patent Document 4 performs DCT encoding on BCI measurements. This approach improves coding efficiency over SQ and DPCM.

[5] proposed an alternative approach from state-of-the-art coding schemes investigated with quantizer dead zones. [5] state that the quality degradation comes mainly from the quantizer, and that the distortion is related to both quantization and SR in the measurement coding scheme. Therefore, in this approach, we designed the system parameters, including measurement matrix selection, quantization and SR trade-offs, and a restoration algorithm using deep learning. Moreover, effective methods can jointly control the quantization step and SR to achieve near-optimal quality at any bitrate.

The directional coding approach was first introduced in [6]. Spatially Directional Predictive Coding (SDPC) is an efficient measurement coding approach that achieves SQ and DPCM coding with high probability of recovery and low bitrates. In this study, we use random Gaussian as the measurement matrix to sample natural images. However, since a Gaussian measurement matrix is applied, it results in unstable utilization of bitrate. Importantly, this kind of measurement matrix could not be used for BCI, which only supports binary.

H. Non-Patent Document 7 proposes a partial modification of the measurement matrix to generate intra prediction candidates by combining the concept of intra prediction borrowed from H.264/AVC with measurement coding (Modified SRM). A four-way intra-prediction approach using a partially modified random binary measurement matrix, this work is mainly focused on conveying coded measurements to endpoints. A specific sampling pattern is embedded in the measurement matrix to obtain boundary information while sensing. In Non-Patent Document 7, the obtained information

を乗算し、予測テンプレートを生成する。ＳＱ符号化、ＤＰＣＭ符号化、ＳＤＰＣ符号化と比較して軽量なアプローチである。しかし、ＤＰＣＭ符号化やＳＤＰＣ符号化よりもビットレートを低くすることができなかった。この提案は、最先端の研究に比べて予測候補生成の複雑さを低減できるが、測定行列の修正により視覚的品質の大幅な劣化を引き起こし、ＲＩＰ条件及びその逆変換を満たすことができなかった。

to generate a prediction template. It is a lightweight approach compared to SQ encoding, DPCM encoding and SDPC encoding. However, the bit rate could not be made lower than that of DPCM encoding and SDPC encoding. Although this proposal can reduce the complexity of predictive candidate generation compared to the state-of-the-art work, the modification of the measurement matrix causes significant visual quality degradation and fails to satisfy the RIP condition and its inverse transform. .

A. Zymnis, S. Boyd and E. Candes, "Compressed Sensing With Quantized Measurements," in IEEE Signal Processing Letters, vol. 17, no. 2, pp. 149-152, Feb. 2010, doi: 10.1109/LSP. 2009.2035667 W. Dai and O. Milenkovic. Information Theoretical and Algorithmic Approaches to Quantized Compressive Sensing. IEEE Transactions on Communications, 59(7):1857-1866, 2011 S. Mun and J. E. Fowler, "DPCM for quantized block-based compressed sensing of images," 2012 Proceedings of the 20th European Signal Processing Conference (EUSIPCO), Bucharest, 2012, pp. 1424-1428 Yifu Zhang, Shunliang Mei, Quqing Chen and Zhibo Chen, "A novel image/video coding method based on Compressed Sensing theory," 2008 IEEE International Conference on Acoustics, Speech and Signal Processing, Las Vegas, NV, 2008, pp. 1361- 1364, doi: 10.1109/ICASSP.2008.4517871 C. Zhao, H. Bai and Y. Zhao, "Optimized Measurements Coding for Compressive Sensing Reconstruction Network," 2018 14th IEEE International Conference on Signal Processing (ICSP), Beijing, China, 2018, pp. 596-599, doi: 10.1109 /ICSP.2018.8652427 J. Zhang, D. Zhao and F. Jiang, "Spatially directional predictive coding for block-based compressive sensing of natural images," 2013 IEEE International Conference on Image Processing, Melbourne, VIC, 2013, pp. 1021-1025, doi: 10.1109/ICIP.2013.6738211 J. Zhou, D. Zhou, L. Guo, Y. Takeshi and S. Goto, "Measurement-domain intra prediction framework for compressively sensed images," 2017 IEEE International Symposium on Circuits and Systems (ISCAS), Baltimore, MD, 2017 , pp. 1-4, doi: 10.1109/ISCAS.2017.8050262

SUMMARY OF THE INVENTION It is an object of the present invention to provide an imaging apparatus and a measurement coding apparatus that can efficiently compress an image while suppressing deterioration in image quality when compressed sensing is used.

An imaging apparatus according to a first aspect includes an image sensor unit that outputs an image in image block units, and a measurement matrix of m rows and n columns (m<n) applied to the pixel values of n pixels that constitute the image block. a compressed sensing unit that performs compressed sensing processing to output m measured values; and an analog/digital converter that outputs measured data by performing analog/digital conversion processing on the m measured values. . The compression sensing unit uses a CoWH (Continuously ordered Walsh-Hadamard) matrix based on a SoWH (Sequence ordered Walsh-Hadamard) matrix in which each row of a NoH (Naturally ordered Hadamard) matrix is arranged in sequence as the measurement matrix, and the compression Perform sensing processing. Each row of the CoWH matrix has a different projection pattern when projected onto the image block. The CoWH matrix is configured such that the number of consecutive elements of the same value in the projection pattern decreases from the upper row to the lower row in the CoWH matrix.

A measurement coding apparatus according to a second aspect includes the imaging apparatus according to the first aspect, a vertical prediction mode using an upper adjacent block adjacent to the upper side of the image block, and a left adjacent block adjacent to the left side of the image block. an intra prediction unit that outputs predicted measurement data obtained by predicting the measurement data of the image block using any one of a plurality of intra prediction modes including a horizontal prediction mode using blocks; A quantization unit that performs quantization processing on residual data representing a difference between the measured data and the predicted measured data output from the intra prediction unit.

Advantageous Effects of Invention According to the present invention, it is possible to provide an imaging device and a measurement coding device that can efficiently compress an image while suppressing image quality deterioration when compressed sensing is used.

1 is a diagram showing the configuration of a measurement transmission system according to an embodiment; FIG. 1 is a diagram showing the configuration of a measurement coding apparatus according to an embodiment; FIG. It is a figure which shows operation|movement of the compression sensing part which concerns on embodiment. It is a figure which shows the vertical prediction mode and horizontal prediction mode which concern on embodiment. It is a figure which shows the structural example of the imaging device which concerns on embodiment. FIG. 4 is a diagram showing projection patterns of measurement matrices according to the embodiment; FIG. 4 is a diagram showing a CoWH matrix generation algorithm according to an embodiment; FIG. 10 is a diagram showing a comparison of reconstruction results using l1 minimization and inverse fast Walsh _- Hadamard transform (IFWHT) when using NoH matrix, SoWH matrix, and CoWH matrix, respectively; It is a figure which shows the example of a structure of a part of encoding apparatus which concerns on embodiment. 1 is a diagram showing the configuration of a measurement decoding device according to an embodiment; FIG. It is a figure for demonstrating the effect|action and effect which concern on embodiment.

A measurement transmission system according to an embodiment will be described with reference to the drawings. In the following description of the drawings, the same or similar parts are denoted by the same or similar reference numerals.

(1) Configuration of measurement transmission system First, the configuration of the measurement transmission system according to this embodiment will be described. FIG. 1 is a diagram showing the configuration of a measurement transmission system 1 according to this embodiment. As shown in FIG. 1 , the measurement transmission system 1 has a measurement coding device 100 and a measurement decoding device 200 .

The measurement coding device 100 generates image data by image sensing and compresses the image data by compression sensing. Then, measurement coding apparatus 100 further compresses the measurement data compressed by compressed sensing by compression coding, and transmits the compressed coded data (bit stream) via a transmission line. For example, the transmission line is a wireless transmission line. The transmission line may include a wired transmission line.

The measurement decoding device 200 receives the compressed encoded data from the measurement encoding device 100 and decodes the measurement data from the received encoded data. The measurement decoding device 200 then restores the original image data from the measurement data using a restoration algorithm, and outputs the image data to, for example, a display.

The measurement transmission system 1 may constitute a remote monitoring system. For example, the measurement coding device 100 may be a battery-powered sensing device and the measurement decoding device 200 may be provided in a server. The server may collect image data from the measurement coding device 100 and perform remote monitoring of the monitored object. Such sensing devices are required to have low power consumption in order to extend the driving time.

Although only one measurement coding apparatus 100 is illustrated in FIG. 1 , the measurement decoding apparatus 200 may receive coded data from a plurality of measurement coding apparatuses 100 .

Also, the transmission of encoded data from measurement encoding device 100 to measurement decoding device 200 is not limited to the case where it is performed via a transmission line, but is transmitted from measurement encoding device 100 to measurement decoding device 200 via a storage medium. Encoded data may be transmitted indirectly to device 200 .

(2) Measurement Coding Apparatus Next, the measurement coding apparatus 100 according to this embodiment will be described.

(2.1) Overall Configuration of Measurement Coding Apparatus FIG. 2 is a diagram showing the configuration of the measurement coding apparatus 100 according to this embodiment.

As shown in FIG. 2, the measurement coding device 100 has an imaging device 110, a coding device 120, and a transmitting section . The imaging device 110, the encoding device 120, and the transmission unit 130 may be configured separately, or may be configured integrally. The measurement coding device 100 may have a battery (not shown).

The imaging device 110 has an image sensor portion 111 , a compression sensing portion 112 and an analog/digital (A/D) conversion portion 113 .

The image sensor unit 111 outputs an image in units of image blocks. The image sensor unit 111 includes a CMOS image sensor (CIS). However, the image sensor unit 111 may be configured including a CCD image sensor.

The compression sensing unit 112 applies a measurement matrix of m rows and n columns (m<n) to pixel values of n pixels forming an image block output from the image sensor unit 111, and outputs m measured values. Such processing corresponds to conversion processing from the pixel domain to the measurement domain.

FIG. 3 is a diagram showing the operation of the compression sensing unit 112 according to this embodiment. As shown in FIG. 3, one image is divided into a plurality of image blocks, and the image blocks are acquired in raster scan order. Each image block consists of pixel values for n pixels (n pixels). The compression sensing unit 112 acquires pixel values from each image block in raster scan order.

In FIG. ₃ , a pixel value sequence X1 corresponding to image block _B1 , a pixel value sequence X2 corresponding to image block B2, _a pixel value sequence X3 corresponding to image block _{B3 ,} and an image block _{B i} , and pixel value sequences X _i corresponding to . A pixel value sequence consists of n pixel values (x ₁ to x _n ).

The compressed sensing unit 112 multiplies each pixel value sequence X by a measurement matrix of m rows and n columns, and outputs the result. As a result, each pixel value sequence X is converted into a measured value sequence Y consisting of m measured values (y ₁ to y _m ). Here, since m is smaller than n, the amount of data is compressed.

Returning to FIG. 2, the A/D conversion unit 113 performs A/D conversion processing on the m measured values Y output from the compression sensing unit 112, thereby outputting measurement data Y as a digital signal. Since the number of measured values is reduced (compressed) by compressed sensing, the power consumption of the A/D converter 113 can be kept low.

The measurement data Y of each image block output from the A/D conversion unit 113 is further compressed by compression encoding using intra prediction by the encoding device 120 . The encoding device 120 includes an intra prediction unit 121, a residual data generation unit 122, a quantization unit 123, a Huffman encoding unit 124, a Huffman decoding unit 125, an inverse quantization unit 126, and a synthesis unit 127. have

The intra prediction unit 121 outputs predicted measurement data _Yp obtained by predicting the measurement data Y of the current image block using any one of a plurality of intra prediction modes. As shown in FIG. 4, the multiple intra prediction modes are a vertical prediction mode using the upper neighboring block adjacent to the upper side of the current image block and a horizontal prediction mode using the left neighboring block adjacent to the left side of the current image block. including. The multiple prediction modes further include an average prediction mode that calculates an average of prediction measurement data in the vertical prediction mode and prediction measurement data in the horizontal prediction mode as prediction measurement data.

The intra prediction unit 121 has a reference data generation unit 1211 , a prediction data generation unit 1212 and a selection unit 1213 .

The reference data generation unit 1211 generates reference data for the upper adjacent block and reference data for the left adjacent block in order to obtain prediction measurement data.

The prediction data generation unit 1212 uses the reference data generated by the reference data generation unit 1211 to generate the prediction measurement data _Yv in the vertical prediction mode, the prediction measurement data _Yh in the horizontal prediction mode, and the prediction measurement data in the average prediction mode. Y _avg .

The selection unit 1213 compares each of the prediction measurement data Y _v in the vertical prediction mode, the prediction measurement data Y _h in the horizontal prediction mode, and the prediction measurement data Y _avg in the average prediction mode with the measurement data Y of the current image block. By calculating the degree of similarity, the predicted measured data with the highest degree of similarity is selected, and the selected predicted measured data _Yp is output to the residual data generator 122 . For example, the sum of absolute differences (SAD) can be used as the index value representing the degree of similarity.

However, if any prediction measurement data has a low degree of similarity with the measurement data Y of the current image block, the selection unit 1213 may directly output the measurement data Y of the current image block to the quantization unit 123 . For example, when the SAD of each of the predicted measured data Y _v in the vertical prediction mode, the predicted measured data Y _h in the horizontal prediction mode, and the predicted measured data Y _avg in the average prediction mode is larger than the threshold, The measurement data Y of the current image block is directly output to the quantization section 123 .

The residual data generation unit 122 generates residual data Y _r representing the difference between the measured data Y of the current image block and the predicted measured data Y _p output by the intra prediction unit 121, and the generated residual data Y _r Output to quantization section 123 .

The quantization unit 123 performs quantization processing on the residual data Y _r (or the measurement data Y of the current image block) and outputs the quantized data Y _q to the Huffman coding unit 124 .

The Huffman encoding unit 124 generates encoded data by performing Huffman encoding processing on the quantized data _Yq , and outputs the generated encoded data to the transmission unit 130 and the Huffman decoding unit 125 . Entropy coding other than Huffman coding may be used instead of Huffman coding.

Transmitting section 130 transmits the encoded data output from Huffman encoding section 124 to measurement decoding apparatus 200 via a transmission path. Transmitter 130 may include a radio transmitter that transmits radio signals containing encoded data. In addition, the transmission unit 130 transmits encoded data including mode information indicating the prediction mode selected by the selection unit 1213 .

Huffman decoding section 125 restores quantized data Y _q by performing Huffman decoding processing on the encoded data output from Huffman coding section 124 , and sends the restored quantized data Y _q to inverse quantization section 126 . Output.

The inverse quantization unit 126 restores the residual data Y _r by performing inverse quantization processing on the restored quantized data Y _q and outputs the restored residual data Y _deq to the synthesizing unit 127 .

The synthesizing unit 127 restores the measured data Y by synthesizing the restored residual data Y _deq and the predicted measured data Y _p , and outputs the restored measured data to the reference data generation unit 1211 of the intra prediction unit 121 .

In addition, in the configuration example of the encoding device 120 shown in FIG. However, the Huffman decoding unit 125 may be unnecessary. In this case, instead of inputting the encoded data output by the Huffman encoding unit 124 to the Huffman decoding unit 125, the quantized data _Yq output by the quantization unit 123 is input to the inverse quantization unit 126. Become.

(2.2) Imaging Device Next, the imaging device 110 according to this embodiment will be described.

FIG. 5 is a diagram showing a configuration example of the imaging device 110 according to this embodiment. The imaging device 110 according to the present embodiment is a CMOS image sensor using compression sensing, ie CS-CIS.

As shown in FIG. 5, the imaging device 110 includes a pixel array 111a, a row selector 111b, a column selector 111c, a matrix operation trigger 112a, and a matrix operation. 112 b and A/D converter 113 .

The pixel array 111a, the row selection section 111b, and the column selection section 111c constitute the image sensor section 111 shown in FIG. The pixel array 111a has a plurality of pixels arranged in an array. Each pixel receives light and outputs a pixel value, which is an electrical signal. The pixel array 111a is divided into blocks of n pixels. The row selection unit 111b and the column selection unit 111c select pixels from which pixel values are to be read, and read pixel values from the selected pixels.

The matrix calculation trigger section 112a and the matrix calculation section 112b constitute the compression sensing section 112 shown in FIG. The matrix calculation trigger unit 112a triggers matrix calculation for a pixel value sequence X consisting of n pixels. The matrix calculator 112 b performs matrix calculation on the pixel value sequence X using the measurement matrix, and outputs a measured value sequence Y made up of m measured values to the A/D converter 113 . The A/D converter 113 converts the measured value sequence Y into a digital signal and outputs measured data Y. FIG.

FIG. 6 is a diagram showing projection patterns of measurement matrices according to the present embodiment. Here, it is assumed that the size of the image block is 8 pixels×8 pixels, that is, the value of n is 64. In addition, the value of m is 8 (there are 8 patterns from low frequency to high frequency in the figure), that is, converting from n = 64 to 8 measured values (m = 8), 12.5% (m/ 8 patterns used for compression up to n=8/64=12.5%). In FIG. 6 , each white area represents an element value of “1” in the measurement matrix, and a black area represents an element value of “0” in the measurement matrix.

As shown in FIG. 6, the compression sensing unit 112 measures the CoWH (Continuously ordered Walsh-Hadamard) matrix based on the SoWH (Sequence ordered Walsh-Hadamard) matrix in which each row of the NoH (Naturally ordered Hadamard) matrix is arranged in sequence. to perform the compressed sensing process. Note that the sequence means the frequency (frequency) at which values are switched in one row of the matrix.

Each row of the CoWH matrix has a different projection pattern when projected onto an image block. The CoWH matrix is configured such that the number of consecutive equivalent elements in the projection pattern decreases from the upper row to the lower row in the CoWH matrix.

For example, the uppermost row (row #1) of the CoWH matrix has 64 consecutive pixels of "1" in the projection pattern. The measurement calculated using row # ₁ is called y1.

Each of the second row (row #2) and the third row (row #3) from the top of the CoWH matrix has 32 consecutive pixels of element “1” and element “0” in the projection pattern. ” continues for 32 pixels. The measurement calculated using row # ₂ is called y2, and the measurement calculated using row # ₃ is called y3.

The fourth row from the top of the CoWH matrix (row #4) has 16 consecutive pixels of element "1" and 32 consecutive pixels of element "0" in the projection pattern.

The fifth row from the top of the CoWH matrix (row #5) has 16 consecutive pixels of element "1" and 16 consecutive pixels of element "0" in the projection pattern.

The sixth row from the top of the CoWH matrix (row #6) has 16 consecutive pixels of element "1" and 32 consecutive pixels of element "0" in the projection pattern.

The seventh row from the top of the CoWH matrix (row #7) has 16 consecutive pixels of element "1" and 16 consecutive pixels of element "0" in the projection pattern.

The eighth row from the top of the CoWH matrix (row #8) has an area of 8 consecutive pixels of "1" and an area of 16 consecutive pixels of "1" in the projection pattern. It also has an area in which the element "0" continues for 8 pixels and an area in which the element "0" continues for 16 pixels.

The CoWH matrix includes rows forming a first projection pattern for obtaining all pixel values of an image block, rows forming a second projection pattern for obtaining pixel values in the upper half of the image block, and rows forming the left half of the image block. and rows forming a third projection pattern from which pixel values are obtained. This makes it possible to smoothly perform intra prediction in vertical prediction mode and horizontal prediction mode in the measurement domain. Specifically, the uppermost row (row #1) in the CoWH matrix constitutes the first projection pattern, and the second and third rows (rows #2 and #3) from the top in the CoWH matrix constitute the first projection pattern. A second projection pattern and a third projection pattern are constructed.

By adopting the CS theory to the CIS, the measurement matrix

は、スクランブルマスクの確率が高いバイナリ｛０，１｝又はワーストケースでも相補｛－１，１｝でなければならないという実装上の制約がある。

has the implementation constraint that the scrambling mask must be binary {0,1} with high probability or even complementary {-1,1} in the worst case.

The continuous-order Walsh-Hadamard (CoWH) according to the present embodiment is an optimization of the binary-sequence-order Walsh-Hadamard (SoWH) that can be derived by the natural-order Hadamard (NoH). Note that the columns of the NoH and SoWH matrices are orthogonal. These matrices are matrices of order n

を与えることによって得ることができる。ＮｏＨ行列は次のように表現できる。

can be obtained by giving A NoH matrix can be expressed as follows.

ここで、Ｉ_ｎ×ｎは単位行列である。

where I _n×n is the identity matrix.

は、

の転置であり、その次数はＸの次元に等しくなければならない。これは、図６（ａ）に示すような投影パターンを生じる。これは繰り返し適用することができ、次のように、ｎ次の

teeth,

, whose degree must equal the dimension of X. This produces a projected pattern as shown in FIG. 6(a). This can be applied iteratively, as follows:

に対してSylvester's constructionを適用することにより次の行列のシーケンスにつながり、これはウォルシュ・アダマール行列と呼ばれ、次のように表現できる。

Applying Sylvester's construction to leads to the following sequence of matrices, called Walsh-Hadamard matrices, which can be expressed as:

但し、

however,

であり、

and

はクロネッカー積である。その結果、図６（ｂ）に示すような投影パターンを生じる。

is the Kronecker product. As a result, a projection pattern as shown in FIG. 6(b) is produced.

The sequential rows of the Walsh-Hadamard matrix are derived from the NoH matrix by applying bit-reversal and gray-code permutation. This will

で表現されるＳｏＷＨ測定行列が得られる。

We obtain the SoWH measurement matrix expressed as

In this embodiment, we rearrange the projection pattern of the SoWH measurement matrix by grouping the low-frequency coefficients at the top of the measurement matrix. In the CoWH arrangement, the projected pattern is organized to increase the number of connected elements, as shown in Fig. 6(c). Thereby, the measurement quality can be improved while sensing. The relevant component corresponds to a group of contiguous pixels with corresponding values and influences the reconstruction algorithm and the inverse transform in terms of richer prior knowledge.

Also, the good continuous pattern of the CoWH matrix shown in FIG. Limited.

の次数とし、探索基準位置（search criteria position）ｄはｎ×ｎに等しい。次に、各ｄについて、ｒ_ｉ，ｊの除算後の剰余を求める。次数が奇数か偶数かを判定するために、

を計算する。各次数についてｒ_ｉ，ｊを求めた後、

を再配置するために、Ａを空の行列

とすると、

である。 A CoWH matrix is generated by the algorithm shown in FIG. Specifically, i and j are

and the search criteria position d is equal to n×n. Then, for each d, find the remainder after division of r _i,j . To determine if the degree is odd or even,

to calculate After obtaining r _i,j for each order,

to rearrange A with an empty matrix

and

is.

A (CoWH matrix) satisfies the RIP with probability of at least

where β is a positive constant.

For hardware implementation, this extended matrix A is binary binarized from the complementary matrix. The sampling mask (projection pattern) of the CoWH matrix is arranged gradually from low frequency coefficients to high frequency coefficients compared to NoH and SoWH matrices.

FIG. 8 is a diagram showing a comparison of reconstruction results using l1 minimization and inverse fast Walsh _- Hadamard transform (IFWHT) when using NoH matrix, SoWH matrix, and CoWH matrix, respectively. As shown in FIG. 8, the CoWH matrix outperforms the NoH and SoWH matrices in terms of visual quality at the same SR. 8(a) shows the constructed measurement matrix, FIG. 8(b) shows the restored image, and FIG. 8(c) shows the result of 400% zoom. Note that n=65536 (256 pixels by 256 pixels, m=8192 (compressed to 12.5%), and B (ie, size of block used in BCI)=8.

(2.3) Encoding Device Next, the encoding device 120 according to this embodiment will be described. FIG. 9 is a diagram showing a configuration example of part of the encoding device 120 according to this embodiment.

As shown in FIG. 9, the reference data generation unit 1211, the prediction data generation unit 1212, and the selection unit 1213 constitute the intra prediction unit 121 shown in FIG. An input buffer 141 is provided on the input side of the intra prediction unit 121 . An output buffer 142 is provided on the output side of the quantization section 123 .

The reference data generating unit 1211 includes a package splitter _1211a that separates the measured values y1, y2 _, and y3 from the restored measured data Y _output by the synthesizing unit ₁₂₇ , and _a subtractor that subtracts the measured value y2 from the measured value y1. 1211b, _a subtractor _1211c for subtracting the measured value y3 from the measured value y1, a register 1211d for storing the output of the subtractor 1211b, and a register 1211e for storing the output of the subtractor 1211c.

where the measurement y1 is the measurement obtained by row #1, which constitutes the projection pattern that captures all pixel values of the image block in the _CoWH matrix, and corresponds to the sum of all pixel values of the image block. .

The measurement y2 is the measurement obtained by row # ₂ , which constitutes the projection pattern that captures the pixel values of the left half of the image block in the CoWH matrix and corresponds to the sum of the pixel values of the left half of the image block. do.

The measurement y3 is the measurement obtained by row #3, which constitutes the projection pattern that captures the pixel values of the upper half of the image block in the _CoWH matrix and corresponds to the sum of the pixel values of the upper half of the image block. do.

Thus, the _output of subtractor _1211b , which subtracts measurement y2 from measurement y1, corresponds to the sum of the pixel values of the right half of the image block, which is held in register 1211d. The sum of pixel values in the right half of this image block is used as reference data in the horizontal prediction mode shown in FIG. Register 1211d only needs to hold the reference data corresponding to the left-hand neighboring block of the current image block, and here its size is 16 bits.

On the other hand, the output of subtractor _{1211c ,} which subtracts measurement y1 from measurement y3, corresponds to the sum of the pixel values of the lower half of the image block, which is held in register 1211e. The sum of pixel values in the lower half of this image block is used as reference data in the vertical prediction mode shown in FIG. The register 1211e holds reference data for each image block belonging to the row above the row to which the current image block belongs. Assuming that the image width is W and the image block size is B, the register 1211e holds reference data for W/B blocks.

The prediction data generation unit 1212 calculates the pixel value (average value) per pixel from the sum of the pixel values in the lower half of the image block output from the register 1211e, and transposes the CoWH matrix to this average value.

を乗算することにより、垂直予測モードの予測測定データＹ_ｖを出力する。このようにして、イントラ予測部１２１は、垂直予測モードについて、上側隣接ブロックについて得られた測定値ｙ_１から、当該上側隣接ブロックについて得られた測定値ｙ_３を減算することにより、画素ドメインにおける当該上側隣接ブロックの下半分の合計の画素値を得る処理と、下半分の合計の画素値を有効画素数で除算することにより、当該上側隣接ブロックの下半分の画素値の平均を得る処理と、この平均からＣｏＷＨ行列に基づいて垂直予測モードの予測測定データＹ_ｖを得る処理とを実行する。

to output the predicted measurement data _Yv in the vertical prediction mode. _In this way, the intra prediction unit 121 subtracts the measured value y3 obtained for the _upper neighboring block from the measured value y1 obtained for the upper neighboring block for the vertical prediction mode, thereby obtaining A process of obtaining the total pixel value of the lower half of the upper neighboring block, and a process of obtaining the average of the pixel values of the lower half of the upper neighboring block by dividing the total pixel value of the lower half by the number of effective pixels. , and obtaining the predicted measurement data Yv of the vertical prediction mode from this average based on the _CoWH matrix.

In addition, the prediction data generation unit 1212 calculates the pixel value (average value) per pixel from the sum of the pixel values of the right half of the image block output by the register 1211d, and transposes the CoWH matrix to this average value.

を乗算することにより、水平予測モードの予測測定データＹ_ｈを出力する。このようにして、イントラ予測部１２１は、水平予測モードについて、左側隣接ブロックについて得られた測定値ｙ_１から、左側隣接ブロックについて得られた測定値ｙ_２を減算することにより、画素ドメインにおける左側隣接ブロックの右半分の合計の画素値を得る処理と、右半分の合計の画素値を有効画素数で除算することにより、左側隣接ブロックの右半分の画素値の平均を得る処理と、この平均からＣｏＷＨ行列に基づいて水平予測モードの予測測定データＹ_ｈを得る処理とを実行する。

to output the predicted measurement data _Yh in the horizontal prediction mode. _In this way, the intra prediction unit ₁₂₁ subtracts the measured value y2 obtained for the left neighboring block from the measured value y1 obtained for the left neighboring block for the horizontal prediction mode, thereby obtaining the left side in the pixel domain. A process of obtaining the total pixel value of the right half of the adjacent block, a process of obtaining the average of the pixel values of the right half of the left adjacent block by dividing the total pixel value of the right half by the number of effective pixels, and this average to obtain predicted measurement data Yh in the horizontal prediction mode based on the _CoWH matrix from .

Further, the predicted data generator 1212 obtains the average of the predicted measured data _Yv and the predicted measured data _{Yh as the predicted measured data Yavg in} the average prediction mode.

Such an intra prediction method can be expressed as follows.

Vertical prediction mode:

水平予測モード:

Horizontal prediction mode:

平均予測モード:

Average prediction mode:

To select the final intra-prediction mode prediction measurement data Y _p , the selection unit 1213 uses SAD as follows to determine the minimum difference between the prediction candidate and Y

を検索し、次のように予測測定データＹ_ｐを選択する。

and select the predicted measurement data _Yp as follows.

However, if no prediction candidate is selected, the selection unit 1213 assigns zero to the prediction measurement data _Yp .

Then, the residual data generator 122 calculates the residual data _Yr as follows.

The quantization unit 123 quantizes the residual data _Yr by scalar quantization as follows and outputs quantized data _Yq .

where _Bp is a bit depth parameter, eg 4 bits.

Note that the inverse quantization unit 126 performs the following calculations.

Among these processes, quantization is lossy, but intra prediction and entropy coding are lossless.

(3) Measurement decoding device Next, the measurement decoding device 200 according to this embodiment will be described. FIG. 10 is a diagram showing the configuration of a measurement decoding apparatus 200 according to this embodiment.

As shown in FIG. 10 , measurement decoding apparatus 200 has reception section 210 , decoding apparatus 220 and restoration section 230 .

Receiving section 210 receives encoded data from measurement encoding apparatus 100 via a transmission path, and outputs the received encoded data to decoding apparatus 220 . The encoded data includes mode information indicating the prediction mode selected by measurement encoding device 100 . Receiving section 210 also outputs mode information to decoding device 220 .

The decoding device 220 has an intra prediction section 221 , a Huffman decoding section 222 , an inverse quantization section 223 and a synthesis section 224 . The intra prediction section 221 has a reference data generation section 2211 and a prediction data generation section 2212 .

Reference data generation section 2211 is configured similarly to reference data generation section 1211 of measurement encoding apparatus 100 , and prediction data generation section 2212 is configured similarly to reference data generation section 1211 of measurement encoding apparatus 100 . However, prediction data generation section 2212 generates prediction measurement data Y _p only for the prediction mode indicated by the mode information transmitted from measurement coding apparatus 100 , and outputs the generated prediction measurement data Y _p to combining section 224 .

The Huffman decoding unit 222 restores the quantized data Y _q by performing Huffman decoding processing on the encoded data output from the receiving unit 210 , and outputs the restored quantized data Y _q to the inverse quantization unit 223 . .

The inverse quantization unit 223 restores the residual data Y _r by performing inverse quantization processing on the restored quantized data Y _q and outputs the restored residual data Y _deq to the synthesizing unit 224 .

Synthesis section 224 reconstructs measurement data Y by combining the reconstructed residual data Y _deq and predicted measurement data Y _p , and outputs the reconstructed measurement data to reconstruction section 230 .

The restoration unit 230 restores the original pixel value X (image block) for n pixels from the measurement data Y using a restoration algorithm and outputs the restored pixel value. CS theory asserts that m-dimensional measurement data is sufficient to reconstruct the original signal measurement via an efficient reconstruction algorithm. For example, the restoration unit 230 restores the pixel value X (image block) for the original n pixels using the l1 algorithm and _IFWHT .

(4) Actions and Effects According to the present embodiment, a new high-resolution BCI including an optimized binary structural measurement matrix (SRM) based on the Walsh-Hadamard transform (WHT) It is possible to realize a lightweight encoding framework for In addition, spatial redundancy can be reduced by performing intra-prediction utilizing the features of the CoWH matrix in the measurement domain. Therefore, images can be efficiently compressed and stored and transmitted.

In the following, simulation results for efficient storage in memory space and transmission of encoded measurements are described. Quantitative metrics are PSNR (peak-to-noise-ratio), SSIM (structural similarity index measure), bpp (bits-per-pixel).

"A. Mercat, M. Viitanen, and J. Vanne, ``UVG dataset: 50/120fps 4K sequences for video codec analysis and development,'' in Proc. ACM Multimedia Syst. Conf., Istanbul, Turkey, June 2020" We use a 4K dataset in a color space composed of Beauty, ReadySetGo, Bosphorus, and HoneyBee. where n=24883200 (3840×2160×3), SRε{4/4, 1/8, 1/16, 1/32, 1/64} and B=16.

Table 1 below shows a comparison of minimum process BCI measurements for PSNR (dB) without encoding, SSIM, bpp. A comparison of the minimum process BCI measurements in Table 1 shows that the CoWH matrix provides quality degradation characteristics of 5-8% on average for PSNR and 0.2-0.5 for SSIM for each decrease in SR.

Furthermore, when SR equals 1/16, the CoWH matrix provides a lower bound for better results with less undersampling artifacts compared to 1/32 and 1/64. In the following, with SR fixed at 1/16, the coding performance is evaluated and compared with state-of-the-art research.

Furthermore, we prove the hypothesis that a higher visual quality can be obtained by increasing the relationship between each projection pattern. However, this results in loss of edge detail.

Furthermore, intra prediction (IPMC: Intra Prediction-based Measurement Coding) by the measurement coding apparatus 100 according to the present embodiment is compared with state-of-the-art work on spatially coding minimal process BCI measurements. For these comparisons, the state-of-the-art code was selected. However, the quantization method and parameters were changed to be the same in order to make the comparison of coding performance fair. Using SQ as a baseline

を用いて測定値をサンプリングした。

was used to sample the measurements.

As shown in the simulation results in Table 2, IPMC as a whole achieved SQ with PSNR of 19.96% dB, SSIM of 20.25% and bpp of 10.34%.

Compared with DPCM, it achieved PSNR of 14.50%, SSIM of 9.19% and bpp of 8.77%, outperforming DPCM in all aspects.

PSNR was 14.37%, SSIM was 9.19% and bpp was 7.19% when compared to similar encodings such as SDPC.

Furthermore, when compared with Modified SRM, which is more advanced in sampling and encoding, PSNR is 11.73%, SSIM is 13.09%, and bpp is 6.74%, which are higher in this embodiment. Therefore, the Modified SRM modifies the binary SRM to obtain information from neighboring blocks, and the H. H.264 to generate the final intra-prediction candidates, so it is not very efficient.

Thus, IPMC according to the present embodiment is an overall superior coding in terms of PSNR of 15.06%, SSIM of 18.74% and bpp of 3.7% compared to the state-of-the-art work. showed performance.

FIG. 11 shows an example of detail zoomed 400% using the HoneyBee dataset. In FIG. 11, (a) is SQ (baseline), (b) is DPCM, (c) is SDCP, (d) is Modified SRM, and (e) is this embodiment. As shown in Fig. 11, the trimmed results using the HoneyBee dataset show that at SR of 1/16, the present embodiment provided richer visual quality compared to the state-of-the-art work.

Thus, according to the overall performance in Table 2, the present embodiment provided trade-off performance performance on all fronts for state-of-the-art research.

In addition, we have implemented the IPMC algorithm on the FPGA. Here, the resolution is 3820×2140, the total logic utilization is 3,421, and the maximum operating frequency is 102.65 MHz. This architecture consumed 28.32 mW of power. This provided the potential for real-time encoding that is 100 times faster than software implementations, as summarized in Table 3.

(5) Other Embodiments A program that causes a computer to execute each process according to the above-described embodiments may be provided. The program may be recorded on a computer readable medium. A computer readable medium allows the installation of the program on the computer. Here, the computer-readable medium on which the program is recorded may be a non-transitory recording medium. The non-transitory recording medium is not particularly limited, but may be, for example, a recording medium such as CD-ROM or DVD-ROM.

A circuit for executing each process performed by the measurement encoding device 100 (encoding device 120) may be integrated, and the measurement encoding device 100 (encoding device 120) may be configured by a semiconductor integrated circuit (chipset, SoC). .

Although the embodiments have been described in detail with reference to the drawings, the specific configuration is not limited to the above, and various design changes can be made without departing from the spirit of the invention.

1: measurement transmission system 100: measurement coding device 110: imaging device 111: image sensor section 111a: pixel array 111b: row selection section 111c: column selection section 112: compression sensing section 112a: matrix calculation trigger section 112b: matrix calculation section 113: A/D conversion unit 120: Encoding device 121: Intra prediction unit 122: Residual data generation unit 123: Quantization unit 124: Huffman coding unit 125: Huffman decoding unit 126: Inverse quantization unit 127: Synthesis unit 130 : Transmitter 141 : Input buffer 142 : Output buffer 200 : Measurement decoder 210 : Receiver 220 : Decoder 221 : Intra predictor 222 : Huffman decoder 223 : Inverse quantizer 224 : Synthesizer 230 : Restorer 1211 : Reference data generator 1211a : Package splitter 1211b : Subtractor 1211c : Subtractor 1211d : Register 1211e : Register 1212 : Predicted data generator 1213 : Selector 2211 : Reference data generator 2212 : Predicted data generator

Claims (8)

an image sensor unit that outputs an image in units of image blocks;
a compression sensing unit that performs a compression sensing process of applying a measurement matrix of m rows and n columns (m<n) to pixel values of n pixels constituting the image block and outputting m measured values;
an analog/digital conversion unit that outputs measurement data by performing analog/digital conversion processing on the m measured values;
The compression sensing unit uses a CoWH (Continuously ordered Walsh-Hadamard) matrix based on a SoWH (Sequence ordered Walsh-Hadamard) matrix in which each row of a NoH (Naturally ordered Hadamard) matrix is arranged in sequence as the measurement matrix, and the compression perform sensing processing,
Each row of the CoWH matrix has a different projection pattern when projected onto the image block,
The imaging apparatus according to claim 1, wherein the CoWH matrix is configured such that the number of consecutive elements having the same value in the projection pattern decreases from an upper row to a lower row in the CoWH matrix. The CoWH matrix is
rows forming a first projection pattern for obtaining all pixel values of the image block;
rows forming a second projection pattern for obtaining pixel values of the upper half of the image block;
and a row forming a third projection pattern for obtaining pixel values of the left half of the image block. the uppermost row in the CoWH matrix constitutes the first projection pattern;
3. The imaging apparatus of claim 2, wherein the second and third rows from the top in the CoWH matrix constitute the second projection pattern and the third projection pattern. An imaging device according to claim 2 or 3;
using any one of a plurality of intra prediction modes including a vertical prediction mode using an upper adjacent block adjacent to the upper side of the image block and a horizontal prediction mode using a left adjacent block adjacent to the left side of the image block; an intra prediction unit that outputs predicted measured data obtained by predicting the measured data of the image block;
a quantization unit that performs quantization processing on residual data representing a difference between the measurement data of the image block and the prediction measurement data output from the intra prediction unit. Device. The intra prediction unit, for the vertical prediction mode,
By subtracting the measurement value obtained by applying the second projection pattern to the upper adjacent block from the measurement value obtained by applying the first projection pattern to the upper adjacent block, obtaining the sum pixel value of the lower half of the upper neighboring block in the pixel domain;
obtaining an average pixel value of the lower half of the upper adjacent block by dividing the total pixel value of the lower half by the number of effective pixels;
obtaining prediction measurement data for the vertical prediction mode from the average based on the CoWH matrix. The intra prediction unit, for the horizontal prediction mode,
By subtracting the measured value obtained by applying the third projection pattern to the left adjacent block from the measured value obtained by applying the first projection pattern to the left adjacent block, obtaining the sum pixel value of the right half of the left neighboring block in the pixel domain;
obtaining an average pixel value of the right half of the left adjacent block by dividing the total pixel value of the right half by the number of valid pixels;
and obtaining prediction measurement data for the horizontal prediction mode from the average based on the CoWH matrix. The plurality of intra prediction modes further include an average prediction mode for calculating an average of prediction measurement data in the vertical prediction mode and prediction measurement data in the horizontal prediction mode as prediction measurement data,
The intra prediction unit calculates a similarity between each of the prediction measurement data in the vertical prediction mode, the prediction measurement data in the horizontal prediction mode, and the prediction measurement data in the average prediction mode and the measurement data of the image block. 7. The measurement coding apparatus according to any one of claims 4 to 6, wherein the process of selecting the predicted measurement data with the highest degree of similarity is performed by calculating. an encoding unit that encodes the residual data quantized by the quantization unit;
A transmitting unit that transmits residual data encoded by the encoding unit to a decoding side,
The measurement coding apparatus according to claim 7, wherein the transmission unit further transmits mode information indicating a prediction mode of the prediction measurement data selected by the intra prediction unit.

Images