JP2022028085A - Image processing device and method - Google Patents

Abstract

To make it possible to suppress an increase in prediction error.SOLUTION: Captured image data are encoded using prediction by a prediction scheme which is set on the basis of a focusing-related parameter. Encoded data of the captured image data are decoded using prediction by a prediction scheme which is set on the basis of a focusing-related parameter. The present disclosure can be applied to, for example, an image processing device, an image encoding device, an image decoding device, an imaging element, an imaging device, or the like.SELECTED DRAWING: Figure 1

Description

The present disclosure relates to image processing devices and methods, and more particularly to image processing devices and methods capable of suppressing an increase in prediction error.

Conventionally, a plurality of signals corresponding to light flux emitted from different exit pupils are detected in a pixel, a phase difference is detected from the detected multiple signals, and a focal length is controlled based on the detected phase difference to control a subject. A method of focusing on is considered (see, for example, Patent Document 1 and Patent Document 2).

Japanese Unexamined Patent Publication No. 2014-215551 JP-A-2017-79407

However, when an image pickup unit having such pixels is used to image a subject and obtain captured image data, there is a possibility that the spatial correlation of the pixels may change depending on the degree of focusing. Therefore, when the pixel value is predicted from the captured image data obtained by such an imaging unit by a single prediction method and the captured image data is encoded using the prediction error, the pixel value is determined depending on the degree of focusing. There was a risk that the prediction error would increase.

The present disclosure has been made in view of such a situation, and makes it possible to suppress an increase in prediction error.

The image processing device on one aspect of the present technology is an image processing device including a coding unit that encodes captured image data using prediction by a prediction method set based on parameters related to focusing.

The image processing method of one aspect of the present technology is an image processing method for encoding captured image data by using prediction by a prediction method set based on parameters related to focusing.

The image processing device of another aspect of the present technology is an image processing device including a decoding unit that decodes the encoded data of the captured image data by using the prediction by the prediction method set based on the parameters related to focusing.

The image processing method of another aspect of the present technology is an image processing method for decoding the encoded data of the captured image data by using the prediction by the prediction method set based on the parameters related to focusing.

In the image processing apparatus and method of one aspect of the present technology, the captured image data is encoded using the prediction by the prediction method set based on the parameters related to focusing.

In the image processing apparatus and method of another aspect of the present technology, the encoded data of the captured image data is decoded by using the prediction by the prediction method set based on the parameters related to focusing.

It is a figure which shows the list of the prediction method, etc. to which this technique is applied. It is a block diagram which shows the main configuration example of the image processing apparatus which realizes the method 1. It is a block diagram which shows the main configuration example of a coding part. It is a block diagram which shows the main configuration example of a decoding part. It is a figure explaining the main configuration example of a pixel array. It is a figure explaining the main composition example of a pixel. It is a figure explaining the example of the prediction method. It is a figure explaining another example of a prediction method. It is a figure explaining another configuration example of a pixel array. It is a figure explaining the example of the prediction method. It is a figure explaining the example of the operation which derives a predicted value. It is a figure explaining another example of a prediction method. It is a flowchart explaining an example of the flow of image processing. It is a flowchart explaining the example of the flow of the simple coding process. It is a flowchart explaining the example of the flow of the simple decoding process. It is a block diagram which shows the main configuration example of the image pickup apparatus which realizes the method 1. It is a flowchart explaining the example of the flow of the image pickup processing. It is a flowchart following FIG. 17 explaining an example of the flow of the image pickup processing. It is a block diagram which shows the main configuration example of the image processing apparatus which realizes the method 2. It is a figure explaining the main configuration example of a bit stream. It is a block diagram which shows the main configuration example of a coding part. It is a block diagram which shows the main configuration example of a decoding part. It is a flowchart explaining an example of the flow of image processing. It is a flowchart explaining the example of the flow of a coding process. It is a flowchart explaining the example of the flow of a decoding process. It is a block diagram which shows the main configuration example of the image pickup apparatus which realizes the method 2. It is a flowchart explaining the example of the flow of the image pickup processing. It is a flowchart following FIG. 27 which explains the example of the flow of the image pickup processing. It is a block diagram which shows the main configuration example of the image processing apparatus which realizes method 2-1. It is a block diagram which shows the main block diagram of the auxiliary information generation part. It is a flowchart explaining an example of the flow of image processing. It is a flowchart explaining the example of the flow of auxiliary information generation processing. It is a block diagram which shows the main configuration example of the image pickup apparatus which realizes method 2-1. It is a flowchart explaining the example of the flow of the image pickup processing. It is a flowchart following FIG. 34 explaining an example of the flow of the image pickup processing. It is a block diagram which shows the main configuration example of the image processing apparatus which realizes the method 3. It is a figure explaining the error information. It is a block diagram which shows the main block diagram of the reliability information generation part. It is a figure explaining the error information. It is a flowchart explaining an example of the flow of image processing. It is a flowchart explaining the example of the flow of the reliability information generation processing. It is a block diagram which shows the main configuration example of the image pickup apparatus which realizes the method 3. It is a flowchart explaining the example of the flow of the image pickup processing. It is a flowchart following FIG. 43 which explains the example of the flow of the image pickup processing. It is a figure which shows the main configuration example of a laminated image sensor. It is a figure which shows the main configuration example of a laminated image sensor. It is a block diagram which shows the main configuration example of a computer.

Hereinafter, embodiments for carrying out the present disclosure (hereinafter referred to as embodiments) will be described. The explanation will be given in the following order.
1. 1. Prediction using pixels that detect phase difference 2. First Embodiment (Details of Method # 1)
3. 3. Second embodiment (details of method # 2)
4. Third Embodiment (Details of Method # 2-1)
5. Fourth Embodiment (Details of Method # 3)
6. Fifth Embodiment (stacked image sensor)
7. Addendum

<1. Prediction using pixels that detect phase difference>
<References that support technical content and terminology>
The scope disclosed in the present technology includes not only the contents described in the embodiments but also the contents described in the following documents known at the time of filing.

Patent Document 3: US2011 / 092247
Patent Document 4: US2012 / 0219231
Patent Document 5: Japanese Unexamined Patent Publication No. 2014-103543

In other words, the content described in the above-mentioned literature is also a basis for determining support requirements.

<DPCM coding>
Conventionally, as described in Patent Document 1 and Patent Document 2, for example, a plurality of signals corresponding to light fluxes emitted from different exit pupils in a pixel are detected, a phase difference is detected from the detected plurality of signals, and the phase difference is detected. A method of controlling the focal length based on the detected phase difference to focus on the subject has been considered.

Further, for example, as described in Patent Documents 3 to 5, at the time of transmission or recording of captured image data (also referred to as captured image data) for the purpose of suppressing an increase in the band used or memory capacity of the interface. A method of compressing the captured image data by coding using DPCM (Differential Pulse Code Modulation) has been considered. In DPCM coding, the pixel value of the pixel to be processed is predicted using the pixel value of the pixel (also referred to as peripheral pixel) located around the pixel value, and the difference (prediction error) from the predicted value is encoded. Improve coding efficiency.

However, when a subject is imaged using an image pickup unit having pixels capable of detecting the phase difference as described above and the captured image data is obtained, the spatial correlation of the pixels is determined according to the degree of focusing. Was in danger of changing.

Therefore, when the captured image data obtained by such an imaging unit is DPCM coded by a single prediction method, the prediction error may increase depending on the degree of focusing. That is, the pixel value of the pixel to be processed is predicted using the pixel values of the peripheral pixels having a predetermined positional relationship with respect to the pixel to be processed without considering the degree of focusing, and the prediction error is used to predict the code. In the case of conversion, the spatial correlation of the pixels changes depending on the degree of focusing, so that the prediction error may increase. In the case of variable-length coding, if the prediction error increases, the coding efficiency may decrease. Further, in the case of fixed-length coding, if the prediction error increases, there is a possibility that the image quality of the decoded image obtained by encoding / decoding the captured image data may decrease.

<Prediction method control>
Therefore, the prediction method is controlled according to the degree of focusing.

For example, the captured image data is encoded using the prediction by the prediction method set based on the parameters related to focusing. For example, the image processing apparatus is provided with a coding unit that encodes the captured image data by using the prediction by the prediction method set based on the parameters related to focusing.

By doing so, it is possible to encode the captured image data so as to suppress the change in the spatial correlation of the pixels according to the degree of focusing. Therefore, it is possible to suppress an increase in prediction error. As a result, in the case of variable-length coding, it is possible to suppress a decrease in coding efficiency. Further, in the case of fixed-length coding, it is possible to suppress a decrease in the image quality of the decoded image.

Further, for example, the encoded data of the captured image data is decoded by using the prediction by the prediction method set based on the parameters related to focusing. For example, the image processing apparatus is provided with a decoding unit that decodes the coded data of the captured image data by using the prediction by the prediction method set based on the parameters related to focusing.

By doing so, it is possible to correctly decode the captured image data encoded so as to suppress the change in the spatial correlation of the pixels according to the degree of focusing. Therefore, it is possible to suppress an increase in prediction error. As a result, in the case of variable-length coding, it is possible to suppress a decrease in coding efficiency. Further, in the case of fixed-length coding, it is possible to suppress a decrease in the image quality of the decoded image.

The prediction method used for such coding or decoding may be set according to whether or not the subject is in focus, for example, based on the parameters related to focusing. By doing so, it is possible to select a prediction method according to the degree of focusing.

For example, when the subject is in focus, a prediction method in which the pixel value of the pixel located closest to the processing target pixel among the pixels of the same color as the processing target pixel is used as the predicted value of the processing target pixel. It may be set. Here, the pixel having the same color as the processing target pixel means a pixel provided with a color filter of the same type as the processing target pixel. Further, here, the same type of color filter refers to a color filter that transmits light having the same (or substantially similar) wavelength. By doing so, it is possible to suppress an increase in prediction error in a state where the subject is in focus.

Further, for example, when the subject is not in focus, the position of the photodiode that detects the pixel value is the same as that of the pixel to be processed, and is the closest to the pixel to be processed among the pixels of the same color as the pixel to be processed. A prediction method may be set in which the pixel value of the positioned pixel is used as the predicted value of the pixel value of the pixel to be processed. Here, the position of the photodiode for detecting the pixel value indicates the position of the photodiode in the pixel. That is, the peripheral pixel "the position of the photodiode that detects the pixel value is the same as the pixel to be processed" is the position in the pixel of the photodiode that detects the pixel value among the peripheral pixels of the pixel to be processed. Indicates a pixel that is identical to a pixel. The "pixels of the same color as the pixel to be processed" and the "color filter of the same type" are as described above. By doing so, it is possible to suppress an increase in the prediction error when the subject is not in focus.

By setting the prediction method as described above in each of the case where the subject is in focus and the case where the subject is not in focus, the subject is in focus regardless of the degree of focus (the subject is in focus). In some cases and out of focus), the increase in prediction error can be suppressed.

Further, for example, the above-mentioned parameters related to focusing may be derived from the pixel values used for the phase difference detection for the focusing. By doing so, it is possible to more easily suppress the increase in the prediction error by utilizing the phase difference detection. For example, by utilizing the pixel structure for detecting the phase difference of the image pickup unit (imaging element), it is possible to more easily suppress the increase in the prediction error.

For example, as in "Method 1" of Table 10 shown in FIG. 1, the prediction method may be controlled according to the phase difference detection result. That is, for example, the result of phase difference detection performed using the pixel value of the captured image data (that is, the pixel value of the decoded image) obtained by decoding the encoded data generated by encoding the captured image data is described above. It may be used as a parameter related to focusing, and a prediction method applied to coding or decoding may be set.

By doing so, for example, the prediction method applied to the coding and decoding can be more easily obtained by utilizing the result of the phase difference detection for the focal length control. That is, it is possible to suppress an increase in the load of coding and decoding due to the processing related to the setting of this prediction method. Further, since it is not necessary to transmit the prediction method information indicating the prediction method applied in the coding to the decoding side, it is possible to suppress the reduction of the coding efficiency. Further, by using the result of the phase difference detection for the focal length control, the prediction method can be set more accurately according to the degree of focusing. That is, it is possible to further suppress an increase in prediction error.

Further, for example, as in the “method 2” of Table 10 shown in FIG. 1, the prediction method may be controlled according to the captured image. That is, for example, the result of prediction by each prediction method performed by using the pixel value used for the phase difference detection for focusing of the captured image data is used as the above-mentioned parameter related to focusing and applied to coding. The prediction method may be set.

By doing so, for example, it is possible to set a prediction method applied to coding from captured image data. That is, it is possible to set a prediction method that is more easily applied to coding without requiring feedback of the decoded image. That is, it is possible to suppress an increase in the coding load due to the processing related to the setting of this prediction method.

Further, in that case, for example, the prediction method set as such may be transmitted to the decoding side. For example, the prediction method information indicating the prediction method applied to the coding may be added to the coded data and transmitted to the decoding side. In other words, the prediction method information added to the coded data may be used as the above-mentioned focusing parameter to set the prediction method applied to decoding.

By doing so, it is possible to suppress an increase in the amount of calculation required for setting the prediction method applied to decoding. That is, it is possible to more easily set a prediction method applied to decoding. That is, it is possible to suppress an increase in the decoding load due to the processing related to the setting of this prediction method.

Further, in the case of this "method 2", as in the "method 2-1" of Table 10 shown in FIG. 1, the above-mentioned prediction method information (prediction method applied to the coding of the captured image data) is further applied. Based on this, phase difference detection (for focal length control) may be performed. For example, the prediction method information may be used as auxiliary information to detect the phase difference of the decoded image. By doing so, the phase difference detection process can be performed more easily. For example, based on the prediction method information, if the subject is not in focus at all, the focal length control accuracy is coarsened, and if the subject is substantially in focus, the accuracy is made finer to focus more efficiently. The distance can be controlled (the focal length control load and processing time can be reduced, or the focal length can be controlled with higher accuracy).

Further, as in "Method 3" of Table 10 shown in FIG. 1, the phase difference (for focal length control) is based on the error information indicating the error due to the coding and decoding of the captured image data as described above. May be detected. For example, the error generated by encoding the captured image data and decoding the encoded data (error information indicating the error) is used as reliability information indicating the reliability of the phase difference detection result, and the phase difference of the decoded image is used. Detection may be performed. For example, the phase difference detection result derived by using a pixel value having a large error caused by coding / decoding is likely to have a relatively low reliability. Therefore, based on this reliability information, the pixel value in the portion having a large error may be controlled so as not to be used for phase difference detection. By doing so, the focal length can be controlled more accurately.

As shown in "Method 3-1" of Table 10 shown in FIG. 1, this "method 3" can also be used in combination with the above-mentioned "method 1". Further, as shown in "Method 3-2" of Table 10 shown in FIG. 1, this "method 3" can also be used in combination with the above-mentioned "method 2" and "method 2-1".

<2. First Embodiment>
<Image processing device>
Next, each method of FIG. 1 will be described more specifically. In this embodiment, "Method 1" will be described. FIG. 2 is a block diagram showing an example of the configuration of an image processing apparatus to which the present technology is applied. The image processing device 100 shown in FIG. 2 is a device that encodes the input captured image data to generate encoded data, further decodes the encoded data to generate captured image data, and outputs the encoded image data. .. The image processing apparatus 100 applies "method 1" as a control method of a prediction method in coding / decoding of captured image data. As shown in FIG. 2, the image processing apparatus 100 includes a coding unit 101, a decoding unit 102, a phase difference detection unit 103, and a prediction method setting unit 104.

The coding unit 101 performs processing related to image coding. For example, the coding unit 101 acquires the captured image data input to the image processing device 100. Further, the coding unit 101 encodes the captured image data obtained by using the prediction by the prediction method set based on the parameters related to focusing. That is, the coding unit 101 encodes by applying the prediction method specified by the prediction method setting unit 104. Therefore, the coding unit 101 can more easily set the prediction method applied to the coding. That is, it is possible to suppress an increase in the load of the coding process.

Although the details will be described later, the coding unit 101 encodes the captured image data by a method simpler than general image coding such as so-called AVC or HEVC. Hereinafter, the coding by the coding unit 101 is also referred to as simple coding.

The coding unit 101 supplies the coded data to the decoding unit 102 via a recording medium or a transmission medium. For example, the coding unit 101 records the coded data on a recording medium such as a semiconductor memory or a hard disk. Further, for example, the coding unit 101 transmits the coded data to the decoding unit 102 via a transmission path such as a bus.

The decoding unit 102 performs processing related to image decoding. For example, the decoding unit 102 acquires the coded data supplied from the coding unit 101 via the recording medium or the transmission medium. For example, the decoding unit 102 reads out the coded data recorded by the coding unit 101 from a recording medium such as a semiconductor memory or a hard disk. Further, for example, the decoding unit 102 receives the coded data transmitted from the coding unit 101 via a transmission line such as a bus. Further, the decoding unit 102 decodes the acquired coded data by using the prediction by the prediction method set based on the parameters related to focusing. That is, the decoding unit 102 decodes by applying the prediction method set by the prediction method setting unit 104. Therefore, the decoding unit 102 can apply the same prediction method that was applied at the time of coding even if the coded data does not include information about the prediction method. Therefore, it is possible to suppress a decrease in coding efficiency.

Although the details will be described later, the decoding unit 102 decodes the encoded data in which the captured image data is encoded by a method simpler than general image decoding such as so-called AVC or HEVC. Hereinafter, the decoding by the decoding unit 102 is also referred to as a simple decoding.

The decoding unit 102 outputs the generated captured image data to the outside of the image processing device 100. Further, the decoding unit 102 supplies the captured image data to the phase difference detection unit 103.

The phase difference detection unit 103 performs processing related to detection of the phase difference. For example, the phase difference detection unit 103 acquires captured image data (decoding result) supplied from the decoding unit 102. Further, the phase difference detection unit 103 detects the phase difference for focusing (to the subject) by using the acquired image data. The captured image data is generated by an image pickup element capable of detecting the phase difference on the image plane, and the phase difference detection unit 103 can detect the phase difference from the captured image data. Further, the phase difference detection unit 103 supplies the derived phase difference detection result to the prediction method setting unit 104.

The prediction method setting unit 104 performs processing related to the setting of the prediction method. For example, the prediction method setting unit 104 acquires the phase difference detection result supplied from the phase difference detection unit 103. Further, the prediction method setting unit 104 sets a prediction method applied to coding or decoding. For example, the prediction method setting unit 104 may set the prediction method based on the parameters related to focusing. For example, the prediction method setting unit 104 may set the acquired phase difference detection result as a parameter related to focusing, and set the prediction method based on the parameter.

For example, the prediction method setting unit 104 may set the prediction method based on the parameter depending on whether or not the subject is in focus. By doing so, the prediction method setting unit 104 can set the prediction method according to the degree of focusing.

For example, when the prediction method setting unit 104 is in focus on the subject, as the prediction method, the pixel value of the pixel located closest to the processing target pixel among the pixels of the same color as the processing target pixel is predicted. The prediction method may be set. By doing so, the prediction method setting unit 104 can suppress an increase in the prediction error in a state where the subject is in focus.

Further, for example, when the prediction method setting unit 104 is not in focus on the subject, the position of the photodiode that detects the pixel value is the same as the processing target pixel and the same color as the processing target pixel as the prediction method. A prediction method may be set in which the pixel value of the pixel located closest to the processing target pixel among the pixels of the above is used as the prediction value. By doing so, the prediction method setting unit 104 can suppress an increase in the prediction error in a state where the subject is not in focus.

The prediction method setting unit 104 may set the prediction method as described above in each of the case where the subject is in focus and the case where the subject is not in focus. By doing so, the prediction method setting unit 104 can suppress an increase in the prediction error regardless of the degree of focusing (whether the subject is in focus or not). ..

Further, for example, the prediction method setting unit 104 may set the prediction method by using the result of the phase difference detection for focusing as the parameter related to the above-mentioned focusing. That is, even if the prediction method setting unit 104 uses the detection result of the phase difference performed by the phase difference detection unit 103 as the parameter related to the above-mentioned focusing, the prediction method applied to coding or decoding is set. good. By doing so, the prediction method setting unit 104 can more easily obtain a prediction method applied to coding and decoding by using, for example, the result of phase difference detection for focal length control. .. That is, the prediction method setting unit 104 can suppress an increase in the load of coding and decoding due to the processing related to the setting of the prediction method. Further, since the prediction method setting unit 104 does not need to transmit the prediction method information indicating the prediction method applied in the coding to the decoding side, it is possible to suppress the reduction of the coding efficiency. Further, the prediction method setting unit 104 can set the prediction method more accurately according to the degree of focusing by using the result of the phase difference detection for the focal length control. That is, the prediction method setting unit 104 can further suppress the increase in the prediction error.

Further, the prediction method setting unit 104 supplies the set prediction method to the coding unit 101 and the decoding unit 102.

Each processing unit (encoding unit 101 to prediction method setting unit 104) described above may be configured as one device (for example, an image processing device 100), or may be configured as a plurality of devices. You may do it. For example, the coding unit 101 and the decoding unit 102 may be configured as different devices from each other. In that case, the phase difference detection unit 103 and the prediction method setting unit 104 may be provided in the device having the coding unit 101, may be provided in the device having the decoding unit 102, or may be provided in the device. It may be provided in a device different from the device having the conversion unit 101 and the device having the decoding unit 102. Further, the phase difference detection unit 103 and the prediction method setting unit 104 may be provided in different devices. In that case, for example, the phase difference detection unit 103 may be provided in the device having the decoding unit 102, and the prediction method setting unit 104 may be provided in the device having the coding unit 101. Further, either the phase difference detection unit 103 or the prediction method setting unit 104 may be provided in a device different from the device having the coding unit 101 and the device having the decoding unit 102. Further, the coding unit 101 and the prediction method setting unit 104 may be configured as different devices from each other.

<Code-coded part>
FIG. 3 is a block diagram showing a main configuration example of the coding unit 101 of FIG. As shown in FIG. 3, the coding unit 101 in this case includes a DPCM (Differential Pulse Code Modulation) processing unit 121, a Golomb coding unit 122, and a compression rate adjusting unit 123.

The DPCM processing unit 121 performs processing related to DPCM. For example, the DPCM processing unit 121 acquires captured image data input to the image processing device 100. Further, the DPCM processing unit 121 acquires the prediction method set by the prediction method setting unit 104. The DPCM processing unit 121 performs DPCM processing using such information. In DPCM processing, the pixel value of the pixel to be processed is predicted using the pixel value of the peripheral pixel (that is, the predicted value of the pixel value of the pixel to be processed is derived), and the predicted value and the pixel value of the pixel to be processed are used. It is a process to obtain the difference (prediction error) of.

The DPCM processing unit 121 performs such processing on the acquired captured image data to derive a prediction error. For example, the DPCM processing unit 121 performs such DPCM processing for each block of a predetermined size of captured image data. In that case, the DPCM processing unit 121 processes the pixel value of the first pixel of the block by PCM processing and outputs it (that is, outputs it as it is), and obtains a prediction error for the other pixels as described above. Since the pixel value processed by PCM can also be said to be a prediction error from the predicted value 0 (initial value), in the following, the processing results will be referred to as a prediction error without distinction. By deriving the prediction error (difference value) in this way, the amount of information can be reduced.

Further, the DPCM processing unit 121 may perform DPCM processing on only some (upper) bits of each pixel value as described above. By doing so, the pixel value can be quantized. For example, the DPCM processing unit 121 may perform DPCM processing on the upper 5 bits with respect to a pixel value having a bit length of 10 bits. By doing so, the DPCM processing unit 121 can further reduce the amount of information.

Then, in such DPCM processing, the DPCM processing unit 121 applies the prediction method set by the prediction method setting unit 104. That is, the DPCM processing unit 121 derives the predicted value of the processing target pixel using the peripheral pixels designated by the prediction method setting unit 104. By doing so, the DPCM processing unit 121 derives a predicted value using peripheral pixels set based on the parameters related to focusing (for example, depending on whether or not the subject is in focus). Can be done. Therefore, the DPCM processing unit 121 can derive the predicted value by the prediction method according to the degree of focusing.

The DPCM processing unit 121 supplies the predicted values derived as described above and the captured image data before the DPCM processing to the Golomb coding unit 122.

The Golomb coding unit 122 performs processing related to Golomb coding. For example, the Golomb coding unit 122 acquires the prediction error and the captured image data supplied from the DPCM processing unit 121. The Golomb coding unit 122 encodes the acquired prediction error into a Golomb Coding. By such processing, the Golomb coding unit 122 can generally reduce the amount of information. The Golomb coding unit 122 supplies the Golomb code (encoded data) and the captured image data to the compression rate adjusting unit 123.

The compression rate adjusting unit 123 performs processing related to the adjustment of the compression rate. For example, the compression rate adjusting unit 123 acquires the Golomb code and the captured image data supplied from the Golomb coding unit 122. The compression rate adjusting unit 123 controls the code amount (that is, the compression rate) of the Golomb code by using the captured image data.

The compression rate adjusting unit 123 extracts information of an arbitrary number of bits that have not been DPCM processed by the DPCM processing unit 121 from the captured image data, and adds it as a refinement to the acquired Golomb code to obtain a code amount. That is, the compression ratio is adjusted. For example, the compression rate adjusting unit 123 can make the code amount of each block a fixed length (that is, realize fixed length coding) by adding the refinement in this way.

The compression ratio adjusting unit 123 outputs the Golomb code (Golomb code having the adjusted amount of code) to which the refinement is added as described above to the outside of the coding unit 101 as coding data. This coded data is transmitted to the decoding unit 102 via, for example, a recording medium or a transmission medium (FIG. 2).

As described above, the DPCM processing unit 121 applies the prediction method set by the prediction method setting unit 104 to perform the DPCM processing, so that the coding unit 101 spatially correlates the pixels according to the degree of focusing. The captured image data can be simply coded so as to suppress the change in. Therefore, the coding unit 101 can suppress an increase in the prediction error regardless of the degree of focusing. As a result, in the case of variable-length coding, the coding unit 101 can suppress a decrease in coding efficiency. Further, in the case of fixed-length coding, the coding unit 101 can suppress a decrease in the image quality of the decoded image.

<Decoding unit>
FIG. 4 is a block diagram showing a main configuration example of the decoding unit 102 of FIG. As shown in FIG. 4, the decoding unit 102 in this case includes a compression rate reverse adjustment unit 131, a Golomb decoding unit 132, and a reverse DPCM (Differential Pulse Code Modulation) processing unit 133.

The compression rate reverse adjustment unit 131 performs the reverse processing of the processing performed by the compression rate adjustment unit 123 of the coding unit 101. For example, the compression rate reverse adjustment unit 131 acquires the coded data supplied from the coding unit 101 to the decoding unit 102. Further, the compression rate reverse adjustment unit 131 removes the refinement added by the compression rate adjustment unit 123 from the coded data. In other words, the compressibility reverse adjustment unit 131 extracts the Golomb code generated by the Golomb coding unit 122 from the acquired coded data. The compression rate reverse adjustment unit 131 supplies the Golomb code to the Golomb decoding unit 132.

The Golomb decoding unit 132 performs a process related to decoding the Golomb code. For example, the Golomb decoding unit 132 acquires the Golomb code supplied from the compression rate reverse adjustment unit 131. Further, the Golomb decoding unit 132 decodes the acquired Golomb code by a method corresponding to the coding method of the Golomb coding unit 122 (also referred to as Golomb decoding) to generate a prediction error. The Golomb decoding unit 132 supplies the generated prediction error to the inverse DPCM processing unit 133.

The reverse DPCM processing unit 133 performs processing related to the reverse DPCM processing, which is the reverse processing of the DPCM processing performed by the DPCM processing unit 121. For example, the inverse DPCM processing unit 133 acquires a prediction error (difference value) supplied from the Golomb decoding unit 132. Further, the reverse DPCM processing unit 133 performs reverse DPCM processing on the acquired prediction error and restores each pixel data.

At that time, the reverse DPCM processing unit 133 applies the prediction method set by the prediction method setting unit 104. That is, the reverse DPCM processing unit 133 restores each pixel data from the prediction error by using the peripheral pixels designated by the prediction method setting unit 104. By doing so, the inverse DPCM processing unit 133 derives a predicted value using peripheral pixels set based on the parameters related to focusing (for example, depending on whether or not the subject is in focus). , The predicted value can be used to recover the pixel data from the prediction error. Therefore, the inverse DPCM processing unit 133 can correctly restore the pixel data from the prediction error derived by using the prediction value of the prediction method according to the degree of focusing.

The reverse DPCM processing unit 133 outputs the pixel data derived as described above to the outside of the decoding unit 102 as captured image data. The captured image data is, for example, output to the outside of the image processing device 100 or supplied to the phase difference detection unit 103 (FIG. 2).

As described above, the reverse DPCM processing unit 133 applies the prediction method set by the prediction method setting unit 104 to perform the reverse DPCM processing, so that the decoding unit 102 spatially focuses the pixels according to the degree of focusing. The captured image data simply encoded so as to suppress the change in correlation can be correctly and simply decoded. Therefore, the decoding unit 102 can suppress an increase in the prediction error. As a result, in the case of variable-length coding, the decoding unit 102 can suppress a decrease in coding efficiency. Further, in the case of fixed-length coding, the decoding unit 102 can suppress a decrease in the image quality of the decoded image.

<Captured image data>
Next, the captured image data will be described. The captured image data input to the image processing device 100 is composed of pixel values capable of detecting a phase difference. This captured image data is generated using, for example, the pixel array 140 shown in FIG. A Bayer array color filter is arranged in each pixel 141 of the pixel array 140. For example, the rectangle of the pixel array 140 indicates the pixel 141. The pixel 141-1 shown in white in the figure is a pixel in which a red (R) color filter is arranged, and the pixel 141-2 shown in light gray in the figure is a pixel in which a green (G) color filter is arranged. The pixels 141-3, which are shown in dark gray in the figure, are the pixels in which the blue (B) color filter is arranged. The circle in each pixel 141 indicates the on-chip lens 142.

As shown in FIG. 6A, each pixel 141 is provided with two photoelectric conversion elements (for example, a photodiode) of the left photoelectric conversion element 151 and the right photoelectric conversion element 152. The left photoelectric conversion element 151 is provided on the left side of the figure of the pixel 141, and the right photoelectric conversion element 152 is provided on the right side of the figure of the pixel 141.

As shown in B of FIG. 6, the light from the subject obliquely incident from the right side in the figure as shown by the arrow 156 is received mainly by the left photoelectric conversion element 151 via the on-chip lens 142 and the color filter 154. (Photoelectrically converted and detected as a pixel value). On the other hand, the light from the subject obliquely incident from the left side in the figure as shown by the arrow 157 is received mainly by the right photoelectric conversion element 152 via the on-chip lens 142 and the color filter 154 (photoelectric conversion). Is detected as a pixel value).

That is, the left photoelectric conversion element 151 and the right photoelectric conversion element 152 mainly detect incident light incident at different angles. That is, there is a difference (parallax) in the angle deviation of the incident light between the pixel value of the left photoelectric conversion element 151 and the pixel value of the right photoelectric conversion element 152. The phase difference (image shift amount) can be detected by using such parallax. By adding the pixel value of the left photoelectric conversion element 151 and the pixel value of the right photoelectric conversion element 152, the pixel value for one pixel can be obtained. Each pixel value of the captured image data input to the image processing device 100 is composed of such a pixel value of the left photoelectric conversion element 151 and a pixel value of the right photoelectric conversion element 152.

<Prediction method>
<First prediction method>
Next, the prediction method applied in the DPCM processing by the DPCM processing unit 121 will be described. FIG. 7 is a diagram illustrating the first prediction method. Pixels 141-1 to 141-8 shown in FIG. 7 are pixels of the pixel array 140 of FIG. 5 in which the red (R) color filter is arranged and the green (G) color filter is arranged. A part of the line 141 is shown. Pixels 141-1, pixels 141-3, pixels 141-5, and pixels 141-7 are pixels 141 in which a red (R) color filter is arranged, and pixels 141-2, 141-4, and 141. -6 and pixels 141-8 are pixels 141 in which a green (G) color filter is arranged.

As described above, each pixel 141 is provided with a left photoelectric conversion element 151 and a right photoelectric conversion element 152, and as shown in FIG. 7, the captured image data is a pixel value detected by each photoelectric conversion element. Has. R _L indicates the pixel value detected in the left photoelectric conversion element 151 of the pixel 141 in which the red (R) color filter is arranged, and R _R is the pixel 141 in which the red (R) color filter is arranged. The pixel value detected in the right photoelectric conversion element 152 is shown. Similarly, G _L indicates the pixel value detected in the left photoelectric conversion element 151 of the pixel 141 in which the green (G) color filter is arranged, and G _R indicates the pixel value in which the green (G) color filter is arranged. The pixel value detected by the right photoelectric conversion element 152 of the pixel 141 is shown.

In general, the prediction accuracy of a predicted value is higher as the filter color (transmitted wavelength range) of the pixel (predicted pixel) that detects the pixel value to be the predicted value is closer to the filter color of the pixel to be processed. Become. That is, in the case of the RGB color filter as described above, when the pixel having the same color filter as the processing target pixel is used as the predicted pixel, the pixel having the color filter having a color different from the processing target pixel is referred to as the predicted pixel. The prediction accuracy is the highest.

Further, in general, the prediction accuracy of the predicted value becomes higher as the position of the predicted pixel is closer to the processing target pixel.

Therefore, in the first prediction method, the closest pixel to the processing target pixel among the pixels of the same color as the processing target pixel (processed pixel) can be expected to improve the prediction accuracy based on these characteristics. The pixel value of the pixel located at (the closest pixel of the same color) is used as the predicted value. In the case of the example of FIG. 7, when DPCM processing the pixel value R _R of the pixel 141-1, the DPCM processing unit 121 refers to the pixel value R _L of the pixel 141-1 as a predicted value as shown by the arrow 161-1. do. Further, when the pixel value R _L of the pixel 141-3 is DPCM processed, the DPCM processing unit 121 refers to the pixel value R _R of the pixel 141-1 as a predicted value as shown by the arrow 161-2, and the pixel 141- When the pixel value R _R of 3 is subjected to DPCM processing, the pixel value R _L of the pixel 141-3 is referred to as a predicted value as shown by the arrow 161-3. Similarly, when the pixel value R _L of the pixel 141-5 is DPCM processed, the DPCM processing unit 121 refers to the pixel value R _R of the pixel 141-3 as a predicted value as shown by the arrow 161-4, and the pixel 141 When the pixel value R _R of −5 is DPCM processed, the pixel value R _L of the pixel 141-5 is referred to as a predicted value as shown by arrows 161-5. Similarly, when the pixel value R _L of the pixel 141-7 is DPCM processed, the DPCM processing unit 121 refers to the pixel value R _R of the pixel 141-5 as a predicted value as shown by the arrow 161-6, and the pixel 141 When the pixel value R _R of -7 is subjected to DPCM processing, the pixel value R _L of the pixel 141-7 is referred to as a predicted value as shown by arrows 161-7.

Further, when the pixel value G _R of the pixel 141-2 is DPCM processed, the DPCM processing unit 121 refers to the pixel value G _L of the pixel 141-2 as a predicted value as shown by arrow 162-1. Further, when the pixel value G _L of the pixel 141-4 is DPCM processed, the DPCM processing unit 121 refers to the pixel value G _R of the pixel 141-2 as a predicted value as shown by the arrow 162-2, and the pixel 141- When the pixel value G _R of 4 is subjected to DPCM processing, the pixel value G _L of the pixel 141-4 is referred to as a predicted value as shown by the arrow 162-3. Similarly, when the pixel value G _L of the pixel 141-6 is DPCM processed, the DPCM processing unit 121 refers to the pixel value G _R of the pixel 141-4 as a predicted value as shown by the arrow 162-4, and the pixel 141 When the pixel value G _R of -6 is subjected to DPCM processing, the pixel value G _L of the pixels 141-6 is referred to as a predicted value as shown by the arrow 162-5. Similarly, when the pixel value G _L of the pixel 141-8 is DPCM processed, the DPCM processing unit 121 refers to the pixel value G _R of the pixel 141-6 as a predicted value as shown by the arrow 162-6, and the pixel 141 When the pixel value G _R of −8 is DPCM processed, the pixel value G _L of the pixel 141-8 is referred to as a predicted value as shown by the arrow 162-7.

<Second prediction method>
FIG. 8 is a diagram illustrating a second prediction method. Pixels 141-1 to 141-8 shown in FIG. 8 indicate pixels similar to those in FIG. 7. The pixel value R _L , the pixel value R _R , the pixel value G _L , and the pixel value G _R are also the same as in FIG. 7.

As described above, in general, the prediction accuracy of the predicted value becomes higher as the color of the filter of the predicted pixel (transmissive wavelength range) is closer to the color of the filter of the pixel to be processed.

Further, as described above, in the left photoelectric conversion element 151 and the right photoelectric conversion element 152, the incident direction of the light mainly detected differs from each other depending on the position in the pixel. Therefore, in general, the prediction accuracy of the predicted value is such that the position in the predicted pixel of the photoelectric conversion element that detects the predicted value is close to the position in the processing target pixel of the photoelectric conversion element that detects the pixel value to be processed. The higher it gets. That is, in the case of the example of FIG. 8, when the pixel value of the left photoelectric conversion element 151 is the processing target, it is better to use the pixel value of the left photoelectric conversion element 151 as the predicted value to predict the pixel value of the right photoelectric conversion element 152. The prediction accuracy is higher than when it is a value. On the contrary, when the pixel value of the right photoelectric conversion element 152 is targeted for processing, the pixel value of the right photoelectric conversion element 152 as the predicted value is better than the pixel value of the left photoelectric conversion element 151 as the predicted value. , The prediction accuracy is high.

Therefore, in the second prediction method, the position of the photodiode (photoelectric conversion element) that detects the pixel value to be processed is the pixel to be processed so that the prediction accuracy can be expected to be improved based on these characteristics. The pixel value of the pixel that is the same and is located closest to the pixel to be processed (the nearest pixel of the same color and the same arrangement) among the pixels of the same color as the pixel to be processed is used as the predicted value.

In the case of the example of FIG. 8, when the DPCM processing unit 121 performs DPCM processing on the pixel value R _L of the pixel 141-3, the pixel value R _L of the pixel 141-1 is referred to as a predicted value as shown by the arrow 163-1. Then, when the pixel value R _R of the pixel 141-3 is subjected to DPCM processing, the pixel value R _R of the pixel 141-1 is referred to as a predicted value as shown by the arrow 163-2. Similarly, when the pixel value R _L of the pixel 141-5 is DPCM processed, the DPCM processing unit 121 refers to the pixel value R _L of the pixel 141-3 as a predicted value as shown by the arrow 163-3, and the pixel 141 When the pixel value R _R of −5 is DPCM processed, the pixel value R _R of the pixel 141-3 is referred to as a predicted value as shown by the arrow 163-4. Similarly, when the pixel value R _L of the pixel 141-7 is DPCM processed, the DPCM processing unit 121 refers to the pixel value R _L of the pixel 141-5 as a predicted value as shown by the arrow 163-5, and the pixel 141 When the pixel value R _R of -7 is subjected to DPCM processing, the pixel value R _R of the pixel 141-5 is referred to as a predicted value as shown by the arrow 163-6.

Further, when the pixel value G _L of the pixel 141-4 is DPCM processed, the DPCM processing unit 121 refers to the pixel value G _L of the pixel 141-2 as a predicted value as shown by the arrow 164-1, and the pixel 141- When the pixel value G _R of 4 is subjected to DPCM processing, the pixel value G _R of the pixel 141-2 is referred to as a predicted value as shown by the arrow 164-2. Similarly, when the pixel value G _L of the pixel 141-6 is DPCM processed, the DPCM processing unit 121 refers to the pixel value G _L of the pixel 141-4 as a predicted value as shown by the arrow 164-3, and the pixel 141 When the pixel value G _R of -6 is subjected to DPCM processing, the pixel value G _R of the pixel 141-4 is referred to as a predicted value as shown by the arrow 164-4. Similarly, when the pixel value G _L of the pixel 141-8 is DPCM processed, the DPCM processing unit 121 refers to the pixel value G _L of the pixel 141-6 as a predicted value as shown by the arrow 164-5, and the pixel 141 When the pixel value G _R of −8 is DPCM processed, the pixel value G _R of the pixel 141-6 is referred to as a predicted value as shown by the arrow 164-6.

<Comparison of prediction accuracy>
The prediction accuracy is compared between the first prediction method and the second prediction method as described above. When the subject is in focus (when the distance to the subject and the distance to the in-focus position are the same or close to each other), the difference between the pixel value of the left photoelectric conversion element 151 and the pixel value of the right photoelectric conversion element 152 is small. The effect of the misalignment on the prediction accuracy is smaller than when the subject is not in focus (the distance to the subject and the distance to the in-focus position are far apart). That is, the influence of the reference distance (distance between the processing target pixel and the predicted pixel) is larger than the influence of the parallax. Therefore, the first prediction method having a shorter reference distance has higher prediction accuracy than the second prediction method.

On the other hand, when the subject is not in focus, the difference between the pixel value of the left photoelectric conversion element 151 and the pixel value of the right photoelectric conversion element 152 is large, and the influence of parallax on the prediction accuracy is focused on the subject. It will be larger than if it is. That is, the effect of parallax is greater than the effect of reference distance. Since the second prediction method using the pixel value detected by the photoelectric conversion elements having the same in-pixel position as the prediction value is not affected by this parallax, the second prediction method is better than the first prediction method. Prediction accuracy is high.

<Control of prediction method>
The prediction method setting unit 104 selects the first prediction method and the second prediction method as candidates, and sets one of these as a prediction method to be applied to coding / decoding based on the phase detection result. You may do it. In that case, the prediction method setting unit 104 sets the prediction method according to whether or not the subject is in focus based on the characteristics of each prediction method as described above (whether the first prediction method is applied). Select whether to apply the second prediction method).

For example, when the subject is in focus, the prediction method setting unit 104 may set the first prediction method as the prediction method. That is, in this case, a prediction method is set in which the pixel value of the pixel located closest to the processing target pixel among the pixels of the same color as the processing target pixel is used as the prediction value. By performing the DPCM processing by the prediction method set in this way by the DPCM processing unit 121, the coding unit 101 can suppress an increase in the prediction error in a state where the subject is in focus.

Further, when the prediction method setting unit 104 is not in focus on the subject, a second prediction method may be set as the prediction method. That is, in this case, the position of the photodiode that detects the pixel value is the same as that of the pixel to be processed, and the pixel value of the pixel located closest to the pixel to be processed among the pixels of the same color as the pixel to be processed is set. The prediction method used as the predicted value is set. By performing the DPCM processing by the prediction method set in this way by the DPCM processing unit 121, the coding unit 101 can suppress an increase in the prediction error in a state where the subject is not in focus.

Further, the prediction method setting unit 104 may set the prediction method as described above in each of the case where the subject is in focus and the case where the subject is not in focus. By performing the DPCM processing by the DPCM processing unit 121 by the prediction method set in this way, the coding unit 101 is not in focus regardless of the degree of focusing (even when the subject is in focus). In some cases), the increase in prediction error can be suppressed.

In the above, the prediction of the coding unit 101 (DPCM processing unit 121) has been described as an example, but the same applies to the decoding unit 102 (reverse DPCM processing unit 133). That is, the reverse DPCM processing unit 133 performs the reverse DPCM processing by the prediction method set in this way, so that the decoding unit 102 is in the state of being in focus on the subject and in the state of not being in focus on the subject. Or, regardless of the degree of focusing (whether the subject is in focus or not), an increase in prediction error can be suppressed.

<Other configuration examples of pixel array>
The captured image data is not limited to the above-mentioned configuration example as long as it can detect the phase difference. For example, the captured image data may be generated by a pixel array in which four photoelectric conversion elements are arranged in one pixel as shown in FIG.

FIG. 9 shows a row of pixels 141 in which a part of the pixel array 140 (a red (R) color filter is arranged and a pixel 141 in which a green (G) color filter is arranged), as in FIGS. 7 and 8. A configuration example of (part) is shown.

In this case, each pixel 141 is provided with photoelectric conversion elements at the upper left, upper right, lower left, and lower right of the pixel, and as shown in FIG. 9, the captured image data is the respective photoelectric conversion elements. Has a pixel value detected in. R ₀₀ indicates the pixel value detected by the photoelectric conversion element arranged in the upper left of the pixel 141 in which the red (R) color filter is arranged, and R ₀₁ indicates the pixel value detected by the red (R) color filter. The pixel value detected by the photoelectric conversion element arranged in the upper right of the pixel 141 is shown, and R ₁₀ is the photoelectric arranged in the lower left of the pixel 141 in which the red (R) color filter is arranged. The pixel value detected in the conversion element is shown, and R ₁₁ shows the pixel value detected in the photoelectric conversion element arranged in the lower right of the pixel 141 in which the red (R) color filter is arranged.

Similarly, G ₀₀ indicates the pixel value detected by the photoelectric conversion element arranged in the upper left of the pixel 141 in which the green (G) color filter is arranged, and G ₀₁ is the green (G) color. The pixel value detected by the photoelectric conversion element arranged in the upper right of the pixel 141 in which the filter is arranged is shown, and G ₁₀ is arranged in the lower left of the pixel 141 in which the green (G) color filter is arranged. The pixel value detected in the photoelectric conversion element is shown, and G ₁₁ indicates the pixel value detected in the photoelectric conversion element arranged in the lower right of the pixel 141 in which the green (G) color filter is arranged. show.

In this case, the pixel values (pixel value R ₀₀ , pixel value R ₀₁ , pixel value G ₀₀ , pixel value G ₀₁ ) in the upper row of FIG. 9 are the same as those described with reference to FIGS. 7 and 8. Predicted values are set and DPCM processed. On the other hand, the pixel values (pixel value R ₁₀ , pixel value R ₁₁ , pixel value G ₁₀ , pixel value G ₁₁ ) in the lower row of FIG. 9 are processed as follows.

<First prediction method>
First, the first prediction method will be described with reference to FIG. As shown in A of FIG. 10, when the DPCM processing unit 121 performs DPCM processing on the pixel value R ₁₀ of the pixel 141-3, the pixel value R ₀₁ and the pixel value R ₁₁ of the pixel 141-1 and the pixel 141 Refer to the pixel value R ₀₀ of -3, and use those pixel values to derive the predicted value (Prediction (R ₁₀ )) of the pixel value R ₁₀ of the pixel 141-3 as shown in the following equation (1). do.

Prediction (R ₁₀ ) = P2 (R ₁₁ , R ₀₀ , R ₁₀ ) ・ ・ ・ (1)

In the equation (1), P2 (a, b, c) is, for example, as shown in FIG. 11, the pixel value a on the left (left) and the pixel on the top (top) of the pixel value X to be processed. Using the value b and the upper right (top-left) pixel value c, it is derived as shown in the following equation (2).

P2 (a, b, c) = Clip ((a + bc), min (a, b), max (a, b))
= Clip ((R ₁₁ + R ₀₀ -R ₀₁ ), min (R ₁₁ , R ₀₀ ), max (R ₁₁ , R ₀₀ ))
... (2)

That is, if (R ₁₁ + R ₀₀ -R ₀₁ ) is smaller than the smaller of R ₁₁ and R ₀₀ , the predicted value is set to the smaller of R ₁₁ and R ₀₀ . If (R ₁₁ + R ₀₀ -R ₀₁ ) is larger than the larger of R ₁₁ and R ₀₀ , the predicted value is set to the larger of R ₁₁ and R ₀₀ . Furthermore, if (R ₁₁ + R ₀₀ -R ₀₁ ) is greater than the smaller of R ₁₁ and R ₀₀ and smaller than the larger of R ₁₁ and R ₀₀ , then the predicted value is (R ₁₁ + R). ₀₀ -R ₀₁ ) is set.

Similarly, as shown in B of FIG. 10, when the DPCM processing unit 121 performs DPCM processing on the pixel value R ₁₁ of the pixel 141-3, the pixel value R ₀₀ , the pixel value R ₁₀ and the pixel value R 10 of the pixel 141-3 are processed. With reference to the pixel value R ₀₁ , the predicted value (Prediction (R ₁₁ )) of the pixel value R ₁₁ of the pixel 141-3 is derived by using those pixel values as in the following equation (3).

Prediction (R ₁₁ ) = P2 (R ₁₀ , R ₀₁ , R ₀₀ )
= Clip ((R ₁₀ + R ₀₁ -R ₀₀ ), min (R ₁₀ , R ₀₁ ), max (R ₁₀ , R ₀₁ ))
... (3)

That is, if (R ₁₀ + R ₀₁ -R ₀₀ ) is smaller than the smaller of R ₁₀ and R ₀₁ , the predicted value is set to the smaller of R ₁₀ and R ₀₁ . Also, if (R ₁₀ + R ₀₁ -R ₀₀ ) is larger than the larger of R ₁₀ and R ₀₁ , the predicted value is set to the larger of R ₁₀ and R ₀₁ . Furthermore, if (R ₁₀ + R ₀₁ -R ₀₀ ) is greater than the smaller of R ₁₀ and R ₀₁ and smaller than the larger of R ₁₀ and R ₀₁ , then the predicted value is (R ₁₀ + R). ₀₁ -R ₀₀ ) is set.

Further, as shown in C of FIG. 10, when the DPCM processing unit 121 performs DPCM processing on the pixel value G ₁₀ of the pixel 141-4, the pixel value G ₀₁ and the pixel value G ₁₁ of the pixel 141-2, and the pixel value G 11 and the like. With reference to the pixel value G ₀₀ of the pixel 141-4 and using those pixel values, as in the following equation (4) (that is, as in the case of the above equation (1)), the pixel 141-4 Derivation of the predicted value (Prediction (G ₁₀ )) of the pixel value G ₁₀ .

Prediction (G ₁₀ ) = P2 (G ₁₁ , G ₀₀ , G ₁₀ )
= Clip ((G ₁₁ + G ₀₀ -G ₁₀ ), min (G ₁₁ , G ₀₀ ), max (G ₁₁ , G ₀₀ ))
... (4)

That is, if (G ₁₁ + G ₀₀ -G ₀₁ ) is smaller than the smaller of G ₁₁ and G ₀₀ , the predicted value is set to the smaller of G ₁₁ and G ₀₀ . Also, if (G ₁₁ + G ₀₀ -G ₀₁ ) is larger than the larger of G ₁₁ and G ₀₀ , the predicted value is set to the larger of G ₁₁ and G ₀₀ . Furthermore, if (G ₁₁ + G ₀₀ -G ₀₁ ) is greater than the smaller of G ₁₁ and G ₀₀ and smaller than the larger of G ₁₁ and G ₀₀ , then the predicted value is (G ₁₁ + G). ₀₀ -G ₀₁ ) is set.

Similarly, as shown in D of FIG. 10, when the pixel value G ₁₁ of the pixel 141-4 is DPCM processed, the DPCM processing unit 121 has a pixel value G ₀₀ , a pixel value G ₁₀ and a pixel value G 10 of the pixel 141-4. With reference to the pixel value G ₀₁ , using those pixel values, as in the following equation (5) (that is, as in the case of the above equation (3)), the pixel value G ₁₁ of the pixel 141-4 Derivation of the predicted value (Prediction (G ₁₁ )).

Prediction (G ₁₁ ) = P2 (G ₁₀ , G ₀₁ , G ₀₀ )
= Clip ((G ₁₀ + G ₀₁ -G ₀₀ ), min (G ₁₀ , G ₀₁ ), max (G ₁₀ , G ₀₁ ))
... (5)

That is, if (G ₁₀ + G ₀₁ -G ₀₀ ) is smaller than the smaller of G ₁₀ and G ₀₁ , the predicted value is set to the smaller of G ₁₀ and G ₀₁ . Also, if (G ₁₀ + G ₀₁ -G ₀₀ ) is larger than the larger of G ₁₀ and G ₀₁ , the predicted value is set to the larger of G ₁₀ and G ₀₁ . Furthermore, if (G ₁₀ + G ₀₁ -G ₀₀ ) is greater than the smaller of G ₁₀ and G ₀₁ and smaller than the larger of G ₁₀ and G ₀₁ , then the predicted value is (G ₁₀ + G). ₀₁ -G ₀₀ ) is set.

<Second prediction method>
Next, the second prediction method will be described with reference to FIG. As shown in A of FIG. 12, when the DPCM processing unit 121 performs DPCM processing on the pixel value R ₁₀ of the pixel 141-3, the pixel value R ₀₀ and the pixel value R ₁₀ of the pixel 141-1 and the pixel 141 Refer to the pixel value R ₀₀ of -3, and use those pixel values to derive the predicted value (Prediction (R ₁₀ )) of the pixel value R ₁₀ of the pixel 141-3 as shown in the following equation (6). do. Also in the case of the equation (6), P2 (a, b, c) is derived as in the above equation (2), for example.

Prediction (R ₁₀ ) = P2 (R ₁₀ , R ₀₀ , R ₀₀ )
= Clip ((R ₁₀ + R ₀₀ -R ₀₀ ), min (R ₁₀ , R ₀₀ ), max (R ₁₀ , R ₀₀ ))
... (6)

That is, if (R ₁₀ + R ₀₀ -R ₀₀ ) is smaller than the smaller of R ₁₁ and R ₀₀ , the predicted value is set to the smaller of R ₁₁ and R ₀₀ . If (R ₁₀ + R ₀₀ -R ₀₀ ) is larger than the larger of R ₁₁ and R ₀₀ , the predicted value is set to the larger of R ₁₁ and R ₀₀ . Furthermore, if (R ₁₀ + R ₀₀ -R ₀₀ ) is greater than the smaller of R ₁₁ and R ₀₀ and smaller than the larger of R ₁₁ and R ₀₀ , then the predicted value is (R ₁₀ + R). ₀₀ -R ₀₀ ) is set.

Similarly, as shown in B of FIG. 12, when the DPCM processing unit 121 performs DPCM processing on the pixel value R ₁₁ of the pixel 141-4, the pixel value R ₀₁ , the pixel value R ₁₁ and the pixel value R 11 of the pixel 141-2 are processed. , Pixel value R ₀₁ of pixel 141-4, and using those pixel values, as shown in the following equation (7), the predicted value of pixel value R ₁₁ of pixel 141-4 (Prediction (R ₁₁ )). ) Is derived. Also in the case of the equation (7), P2 (a, b, c) is derived as in the above equation (2), for example.

Prediction (R ₁₁ ) = P2 (R ₁₁ , R ₀₁ , R ₀₁ )
= Clip ((R ₁₁ + R ₀₁ -R ₀₁ ), min (R ₁₁ , R ₀₁ ), max (R ₁₁ , R ₀₁ ))
... (7)

That is, if (R ₁₁ + R ₀₁ -R ₀₁ ) is smaller than the smaller of R ₁₁ and R ₀₁ , the predicted value is set to the smaller of R ₁₁ and R ₀₁ . Also, if (R ₁₁ + R ₀₁ -R ₀₁ ) is larger than the larger of R ₁₁ and R ₀₁ , the predicted value is set to the larger of R ₁₁ and R ₀₁ . Furthermore, if (R ₁₁ + R ₀₁ -R ₀₁ ) is greater than the smaller of R ₁₁ and R ₀₁ and smaller than the larger of R ₁₁ and R ₀₁ , then the predicted value is (R ₁₁ + R). ₀₁ -R ₀₁ ) is set.

Further, as shown in C of FIG. 12, when the DPCM processing unit 121 performs DPCM processing on the pixel value G ₁₀ of the pixel 141-4, the pixel value G ₀₀ and the pixel value G ₁₀ of the pixel 141-2, and the pixel value G 10 and so on. With reference to the pixel value G ₀₀ of the pixel 141-4 and using those pixel values, as in the following equation (8) (that is, as in the case of the above equation (6)), the pixel 141-4 Derivation of the predicted value (Prediction (G ₁₀ )) of the pixel value G ₁₀ .

Prediction (G ₁₀ ) = P2 (G ₁₀ , G ₀₀ , G ₀₀ )
= Clip ((G ₁₀ + G ₀₀ -G ₀₀ ), min (G ₁₀ , G ₀₀ ), max (G ₁₀ , G ₀₀ ))
... (8)

That is, if (G ₁₀ + G ₀₀ -G ₀₀ ) is smaller than the smaller of G ₁₀ and G ₀₀ , the predicted value is set to the smaller of G ₁₀ and G ₀₀ . Also, if (G ₁₀ + G ₀₀ -G ₀₀ ) is larger than the larger of G ₁₀ and G ₀₀ , the predicted value is set to the larger of G ₁₀ and G ₀₀ . Furthermore, if (G ₁₀ + G ₀₀ -G ₀₀ ) is greater than the smaller of G ₁₀ and G ₀₀ and smaller than the larger of G ₁₀ and G ₀₀ , then the predicted value is (G ₁₀ + G). ₀₀ -G ₀₀ ) is set.

Similarly, as shown in D of FIG. 12, when the pixel value G ₁₁ of the pixel 141-4 is DPCM processed, the DPCM processing unit 121 has the pixel value G ₀₁ and the pixel value G ₁₁ of the pixel 141-2, and the pixel value G 11. , Pixel _141-4 , and using those pixel values, as in the following equation (9) (that is, as in the case of the above equation (7)), pixel 141-4. Prediction (Prediction (G ₁₁ )) of the pixel value G ₁₁ of.

Prediction (G ₁₁ ) = P2 (G ₁₁ , G ₀₁ , G ₀₁ )
= Clip ((G ₁₁ + G ₀₁ -G ₀₁ ), min (G ₁₁ , G ₀₁ ), max (G ₁₁ , G ₀₁ ))
... (9)

That is, if (G ₁₁ + G ₀₁ -G ₀₁ ) is smaller than the smaller of G ₁₁ and G ₀₁ , the predicted value is set to the smaller of G ₁₁ and G ₀₁ . Also, if (G ₁₁ + G ₀₁ -G ₀₁ ) is larger than the larger of G ₁₁ and G ₀₁ , the predicted value is set to the larger of G ₁₁ and G ₀₁ . Furthermore, if (G ₁₁ + G ₀₁ -G ₀₁ ) is larger than the smaller of G ₁₁ and G ₀₁ and smaller than the larger of G ₁₁ and G ₀₁ , the predicted value is (G ₁₁ + G). ₀₁ -G ₀₁ ) is set.

<Comparison of prediction accuracy>
In this case as well, the characteristics of the first prediction method and the second prediction method are the same as in the case where the two photoelectric conversion elements of each pixel described above are provided.

<Control of prediction method>
Therefore, in this case as well, the prediction method setting unit 104 may use the first prediction method and the second prediction method as candidates, and may provide the two photoelectric conversion elements of each pixel described above based on the phase detection result. Similarly, any one of these may be set as a prediction method applied to coding / decoding. By doing so, the same effect as the above-mentioned example can be obtained.

In the above, the equation (2) is given as an example of the operation for deriving the predicted value, but the method for deriving the predicted value is arbitrary and is not limited to this example.

<Flow of image processing>
Next, an example of the flow of image processing executed by the image processing apparatus 100 of FIG. 2 will be described with reference to the flowchart of FIG.

When the image processing is started, the prediction method setting unit 104 initializes the setting of the prediction method applied to the coding / decoding in step S101 of FIG.

In step S102, the coding unit 101 simply encodes the captured image data using the prediction method currently set by the prediction method setting unit 104, and generates the encoded data of the captured image data.

In step S103, the decoding unit 102 simply decodes the coded data generated in step S102 using the prediction method currently set by the prediction method setting unit 104, and generates captured image data.

In step S104, the phase difference detection unit 103 detects the phase difference using the captured image data generated in step S103.

In step S105, the prediction method setting unit 104 uses the phase difference detection result obtained in step S104 as a parameter related to focusing, and sets a prediction method to be applied to coding / decoding based on the phase difference detection result.

For example, the prediction method setting unit 104 sets the prediction method to be applied to coding / decoding among the candidates (for example, the first prediction method and the second prediction method) depending on whether or not the subject is in focus. Select from. For example, when the prediction method setting unit 104 determines that the subject is in focus based on the phase detection result, the first prediction method is selected and it is determined that the subject is not in focus. , Select the second prediction method.

In step S106, the coding unit 101 determines whether or not all the frames have been processed. When it is determined that an unprocessed frame exists, that is, a new frame of captured image data is input to the image processing apparatus 100, the process returns to step S102, and the subsequent processes are repeated.

When the processing of steps S102 to S106 is executed for each frame and it is determined in step S106 that all the frames of the input captured image data have been processed, the image processing ends.

<Flow of simple coding process>
Next, an example of the flow of the simple coding process executed in step S102 of FIG. 13 will be described with reference to the flowchart of FIG.

When the simple coding process is started, the DPCM processing unit 121 performs DPCM processing on the captured image data in step S121 to generate residual data (prediction error) between the captured image data and the predicted value.

At that time, the DPCM processing unit 121 DPCM processes the captured image data using the prediction method set by the prediction method setting unit 104 (the prediction method set in step S101 or step S105 in FIG. 13). For example, the DPCM processing unit 121 DPCM processes the captured image data by the first prediction method or the second prediction method set according to whether or not the subject is in focus.

In step S122, the Golomb coding unit 122 Golom-codes the residual data (prediction error) obtained by the process of step S121.

In step S123, the compression rate adjusting unit 123 adjusts the compression rate of the coded data by, for example, quantizing the data or adding the data to the Golomb code obtained by the process of step S122.

When the coded data having a predetermined compression rate is obtained by the process of step S123, the simple coding process is completed, and the process returns to FIG.

<Flow of simple decryption process>
Next, an example of the flow of the simple decoding process executed in step S103 of FIG. 13 will be described with reference to the flowchart of FIG.

When the simple decoding process is started, in step S131, the compression rate reverse adjustment unit 131 compresses by performing reverse adjustment of the compression rate of the coded data (that is, reverse processing of the process of step S123 in FIG. 14). Restore the Golomb code before adjusting the rate.

In step S132, the Golomb decoding unit 132 Golom-decodes each Golomb code obtained by the process of step S131, and restores the residual data (prediction error).

In step S133, the reverse DPCM processing unit 133 performs reverse DPCM processing (that is, reverse processing of the processing in step S121 of FIG. 14) using the residual data obtained by the processing in step S132. That is, the reverse DPCM processing unit 133 restores each pixel value by adding residual data to each other and generates captured image data.

At that time, the reverse DPCM processing unit 133 performs reverse DPCM processing of the residual data using the prediction method set by the prediction method setting unit 104 (the prediction method set in step S101 or step S105 of FIG. 13). For example, the reverse DPCM processing unit 133 performs reverse DPCM processing of residual data by a first prediction method or a second prediction method set according to whether or not the subject is in focus.

When the captured image data is obtained by the process of step S133, the simple decoding process is completed, and the process returns to FIG.

By executing each process as described above, it is possible to suppress an increase in prediction error in coding / decoding. As a result, in the case of variable-length coding, it is possible to suppress a decrease in coding efficiency. Further, in the case of fixed-length coding, it is possible to suppress a decrease in the image quality of the decoded image.

<Image pickup device>
The present technology of "Method 1" as described above can be applied to any electronic device. For example, it can be applied to an image pickup device. FIG. 16 is a block diagram showing a main configuration example of an image pickup apparatus to which the present technology is applied. The image pickup apparatus 170 shown in FIG. 16 captures a subject, generates captured image data, records it, and outputs it.

As shown in FIG. 16, the image pickup apparatus 170 includes a lens 171, an image sensor 172, a signal processing unit 173, an output unit 174, and a storage unit 175.

The lens 171 is an optical system controlled by the lens control unit 192 of the signal processing unit 173, which will be described later, for adjusting the focal length. It may include an aperture or the like. The light from the subject is incident on the image sensor 172 through the lens 171.

The image sensor 172 receives light from a subject incident on the lens 171 and generates captured image data. Further, the image sensor 172 simply encodes the captured image data and supplies the captured image data to the signal processing unit 173 as encoded data.

The signal processing unit 173 decodes the coded data supplied from the image sensor 172 and generates the captured image data. The signal processing unit 173 supplies the captured image data to the output unit 174 and the storage unit 175. Further, the signal processing unit 173 detects the phase difference from the captured image data, controls the lens 171 based on the phase difference detection result, and controls the focal length. Further, the signal processing unit 173 controls the prediction method applied in the coding / decoding based on the phase difference detection result.

The output unit 174 acquires the captured image data supplied from the signal processing unit 173 and outputs it to the outside of the image pickup device 170. For example, the output unit 174 has a display unit (monitor) and displays the captured image on the display unit. Further, for example, the output unit 174 has an external output terminal, and supplies captured image data to another device via the external output terminal.

The storage unit 175 acquires the captured image data supplied from the signal processing unit 173 and stores it. For example, the storage unit 175 has a storage medium such as a hard disk or a flash memory, and stores the captured image data supplied from the signal processing unit 173 in the storage medium.

The storage unit 175 can appropriately read the captured image data stored in the storage medium and supply it to another processing unit such as the output unit 174. Further, the storage medium may be a medium (removable media) that can be attached to and detached from the image pickup apparatus 170 (storage unit 175).

<Image sensor>
The image sensor 172 includes a light receiving unit 181, an ADC (Analog Digital Converter) 182, a coding unit 101, and a transmitting unit 183.

The light receiving unit 181 has, for example, a pixel array 140 as described with reference to FIGS. 5 and 6, and receives light from a subject, performs photoelectric conversion, and detects each pixel value. The light receiving unit 181 supplies each detected pixel value to the ADC 182.

The ADC 182 A / D-converts an electric signal (pixel value) supplied from the light receiving unit 181 to generate captured image data of digital data. As described with reference to FIGS. 7 to 12, the captured image data has a data structure in which each pixel has a plurality of pixel values and the phase difference can be detected using them. The ADC 182 supplies the generated captured image data to the coding unit 101.

The coding unit 101 acquires the captured image data supplied from the ADC 182. Further, the coding unit 101 acquires information regarding the setting of the prediction method supplied from the prediction method setting unit 104 of the signal processing unit 173 (information indicating the prediction method set by the prediction method setting unit 104). Similar to the case of the image processing apparatus 100, the coding unit 101 has a configuration as shown in FIG. 3, for example, and performs a simple coding process as described with reference to the flowchart of FIG. 14 to obtain an captured image. The data is simply encoded and the encoded data is generated. At that time, the coding unit 101 performs simple coding based on the information supplied from the prediction method setting unit 104 (that is, using the prediction method set by the prediction method setting unit 104). The coding unit 101 supplies the generated coded data to the transmission unit 183.

The transmission unit 183 acquires the coded data supplied from the coding unit 101 and transmits it to the signal processing unit 173 (reception unit 191).

<Signal processing unit>
The signal processing unit 173 includes a receiving unit 191, a decoding unit 102, a phase difference detection unit 103, a prediction method setting unit 104, a lens control unit 192, and an image processing unit 193.

The receiving unit 191 receives the coded data transmitted from the transmitting unit 183 of the image sensor 172. The receiving unit 191 supplies the received coded data to the decoding unit 102.

The decoding unit 102 acquires the coded data supplied from the receiving unit 191. Further, the decoding unit 102 acquires information regarding the setting of the prediction method supplied from the prediction method setting unit 104 (information indicating the prediction method set by the prediction method setting unit 104). Similar to the case of the image processing apparatus 100, the decoding unit 102 has a configuration as shown in FIG. 4, for example, and performs a simple decoding process as described with reference to the flowchart of FIG. 15 to obtain encoded data. Simple decoding is performed to generate captured image data. At that time, the decoding unit 102 uses the information supplied from the prediction method setting unit 104 (the prediction method set by the prediction method setting unit 104, that is, the prediction method applied by the coding unit 101 to the simple coding). Use) to perform a simple decryption. The decoding unit 102 supplies the generated captured image data to the phase difference detection unit 103 and the image processing unit 193.

The phase difference detection unit 103 acquires the captured image data supplied from the decoding unit 102. The phase difference detection unit 103 detects the phase difference for focusing (to the subject) by using the captured image data (decoding result) supplied from the decoding unit 102, as in the case of the image processing device 100. conduct. The captured image data is generated by an image pickup element capable of detecting the phase difference on the image plane, and the phase difference detection unit 103 can detect the phase difference from the captured image data. Further, the phase difference detection unit 103 supplies the derived phase difference detection result to the prediction method setting unit 104 and the lens control unit 192.

The prediction method setting unit 104 acquires the phase difference detection result supplied from the phase difference detection unit 103. The prediction method setting unit 104 sets a prediction method applied to coding or decoding based on the acquired phase difference detection result, as in the case of the image processing device 100. The prediction method setting unit 104 supplies information indicating the set prediction method to the coding unit 101 and the decoding unit 102.

The lens control unit 192 acquires the phase difference detection result supplied from the phase difference detection unit 103. The lens control unit 192 generates a control signal for controlling the lens 171 so as to focus on the subject based on the phase difference detection result, and supplies the control signal to the lens 171. That is, the lens control unit 192 controls the lens 171 so as to focus on the subject.

The image processing unit 193 acquires the captured image data supplied from the decoding unit 102. The image processing unit 193 performs image processing on the captured image data. The content of this image processing is arbitrary. The image processing unit 193 supplies the captured image data after image processing to at least one of the output unit 174 and the storage unit 175. Of course, the image processing unit 193 can also skip the image processing. In that case, the image processing unit 193 supplies the acquired captured image data to at least one of the output unit 174 and the storage unit 175.

As described above, the captured image data is simply encoded and transmitted between the image sensor 172 of the image pickup apparatus 170 and the signal processing unit 173. Then, the coding unit 101 uses a prediction method set by the prediction method setting unit 104 based on the phase difference detection result (for example, a prediction method selected depending on whether or not the subject is in focus). Simple encoding of captured image data is performed. Similarly, the decoding unit 102 uses a prediction method set by the prediction method setting unit 104 based on the phase difference detection result (for example, a prediction method selected depending on whether or not the subject is in focus). Performs simple decoding of coded data.

Therefore, the image pickup apparatus 170 can suppress an increase in prediction error in this simple coding / simple decoding. As a result, in the case of variable-length coding, it is possible to suppress a decrease in coding efficiency. Further, in the case of fixed-length coding, it is possible to suppress a decrease in the image quality of the decoded image.

Further, since the prediction method applied to the coding and decoding is obtained by using the result of the phase difference detection for the focal length control, the image pickup apparatus 170 is loaded with coding and decoding by the processing related to the setting of the prediction method. Can be suppressed from increasing. Further, since it is not necessary to transmit the prediction method information indicating the prediction method applied in the coding from the coding unit 101 to the decoding unit 102, the image pickup apparatus 170 can suppress the reduction of the coding efficiency. Further, the image pickup apparatus 170 can more accurately set the prediction method according to the degree of focusing by using the result of the phase difference detection for the focal length control. That is, the image pickup apparatus 170 can further suppress the increase in the prediction error.

<Flow of imaging process>
An example of the flow of the image pickup process executed by the image pickup apparatus 170 to image the subject will be described with reference to the flowcharts of FIGS. 17 and 18.

When the imaging process is started, in step S151, the lens 171 initializes the in-focus position. In step S152, the prediction method setting unit 104 initializes the prediction method setting.

In step S153, the light receiving unit 181 takes an image of the subject, photoelectrically converts the light from the subject, and obtains each pixel value as an electric signal. In step S154, the ADC 182 A / D-converts the pixel value to generate captured image data of digital data.

In step S155, the coding unit 101 executes a simple coding process as described with reference to the flowchart of FIG. 14, and obtains captured image data by the prediction method set in step S152 or step S164 (FIG. 18). Simple encoding is performed and encoded data is generated.

In step S156, the transmission unit 183 transmits the coded data generated in step S155. In step S161 of FIG. 18, the receiving unit 191 receives the coded data transmitted in step S156.

In step S162, the decoding unit 102 executes the simple decoding process as described with reference to the flowchart of FIG. 15, and is the same as the prediction method set in step S152 or step S164 (that is, the same as applied in step S155). The coded data is simply decoded by the prediction method), and the captured image data is generated.

In step S163, the phase difference detection unit 103 detects the phase difference using the captured image data generated in step S162. In step S164, the prediction method setting unit 104 sets the prediction method applied to the coding / decoding based on the phase difference detection result obtained in step S163.

In step S165, the lens control unit 192 controls the lens 171 based on the phase difference detection result, and updates the in-focus position. In step S166, the image processing unit 193 performs image processing on the captured image data generated in step S162.

In step S167, the output unit 174 outputs the captured image data after the image processing in step S166 to the outside of the image pickup device 170. Further, the storage unit 175 stores the captured image data in the storage medium.

In step S168, the image pickup apparatus 170 determines whether or not to end the image pickup process. If it is determined that the imaging is continued and a new frame image is generated, the process returns to step S153 in FIG. That is, the image pickup apparatus 170 executes each process of step S153 to step S168 for each frame. For example, if it is determined in step S168 that the imaging process is to be completed based on a user instruction or the like, the imaging process is terminated.

By performing the image pickup process as described above, the image pickup apparatus 170 can suppress an increase in the prediction error in the simple coding in step S155 (FIG. 17) and the simple decoding in step S162 (FIG. 18).

<3. Second Embodiment>
<Image processing device>
In the present embodiment, "method 2" will be described. FIG. 19 is a block diagram showing an example of the configuration of an image processing apparatus to which the present technology is applied. Similar to the image processing device 100, the image processing device 200 shown in FIG. 19 encodes the input captured image data to generate coded data, and further decodes the coded data to generate captured image data. It is a device that outputs data. However, the image processing apparatus 200 applies "method 2" as a control method of a prediction method in coding / decoding of the captured image data. As shown in FIG. 19, the image processing device 200 has a coding unit 201 and a decoding unit 102.

The coding unit 201 performs processing related to image coding. For example, the coding unit 201 acquires the captured image data input to the image processing device 200. Further, the coding unit 201 simply encodes the acquired captured image data in the same manner as the coding unit 101. At that time, the coding unit 201 encodes the captured image data by using the prediction by the prediction method set based on the parameters related to focusing. However, the coding unit 201 sets a prediction method based on the captured image data, and applies the prediction method set by itself to perform simple coding. For example, the coding unit 201 makes predictions by each prediction method using captured image data, and uses the result of each prediction as a parameter related to focusing to set the prediction method.

Similar to the coding unit 101, the coding unit 201 supplies the generated coded data to the decoding unit 202 via the recording medium or the transmission medium. However, the coding unit 201 adds the prediction method information indicating the set prediction method to the coded data generated by encoding the captured image data, and the coded data to which the prediction method information is added is added to the decoding unit 202. Supply. That is, in this case, the prediction method information is supplied from the coding unit 201 to the decoding unit 202.

The decoding unit 202 performs processing related to image decoding. For example, the decoding unit 202, like the decoding unit 102, acquires the coded data supplied from the coding unit 201 via the recording medium or the transmission medium. Further, the decoding unit 202, like the decoding unit 102, simply decodes the acquired coded data. At that time, the decoding unit 202 decodes the coded data by using the prediction by the prediction method set based on the parameters related to focusing. However, the decoding unit 202 performs simple decoding by applying the prediction method used at the time of coding, which is indicated by the prediction method information added to the coded data. The decoding unit 202 outputs the generated captured image data to the outside of the image processing device 100.

FIG. 20 is a diagram showing a main configuration example of the coded data. The coded data 210 shown in FIG. 20 is coded data generated by the coding unit 201 and supplied to the decoding unit 202. As shown in FIG. 20, the coded data 210 includes a header 211, prediction method information 212, VLC (Variable Length Code) 213, and refinement 214.

Various header information related to the coded data 210 is stored in the header 211. The prediction method information 212 is information indicating a prediction method (prediction method applied to coding) set by the coding unit 201. The VLC 213 is composed of coded data (for example, Golomb code) of captured image data. The refinement 214 is bit data added for adjusting the compression ratio.

In this way, the prediction method information 212 is added to the coded data of the captured image data and supplied from the coding unit 201 to the decoding unit 202. Therefore, the decoding unit 202 can apply the same prediction method applied to the coding to the decoding without receiving the information from the prediction method setting unit (only by acquiring the coded data).

The coding unit 201 and the decoding unit 202 described above may be configured as one device (for example, an image processing device 200) or may be configured as different devices from each other.

<Code-coded part>
FIG. 21 is a block diagram showing a main configuration example of the coding unit 201 of FIG. As shown in FIG. 21, the coding unit 201 in this case has a prediction method setting unit 221 and a coding unit 222.

The prediction method setting unit 221 acquires the captured image data input to the image processing device 200, and sets the prediction method based on the captured image data. The setting method of this prediction method is arbitrary. For example, the prediction method setting unit 221 may perform prediction by each prediction method prepared as a candidate using captured image data, and use the result of each prediction as a parameter related to focusing to set the prediction method. .. More specifically, for example, the prediction method setting unit 221 encodes the captured image data by applying each candidate of the prediction method, compares the coding efficiencies of each, and encodes with the best coding efficiency. You may choose the method. The prediction method setting unit 221 notifies the coding unit 222 of the set prediction method.

The coding unit 222 acquires the captured image data input to the image processing device 200. Further, the coding unit 222 simply encodes the acquired captured image data by applying the prediction method notified from the prediction method setting unit 221. In addition, the coding unit 222 generates prediction method information indicating the prediction method to which the prediction method is applied. The coding unit 222 adds the generated prediction method information to the generated coded data and supplies it to the decoding unit 202.

The coding unit 222 is a processing unit similar to the coding unit 101, and performs the same processing. That is, the coding unit 222 may have a configuration as shown in FIG. 3, for example, and may perform a simple coding process as described with reference to the flowchart of FIG.

As described in the first embodiment, when the subject is in focus, the first prediction method has higher prediction accuracy than the second prediction method, and the subject is not in focus. The second prediction method has higher prediction accuracy than the first prediction method. Since the prediction method setting unit 221 compares the prediction results of all the candidates and selects a prediction method having better coding efficiency, it can cope with such a change in prediction accuracy according to the degree of focusing. (It is possible to select a prediction method with better coding efficiency regardless of the degree of focusing). That is, the coding unit 222 (coding unit 201) can suppress an increase in the prediction error.

In addition, as in the case of the first embodiment, the prediction method setting unit 221 provides information based on the decoded image data (information indicating the prediction method set based on the decoded image data). ) Can be set without the need for a prediction method.

<Decoding unit>
FIG. 22 is a block diagram showing a main configuration example of the decoding unit 202 of FIG. As shown in FIG. 22, the decoding unit 202 has a prediction method setting unit 231 and a decoding unit 232.

The prediction method setting unit 231 acquires the coded data supplied from the coding unit 201. Prediction method information is added to this coded data. The prediction method setting unit 231 parses the acquired coded data and acquires the prediction method information. The prediction method setting unit 231 notifies the decoding unit 232 of the prediction method specified by the prediction method information (that is, the prediction method applied in the coding).

Upon receiving the notification, the decoding unit 232 simply decodes the coded data supplied from the prediction method setting unit 231 using the prediction method notified from the prediction method setting unit 231 to generate captured image data. By doing so, the decoding unit 232 can apply the prediction method applied in the coding to the decoding. The decoding unit 232 outputs the generated captured image data to the outside of the image processing device 200.

The decoding unit 232 is a processing unit similar to the decoding unit 102, and performs the same processing. That is, the decoding unit 232 may have a configuration as shown in FIG. 4, for example, and may perform a simple decoding process as described with reference to the flowchart of FIG.

Since the prediction method setting unit 231 sets the prediction method based on the prediction method information supplied from the coding unit 201, the decoding unit 232 can perform decoding by the prediction method applied to the coding. Therefore, the decoding unit 232 (decoding unit 202) can cope with a change in the prediction accuracy according to the degree of focusing, and can suppress an increase in the prediction error.

<Flow of image processing>
Next, an example of the flow of image processing executed by the image processing apparatus 200 of FIG. 19 will be described with reference to the flowchart of FIG. 23.

When the image processing is started, the coding unit 201 (prediction method setting unit 221) and the decoding unit 202 (prediction method setting unit 231) apply the prediction method to the coding / decoding in step S201 of FIG. Initialize the settings of.

In step S202, the coding unit 201 executes a coding process, sets a prediction method based on the captured image data, simply encodes the captured image data using the prediction method, and encodes the captured image data. To generate.

In step S203, the decoding unit 202 executes a decoding process, sets a prediction method based on the prediction method information, and simply decodes the coded data generated in step S202 using the prediction method, and captures an image. Generate data.

In step S204, the coding unit 201 determines whether or not all the frames have been processed. When it is determined that an unprocessed frame exists, that is, a new frame of captured image data is input to the image processing apparatus 200, the process returns to step S202, and the subsequent processes are repeated.

When the processing of steps S202 to S204 is executed for each frame and it is determined in step S204 that all the frames of the input captured image data have been processed, the image processing ends.

<Flow of coding process>
Next, an example of the flow of the coding process executed in step S202 of FIG. 23 will be described with reference to the flowchart of FIG. 24.

When the coding process is started, the prediction method setting unit 221 sets the prediction method based on the captured image data in step S211. In step S212, the coding unit 222 simply encodes the captured image data by the prediction method set in step S211 and generates the coded data. This simple coding is the same as the simple coding process described with reference to the flowchart of FIG.

In step S213, the coding unit 222 generates prediction method information indicating the prediction method applied to the simple coding, and adds it to the coding data generated in step S212. When the process of step S213 is completed, the coding process is completed, and the process returns to FIG. 23.

<Flow of decryption process>
Next, an example of the flow of the decoding process executed in step S203 of FIG. 23 will be described with reference to the flowchart of FIG.

When the decoding process is started, the prediction method setting unit 231 parses the coded data and extracts the prediction method information in step S221. In step S222, the prediction method setting unit 231 sets the prediction method based on the prediction method information extracted in step S221.

In step S223, the decoding unit 232 simply decodes the coded data by the prediction method set in step S222 and generates captured image data. This simple decoding is the same as the simple decoding process described with reference to the flowchart of FIG. When the process of step S223 is completed, the decoding process is completed, and the process returns to FIG. 23.

By executing each process as described above, it is possible to suppress an increase in prediction error in coding / decoding. As a result, in the case of variable-length coding, it is possible to suppress a decrease in coding efficiency. Further, in the case of fixed-length coding, it is possible to suppress a decrease in the image quality of the decoded image.

<Image pickup device>
The present technology of "method 2" as described above can be applied to any electronic device. For example, it can be applied to an image pickup device. FIG. 26 is a block diagram showing a main configuration example of an image pickup apparatus to which the present technology is applied. The image pickup device 250 shown in FIG. 26 is the same device as the image pickup device 170.

As shown in FIG. 26, the image pickup apparatus 250 includes a lens 251, an image sensor 252, a signal processing unit 253, an output unit 254, and a storage unit 255.

The lens 251 has the same configuration as the lens 171 and functions in the same manner. The image sensor 252 is a processing unit similar to the image sensor 172. The signal processing unit 253 is a processing unit similar to the signal processing unit 173. The output unit 254 is a processing unit similar to the output unit 174. The storage unit 255 is a processing unit similar to the storage unit 175.

However, the prediction method applied to the simple coding performed in the image sensor 252 is set in the image sensor 252, and the prediction method applied to the simple decoding performed in the signal processing unit 253 is supplied from the image sensor 172. It is set based on the prediction method information.

<Image sensor>
The image sensor 252 includes a light receiving unit 261 and an ADC 262, an encoding unit 201, and a transmitting unit 263.

The light receiving unit 261 is a processing unit similar to the light receiving unit 181 and has the same configuration and performs the same processing. The ADC 262 is a processing unit similar to the ADC 182, has a similar configuration, and performs the same processing. The transmission unit 263 is a processing unit similar to the transmission unit 183, has a similar configuration, and performs the same processing.

The coding unit 201 acquires the captured image data supplied from the ADC 262. Further, the coding unit 201 has a configuration as shown in FIG. 21, for example, as in the case of the image processing device 200, and performs coding processing as described with reference to the flowchart of FIG. 24. A prediction method is set from the captured image data, and the captured image data is simply encoded using the prediction method to generate encoded data. The coding unit 201 supplies the generated coded data to the transmission unit 263.

<Signal processing unit>
The signal processing unit 253 includes a receiving unit 271, a decoding unit 202, a phase difference detection unit 272, a lens control unit 273, and an image processing unit 274.

The receiving unit 271 is a processing unit similar to the receiving unit 191 and has the same configuration and performs the same processing. The phase difference detection unit 272 is a processing unit similar to the phase difference detection unit 103, has a similar configuration, and performs the same processing. The lens control unit 273 is a processing unit similar to the lens control unit 192, has a similar configuration, and performs the same processing. The image processing unit 274 is a processing unit similar to the image processing unit 193, has a similar configuration, and performs the same processing.

The decoding unit 202 acquires the coded data supplied from the receiving unit 271. Similar to the case of the image processing apparatus 200, the decoding unit 202 has a configuration as shown in FIG. 22, for example, and performs decoding processing as described with reference to the flowchart of FIG. 25 from the coded data. Prediction method information is extracted, a prediction method is set based on the prediction method information, coded data is simply decoded using the prediction method, and captured image data is generated. The decoding unit 202 supplies the generated captured image data to the phase difference detection unit 272 and the image processing unit 274.

As described above, the captured image data is simply encoded and transmitted between the image sensor 252 of the image pickup apparatus 250 and the signal processing unit 253. Then, the coding unit 201 and the decoding unit 202 set a prediction method as described above, and perform simple coding / simple decoding using the prediction method.

Therefore, the image pickup apparatus 250 can suppress an increase in prediction error in this simple coding / simple decoding. As a result, in the case of variable-length coding, it is possible to suppress a decrease in coding efficiency. Further, in the case of fixed-length coding, it is possible to suppress a decrease in the image quality of the decoded image.

Further, by doing so, for example, it is possible to set a prediction method applied to coding from captured image data. That is, it is possible to set a prediction method that is more easily applied to coding without requiring feedback of the decoded image. That is, it is possible to suppress an increase in the coding load due to the processing related to the setting of this prediction method.

<Flow of imaging process>
An example of the flow of the image pickup process executed by the image pickup apparatus 250 to image a subject will be described with reference to the flowcharts of FIGS. 27 and 28.

When the imaging process is started, each process of steps S241 to S244 is executed in the same manner as each process of steps S151 to S154 of FIG.

In step S245, the coding unit 201 executes the coding process as described with reference to the flowchart of FIG. 24, sets the prediction method based on the captured image data, and simply encodes the captured image data.

In step S246, the transmission unit 263 transmits the coded data to which the prediction method information is added, which was generated in step S245. In step S251 of FIG. 28, the receiving unit 271 receives the coded data to which the prediction method information is added, which was transmitted in step S246.

In step S252, the decoding unit 202 executes the decoding process as described with reference to the flowchart of FIG. 25, extracts the prediction method information from the coded data, and sets the prediction method based on the prediction method information. , The coded data is simply decoded using the prediction method, and the captured image data is generated.

Each process of steps S253 to S256 is executed in the same manner as each process of step S163 and steps S165 to S167 of FIG.

In step S257, the image pickup apparatus 250 determines whether or not to end the image pickup process. If it is determined that the imaging is continued and a new frame image is generated, the process returns to step S243 in FIG. 27. That is, the image pickup apparatus 250 executes each process of step S243 to step S257 for each frame. For example, if it is determined in step S257 to end the imaging process based on a user instruction or the like, the imaging process ends.

By performing the image pickup process as described above, the image pickup apparatus 250 can suppress an increase in the prediction error in the simple coding in step S245 (FIG. 27) and the simple decoding in step S252 (FIG. 28).

<4. Third Embodiment>
<Image processing device>
In this embodiment, "Method 2-1" will be described. FIG. 29 is a block diagram showing an example of the configuration of an image processing apparatus to which the present technology is applied. Similar to the image processing device 200, the image processing device 300 shown in FIG. 29 encodes the input captured image data to generate coded data, and further decodes the coded data to generate captured image data. It is a device that outputs data. Then, like the image processing device 200, the image processing device 300 applies "method 2" as a control method of a prediction method in coding / decoding of the captured image data. However, the image processing apparatus 300 generates auxiliary information for phase difference detection (also referred to as phase difference detection auxiliary information) of the decoded image based on the prediction method information, and the phase difference is detected using the phase difference detection auxiliary information. Perform detection. As shown in FIG. 29, the image processing device 300 has an auxiliary information generation unit 301, a phase difference detection unit 302, and a lens control unit 303 in addition to the configuration of the image processing device 200.

The auxiliary information generation unit 301 acquires the prediction method information (prediction method information extracted from the coded data by the decoding unit 202) from the decoding unit 202. The prediction method information indicates a prediction method applied to coding / decoding. Then, this prediction method is set according to the degree of focusing as described above. That is, the degree of focusing (possibility of focusing on the subject) can be easily estimated based on the prediction method applied to the coding / decoding. Therefore, the auxiliary information generation unit 301 generates phase difference detection auxiliary information indicating the degree of focusing (possibility of focusing on the subject) based on the prediction method information, and outputs the phase difference detection auxiliary information to the phase difference detection unit 302. Supply.

The phase difference detection unit 302 is a processing unit similar to the phase difference detection unit 103, detects the phase difference using the captured image data generated by the decoding unit 202, and supplies the phase difference detection result to the lens control unit 303. do. However, the phase difference detection unit 302 switches parameters according to the degree of focusing so that the detection accuracy does not decrease depending on the degree of focusing (whether or not the subject is in focus), and the detection accuracy is sufficient. In some cases, the process is simplified. The phase difference detection unit 302 performs such control based on the phase difference detection auxiliary information supplied from the auxiliary information generation unit 301. That is, the phase difference detection unit 302 switches the phase difference detection parameter or simplifies the process according to the degree of focusing estimated from the prediction method information (prediction method applied to coding / decoding). do.

By doing so, the image processing apparatus 300 can suppress a decrease in the accuracy of phase difference detection regardless of the degree of focusing. In addition, the phase difference detection process can be simplified so that the detection accuracy does not decrease, and the increase in the load can be suppressed.

The lens control unit 303 is a processing unit similar to the lens control unit 192, and controls a lens (not shown) based on the phase difference detection result supplied from the phase difference detection unit 302 to control the in-focus position.

<Auxiliary information generation unit>
FIG. 30 is a block diagram showing a main configuration example of the auxiliary information generation unit 301 of FIG. 29. As shown in FIG. 30, the auxiliary information generation unit 301 has a phase difference detection region statistical processing unit 311 and a threshold value processing unit 312.

Prediction method information is generated for each coded block. The phase difference detection area statistical processing unit 311 acquires the prediction method information for each coded block supplied from the decoding unit 202. Further, the phase difference detection area statistical processing unit 311 performs statistical processing of the entire frame using the prediction method information. For example, the phase difference detection area statistical processing unit 311 derives the probability of a predetermined prediction method for the entire frame. For example, the phase difference detection region statistical processing unit 311 derives the probability (N%) that the second prediction method is applied in the entire frame. The phase difference detection area statistical processing unit 311 supplies information indicating the probability (also referred to as a prediction method probability) to the threshold value processing unit 312.

The threshold value processing unit 312 acquires the prediction method probability supplied from the phase difference detection area statistical processing unit 311, and determines the degree of focusing (whether or not the subject is in focus) based on the prediction method probability. .. The method of this determination is arbitrary. For example, the threshold value processing unit 312 compares the value (N%) of the prediction method probability with the threshold value, and determines the degree of focusing (whether or not the subject is in focus) based on the comparison result. For example, the threshold processing unit 312 may not focus on the subject (out focus) when the probability N of the second prediction method selected when the subject is not in focus is larger than the predetermined threshold Th0. It is judged that the sex is large. Further, for example, when the probability N of the second prediction method is smaller than the predetermined threshold value Th1 (Th0> Th1), the threshold value processing unit 312 determines that there is a high possibility that the subject is in focus (In Focus). do. Further, for example, in other cases, that is, when the probability N of the second prediction method is the threshold value Th0 or less and the threshold value Th1 or more, the threshold value processing unit 312 states that the degree of focusing is unknown. judge.

The prediction method for deriving the probability is arbitrary and is not limited to the second prediction method. For example, it may be the first prediction method.

The threshold value processing unit 312 generates phase difference detection auxiliary information indicating the result of the determination as described above (that is, the determined degree of focusing), and supplies it to the phase difference detection unit 302.

<Flow of image processing>
Next, an example of the flow of image processing executed by the image processing apparatus 300 of FIG. 29 will be described with reference to the flowchart of FIG.

When the image processing is started, in step S301, the lens control unit 303 controls a lens (not shown) and initializes the in-focus position.

Each process of steps S302 to S304 is executed in the same manner as each process of steps S201 to S203 of FIG.

In step S305, the auxiliary information generation unit 301 executes the auxiliary information generation process, and generates the phase difference detection auxiliary information based on the prediction method information extracted from the coded data in step S304.

In step S306, the phase difference detection unit 302 detects the phase difference based on the captured image data generated in step S304 and the phase difference detection auxiliary information generated in step S305.

In step S307, the lens control unit 303 controls a lens (not shown) based on the phase difference detection result generated in step S306, and updates the in-focus position.

In step S308, the coding unit 201 determines whether or not all frames have been processed. When it is determined that an unprocessed frame exists, that is, a new frame of captured image data is input to the image processing apparatus 100, the process returns to step S303, and the subsequent processes are repeated.

When the processing of steps S303 to S308 is executed for each frame and it is determined in step S308 that all the frames of the input captured image data have been processed, the image processing ends.

<Flow of auxiliary information generation process>
Next, an example of the flow of the auxiliary information generation process executed in step S305 of FIG. 31 will be described with reference to the flowchart of FIG. 32.

When the auxiliary information generation process is started, the phase difference detection area statistical processing unit 311 performs statistical processing based on the prediction method information in step S321, and derives the probability that the second prediction method is selected.

In step S322, the threshold value processing unit 312 compares the probability of the second prediction method derived in step S321 with the threshold value, and determines the degree of focusing (possibility of focusing on the subject).

In step S323, the threshold value processing unit 312 outputs the determination result obtained in step S322 as phase difference detection auxiliary information.

When the process of step S323 is completed, the auxiliary information generation process is completed, and the process returns to FIG. 31.

By executing each process as described above, the image processing apparatus 300 can suppress a decrease in the accuracy of phase difference detection regardless of the degree of focusing. In addition, the phase difference detection process can be simplified so that the detection accuracy does not decrease, and the increase in the load can be suppressed.

<Image pickup device>
The present technology of "Method 2-1" as described above can be applied to any electronic device. For example, it can be applied to an image pickup device. FIG. 33 is a block diagram showing a main configuration example of an image pickup apparatus to which the present technology is applied. The image pickup apparatus 350 shown in FIG. 33 is the same apparatus as the image pickup apparatus 250.

As shown in FIG. 33, the image pickup apparatus 350 includes a lens 351, an image sensor 352, a signal processing unit 353, an output unit 354, and a storage unit 355.

The lens 351 has the same configuration as the lens 251 and functions in the same manner. The image sensor 352 is a processing unit similar to the image sensor 252. The signal processing unit 353 is a processing unit similar to the signal processing unit 353. The output unit 354 is a processing unit similar to the output unit 254. The storage unit 355 is a processing unit similar to the storage unit 255.

However, the phase difference detection performed by the signal processing unit 353 is performed by using the captured image data and the phase difference detection auxiliary information.

<Image sensor>
The image sensor 352 has a light receiving unit 361, an ADC 362, an encoding unit 201, and a transmitting unit 363.

The light receiving unit 361 is a processing unit similar to the light receiving unit 261 and has the same configuration and performs the same processing. The ADC 362 is a processing unit similar to the ADC 262, has a similar configuration, and performs the same processing. The coding unit 201 performs the same processing as in the case of the image processing apparatus 300. The transmission unit 363 is a processing unit similar to the transmission unit 263, has a similar configuration, and performs the same processing.

<Signal processing unit>
The signal processing unit 353 includes a receiving unit 371, a decoding unit 202, an auxiliary information generation unit 301, a phase difference detection unit 302, a lens control unit 303, and an image processing unit 372.

The receiving unit 371 is a processing unit similar to the receiving unit 271, has a similar configuration, and performs the same processing. The image processing unit 372 is a processing unit similar to the image processing unit 274, has a similar configuration, and performs the same processing.

The decoding unit 202 acquires the coded data supplied from the receiving unit 371. Similar to the case of the image processing apparatus 300, the decoding unit 202 has a configuration as shown in FIG. 22, for example, and performs decoding processing as described with reference to the flowchart of FIG. 25. For example, the decoding unit 202 extracts prediction method information from the coded data, sets a prediction method based on the prediction method information, simply decodes the coded data using the prediction method, and generates captured image data. do. The decoding unit 202 supplies the generated captured image data to the phase difference detection unit 302 and the image processing unit 372. Further, the decoding unit 202 supplies the extracted prediction method information to the auxiliary information generation unit 301.

The auxiliary information generation unit 301 has a configuration as shown in FIG. 30 and performs auxiliary information generation processing as described with reference to the flowchart of FIG. 32, as in the case of the image processing device 300. For example, the auxiliary information generation unit 301 acquires the prediction method information supplied from the decoding unit 202, and generates the phase difference detection auxiliary information based on the prediction method information. The auxiliary information generation unit 301 supplies the phase difference detection auxiliary information to the phase difference detection unit 302.

The phase difference detection unit 302 performs the same processing as in the case of the image processing device 300. For example, the phase difference detection unit 302 acquires captured image data from the decoding unit 202 and acquires phase difference detection auxiliary information from the auxiliary information generation unit 301. The phase difference detection unit 302 detects the phase difference by using the information. The phase difference detection unit 302 supplies the phase difference detection result to the lens control unit 303.

The lens control unit 303 performs the same processing as in the case of the image processing device 300. For example, the lens control unit 303 controls the lens 351 and controls the in-focus position based on the phase difference detection result supplied from the phase difference detection unit 302.

With such a configuration, the image pickup apparatus 350 can suppress a decrease in the accuracy of phase difference detection regardless of the degree of focusing, as in the case of the image processing apparatus 300. In addition, the phase difference detection process can be simplified so that the detection accuracy does not decrease, and the increase in the load can be suppressed.

<Flow of imaging process>
An example of the flow of the image pickup process executed by the image pickup apparatus 350 to image the subject will be described with reference to the flowcharts of FIGS. 34 and 35.

When the imaging process is started, each process of steps S341 to S346 of FIG. 34 is executed in the same manner as each process of steps S241 to S246 of FIG. 27, and the process proceeds to FIG. 35. Then, each process of step S351 and step S352 of FIG. 35 is executed in the same manner as each process of step S251 and step S252 of FIG. 28.

In step S353, the auxiliary information generation unit 301 executes the auxiliary information generation process as described with reference to the flowchart of FIG. 32, and generates the phase difference detection auxiliary information based on the prediction method information.

In step S354, the phase difference detection unit 302 detects the phase difference based on the captured image data generated in step S352 and the phase difference detection auxiliary information generated in step S353.

Each process of steps S355 to S357 is executed in the same manner as each process of steps S254 to S256 of FIG. 28.

In step S358, the image pickup apparatus 350 determines whether or not to end the image pickup process. If it is determined that the imaging is continued and a new frame image is generated, the process returns to step S343 in FIG. That is, the image pickup apparatus 350 executes each process of step S343 to step S358 for each frame. For example, if it is determined in step S358 to end the image pickup process based on a user instruction or the like, the image pickup process ends.

By performing the image pickup process as described above, the image pickup apparatus 350 can suppress an increase in the prediction error in the simple coding in step S345 (FIG. 34) and the simple decoding in step S352 (FIG. 35). Further, the image pickup apparatus 350 can suppress a decrease in the accuracy of phase difference detection regardless of the degree of focusing. In addition, the phase difference detection process can be simplified so that the detection accuracy does not decrease, and the increase in the load can be suppressed.

<5. Fourth Embodiment>
<Image processing device>
In the present embodiment, "method 3" will be described. FIG. 36 is a block diagram showing an example of the configuration of an image processing apparatus to which the present technology is applied. Similar to the image processing device 200, the image processing device 400 shown in FIG. 36 encodes the input captured image data to generate coded data, and further decodes the coded data to generate captured image data. It is a device that outputs data. Then, like the image processing device 200, the image processing device 400 applies "method 2" as a control method of a prediction method in coding / decoding of the captured image data. However, the image processing apparatus 400 derives the coding / decoding error information, and based on the error information, generates the phase difference detection result reliability information indicating the reliability of the phase difference detection result of the decoded image. Phase difference detection is performed using the phase difference detection result reliability information.

As shown in FIG. 36, the image processing device 400 has a reliability information generation unit 401, a phase difference detection unit 402, and a lens control unit 403 in addition to the configuration of the image processing device 200.

The decoding unit 202 extracts the prediction method information from the coded data, sets the prediction method based on the prediction method information, and decodes the coded data using the prediction method. In this case, the decoding unit 202 further derives the error information.

The error information is information indicating the amount of bits lost due to coding / decoding. For example, in FIG. 37, each square indicates a bit, a vertical position in the figure indicates a bit depth, and a horizontal position in the figure indicates a pixel position (horizontal position). That is, the vertical arrangement (column) in the figure of the square indicates the bit string of the pixel value.

Simple coding / decoding is processed in order from the left column (pixel value). As described above, in the simple coding, for example, a part of the bit string is DPCM processed. In the case of the example of FIG. 37, the upper 5 bits are DPCM processed. However, the leftmost pixel value processed first is PCM processed and output as it is (the difference value from 0 is output as a prediction error). The prediction error derived by DPCM processing (or PCM processing) is Golomb-coded. Then, several bits are added as refinement as appropriate. In the case of the example of FIG. 37, 1 bit or 2 bits are added as a refinement in each pixel value. That is, the coded data includes information about the above bits. In simple decoding, these reverse processes are performed and the values of these bits are restored.

In other words, the remaining bits (in the case of the example of FIG. 37, the uncoded bits represented by the white squares) are not transmitted and are lost. In the case of the example of FIG. 37, 3 bits or 4 bits are lost in each pixel (bit loss).

The error information is information indicating this bit amount, and is generated for each coded block, for example. That is, the decoding unit 202 aggregates the lost bit amount (bitloss) for each coded block (for example, calculates the total value or the average value), and supplies it to the reliability information generation unit 401 as error information. do. The data unit for generating the error information is arbitrary. For example, it may be information on a pixel-by-pixel basis.

The reliability information generation unit 401 acquires the error information from the decoding unit 202. The larger the value of the error information, the larger the amount of bit loss due to simple coding / decoding. The larger the bit loss amount, the larger the amount of information loss in the captured image data, and the higher the possibility that the accuracy of phase difference detection will decrease. That is, the larger the value of the error information, the lower the reliability of the phase difference detection result.

Therefore, the reliability information generation unit 401 determines the reliability of the phase difference detection result based on the acquired error information, generates the phase difference detection result reliability information indicating the reliability of the phase difference detection result, and generates the phase difference detection result reliability information. Is supplied to the phase difference detection unit 402. The reliability information generation unit 401 derives the phase difference detection result reliability information for each region (also referred to as a phase difference detection region) that is a processing unit for phase difference detection.

The phase difference detection unit 402 is a processing unit similar to the phase difference detection unit 103, detects the phase difference using the captured image data generated by the decoding unit 202, and supplies the phase difference detection result to the lens control unit 403. do. However, the phase difference detection unit 402 acquires the phase difference detection result reliability information supplied from the reliability information generation unit 401, and detects the phase difference in consideration of the phase difference detection result reliability information. That is, the phase difference detection unit 402 detects the phase difference based on the captured image data and the phase difference detection result reliability information (that is, error information).

For example, the phase difference detection unit 402 excludes the phase difference detection region determined to have a sufficiently low reliability in the phase difference detection result reliability information from the target of the phase difference detection. In other words, the phase difference detection unit 402 detects the phase difference by using the pixel value of the phase difference detection region determined to have sufficiently high reliability in the phase difference detection result reliability information.

By doing so, the phase difference detection unit 402 can suppress a decrease in the accuracy of the phase difference detection (typically, the phase difference can be detected more accurately).

<Reliability information generation unit>
FIG. 38 is a block diagram showing a main configuration example of the reliability information generation unit 401 of FIG. As shown in FIG. 38, the reliability information generation unit 401 has an integration unit 411 and a threshold value processing unit 412.

The integrating unit 411 acquires the error information supplied from the decoding unit 202, integrates the error information, and derives the integrated value for each phase difference detection region. The integration unit 411 supplies the derived integrated value for each phase difference detection region to the threshold value processing unit 412.

The threshold value processing unit 412 acquires the integrated value supplied from the integrating unit 411 and compares the integrated value with the threshold value to determine the reliability of the phase difference detection result in the phase difference detection region corresponding to the integrated value. judge. For example, the threshold value processing unit 412 determines whether or not the phase difference detection result is sufficiently reliable for each phase difference detection region. The threshold value processing unit 412 generates phase difference detection result reliability information indicating the determination result, and supplies it to the phase difference detection unit 402.

Based on such phase difference detection result reliability information, the phase difference detection unit 402 identifies, for example, a phase difference detection region in which the reliability of the phase difference detection result is sufficiently low, as shown in FIG. 39. The area is excluded from the target of phase difference detection. In the case of the example of FIG. 39, each square with a number or the like indicates a phase difference detection region. The square with "x" indicates the phase difference detection region in which the reliability of the phase difference detection result is determined to be sufficiently low based on the phase difference detection result reliability information (that is, error information). The phase difference detection unit 402 excludes such a region from the target and performs phase difference detection (that is, phase difference detection is performed in other regions).

In FIG. 39, the white portion is a region where the pixel value is flat (flat region), and such a region is also excluded from the target of phase difference detection because it is difficult to detect the phase difference. Further, in the figure, the squares shown in dark gray indicate the phase difference detection region where the phase difference detection result was inaccurate. That is, the phase difference detection unit 402 excludes the region where the reliability of the phase difference detection result is low from the target as described above, thereby excluding the region where the phase difference detection result is inaccurate from the target. Can be done. Therefore, the phase difference detection unit 402 can perform more accurate phase difference detection than the case where only the flat region is excluded.

<Flow of image processing>
Next, an example of the flow of image processing executed by the image processing apparatus 400 of FIG. 36 will be described with reference to the flowchart of FIG. 40.

When the image processing is started, the processing of steps S401 to S404 is executed in the same manner as the processing of steps S301 to S304.

In step S405, the decoding unit 202 calculates the amount of lost bits based on the captured image data generated in step S404, and generates error information.

In step S406, the reliability information generation unit 401 executes the reliability information generation process and generates the phase difference detection result reliability information based on the error information derived in step S405.

In step S407, the phase difference detection unit 302 detects the phase difference based on the captured image data generated in step S404 and the phase difference detection result reliability information generated in step S406.

In step S408, the lens control unit 403 controls a lens (not shown) based on the phase difference detection result generated in step S407, and updates the in-focus position.

In step S409, the coding unit 201 determines whether or not all the frames have been processed. When it is determined that an unprocessed frame exists, that is, a new frame of captured image data is input to the image processing apparatus 100, the process returns to step S403, and the subsequent processes are repeated.

When the processing of steps S403 to S409 is executed for each frame and it is determined in step S409 that all the frames of the input captured image data have been processed, the image processing ends.

<Flow of reliability information generation process>
Next, an example of the flow of the reliability information generation process executed in step S406 of FIG. 40 will be described with reference to the flowchart of FIG. 41.

When the reliability information generation process is started, the integration unit 411 integrates the error information derived in step S405 in each phase difference detection region in step S421, and derives the integrated value for each phase difference detection region.

In step S422, the threshold value processing unit 412 compares the integrated value of the error information of each phase difference detection region derived in step S421 with the threshold value, and determines whether or not to exclude that region from the target of phase difference detection.

In step S423, the threshold value processing unit 412 generates phase difference detection result reliability information indicating the determination result obtained in step S422 and outputs it (that is, supplies it to the phase difference detection unit 402). When the process of step S423 is completed, the reliability information generation process is completed, and the process returns to FIG. 40.

By executing each process as described above, the image processing apparatus 400 can suppress a decrease in the accuracy of the phase difference detection (typically, the phase difference can be detected more accurately).

<Image pickup device>
The present technology of "Method 3" as described above can be applied to any electronic device. For example, it can be applied to an image pickup device. FIG. 42 is a block diagram showing a main configuration example of an image pickup apparatus to which the present technology is applied. The image pickup apparatus 450 shown in FIG. 42 is the same apparatus as the image pickup apparatus 250.

As shown in FIG. 42, the image pickup apparatus 450 includes a lens 451, an image sensor 452, a signal processing unit 453, an output unit 454, and a storage unit 455.

The lens 351 has the same configuration as the lens 251 and functions in the same manner. The image sensor 352 is a processing unit similar to the image sensor 252. The signal processing unit 353 is a processing unit similar to the signal processing unit 353. The output unit 354 is a processing unit similar to the output unit 254. The storage unit 355 is a processing unit similar to the storage unit 255.

However, the phase difference detection performed by the signal processing unit 353 is performed by using the captured image data and the phase difference detection auxiliary information.

<Image sensor>
The image sensor 452 has a light receiving unit 461, an ADC 462, an encoding unit 201, and a transmitting unit 463.

The light receiving unit 461 is a processing unit similar to the light receiving unit 361, has a similar configuration, and performs the same processing. The ADC 462 is a processing unit similar to the ADC 362, has a similar configuration, and performs the same processing. The coding unit 201 performs the same processing as in the case of the image processing apparatus 400. The transmission unit 463 is a processing unit similar to the transmission unit 363, has a similar configuration, and performs the same processing.

<Signal processing unit>
The signal processing unit 453 includes a receiving unit 471, a decoding unit 202, a reliability information generation unit 401, a phase difference detection unit 402, a lens control unit 403, and an image processing unit 472.

The receiving unit 471 is a processing unit similar to the receiving unit 371, has a similar configuration, and performs the same processing. The image processing unit 472 is a processing unit similar to the image processing unit 372, has a similar configuration, and performs the same processing. The lens control unit 403 is a processing unit similar to the lens control unit 303, has a similar configuration, and performs the same processing.

The decoding unit 202 acquires the coded data supplied from the receiving unit 471. Similar to the case of the image processing apparatus 400, the decoding unit 202 has a configuration as shown in FIG. 22, for example, and performs decoding processing as described with reference to the flowchart of FIG. 25. For example, the decoding unit 202 extracts prediction method information from the coded data, sets a prediction method based on the prediction method information, simply decodes the coded data using the prediction method, and generates captured image data. do. The decoding unit 202 supplies the generated captured image data to the phase difference detection unit 402 and the image processing unit 472. Further, the decoding unit 202 derives error information based on the generated captured image data. The decoding unit 202 supplies the error information to the reliability information generation unit 401.

Similar to the case of the image processing apparatus 400, the reliability information generation unit 401 has a configuration as shown in FIG. 38, and performs reliability information generation processing as described with reference to the flowchart of FIG. 41. For example, the reliability information generation unit 401 acquires the error information supplied from the decoding unit 202, derives the integrated value for each phase difference detection region, and determines the reliability of the phase difference detection result based on the integrated value. Judgment is made and phase difference detection result reliability information is generated. The reliability information generation unit 401 supplies the phase difference detection result reliability information to the phase difference detection unit 402.

The phase difference detection unit 402 performs the same processing as in the case of the image processing device 400. For example, the phase difference detection unit 402 acquires captured image data from the decoding unit 202, and acquires phase difference detection result reliability information from the reliability information generation unit 401. The phase difference detection unit 402 detects the phase difference by using the information. The phase difference detection unit 402 supplies the phase difference detection result to the lens control unit 403.

With such a configuration, the image pickup apparatus 450 can suppress a decrease in the accuracy of the phase difference detection as in the case of the image processing apparatus 400 (typically, the phase difference is detected more accurately). can do).

<Flow of imaging process>
An example of the flow of the image pickup process executed by the image pickup apparatus 450 to image a subject will be described with reference to the flowcharts of FIGS. 43 and 44.

When the imaging process is started, each process of steps S441 to S446 in FIG. 43 is executed in the same manner as each process of steps S341 to S346 of FIG. 34, and the process proceeds to FIG. 44. Then, each process of step S451 and step S452 of FIG. 44 is executed in the same manner as each process of step S351 and step S352 of FIG. 35.

In step S453, the decoding unit 202 derives error information based on the captured image data generated in step S452.

In step S454, the reliability information generation unit 401 executes the reliability information generation process as described with reference to the flowchart of FIG. 41, and generates the phase difference detection result reliability information based on the error information.

In step S455, the phase difference detection unit 402 detects the phase difference based on the captured image data generated in step S452 and the phase difference detection result reliability information generated in step S454.

Each process of steps S456 to S458 is executed in the same manner as each process of steps S355 to S357 of FIG. 35.

In step S459, the image pickup apparatus 450 determines whether or not to end the image pickup process. If it is determined that the imaging is continued and a new frame image is generated, the process returns to step S443 in FIG. 43. That is, the image pickup apparatus 450 executes each process of steps S443 to S459 for each frame. For example, if it is determined in step S459 that the imaging process is to be completed based on a user instruction or the like, the imaging process is terminated.

By performing the image pickup process as described above, the image pickup apparatus 450 can suppress an increase in prediction error in the simple coding in step S445 (FIG. 43) and the simple decoding in step S452 (FIG. 44). Further, the image pickup apparatus 450 can suppress a decrease in the accuracy of the phase difference detection (typically, the phase difference can be detected more accurately).

<Others>
In the above-mentioned "method 3", the simple coding / simple decoding method is arbitrary. This "method 3" can be applied to "method 2" as described above, or can be applied to "method 1". It can also be applied to other simple coding / decoding methods.

Furthermore, this "method 3" can also be used in combination with the above-mentioned "method 2-1". That is, the phase difference detection unit may detect the phase difference based on the captured image data, the phase difference detection auxiliary information, and the phase difference detection result reliability information.

<6. Fifth Embodiment>
<Application example: Image sensor>
In the above, an example of applying the present technology described in each embodiment to an image pickup apparatus has been described, but the present technology can be applied to any device. For example, this technique can also be applied to an image pickup device. FIG. 45 is a block diagram showing a main configuration example of a stacked image sensor, which is an image pickup device to which the present technology is applied. The stacked image sensor 500 shown in FIG. 45 is an image sensor (image sensor) that captures a subject, obtains digital data (image data) of the captured image, and outputs the image data.

As shown in FIG. 45, the laminated image sensor 500 has three semiconductor substrates, a semiconductor substrate 501 to a semiconductor substrate 503. These semiconductor substrates are sealed in a state of being superimposed on each other and modularized (integrated). That is, these semiconductor substrates form a multilayer structure (laminated structure). Electronic circuits are formed on the semiconductor substrates 501 to 503, respectively, and the circuits formed on the semiconductor substrates are connected to each other by vias (VIA) or the like. The path between each semiconductor substrate (circuit formed in) is also referred to as a bus. For example, the circuit of the semiconductor substrate 501 and the circuit of the semiconductor substrate 502 can exchange data and the like via the bus 511. Further, the circuit of the semiconductor substrate 502 and the circuit of the semiconductor substrate 503 can exchange data and the like via the bus 512.

Further, the interface 513 of the laminated image sensor 500 is formed in the circuit formed on the semiconductor substrate 502. That is, the circuit formed on the semiconductor substrate 502 can exchange data and the like with an external circuit of the laminated image sensor 500 (for example, a circuit formed on the circuit board 520) via the interface 513. The interface 513 is an interface in which communication is performed by a communication method conforming to a predetermined communication standard. This communication standard is optional. For example, MIPI (Mobile Industry Processor Interface) may be used, SLVS-EC (Scalable Low Voltage Signaling Embedded Clock) may be used, or other standards may be used. The specific configuration of the interface 513 is arbitrary. For example, the interface 513 may include not only a configuration for controlling input / output but also a transmission line such as a bus or a cable.

By forming the multilayer structure of the semiconductor substrate inside the module in this way, the laminated image sensor 500 can realize the mounting of a larger-scale circuit without increasing the size of the semiconductor substrate. That is, the stacked image sensor 500 can mount a larger-scale circuit while suppressing an increase in cost.

FIG. 46 shows an example of the circuit configuration formed on each semiconductor substrate. In FIG. 46, for convenience of explanation, the semiconductor substrates 501 to 503 are arranged side by side on a plane, but in reality, they are superimposed on each other as shown in FIG. 45.

A light receiving unit 531 and an A / D conversion unit 532 are formed on the top semiconductor substrate 501. The light receiving unit 531 has a plurality of unit pixels having a photoelectric conversion element such as a photodiode, and the incident light is photoelectrically converted in each unit pixel, and the charge corresponding to the incident light is used as an electric signal (pixel signal) for A / D. It is output to the conversion unit 532.

The A / D conversion unit 532 A / D-converts each pixel signal supplied from the light receiving unit 531 to generate pixel data of digital data. The A / D conversion unit 532 supplies the set of pixel data of each unit pixel thus generated to the semiconductor substrate 502 as image data via the bus 511.

An image processing unit 533, which is a logic circuit for performing image processing and the like, is formed on the semiconductor substrate 502 in the middle stage. When the image processing unit 533 acquires the image data supplied from the semiconductor substrate 501 via the bus 511, the image processing unit 533 performs predetermined image processing on the image data. The content of this image processing is arbitrary. For example, defect pixel correction, phase difference detection for autofocus, pixel addition, digital gain, noise reduction, and the like may be included in this image processing. Of course, processing other than these may be included.

A DRAM (Dynamic Random Access Memory) 534 is formed on the bottom semiconductor substrate 503. The DRAM 534 can store data and the like supplied from the semiconductor substrate 502 (image processing unit 533) via the bus 512. Further, the DRAM 534 can read out the stored data or the like in response to a request from the semiconductor substrate 502 (image processing unit 533) or the like and supply the stored data or the like to the semiconductor substrate 502 via the bus 512. That is, the image processing unit 533 can perform image processing using this DRAM 534, such as temporarily holding the image data being processed. For example, so-called slow motion imaging can be realized by imaging at a high frame rate, storing the captured image of each frame in DRAM 534, and reading and outputting it at a low frame rate.

In such use of the DRAM 534, the image processing unit 533 encodes (compresses) the image data, stores the generated coded data in the DRAM 534, or reads the coded data from the DRAM 534 and decodes the image data ( Decoded image data) is generated. For example, the image processing unit 533 has a coding unit 533A and a decoding unit 533B. The coding unit 533A encodes the image data, supplies the generated coded data to the DRAM 534, and stores the coded data. The decoding unit 533B decodes the coded data read from the DRAM 534 and generates image data (decoded image data). By storing the image data in the DRAM 534 as encoded data (compressed data) in this way, the amount of data stored in the DRAM 534 can be reduced. Therefore, it is possible to improve the utilization efficiency of the storage area of the DRAM 534 and the band of the bus 512. Therefore, it is possible to suppress an increase in the capacity of the DRAM 534 and the bandwidth of the bus 512, and it is possible to suppress an increase in the manufacturing cost.

As the coding unit 533A, the above-mentioned coding unit 101 or coding unit 201 may be applied. Further, the decoding unit 102 and the decoding unit 202 described above may be applied as the decoding unit 533B. By doing so, in writing data to DRAM 534 and reading data from DRAM 534, the above-mentioned effects can be obtained in the first embodiment, the second embodiment, and the like.

Further, the image processing unit 533 may be provided with the auxiliary information generation unit 301 to the lens control unit 303 to form the configuration as described with reference to FIG. 29. By doing so, the above-mentioned effect can be obtained in the third embodiment or the like based on the coding / decoding in the exchange of data with the DRAM 534.

Further, the image processing unit 533 may be provided with the reliability information generation unit 401 to the lens control unit 403 to form the configuration as described with reference to FIG. 36. By doing so, the above-mentioned effect can be obtained in the fourth embodiment or the like based on the coding / decoding in the exchange of data with the DRAM 534.

Further, an image processing unit 535, which is a logic circuit for performing image processing and the like, is formed on the circuit board 520. When the image processing unit 535 acquires the image data supplied from the semiconductor substrate 502 (image processing unit 533) of the laminated image sensor 500 via the interface 513, the image processing unit 535 performs predetermined image processing on the image data. The content of this image processing is arbitrary.

That is, the image processing unit 533 can supply data or the like to the image processing unit 535 (output to the outside of the stacked image sensor 500) via the interface 513. At the time of such output, the image processing unit 533 encodes (compresses) the image data and outputs it as encoded data. For example, the image processing unit 533 has a coding unit 533C, and the image processing unit 535 has a decoding unit 535A. The coding unit 533C encodes the image data and outputs the coded data via the interface 513. The decoding unit 535A decodes the coded data supplied via the interface 513 and generates image data (decoded image data). The image processing unit 535 performs image processing on the generated decoded image data.

By transmitting the coded data (compressed data) via the interface 513 in this way, the amount of data to be transmitted can be reduced. Therefore, it is possible to improve the utilization efficiency of the band of the interface 513. That is, the increase in the bandwidth of the interface 513 can be suppressed, and the increase in the manufacturing cost can be suppressed.

As the coding unit 533C, the above-mentioned coding unit 101 or coding unit 201 may be applied. Further, the decoding unit 102 and the decoding unit 202 described above may be applied as the decoding unit 535A. By doing so, it is possible to obtain the above-mentioned effects in the first embodiment, the second embodiment, and the like in the data output to the outside of the stacked image sensor 500.

Further, the image processing unit 533 may be provided with the auxiliary information generation unit 301 to the lens control unit 303 to form the configuration as described with reference to FIG. 29. By doing so, the above-mentioned effect can be obtained in the third embodiment or the like based on the coding / decoding in the data output to the outside of the stacked image sensor 500.

Further, the image processing unit 533 may be provided with the reliability information generation unit 401 to the lens control unit 403 to form the configuration as described with reference to FIG. 36. By doing so, the above-mentioned effect can be obtained in the fourth embodiment or the like based on the coding / decoding in the data output to the outside of the stacked image sensor 500.

<7. Addendum>
<Computer>
The series of processes described above can be executed by hardware or software. When a series of processes are executed by software, the programs constituting the software are installed in the computer. Here, the computer includes a computer embedded in dedicated hardware and, for example, a general-purpose personal computer capable of executing various functions by installing various programs.

FIG. 47 is a block diagram showing an example of hardware configuration of a computer that executes the above-mentioned series of processes programmatically.

In the computer 900 shown in FIG. 47, the CPU (Central Processing Unit) 901, the ROM (Read Only Memory) 902, and the RAM (Random Access Memory) 903 are connected to each other via the bus 904.

An input / output interface 910 is also connected to the bus 904. An input unit 911, an output unit 912, a storage unit 913, a communication unit 914, and a drive 915 are connected to the input / output interface 910.

The input unit 911 includes, for example, a keyboard, a mouse, a microphone, a touch panel, an input terminal, and the like. The output unit 912 includes, for example, a display, a speaker, an output terminal, and the like. The storage unit 913 is composed of, for example, a hard disk, a RAM disk, a non-volatile memory, or the like. The communication unit 914 is composed of, for example, a network interface. The drive 915 drives a removable medium 921 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

In the computer configured as described above, the CPU 901 loads the program stored in the storage unit 913 into the RAM 903 via the input / output interface 910 and the bus 904 and executes the above-mentioned series. Is processed. The RAM 903 also appropriately stores data and the like necessary for the CPU 901 to execute various processes.

The program executed by the computer (CPU901) can be recorded and applied to the removable media 921 as a package media or the like, for example. In that case, the program can be installed in the storage unit 913 via the input / output interface 910 by mounting the removable media 921 in the drive 915.

The program can also be provided via wired or wireless transmission media such as local area networks, the Internet, and digital satellite broadcasting. In that case, the program can be received by the communication unit 914 and installed in the storage unit 913.

In addition, this program can be pre-installed in the ROM 902 or the storage unit 913.

<Applicable target of this technology>
This technique can be applied to any image coding / decoding method. That is, as long as it does not contradict the above-mentioned technology, the specifications of various processes related to image coding / decoding are arbitrary and are not limited to the above-mentioned examples.

Further, although the case where the present technology is applied to the image pickup apparatus has been described above, the present technology can be applied not only to the image pickup apparatus but also to any apparatus (electronic device). For example, the present technology can be applied to an image processing device or the like that performs image processing on an captured image obtained by high digital gain imaging performed in another device.

In addition, the present technology includes any configuration or a module using a processor (for example, a video processor) as a system LSI (Large Scale Integration), a module using a plurality of processors, or the like (for example, video) mounted on an arbitrary device or a device constituting the system. It can also be implemented as a module), a unit using a plurality of modules (for example, a video unit), a set in which other functions are added to the unit (for example, a video set), or the like (that is, a partial configuration of a device).

Further, the present technology can also be applied to a network system composed of a plurality of devices. For example, it can be applied to cloud services that provide services related to images (moving images) to arbitrary terminals such as computers, AV (Audio Visual) devices, portable information processing terminals, and IoT (Internet of Things) devices. can.

Systems, equipment, processing departments, etc. to which this technology is applied should be used in any field such as transportation, medical care, crime prevention, agriculture, livestock industry, mining, beauty, factories, home appliances, weather, nature monitoring, etc. Can be done. The use is also arbitrary.

For example, the present technology can be applied to systems and devices used for providing ornamental contents and the like. Further, for example, the present technology can be applied to systems and devices used for traffic such as traffic condition supervision and automatic driving control. Further, for example, the present technology can be applied to systems and devices used for security purposes. Further, for example, the present technology can be applied to a system or device used for automatic control of a machine or the like. Further, for example, the present technology can be applied to systems and devices used for agriculture and livestock industry. The present technology can also be applied to systems and devices for monitoring natural conditions such as volcanoes, forests and oceans, and wildlife. Further, for example, the present technology can be applied to systems and devices used for sports.

<Others>
In the present specification, the "flag" is information for identifying a plurality of states, and is not only information used for identifying two states of true (1) or false (0), but also three or more states. It also contains information that can identify the state. Therefore, the value that this "flag" can take may be, for example, 2 values of 1/0 or 3 or more values. That is, the number of bits constituting this "flag" is arbitrary, and may be 1 bit or a plurality of bits. Further, the identification information (including the flag) is assumed to include not only the identification information in the bit stream but also the difference information of the identification information with respect to a certain reference information in the bit stream. In, the "flag" and "identification information" include not only the information but also the difference information with respect to the reference information.

Further, various information (metadata and the like) related to the coded data (bitstream) may be transmitted or recorded in any form as long as it is associated with the coded data. Here, the term "associate" means, for example, to make the other data available (linkable) when processing one data. That is, the data associated with each other may be combined as one data or may be individual data. For example, the information associated with the coded data (image) may be transmitted on a transmission path different from the coded data (image). Further, for example, the information associated with the coded data (image) may be recorded on a recording medium (or another recording area of the same recording medium) different from the coded data (image). good. It should be noted that this "association" may be a part of the data, not the entire data. For example, the image and the information corresponding to the image may be associated with each other in any unit such as a plurality of frames, one frame, or a part within the frame.

In addition, in this specification, "synthesize", "multiplex", "add", "integrate", "include", "store", "insert", "insert", "insert". A term such as "" means combining a plurality of objects into one, for example, combining encoded data and metadata into one data, and means one method of "associating" described above.

Further, the embodiment of the present technology is not limited to the above-described embodiment, and various changes can be made without departing from the gist of the present technology.

Further, for example, the present technology further applies to any configuration constituting a device or a system, for example, a processor as a system LSI (Large Scale Integration), a module using a plurality of processors, a unit using a plurality of modules, and a unit. It can also be implemented as a set or the like with other functions added (that is, a partial configuration of the device).

In the present specification, the system means a set of a plurality of components (devices, modules (parts), etc.), and it does not matter whether or not all the components are in the same housing. Therefore, a plurality of devices housed in separate housings and connected via a network, and a device in which a plurality of modules are housed in one housing are both systems. ..

Further, for example, the configuration described as one device (or processing unit) may be divided and configured as a plurality of devices (or processing units). On the contrary, the configurations described above as a plurality of devices (or processing units) may be collectively configured as one device (or processing unit). Further, of course, a configuration other than the above may be added to the configuration of each device (or each processing unit). Further, if the configuration and operation of the entire system are substantially the same, a part of the configuration of one device (or processing unit) may be included in the configuration of another device (or other processing unit). ..

Further, for example, the present technology can have a cloud computing configuration in which one function is shared by a plurality of devices via a network and jointly processed.

Further, for example, the above-mentioned program can be executed in any device. In that case, the device may have necessary functions (functional blocks, etc.) so that necessary information can be obtained.

Further, for example, each step described in the above-mentioned flowchart can be executed by one device or can be shared and executed by a plurality of devices. Further, when a plurality of processes are included in one step, the plurality of processes included in the one step can be executed by one device or shared by a plurality of devices. In other words, a plurality of processes included in one step can be executed as processes of a plurality of steps. On the contrary, the processes described as a plurality of steps can be collectively executed as one step.

In the program executed by the computer, the processing of the steps for describing the program may be executed in chronological order in the order described in the present specification, or may be called in parallel or called. It may be executed individually at the required timing such as when. That is, as long as there is no contradiction, the processes of each step may be executed in an order different from the above-mentioned order. Further, the processing of the step for describing this program may be executed in parallel with the processing of another program, or may be executed in combination with the processing of another program.

It should be noted that the present techniques described above and below in the present specification can be independently implemented independently as long as there is no contradiction. Of course, any plurality of the present technologies can be used in combination. For example, some or all of the techniques described in any of the embodiments may be combined with some or all of the techniques described in other embodiments. In addition, a part or all of any of the above-mentioned techniques may be carried out in combination with other techniques not described above.

The present technology can also have the following configurations.
(1) An image processing device including a coding unit that encodes captured image data using prediction by a prediction method set based on parameters related to focusing.
(2) The image processing apparatus according to (1), wherein the prediction method is set according to whether or not the subject is in focus based on the parameters related to the in-focus.
(3) When the subject is in focus, a prediction method is set in which the pixel value of the pixel located closest to the processing target pixel among the pixels of the same color as the processing target pixel is used as the prediction value (2). ). The image processing apparatus.
(4) When the subject is not in focus, the position of the photodiode that detects the pixel value is the same as that of the pixel to be processed, and the pixel of the same color as the pixel to be processed is the most of the pixel to be processed. The image processing apparatus according to (2) or (3), wherein a prediction method is set in which the pixel value of pixels located in the vicinity is used as the prediction value.
(5) The image processing apparatus according to any one of (1) to (4), wherein the parameters related to focusing are derived from pixel values used for phase difference detection for focusing.
(6) The parameter related to the focusing is the phase difference detection performed by using the pixel value of the captured image data obtained by decoding the coded data generated by the coding unit encoding the captured image data. The image processing apparatus according to (5), which is the result of the above.
(7) The image processing apparatus according to (5), wherein the parameter related to focusing is the result of prediction by each prediction method performed using the pixel value of the captured image data.
(8) Further provided with a setting unit for setting the prediction method.
The image processing apparatus according to any one of (1) to (7), wherein the coding unit encodes the captured image data by using the prediction by the prediction method set by the setting unit.
(9) The image processing apparatus according to (8), wherein the setting unit sets the prediction method according to whether or not the subject is in focus based on the parameters related to the in-focus.
(10) When the setting unit is in focus on the subject, as the prediction method, the setting unit predicts the pixel value of the pixel located closest to the processing target pixel among the pixels of the same color as the processing target pixel. The image processing apparatus according to (9).
(11) When the setting unit is not in focus on the subject, the position of the photodiode that detects the pixel value is the same as the processing target pixel and the same color as the processing target pixel as the prediction method. The image processing apparatus according to (9) or (10), wherein a prediction method is set in which the pixel value of the pixel located closest to the processing target pixel among the pixels is used as the prediction value.
(12) The setting unit detects the phase difference for focusing by using the captured image data obtained by decoding the coded data generated by the coding unit encoding the captured image data. The image processing apparatus according to any one of (9) to (11), wherein the result is used as a parameter related to focusing and the prediction method is set.
(13) The setting unit sets the prediction method by using the captured image data to make a prediction by each prediction method and using the result of each prediction as a parameter related to the focusing. The image processing apparatus according to any one.
(14) The image processing apparatus according to (13), wherein the coding unit adds prediction method information indicating a prediction method set by the setting unit to the coded data generated by encoding the captured image data. ..
(15) The coding unit is
Using the prediction method, a difference derivation unit that predicts the processing target pixel of the captured image data and derives the difference between some bits of the pixel value of the processing target pixel and the predicted value, and
A Golomb coding unit that Golom-codes the difference derived by the difference deriving unit, and a Golomb coding unit.
(1) to (1) to (1) to (1) to (1) to (1) to (1) to (1) to (1) to (1) to (1) to (1) to (1) to (1) to (1) to (1) to (1) to (1) to (1) to (1) to (1) The image processing apparatus according to any one of 14).
(16) The image processing apparatus according to any one of (1) to (15), further comprising a transmission unit that transmits the coded data generated by encoding the captured image data by the coding unit.
(17) The image processing apparatus according to any one of (1) to (16), further comprising a decoding unit that decodes the coded data generated by encoding the captured image data by the coding unit.
(18) The image according to (17), further comprising a phase difference detection unit that performs phase difference detection for focusing using the captured image data generated by decoding the coded data by the decoding unit. Processing device.
(19) The image processing apparatus according to (18), further comprising a control unit that controls an optical system so as to focus on a subject based on the result of the phase difference detection by the phase difference detection unit.
(20) An image processing method for encoding captured image data using prediction by a prediction method set based on parameters related to focusing.

(21) An image processing apparatus including a decoding unit that decodes encoded data of captured image data by using prediction by a prediction method set based on parameters related to focusing.
(22) The image processing apparatus according to (21), wherein the prediction method is set according to whether or not the subject is in focus based on the parameters related to the in-focus.
(23) When the subject is in focus, a prediction method is set in which the pixel value of the pixel located closest to the processing target pixel among the pixels of the same color as the processing target pixel is used as the prediction value (22). ). The image processing apparatus.
(24) When the subject is not in focus, the position of the photodiode that detects the pixel value is the same as that of the pixel to be processed, and the pixel of the same color as the pixel to be processed is the most of the pixel to be processed. The image processing apparatus according to (22) or (23), wherein a prediction method is set in which the pixel value of pixels located in the vicinity is used as the prediction value.
(25) The parameters related to the focusing are the results of the phase difference detection for the focusing performed by the decoding unit using the captured image data obtained by decoding the coded data (21) to (21). 24) The image processing apparatus according to any one of.
(26) The image according to any one of (21) to (25), wherein the parameter related to focusing is prediction method information indicating the prediction method used at the time of coding added to the coded data. Processing device.
(27) Further provided with a setting unit for setting the prediction method.
The image processing apparatus according to any one of (21) to (26), wherein the decoding unit decodes the coded data by using the prediction by the prediction method set by the setting unit.
(28) The image processing apparatus according to (27), wherein the setting unit sets the prediction method according to whether or not the subject is in focus based on the parameters related to the in-focus.
(29) When the setting unit is in focus on the subject, as the prediction method, the setting unit predicts the pixel value of the pixel located closest to the processing target pixel among the pixels of the same color as the processing target pixel. The image processing apparatus according to (28).
(30) When the setting unit is not in focus on the subject, the position of the photodiode that detects the pixel value is the same as the processing target pixel and the same color as the processing target pixel as the prediction method. The image processing apparatus according to (28) or (29), which sets a prediction method in which the pixel value of the pixel located closest to the processing target pixel among the pixels is used as the prediction value.
(31) The setting unit uses the result of the phase difference detection for focusing performed by the decoding unit using the captured image data obtained by decoding the coded data as a parameter for focusing. The image processing apparatus according to (28), which sets the prediction method.
(32) The decoding unit is
A reverse adjustment unit that reversely adjusts the code amount of the coded data by removing the refinement added to the coding unit, and a reverse adjustment unit.
A Golomb decoding unit that Golom-decodes the coded data whose code amount is reverse-adjusted by the reverse adjustment unit, and a Golomb decoding unit.
It is provided with a captured image data generation unit that generates the captured image data by using the difference between the pixel value and the predicted value of the captured image data generated by the gorom decoding of the coded data by the gorom decoding unit. 21) The image processing apparatus according to any one of (31).
(33) Further, a receiving unit for receiving the coded data is provided.
The image processing apparatus according to any one of (21) to (32), wherein the decoding unit decodes the coded data received by the receiving unit.
(34) Further provided with a phase difference detection unit that performs phase difference detection for focusing using the captured image data generated by decoding the coded data by the decoding unit (21) to (33). The image processing apparatus according to any one of.
(35) The phase difference detection unit further performs the phase difference detection based on the prediction method information indicating the prediction method used when the captured image data is encoded to generate the coded data (34). ). The image processing apparatus.
(36) Further provided with an auxiliary information generation unit that generates auxiliary information for the phase difference detection based on the prediction method information.
The phase difference detection unit performs the phase difference detection based on the captured image data generated by decoding the coded data by the decoding unit and the auxiliary information generated by the auxiliary information generation unit. The image processing apparatus according to (35).
(37) The image according to any one of (34) to (36), wherein the phase difference detection unit further detects the phase difference based on error information indicating an error due to coding and decoding of captured image data. Processing device.
(38) Further provided with a reliability information generation unit that generates reliability information indicating the reliability of the result of the phase difference detection based on the error information.
The phase difference detection unit detects the phase difference based on the captured image data generated by decoding the coded data by the decoding unit and the reliability information generated by the reliability information generation unit. The image processing apparatus according to (37).
(39) The image according to any one of (34) to (38), further comprising a control unit that controls the optical system so as to focus on the subject based on the result of the phase difference detection by the phase difference detection unit. Processing equipment.
(40) An image processing method for decoding encoded data of captured image data by using prediction by a prediction method set based on parameters related to focusing.

100 Image processing device, 101 Coding unit, 102 Decoding unit, 103 Phase difference detection unit, 104 Prediction method setting unit, 121 DPCM processing unit, 122 Golom coding unit, 123 Compression rate adjustment unit, 131 Compression rate reverse adjustment unit, 132 Golom decoding unit, 133 reverse DPCM processing unit, 140 pixel array, 141 pixels, 142 on-chip lens, 151 left photoelectric conversion element, 152 right photoelectric conversion element, 154 color filter, 170 image pickup device, 171 lens, 172 image sensor, 173 Signal processing unit, 174 output unit, 175 storage unit, 181 light receiving unit, 182 ADC, 183 transmitter unit, 191 receiver unit, 192 lens control unit, 193 image processing unit, 200 image processing unit, 201 encoding unit, 202 decoding Unit, 221 Prediction method setting unit, 222 Coding unit, 231 Prediction method setting unit, 232 Decoding unit, 250 Imaging device, 251 lens, 252 image sensor, 253 signal processing unit, 254 output unit, 255 storage unit, 261 light receiving unit , 262 ADC, 263 transmitter, 271 receiver, 272 phase difference detector, 273 lens control unit, 274 image processing unit, 300 image processing device, 301 auxiliary information generation unit, 302 phase difference detection unit, 303 lens control unit, 311 Phase difference detection area statistical processing unit, 312 threshold processing unit, 350 image pickup device, 351 lens, 352 image sensor, 353 signal processing unit, 354 output unit, 355 storage unit, 361 light receiving unit, 362 ADC, 363 transmitter, 371 Receiver, 372 image processing unit, 400 image processing unit, 401 reliability information generation unit, 402 phase difference detection unit, 403 lens control unit, 411 integration unit, 412 threshold processing unit, 450 image pickup device, 451 lens, 452 image sensor. , 453 Signal processing unit, 461 Light receiving unit, 462 ADC, 463 transmitter, 471 receiver, 472 image processing unit, 500 stacked image sensor, 501 to 503 semiconductor substrate, 520 circuit board, 531 light receiving unit, 532 A / D conversion unit, 533 image processing unit, 534 DRAM, 535 Image processing unit

Claims (20)

An image processing device including a coding unit that encodes captured image data using prediction by a prediction method set based on parameters related to focusing. The image processing apparatus according to claim 1, wherein the prediction method is set according to whether or not the subject is in focus based on the parameters related to focusing. The second aspect of claim 2, wherein when the subject is in focus, a prediction method is set in which the pixel value of the pixel located closest to the processing target pixel among the pixels of the same color as the processing target pixel is set as the prediction value. Image processing equipment. When the subject is not in focus, the position of the photodiode that detects the pixel value is the same as that of the pixel to be processed, and the position is closest to the pixel to be processed among the pixels of the same color as the pixel to be processed. The image processing apparatus according to claim 2, wherein a prediction method is set in which the pixel value of the pixel to be used is used as the prediction value. The image processing apparatus according to claim 1, wherein the parameters related to focusing are derived from pixel values used for phase difference detection for focusing. The parameter related to the focusing is the result of the phase difference detection performed by using the pixel value of the captured image data obtained by decoding the coded data generated by the coding unit encoding the captured image data. The image processing apparatus according to claim 5. The image processing apparatus according to claim 5, wherein the parameter related to focusing is the result of prediction by each prediction method performed by using the pixel value of the captured image data. Further provided with a setting unit for setting the prediction method,
The image processing apparatus according to claim 1, wherein the coding unit encodes the captured image data by using the prediction by the prediction method set by the setting unit. The image processing apparatus according to claim 8, wherein the coding unit adds prediction method information indicating a prediction method set by the setting unit to the coded data generated by encoding the captured image data. An image processing method that encodes captured image data using prediction by a prediction method set based on parameters related to focusing. An image processing device including a decoding unit that decodes the encoded data of the captured image data by using the prediction by the prediction method set based on the parameters related to focusing. The image processing apparatus according to claim 11, wherein the prediction method is set according to whether or not the subject is in focus based on the parameters related to focusing. The twelfth aspect of claim 12, wherein when the subject is in focus, a prediction method is set in which the pixel value of the pixel located closest to the processing target pixel among the pixels of the same color as the processing target pixel is set as the prediction value. Image processing equipment. When the subject is not in focus, the position of the photodiode that detects the pixel value is the same as that of the pixel to be processed, and the position is closest to the pixel to be processed among the pixels of the same color as the pixel to be processed. The image processing apparatus according to claim 12, wherein a prediction method is set in which the pixel value of the pixel to be used is used as the prediction value. The image processing according to claim 11, wherein the parameter relating to the focusing is the result of the phase difference detection for the focusing performed by the decoding unit using the captured image data obtained by decoding the coded data. Device. The image processing apparatus according to claim 11, wherein the parameter related to focusing is predictive method information indicating a predictive method used at the time of coding, which is added to the coded data. Further provided with a setting unit for setting the prediction method,
The image processing apparatus according to claim 11, wherein the decoding unit decodes the coded data by using the prediction by the prediction method set by the setting unit. Auxiliary information generation unit that generates auxiliary information for phase difference detection for focusing based on the prediction method information indicating the prediction method used when the captured image data is encoded to generate the coded data. When,
A phase difference detection unit that performs phase difference detection based on the captured image data generated by decoding the coded data by the decoding unit and the auxiliary information generated by the auxiliary information generation unit is further added. The image processing apparatus according to claim 11. A reliability information generation unit that generates reliability information indicating the reliability of the result of the phase difference detection for focusing based on the error information indicating the error due to the coding and decoding of the captured image data.
A phase difference detection unit that performs phase difference detection based on the captured image data generated by decoding the coded data by the decoding unit and the reliability information generated by the reliability information generation unit. 11. The image processing apparatus according to claim 11. An image processing method for decoding encoded data of captured image data by using prediction by a prediction method set based on parameters related to focusing.

Images