JP2019068386A - Encoding device, control method of the same, control program, and imaging apparatus - Google Patents

Abstract

To suppress a decrease in a frame rate of continuous shooting of still images and moving images when encoding a captured image (RAW data) and a depth map.SOLUTION: When encoding processing is performed on RAW data obtained by imaging and a depth map indicating a depth of each pixel of the RAW data to obtain encoded data, an encoding processing unit 103a divides the depth map into a plurality of divided areas on the basis of a ratio of a frame rate of the RAW data and an update interval of the depth map and performs encoding processing on the RAW data and the depth map having the plurality of divided areas by time division.SELECTED DRAWING: Figure 2

Description

  The present invention relates to an encoding apparatus, a control method thereof, a control program, and an imaging apparatus, and more particularly to an imaging apparatus capable of encoding a video signal.

  With the widespread use of digital cameras or digital camcorders, which are one of the imaging devices, still images or moving images are captured digitally. An image obtained by the digital method can be easily subjected to image processing by arithmetic processing using software or firmware. As one of image processing, for example, aberration correction of a lens or adjustment of a dynamic range can be mentioned. Such image processing technology is called computational photography.

  For example, an imaging apparatus that records depth information (called a depth map) generated by a depth estimation method represented by a DFD (Depth from Defocus) method or a stereo method, and performs image processing based on the depth map (See Patent Document 1).

JP, 2014-168227, A

  By the way, since generation of a depth map requires a large amount of calculation, in the case of continuous shooting of a still image or recording of a moving image, the generation timing of the depth map is discrete with respect to the photographed image.

  The depth map is two-dimensional data having depth information corresponding to each point in the captured image, and the amount of information also increases in proportion to the number of pixels, so it is desirable to compress and record. Therefore, if the depth map is treated as a monochrome image, it is possible to apply the compression technique used for image data, and compression recording without adding a new encoder by applying the same compression method as the captured image. can do.

  However, when the captured image and the depth map are encoded in a time division manner by one encoder, so-called processing latency is generated along with the encoding process of the depth map. As a result, the encoding process for the photographed image is locally reduced, and it becomes difficult to keep the photographing interval constant, for example, in continuous shooting of still images. In addition, the frame rate may be reduced at the time of moving image shooting.

  Therefore, it is an object of the present invention to provide an imaging apparatus capable of suppressing continuous shooting of still images and lowering of a frame rate of moving images when encoding captured images and depth maps, a control method thereof, and a control program It is to do.

  In order to achieve the above object, the encoding apparatus according to the present invention performs encoding processing on RAW data obtained by imaging and a depth map indicating the depth for each pixel of the RAW data to obtain encoded data. The apparatus, comprising: dividing means for dividing the depth map into a plurality of divided areas based on a ratio of an update interval of the RAW data and an update interval of the depth map; the RAW data and the plurality of divided areas And encoding means for encoding the depth map by time division.

  According to the present invention, it is possible to suppress the lowering of the frame rate of continuous shooting of still images and moving images.

It is a block diagram for demonstrating the structure about an example of the imaging device by the 1st Embodiment of this invention. It is a figure which shows the structure of RAW data which is an output of the imaging part shown in FIG. It is a figure for demonstrating the example about the structure of the image pick-up element with which the imaging part shown in FIG. 1 was equipped. It is a flowchart for demonstrating the encoding process of RAW data and a depth map which are performed by the RAW data encoding / decoding process part shown in FIG. It is a figure which shows an example of the RAW data and depth map which are encoded by the RAW data encoding / decoding process part shown in FIG. It is a figure for demonstrating an example of area | region division | segmentation of the fats map performed by the RAW data encoding / decoding process part shown in FIG. It is a figure which shows the example about the result of the encoding process performed by the RAW data encoding / decoding process part 103 shown in FIG. It is a block diagram which shows the example about the structure of the RAW data encoding / decoding process part used with the camera by the 2nd Embodiment of this invention. It is a flowchart for demonstrating the encoding process of RAW data and a depth map which are performed by the RAW data encoding / decoding process part with which the camera by the 2nd Embodiment of this invention was equipped. It is a figure which shows an example of the re-region division of the fats map performed by the RAW data encoding / decoding process part with which the camera by the 2nd Embodiment of this invention was equipped. It is a figure which shows the example about the result of the encoding process performed by the RAW data encoding / decoding process part shown in FIG. It is a figure which shows an example which rearranged the partial area | region shown in FIG.

  Hereinafter, an example of an imaging device according to an embodiment of the present invention will be described with reference to the drawings.

First Embodiment
FIG. 1 is a block diagram for describing a configuration of an example of an imaging device according to a first embodiment of the present invention. FIG. 1 (a) is a block diagram showing the configuration of the imaging apparatus, and FIG. 1 (b) is a block diagram showing the configuration of the RAW data encoding / decoding processing unit shown in FIG. 1 (a).

  The illustrated imaging apparatus is, for example, a digital camera (hereinafter simply referred to as a camera) 100 capable of capturing still images and moving images. The camera 100 includes a control unit 101. The control unit 101 controls the entire camera 100.

  The imaging unit 102 includes a lens optical system and an imaging device. The lens optical system includes an optical lens and a stop, and also includes a focus control and a lens driving unit, and can perform optical zoom. The imaging device is, for example, a CCD image sensor or a CMOS sensor, and generates an electrical signal according to an optical image (object image) formed through a lens optical system. Then, the imaging device converts the electric signal into a digital signal and sends the digital signal to the RAW data encoding / decoding processing unit 103 and the depth map generation unit 104 as RAW data.

  FIG. 2 is a diagram showing a configuration of RAW data which is an output of the imaging unit shown in FIG.

  As shown, the RAW data is composed of four color elements of Bayer arrangement R (red), G1 (green), G2 (green), and B (blue).

  FIG. 3 is a diagram for explaining an example of the configuration of an imaging element provided in the imaging unit shown in FIG. And FIG. 3 (a) is a top view, FIG.3 (b) is the figure seen from the side.

  In the imaging device, each of the pixels includes one microlens (ML) 200 and two photoelectric conversion units 201 and 202 associated with the ML 200. Then, each pixel of the imaging device receives a light flux emitted from a different area of the exit pupil of the lens optical system. Here, RAW data (referred to as A image) is output according to the output of the photoelectric conversion unit 20, and RAW data (referred to as B image) is output according to the output of the electrical conversion unit 202.

  Referring again to FIG. 1, as shown in FIG. 1 (b), the RAW data encoding / decoding processing unit 103 includes a RAW data encoding processing unit 103a and a RAW data decoding processing unit 103b. Performs coding processing and decoding processing independently.

  The RAW data encoding processing unit 103a outputs, to the memory 105, mainly the RAW data input from the imaging unit 102 and the encoded data obtained by encoding the depth map generated by the depth map generation unit 104 described later. . The RAW data decoding processing unit 103 b transmits, to the image processing unit 107, RAW data that is decoded data obtained by decoding the encoded data stored in the memory 105, and outputs the decoded depth map to the memory 105.

  The RAW data and the depth map are encoded / decoded in a time division manner as described later.

  The depth map generation unit 104 generates a depth map by using the A image and the B image to calculate the depth distance of each point (for each pixel) in the image using a known DFD method or stereo method. Then, the depth map generation unit 104 outputs the depth map to the memory 105.

  The memory 105 is used to store various data output from each unit constituting the camera 100. The memory 105 is, for example, a storage area configured of volatile memory. A memory I / F unit arbitrates memory access requests from the respective units of the camera 100 and performs read and write control on the memory 105.

  The image processing unit 107 performs image processing such as demosaicing processing, noise removal processing, optical distortion correction processing, and color correction processing on the RAW data input from the RAW data encoding / decoding processing unit 103 to generate image data. Generate Then, the image processing unit 107 outputs the image data to the display unit 108 and the image data encoding unit 109.

  The image processing unit 107 performs image processing at timing independent of the generation of encoded data by the RAW data encoding / decoding processing unit 103. That is, the image processing unit 107 performs image processing of the RAW data obtained by decoding the encoded data stored in the memory 105 by the RAW data encoding / decoding unit 103 at independent timing. Further, the image processing unit 107 performs image alignment processing and image combining processing on the A image and the B image using the decoded depth map, and executes refocusing processing on the virtual focus distance.

  The display unit 108 is, for example, a liquid crystal display or an organic EL display, and displays an image according to image data output from the image processing unit 107.

  The image data encoding unit 109a performs coding on the image data output from the image processing unit 107, for example, in the case of a still image, in the case of JPEG and in the case of a moving image, for example, The encoded data obtained by the quantization processing is output to the memory 105.

  The recording processing unit 110 records various data such as encoded data stored in the memory 105 in the recording medium 111. The recording medium 111 is, for example, a recording medium configured of a non-volatile memory.

  FIG. 4 is a flowchart for explaining the encoding process of the RAW data and the depth map performed by the RAW data encoding / decoding unit shown in FIG. Note that the process according to the illustrated flowchart is performed under the control of the control unit 101.

  FIG. 5 is a view showing an example of RAW data and a depth map encoded by the RAW data encoding / decoding processing unit shown in FIG.

  Here, RAW data of 4096 × 2160 pixels is recorded as a moving image of 60 frames / second. In addition, the depth map is generated at an update interval of 15 frames per second, which is a quarter of the frame rate of the raw data. Note that RAW data can also be recorded by encoding both A and B images, but here, one RAW data is encoded for simplification of the description.

  When the encoding process is started, the RAW data encoding / decoding processing unit 103 acquires an update interval (frame rate) of the depth map with respect to the frame rate of the RAW data from the depth map generation unit 104 (step S401). Here, although the update interval is acquired from the depth map generation unit 104, the update interval may be obtained by measuring the time when the first depth map is generated from the start of recording of RAW data. .

  Subsequently, the RAW data encoding / decoding processing unit 103 divides the area of the depth map based on the ratio of the frame rate of the RAW data to the update interval of the depth map (step S402).

  FIG. 6 is a diagram for explaining an example of area division of a fats map performed by the RAW data encoding / decoding unit shown in FIG. 6 (a) shows an example divided in the horizontal direction and the vertical direction, and FIG. 6 (b) shows an example divided in the horizontal direction. Further, FIG. 6C is a diagram showing an example of division in the vertical direction.

  Here, since one depth map is generated for RAW data of four frames, the depth map is divided into four divided areas d [0] to d [3] as shown in FIG. When dividing, as shown in FIG. 6A, it is divided in the horizontal and vertical directions or divided only in the horizontal direction as shown in FIG. 6B. Furthermore, as shown in FIG. 6C, the division may be performed only in the vertical direction. In addition, as long as the area of each division area is uniform, you may divide | segment into any of FIG. 6 (a)-FIG.6 (c).

  Subsequently, the control unit 101 determines whether the recording of the encoded data by the RAW data encoding / decoding processing unit 103 is stopped (step S403). When the recording of the encoded data is stopped (YES in step S403), the control unit 101 ends the encoding process by the RAW data encoding / decoding unit 103.

  On the other hand, when the recording of the encoded data is not stopped (NO in step S403), the control unit 101 controls the RAW data encoding / decoding processing unit 103 to perform encoding processing of the RAW data (step S404). Then, when the RAW data encoding process is completed, the RAW data encoding / decoding processing unit 103 determines divided areas of the depth map to be encoded (step S405).

  As shown in FIG. 6, an index number is added to each of the divided areas, and the RAW data encoding / decoding processing unit 103 has an index counter that increments the index number. The RAW data encoding / decoding processing unit 103 increments the index counter for each frame of RAW data. In addition, the RAW data encoding / decoding processing unit 103 resets the index counter when the index number becomes the division number. Then, the RAW data encoding / decoding processing unit 103 determines divided areas based on the count value of the index counter.

  The depth map is stored in the memory 105 as described above. Therefore, the RAW data encoding / decoding processing unit 103 sets information such as the start address of the dismap, the origin coordinates, and the number of horizontal and vertical pixels, and performs the depth map encoding process of the divided area determined according to the information. (Step S406). Then, the process returns to step S403.

  FIG. 7 is a diagram showing an example of the result of the encoding process performed by the RAW data encoding / decoding unit shown in FIG.

  As described above, the RAW data encoding / decoding processing unit 103 divides the depth map into areas based on the ratio between the frame rate of the RAW data and the update interval of the depth map. As a result, control is performed so as to distribute and encode one depth map in the time axis direction. Therefore, as shown in FIG. 7, it is possible to equalize the encoding process of the depth map and perform the recording of the RAW data to the maximum.

  Furthermore, here, an example was described in which the frame rate of RAW data and the update interval of the depth map were fixed. The update interval of the depth map may fluctuate with respect to the RAW data frame due to factors such as the degree of congestion of access to the memory 105. In this case, if buffer areas for temporarily storing a plurality of depth maps are provided in the memory 105, it is possible to easily adjust the leveling timing of the depth maps.

  However, since the update interval of the depth map fluctuates and the buffer area may overflow, it is desirable to measure the accumulation amount in the buffer area to control the encoding process of the depth map. For example, in the case of continuous shooting of still images, priority is given to the recording of the depth map, and the update interval of the RAW data is changed when the storage amount of the buffer area exceeds a predetermined threshold. This prevents the buffer area for the depth map from overflowing.

  Also, in the case of a moving image, it is not desirable to change the frame rate of RAW data, so if the storage amount of the buffer area exceeds a predetermined threshold, discard the newly input depth map You may control.

  As described above, in the first embodiment of the present invention, the depth map is area-divided based on the ratio of the frame rate of the RAW data to the update interval of the depth map, and the depth map is encoded. By this, it is possible to equalize the encoding load relating to the depth map, and to suppress the continuous shooting of still images and the decrease of the frame rate of moving images.

Second Embodiment
Subsequently, a camera according to a second embodiment of the present invention will be described. The configuration of the camera according to the second embodiment is the same as that of the camera shown in FIG. 1, but the configuration of the RAW data encoding / decoding processing unit 103 is different.

  FIG. 8 is a block diagram showing an example of the configuration of the RAW data encoding / decoding processing unit used in the camera according to the second embodiment of the present invention.

  The illustrated RAW data encoding / decoding processing unit 800 has a RAW data encoding processing unit 800 a and a RAW data decoding processing unit 800 b. Each of the RAW data encoding processing unit 800a and the RAW data decoding processing unit 800b has a four-core configuration capable of parallel processing of color elements (R, G1, G2, and B) in the RAW data. Thus, when processing the depth map, the divided regions of the debs map are processed in parallel.

  As illustrated, the RAW data encoding processing unit 800a includes a plane separation unit 801 and first to fourth encoding processing cores 802 to 805. Also, the RAW data decoding processing unit 800 b includes a plane combining unit 810 and first to fourth decoding processing cores 812 to 815.

  The plane separation unit 801 separates the RAW data into color elements R, G1, G2, and B, and sends them to the first to fourth encoding processing cores 802 to 805, respectively. The first, second, third, and fourth encoding processing cores 802, 803, 804, and 805 perform encoding processing on color elements R, G1, G2, and B, respectively.

  The first, second, third and fourth decoding processing cores 812, 813, 814 and 815 receive the coded data and perform decoding on the color elements R, G1, G2 and B respectively. Then, the plane combining unit 811 performs combining processing on the color elements R, G1, G2, and B subjected to the decoding processing.

  FIG. 9 is a flowchart for describing the encoding process of the RAW data and the depth map performed by the RAW data encoding / decoding unit provided in the camera according to the second embodiment of the present invention.

  Note that the process according to the illustrated flowchart is performed under the control of the control unit 101. In FIG. 9, the same steps as the steps of the flowchart shown in FIG. Here, as in the first embodiment, RAW data of 4096 × 2160 pixels is recorded as a moving image of 60 frames / second. In addition, the depth map is generated at an update interval of 15 frames per second, which is a quarter of the frame rate of the raw data.

  In step S402, the RAW data encoding / decoding processing unit 800 divides the depth map into, for example, four regions as shown in FIG.

  After the process of step S405, the RAW data encoding / decoding processing unit 800 further re-divides the dismap divided in step S402 in order to perform encoding processing in four parallels (step S906).

  FIG. 10 is a diagram showing an example of re-region division of a fats map performed by the RAW data encoding / decoding processing unit provided in the camera according to the second embodiment of the present invention.

  In the process of step S906, each of divided areas d [0] to d [3] is further divided into four partial areas. In the illustrated example, divided area d [0] is divided into partial areas d [00] to d [03], and divided area d [1] is divided to partial areas d [10] to d [13]. Further, divided area d [2] is divided into partial areas d [20] to d [23], and divided area d [3] is divided into partial areas d [30] to d [33].

  As described above, the depth map is stored in the memory 105. Therefore, the RAW data encoding / decoding processing unit 800 sets information such as the start address of the dismap, the origin coordinates, the number of horizontal and vertical pixels into four divided areas, and determines the division according to the information. A region depth map encoding process is performed (step S 907). Then, the process returns to step S403.

  FIG. 11 is a diagram showing an example of the result of the encoding process performed by the RAW data encoding / decoding unit shown in FIG.

  As described above, the RAW data encoding / decoding processing unit 800 divides the depth map into areas based on the ratio between the frame rate of the RAW data and the update interval of the depth map. Furthermore, the RAW data encoding / decoding processing unit 800 re-divides the divided area based on the number of parallel processing of the encoding processing. Thus, control is performed to disperse one depth map in the time axis direction and to perform parallel encoding processing. Therefore, as shown in FIG. 11, the encoding process of the depth map can be equalized more than the first embodiment, and the recording of the RAW data can be maximally performed.

  Here, the divided area is re-divided as shown in FIG. 10 in accordance with the number of parallel processing of the RAW data encoding / decoding processing unit 800. On the other hand, the depth map having four divided areas d [0] to d [3] shown in FIG. 10 may be rearranged.

  FIG. 12 is a diagram showing an example in which partial areas shown in FIG. 10 are rearranged.

  As shown in FIG. 12, a partial area is interleaved in pixel units at the positions of R, G1, G2, and B in the Bayer array to generate a rearranged depth map. In this way, parallel encoding processing can be performed as if it were RAW data without being aware of the four partial regions.

  As described above, in the second embodiment of the present invention, the depth map is divided according to the ratio of the frame rate of the RAW data to the update interval of the depth map, and the division is further performed based on the number of parallel processing of encoding processing. The region is subdivided and the depth map is encoded. By this, it is possible to equalize the encoding load relating to the depth map, and to suppress the continuous shooting of still images and the decrease of the frame rate of moving images.

  As mentioned above, although this invention was demonstrated based on embodiment, this invention is not limited to these embodiment, The various form of the range which does not deviate from the summary of this invention is also included in this invention .

  For example, the control method may be executed by the encoding apparatus as the control method of the above-described embodiment. Alternatively, a program having the functions of the above-described embodiments may be used as a control program to cause a computer provided with the encoding apparatus to execute the control program. The control program is recorded, for example, on a computer readable recording medium.

Other Embodiments
The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or apparatus via a network or storage medium, and one or more processors in a computer of the system or apparatus read and execute the program. Can also be realized. It can also be implemented by a circuit (eg, an ASIC) that implements one or more functions.

101 control unit 102 imaging unit 103, 800 RAW data encoding / decoding processing unit 104 depth map generation unit 105 memory 106 memory I / F unit 107 image processing unit 108 display unit 110 recording processing unit 111 recording medium

Claims (11)

1. An encoding apparatus for encoding raw data obtained by imaging and a depth map indicating the depth of each pixel of the raw data to obtain coded data,
A division unit that divides the depth map into a plurality of divided areas based on a ratio of the update interval of the RAW data and the update interval of the depth map;
Encoding means for encoding the RAW data and the depth map having the plurality of divided areas by time division;
An encoding apparatus comprising:   The encoding apparatus according to claim 1, wherein the depth map is temporarily stored in a memory. The dividing means further divides the divided area into a plurality of partial areas,
The encoding apparatus according to claim 1 or 2, wherein the encoding means performs encoding processing in parallel for each of the color elements, with respect to each of the color components, a debs map having the plurality of partial regions.   4. The encoding apparatus according to claim 3, wherein the dividing unit divides the divided area in the depth map into a plurality of partial areas based on the number of parallel processes in the encoding unit.   5. The encoding apparatus according to claim 4, wherein the color elements are arranged in a Bayer array, and the dividing unit rearranges the partial areas in the depth map to correspond to the Bayer array.   In encoding processing of the RAW data at a predetermined update interval, there is provided a control means for changing the update interval of the RAW data when the amount of the depth map stored in the memory exceeds a predetermined threshold. The encoding device according to claim 2.   It has control means for discarding the depth map newly stored in the memory if the amount of depth map stored in the memory exceeds a predetermined threshold when encoding the RAW data at a predetermined update interval The encoding apparatus according to claim 2, characterized in that: Imaging means for imaging a subject to obtain RAW data;
Generation means for generating a depth map based on the RAW data;
An encoding apparatus according to any one of claims 1 to 7;
Recording means for recording encoded data obtained by the encoding device on a recording medium;
An imaging apparatus characterized by having:   9. The image pickup apparatus according to claim 8, further comprising decoding means for decoding the encoded data recorded on the recording medium to obtain decoded data. A control method of an encoding apparatus for obtaining encoded data by encoding RAW data obtained by imaging and a depth map indicating a depth of each pixel of the RAW data,
A division step of dividing the depth map into a plurality of divided areas based on a ratio of the update interval of the RAW data and the update interval of the depth map;
An encoding step of encoding the RAW data and the depth map having the plurality of divided areas by time division;
A control method characterized by comprising: A control program for use in an encoding apparatus for obtaining encoded data by encoding RAW data obtained by imaging and a depth map indicating the depth of each pixel of the RAW data, and obtaining encoded data,
In a computer included in the encoding device,
A division step of dividing the depth map into a plurality of divided areas based on a ratio of the update interval of the RAW data and the update interval of the depth map;
An encoding step of encoding the RAW data and the depth map having the plurality of divided areas by time division;
A control program characterized by causing

Images