JP2019087883A - Encoding device and program, and image processing system - Google Patents

Abstract

To provide an encoding device capable of alleviating image quality degradation due to interaction between pixels in a boundary region between a background region and a target region.SOLUTION: An encoding device that encodes an input image according to the present invention includes a target region setting unit that sets a target region corresponding to the input image and outputs target region information, a target region coordinate correction unit that corrects the target region information and outputs the corrected target region information, a pre-processing unit that identifies pixels belonging to a background region in the input image on the basis of the corrected target region information, executes filtering for limiting the dynamic range for pixels belonging to the identified background region, and output the pixels as a pre-encoded image, and an encoding unit that compresses the pre-encoded image by using a predetermined image encoding method and outputs the compressed encoded data.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an encoding apparatus and program, and an image processing system, and is applicable to, for example, a system for compressing an image or video.

  In recent years, surveillance cameras have become widespread, and it is also desired to further increase the resolution, increase the frame rate, and increase the number of viewpoints. However, higher resolution, higher frame rate, and multiple viewpoints cause a significant increase in the code amount of moving images, resulting in an increase in communication costs and storage costs.

  Therefore, in order to alleviate this problem, a method has been proposed in which a region of interest is detected from a moving image and a large number of bits are allocated to the region of interest. The attention area is, for example, a face area. Hereinafter, the region of interest will be referred to as "ROI" which is an abbreviation of Region of Interest. Further, a method of allocating a large number of bits to a region of interest is called “ROI coding”.

  For example, in Patent Document 1, it is an area other than the ROI by controlling the parameters given to the encoder for each block using the function of the encoder that the code amount and the parameter for controlling the video quality are given for each block. We propose a system configuration that reduces the amount of information in non-interest areas more than the amount of information in ROI. Hereinafter, in this specification, the non-interesting area will be referred to as a “background area”. In addition, the background area is not necessarily limited to the stationary state.

  By reducing the amount of information using the function provided by the encoder in this manner, it is possible to reduce the amount of code efficiently and reliably. H. H.264 / MPEG-4 AVC (Advanced Video Coding: hereinafter also referred to as "AVC") or H.264. As major standards such as H.265 / MPEG-H HEVC (High Efficiency Video Coding: hereinafter referred to as "HEVC") have a function to control the quality for each area, it is necessary to use this function as follows. It can be said that this is a typical method for allocating a large number of bits to the ROI.

  On the other hand, for example, due to various reasons such as cost and compatibility, when the use of an encoder not having a function of providing the video quality control parameter for each block is unavoidable, the above-described system can not be adopted. Even in such a case, for example, the technology described in Patent Document 2 or Non-Patent Document 1 is proposed as a technology capable of controlling the video quality for each area independently of the encoder.

  In order to reduce the amount of information in the background region without relying on an encoder, Patent Document 2 performs preprocessing on the background region with a low-pass filter to remove information in high-frequency components, thereby suppressing the amount of information in the background region. It is proposed to do.

  On the other hand, in Non-Patent Document 1, in order to reduce the amount of information in the background region without depending on the encoder, the amount of information in the background region is suppressed by performing preprocessing for limiting the dynamic range of pixel signals in the background region. Suggest that.

  FIG. 8 is a view showing a configuration example of a conventional image processing system represented by Non-Patent Document 1. As shown in FIG.

  In FIG. 8, the image processing system Z receives an image encoding device X1 that compresses pixels belonging to the background region of the input image by a smaller number of bits than the information of the ROI and outputs as a bit stream. And an image decoding apparatus X2 that decodes the bit stream and outputs an output image.

  The image encoding device X1 sets an ROI corresponding to an input image, and an ROI setting unit 301 that outputs ROI information, identifies a pixel belonging to a background region based on the ROI information, and belongs to the background region among the input image The pre-processing unit U1 performs filtering for limiting (suppressing) the dynamic range to the pixels, and the pre-coding unit is compressed using an image coding method such as AVC or the like, and a bit stream is output. And an encoder unit 306 for outputting. Here, the preprocessing is a process of limiting the dynamic range of the pixel signal, and also includes the case where the pixel signal is limited to a fixed value such as 128. Note that setting the pixel signal to a fixed value is equivalent to setting the dynamic range to 1.

  The ROI setting unit 301 is a function of setting an ROI corresponding to an input image, but the means for specifying the ROI is not particularly limited. For example, the input image is given as ROI auxiliary information, and by applying a face detection algorithm, a person detection algorithm, a license plate detection algorithm, a vehicle body detection algorithm, etc. to the input image, the position and size of the ROI from the input image There is a way to identify by detecting. Further, as auxiliary ROI information, for example, a method of specifying by the position and size of the ROI manually input in advance may be considered, or it may be specified by the position and size of the ROI input through the user interface. Furthermore, as the ROI auxiliary information, for example, an infrared camera image corresponding to the input image or a depth sensor image may be used to specify the ROI.

  The image decoding apparatus X2 decodes the input bit stream data according to a method corresponding to the encoder unit 306, and specifies a pixel belonging to the background area based on the ROI information and a decoder unit 307 that outputs the decoded image (output image). And a post-processing unit U2 that applies filtering (corresponding to amplification of the amplitude of the signal) for restoring the dynamic range to the pixels belonging to the background area in the decoded image and outputs the result as an output image. The preprocessing unit U1 and the post-processing unit U2 share ROI information corresponding to an image using any means such as communicating via a network using information multiplexing means .

JP, 2009-49979, A Patent No. 3046379

“Filtering Scheme for ROI Coding by Dynamic Range Compression and Updating Source Picture Filter, Proceedings of the 4th II AE International Conference on Intelligent Systems and Image Processing 2016”

  As described above, it is possible to allocate a large number of bits to the ROI by changing the dynamic range of the background region and the ROI using the preprocessing unit U1.

  However, when the dynamic range of the background region and the ROI is changed using the preprocessing unit U1, the texture of the background region and the ROI is largely different in the image before encoding. For example, it is assumed that the pixel value is aligned with “200 210 220 240” in a part of the input image, and is divided into a background area and an ROI at the boundary between 210 and 220. At this time, the compression ratio of the dynamic range is set to 0.5 (half the dynamic range), and the dynamic range is compressed around the pixel value 0. The pixel value of the image before encoding is aligned with “100 105 220 240”. It will be Compared to the difference between 210 and 220, the difference between 105 and 220 is naturally large, and this difference is expressed as that the texture is greatly different.

  On the other hand, most video encodings in recent years such as Discrete Cosine Transform and other encoding based on transformation and quantization affect each other and compress between neighboring pixels, and in the border region between background region and ROI Interaction between pixels occurs. Here, in the interaction between pixels occurring in the boundary region of the background region and the ROI, the pixel values of the background region affect the pixel values of the ROI, and conversely, the pixel values of the ROI affect the pixel values of the background region ( Pixel values leak out of the area). Also, an in-loop filter such as a deblocking filter adopted by AVC also causes an interaction between pixels in the boundary region between the background region and the ROI.

  In a system in which the dynamic range of the background region and the ROI is changed using the preprocessing unit U1, when the interaction between pixels occurs in the boundary region of the background region and the ROI, pixel values of different textures exude each other. Become. In particular, although the ROI is an area where high quality is desired to be guaranteed, the image quality is degraded due to the influence of the pixel values of the low quality background area passed through the filter.

  Therefore, there is a need for an encoding apparatus and program, and an image processing system capable of alleviating image quality degradation due to interaction between pixels in the border region between the background region and the ROI.

  According to a first aspect of the present invention, in an encoding apparatus for encoding an input image, (1) an attention area setting unit for setting an attention area corresponding to the input image and outputting attention area information; and (2) the attention A region of interest is corrected, and a region of interest coordinate correction unit for outputting corrected region of interest information, and (3) based on the corrected region of interest region, the pixels belonging to the background region in the input image are identified and identified The pre-processing unit applies filtering for limiting the dynamic range to the pixels belonging to the background area, and outputs the pre-encoding image, and (4) compresses the pre-encoding image using a predetermined image encoding method. And an encoding unit for outputting compressed encoded data.

  A second present invention is characterized in that the coding device of the first present invention is applied as the coding device in an image processing system in which the coding device and the decoding device face each other.

  According to a third aspect of the present invention, in the picture processing system in which the coding device and the decoding device face each other, the coding device of the first present invention is applied as the coding device, and (1-1) the coding Among the pixels belonging to the background area, the input image of the device is a boundary area which is a set of M (M is an integer of 1 or more) double pixels surrounding the area of interest, and of the pixels belonging to the background area The preprocessing unit of the encoding device includes (1-2) the external region of the input image based on the corrected attention region information. An external area preprocessing unit for specifying pixels belonging to the external area and performing filtering for limiting a dynamic range to the pixels belonging to the specified external area; (1-3) the input image based on the corrected attention area information Belongs to the boundary area And a preprocessing unit for boundary area that performs filtering to limit the dynamic range to the pixels that belong to the specified boundary area, and the decoding apparatus is configured to (2-1) input the encoded A decoding unit that decodes data according to a method corresponding to the predetermined image coding method used in the coding unit and outputs the decoded image as a decoded image; (2-2) the decoding based on the corrected attention area information An external area post-processing unit which identifies pixels belonging to the external area in the image and applies filtering for recovering the dynamic range to the pixels belonging to the identified external area; (2-3) the corrected attention A filter for identifying a pixel belonging to the boundary area among the decoded image based on area information, and restoring a dynamic range to the pixel belonging to the specified boundary area And having a for border region post-processing unit that performs ring.

  A coding program according to a fourth aspect of the present invention sets a computer mounted on a coding apparatus for coding an input image, (1) sets a region of interest corresponding to the input image, and outputs a region of interest information A setting unit, (2) an attention area coordinate correction unit that corrects the attention area information and outputs the corrected attention area information, and (3) a background area of the input image based on the corrected attention area information. And a pre-processing unit that performs filtering to limit the dynamic range to the pixels belonging to the identified background area, and outputs the image as a pre-encoding image, and (4) using a predetermined image encoding method It is characterized in that it functions as an encoding unit that compresses the pre-encoding image and outputs compressed encoded data.

  According to the present invention, it is possible to alleviate the image quality deterioration due to the interaction between pixels in the boundary region between the background region and the ROI.

It is a block diagram shown about composition of an image processing system concerning a 1st embodiment. It is a flowchart which shows operation | movement of the image processing system which concerns on 1st Embodiment. It is a figure which shows the example of the process which expands the magnitude | size of ROI which concerns on 1st Embodiment. It is a figure which shows the normalization example of ROI in the block which concerns on 1st Embodiment. It is a block diagram shown about composition of an image processing system concerning a 2nd embodiment. It is a flowchart which shows operation | movement of the image processing system which concerns on 2nd Embodiment. It is a figure which shows the example at the time of calculating | requiring the double boundary area which concerns on 2nd Embodiment. FIG. 1 is a diagram showing a configuration example of a conventional image processing system represented by Non-Patent Document 1.

(A) First Embodiment Hereinafter, a first embodiment of an encoding device and program according to the present invention and an image processing system will be described in detail with reference to the drawings. Hereinafter, an example in which the encoding device of the present invention is applied to an image encoding device and the decoding device of the present invention is applied to an image decoding device will be described.

(A-1) Configuration of First Embodiment FIG. 1 is a block diagram showing the configuration of an image processing system 1 according to the first embodiment.

  An image processing system 1 encodes an input image and outputs a stream (bit string), and an image encoding apparatus 10 decodes a stream (bit stream) encoded by the image encoding apparatus 10 and outputs a decoded image. And an image decoding apparatus 20.

  The medium for inputting the stream output from the image encoding device 10 to the image decoding device 20 is not limited. For example, the stream output from the image encoding device 10 may be transmitted to the image decoding device 20 by communication (for example, the Internet or the like), or the stream data output from the image encoding device 10 may be data It may be recorded on a recording medium (for example, a medium such as a DVD or HDD) and input to the image decoding apparatus 20 offline.

  In this embodiment, the image encoding device 10 is described as performing encoding for each input image, but for the moving image encoding process by causing the image encoding device 10 to process a plurality of input images continuously. It may be applied. Similarly, the image decoding apparatus 20 may be applied to the decoding process of a moving image by decoding a stream of a plurality of encoded data successively.

  Next, the internal configuration of the image coding device 10 will be described. FIG. 1 is also a block diagram showing a functional configuration of the image coding device 10 according to the first embodiment. The image coding apparatus may be configured by hardware (for example, a dedicated semiconductor chip or the like), or some or all may be configured as software.

  The image processing system 1 is different from the conventional configuration (image processing system Z) in that the image coding device X1 of FIG. 8 is replaced with the image coding device 10. In the following, the first embodiment will be described focusing on differences from the prior art.

  The image coding apparatus 10 sets an ROI corresponding to an input image, an ROI setting unit 301 that outputs ROI information, corrects ROI information, and an ROI coordinate correction unit U3 that outputs corrected ROI information; Based on the information, identify the pixels belonging to the background area, apply filtering to reduce the dynamic range to the pixels belonging to the background area of the input image, and output the pre-encoding image as the pre-processing unit U1 and the encoder unit 306 doing. The function of the encoder unit 306 has the same function as that described above with reference to FIG.

  The ROI coordinate correction unit U3 enlarges the region of the ROI based on the ROI information and outputs the ROI enlargement unit 102 that outputs it as enlarged ROI information, and the ROI coordinate normalization unit 303 that normalizes the enlarged ROI information and outputs it as corrected ROI information. And.

  The functions and purposes of the ROI enlargement unit 102 and the ROI coordinate normalization unit 303 will be described in detail in the operation section.

  Further, as described later, since the function of the ROI coordinate normalization unit 303 is a function to improve the coding efficiency, the ROI coordinate correction unit U3 has only the function of the ROI enlargement unit 102, and the ROI enlargement unit 102. Even if the output of is the corrected ROI information, a certain effect exists.

  The preprocessing unit U1 and the post-processing unit U2 share the corrected ROI information corresponding to the image using any means such as communicating via a network etc using the information multiplexing means. It assumes.

(A-2) Operation of First Embodiment Next, the operation of the image processing system 1 according to the first embodiment will be described in detail with reference to the drawings. FIG. 2 is a flowchart showing the operation of the image processing system according to the first embodiment. The operations of the image encoding device 10 and the image decoding device 20 will be separately described below.

(A-2-1) Operation of Image Encoding Device 10 FIG. 2A is a flowchart showing the operation of the image encoding device according to the first embodiment.

  The ROI setting unit 301 determines an initial value of the ROI based on the ROI auxiliary information, and outputs it as ROI information (S11).

  Next, the ROI enlargement unit 102 enlarges the size of the ROI by N pixels in the peripheral direction, and outputs it as enlargement ROI information (S12).

  FIG. 3 is a diagram showing an example of processing of enlarging the size of the ROI according to the first embodiment. FIG. 3 shows an example in which the size of the ROI is enlarged (expanded) by 2 pixels in the peripheral direction with respect to the input ROI portion (hatched portion). Each number in the dilated ROI portion indicates a multiple structure expanded (dilated) from the ROI portion (for example, a region directly in contact with the ROI portion is shown as “1”). There are various methods of dilation, but a typical example is dilation often used in image processing.

  The value of N is arbitrary, and in the encoder unit 306 and the decoder unit 307, the optimal value changes depending on how many pixels in the vicinity affect and influence each other.

  Although the value of N is not limited in this embodiment, many codecs perform compression for each block formed by 4 × 4 pixels and each block formed by 8 × 8 pixels. It can be expected that good results can be obtained by setting 1 ≦ N ≦ 8 even if ≦ N ≦ 4 or more. In addition, when the encoder unit 306 compresses with a sufficiently small quantization step width, the interaction associated with conversion and quantization may be ignored. As described above, in the case where compression is performed with a sufficiently small quantization step width and it is sufficient to consider only the interaction by the in-loop filter, and in the case of using AVC or HEVC, these video encoding schemes Since the loop filter exudes the pixel values of adjacent two pixels, it can be said that N = 2 or around is a preferable value.

  If N is too large, it will lead to an increase in the area to divide the code amount by a large amount needlessly, so it is necessary to set an appropriate value according to the encoder unit 306 and decoder unit 307 to be used.

  The purpose of this process is to widen the range of the ROI to prevent the image quality of the ROI from being degraded due to the influence of the pixel values of the low quality background area passed through the filter. In the image processing system 1 in which a large amount of code is allocated to the ROI, the ROI is often an area where it is desired to guarantee particularly high quality, so the ROI is expanded to guarantee the quality of the ROI.

  In particular, in the case of an application in which a zoomed area is set as an ROI in the ROI setting unit 301, the ROI is displayed in an enlarged manner. Here, if the ROI is degraded due to the exudation of the background area, the viewer will see a noticeable artifact. By using the image processing system 1 of this embodiment, it is possible to provide the viewer with an ROI with guaranteed quality even in such a scene.

  Next, the ROI coordinate normalization unit 303 normalizes the boundary between the background area and the ROI so as to be aligned with the block boundary at the time of compression by the codec, and outputs it as corrected ROI information (S13).

  The normalization is performed, for example, on the basis of any one of the following (1) to (3).

  (1) If even one pixel among the pixels belonging to a block of a predetermined size is included in the ROI, it is assumed that all the pixels included in the block belong to the ROI.

  (2) If half or more of the pixels belonging to a block of a predetermined size are included in the ROI, it is assumed that all the pixels included in the block belong to the ROI.

  (3) The pixels included in the block are considered to belong to the ROI only when all the pixels belonging to the block of the predetermined size are included in the ROI.

  FIG. 4 is a diagram showing an example of ROI normalization in the block according to the first embodiment. FIG. 4 is an image of the case where normalization is performed based on (1) of the reference examples described above.

  In FIG. 4, the size of each block (B1 to B3) is 4 pixels wide and 4 pixels high. For example, it is assumed that the pixels included in the block B2 belong to the ROI because the number of pixels included in the ROI before normalization is 4 (that is, because it is 1 or more).

  In many image coding methods that can be adopted by the encoder unit 306, the amount of information is reduced by performing prediction or conversion for each block of a specific size. If a sharp edge is generated in the block, many AC components are generated, which causes an increase in the code amount. That is, it is desirable not to generate an unnecessary edge in the block, so that the boundary between the background region and the ROI is matched with the boundary of the block adopted by the image coding method by normalization.

  As described above, various methods can be considered for normalization, all of which have a certain effect, but the pixels selected as the ROI are treated as a background region in comparison with the function and purpose of the ROI enlargement unit 102 That is not preferable because it causes the possibility that the pixel values in the background area may leak into the original ROI, and in this aspect, the criterion (1) is most desirable among the above three criteria.

  Next, the preprocessing unit U1 specifies the pixels belonging to the background area based on the corrected ROI information, performs filtering for limiting the dynamic range to the pixels belonging to the background area in the input image, and outputs as an image before encoding To do (S14). For the filtering that limits the dynamic range, for example, the technology described in Non-Patent Document 1 can be applied. However, in this embodiment, the central pixel value at the time of limiting the dynamic range is not particularly limited, and even if the dynamic range is set to 1, the present invention exhibits certain effects.

  Next, the encoder unit 306 compresses the uncoded image using an image coding method such as AVC, and outputs a bit stream (S15).

(A-2-2) Operation of Image Decoding Device X2 FIG. 2 (B) is a flowchart showing the operation of the image decoding device X2 according to the first embodiment.

  First, the decoder unit 307 receives a bit stream from the image coding apparatus 10, decodes it using an image coding method such as AVC, and outputs a decoded image (S21).

  The post-processing unit U2 specifies pixels belonging to the background area based on the corrected ROI information, performs filtering for recovering the dynamic range on pixels belonging to the background area in the decoded image, and outputs the result as an output image (S22). The filtering for recovering the dynamic range can be applied to the technique described in Non-Patent Document 1 as in the case of the preprocessing unit U1, but the filter is not limited to the method described in Non-Patent Document 1 as long as it is a filter for recovering the dynamic range.

(A-3) Effects of the First Embodiment As described above, according to the first embodiment, in the ROI enlargement unit 102, the size of the ROI is greatly enlarged in the peripheral direction by N pixels. It is possible to prevent the image quality of the original ROI from being degraded due to the influence of the pixel values of the low quality background area passed through the filter.

(B) Second Embodiment Hereinafter, a second embodiment of the encoding device and program according to the present invention and the image processing system will be described in detail with reference to the drawings. Hereinafter, an example in which the encoding device of the present invention is applied to an image encoding device and the decoding device of the present invention is applied to an image decoding device will be described.

(B-1) Configuration of Second Embodiment FIG. 5 is a block diagram showing the configuration of an image processing system 1A according to the second embodiment.

  The image processing system 1A is different from the first embodiment in that the image coding device 10 and the image decoding device X2 in FIG. 1 are replaced with the image coding device 10A and the image decoding device 20. Hereinafter, the second embodiment will be described focusing on differences from the first embodiment.

  In the image encoding device 10A of the second embodiment, the preprocessing unit U4 specifies pixels belonging to the external region based on the corrected ROI information, and limits the dynamic range to pixels belonging to the external region in the input image. An external area preprocessing unit 304 that performs filtering and outputs as a preprocessing intermediate image, and identifies pixels belonging to the boundary area based on the corrected ROI information, and dynamic range to pixels belonging to the boundary area of the preprocessing intermediate image And a boundary region pre-processing unit 205 that performs filtering to limit and outputs as a pre-coding image.

  The boundary region is a set of M-fold pixels surrounding the ROI among the pixels belonging to the background region, and the outer region is a set of pixels other than the boundary region among the pixels belonging to the background region. The details of the external area preprocessing unit 304 and the boundary area preprocessing unit 205 will be described in the section of operation.

  In this embodiment, the external area preprocessing unit 304 is configured to be applied earlier than the boundary area preprocessing unit 205. However, the pixels and the boundary area to be processed by the external area preprocessing unit 304 are to be processed. Since the pixels to be processed by the pre-processing unit 205 are different, the boundary area pre-processing unit 205 may be applied prior to the external area pre-processing unit 304, and the external area pre-processing unit 304 and the boundary area may be applied. The pre-processing unit 205 may be applied simultaneously. That is, it has a meaning that the preprocessing unit U4 is configured by the functions of the external region preprocessing unit 304 and the boundary region preprocessing unit 205.

  In the image decoding apparatus 20 according to the second embodiment, the post-processing unit U5 identifies pixels belonging to the external area based on the corrected ROI information, and restores the dynamic range to pixels belonging to the external area of the decoded image. And an external region post-processing unit 308 that outputs the post-processing intermediate image, and identifies pixels belonging to the boundary region based on the corrected ROI information, and dynamically applies pixels belonging to the boundary region of the post-processed intermediate image It has a boundary area post-processing unit 209 which performs filtering for recovering the range and outputs it as an output image. The details of the external area post-processing unit 308 and the boundary area post-processing unit 209 will be described in the section of operation.

  In this embodiment, the external area post-processing unit 308 is configured to be applied before the boundary area post-processing unit 209, but the pixels and the boundary area to be processed by the external area post-processing unit 308 Since the pixels to be processed by the post-processing unit 209 are different, the boundary region post-processing unit 209 may be applied before the external region post-processing unit 308, and the external region post-processing unit 308 and the boundary region The post-processing unit 209 may be applied simultaneously. That is, it has meaning that the post-processing unit U5 is configured by the functions of the external region post-processing unit 308 and the boundary region post-processing unit 209.

  The preprocessing unit U4 and the post-processing unit U5 share the corrected ROI information corresponding to the image using any means such as communicating via a network etc using the information multiplexing means I assume.

  As described later, one of the purposes for introducing the combination of the external area preprocessing unit 304 and the boundary area preprocessing unit 205 and the combination of the external area post processing unit 308 and the boundary area post processing unit 209 is It is to provide a better alternative to the ROI coordinate normalization unit 303. Therefore, even if the ROI coordinate correction unit U3 does not have the function of the ROI coordinate normalization unit 303 and outputs the enlarged ROI information as the corrected ROI information, the present invention has a certain effect.

  Further, the purpose of introducing the combination of the external region preprocessing unit 304 and the boundary region preprocessing unit 205 and the combination of the external region post-processing unit 308 and the boundary region post-processing unit 209 is that the ROI coordinate normalization unit 303 In addition to providing a better alternative, in the case where the ROI coordinate correction unit U3 includes the function of the ROI enlargement unit 102 and the function of the ROI coordinate normalization unit 303 as in the first embodiment, Also, the present invention has certain effects.

  Also, as described later, even if ROI coordinates are obtained by introducing the combination of the external region preprocessing unit 304 and the boundary region preprocessing unit 205 and the combination of the external region post processing unit 308 and the boundary region post processing unit 209. Even if the correction unit U3 does not have the function of the ROI enlargement unit 102 or the function of both the ROI enlargement unit 102 and the ROI coordinate normalization unit 303, sharp edges are not generated at the boundary between the background region and the ROI. (Because the amount of change in the edge becomes loose), deterioration of the quality of the background area and the ROI due to the exudation due to the interaction between the pixels occurring at the boundary area of the background area and the ROI can be suppressed. Therefore, even if the ROI coordinate correction unit U3 does not have the function of the ROI enlargement unit 102 or the function of both the ROI enlargement unit 102 and the ROI coordinate normalization unit 303, the present invention exerts a certain effect.

(B-2) Operation of Second Embodiment FIG. 6 is a flowchart showing an operation of the image processing system according to the second embodiment. In FIG. 6 and FIG. 2 described above, the processes with the same reference numerals are the same processes, so only the process specific to the image processing system 1A of the second embodiment will be described below.

(B-2-1) Operation of Image Encoding Device 10A FIG. 6A is a flowchart showing the operation of the image encoding device 10A according to the second embodiment.

After the above-described step S13, the external region preprocessing unit 304 identifies the pixels belonging to the external region based on the corrected ROI information, and applies filtering for limiting the dynamic range to the pixels belonging to the external region in the input image. , And output as a preprocessed intermediate image (S31). The process of step S31 has a function corresponding to the pre-processing unit U1 in the first embodiment. The external area preprocessing unit 304 limits the dynamic range of the pixels belonging to the external area to the dynamic range (hereinafter referred to as “R ₀ ”) for the pixels belonging to the external area.

  Next, the boundary region pre-processing unit 205 identifies pixels belonging to the boundary region based on the corrected ROI information, performs filtering to limit the dynamic range to pixels belonging to the boundary region among the preprocessing intermediate images, and Output as an image before conversion (S32).

  Here, as described above, the boundary area is a set of M-fold pixels surrounding the ROI among the pixels belonging to the background area. The set of pixels surrounding the ROI is, for example, determined using dilation often used in image processing. The first dilation is applied to the ROI, and the newly expanded region is the first boundary region. The expansion is applied one more time, and the newly expanded area is the second boundary area. FIG. 7 is a diagram showing an example in the case where double boundary regions according to the second embodiment are obtained.

Note that the boundary area preprocessing unit 205 limits the dynamic range of the pixels belonging to the boundary area to a dynamic range wider than the dynamic range R ₀ for pixels belonging to the external area (that is, higher image quality than the external area). Compressed). That is, it can be said that the boundary area is a buffer zone of the gap between the ROI and the texture of the external area. When the border area is double or more, the dynamic range may be different between the x-th border area and the x + 1-th border area. In this case, the dynamic range of the border area located on the ROI side It is assumed that R _x is a dynamic range of a size greater than or equal to the dynamic range R _{x + 1 of} the boundary region located outside of it.

That is, assuming that the dynamic range of the ROI is R _r , the relationship of the following equation (1) is satisfied.
R ₀ <. . . ≦ R _{x + 1} ≦ R _x ≦. . . , R R ₁ <R _r (1)

The numerical values described in each pixel in FIG. 7 are an example of the dynamic range of the pixel, and the dynamic range of the ROI is maximized to R _r = 256 and the dynamic range of the first boundary region to R 1 = 128. The dynamic range of the _second boundary region is R ₂ = 64, and the dynamic range of the outer region is R ₀ = 32. In addition, although it is variously considered how many dynamic ranges are given to what number of layers, such as a linear change and a non-linear change, in any case, for example, it shall be decided in advance, the pre-processing unit U4 and the post-processing unit It shall be shared with U5.

  In the second embodiment, the reason for providing the boundary area is mainly as follows.

  In the first embodiment, the increase in the code amount due to the entry of an edge due to the boundary between the background region and the ROI in the block is considered as a problem, and the ROI coordinate normalization unit 303 is used as a solution. However, although this solution contributes to the alleviation of the problem, it still leaves room for efficiency improvement in that it increases the number of pixels that must be allocated code amounts equivalent to the ROI and generates unnecessary code amounts. ing. In the second embodiment, the external area and the ROI are connected smoothly through the boundary area, thereby minimizing the increase in the number of pixels for which the code equivalent to the ROI must be allocated, and the background area in the block. It also prevents an increase in the code amount due to the edge entering due to the boundary between and and the ROI.

  When a codec using an in-loop filter such as a deblocking filter is used for the encoder unit 306 and the decoder unit 307, the in-loop filter is a background region and ROI even if the boundary between the background region and the ROI is aligned with the block boundary. Generate inter-pixel interactions in the border region of If the ROI enlargement unit 102 is present, the quality of the ROI is not impaired, but the pixel value of the ROI exudes to the background area. For pixels in the background area, pixel values of ROIs with different textures become noise. Further, when the post-processing unit U2 is applied, since the post-processing unit U2 corresponds to the amplification processing of the amplitude, the noise due to the exudation of the pixel value of the ROI is amplified in the background region near the ROI. As a result, the quality of the background area is lost more than necessary. Since the in-loop filter becomes stronger as the gap between adjacent pixel values becomes larger, in the second embodiment, this problem is alleviated by smoothly connecting the external region and the ROI via the boundary region. .

  Then, by connecting the external area and the ROI smoothly through the boundary area, the block of the ROI adjacent to the background area or the block of the background area adjacent to the ROI is in the intra block mode (AVC or HEVC Etc. intra-prediction becomes more effective and coding efficiency improves.

  Although the size of M is arbitrary, it is assumed that they are shared by the pre-processing unit U4 and the post-processing unit U5. Further, like the value of N in the first embodiment, in the encoder unit 306 and the decoder unit 307, it is optimal depending on how many pixels of adjacent pixels affect each other (that is, kernel size of in-loop filter). The value changes. In many codecs, compression is often performed for each block formed by 4 × 4 pixels and each block formed by 8 × 8 pixels, so 1 ≦ M ≦ 4 or at most 1 ≦ M ≦ 8. Good results can be expected. Further, in the in-loop filter of HAVC or HEVC, the pixel values of two neighboring pixels exude, so it can be said that M = 2 or around is the most preferable value.

  As in the case of N, if M is too large, it will lead to an increase in the area for dividing the code amount by much, so it is necessary to set an appropriate value according to the encoder unit 306 and decoder unit 307 used.

(B-2-2) Operation of Image Decoding Device 20 FIG. 6 (B) is a flowchart showing the operation of the image decoding device 20 according to the second embodiment.

  After the above-described step S21, the external region post-processing unit 308 identifies the pixels belonging to the external region based on the corrected ROI information, and applies filtering for recovering the dynamic range to the pixels belonging to the external region in the decoded image. , And output as a post-processing intermediate image (S41). The process of step S41 corresponds to the inverse operation of the external region preprocessing unit 304, and has a function corresponding to the post-processing unit U2 in the first embodiment.

  The boundary region post-processing unit 209 identifies pixels belonging to the boundary region based on the corrected ROI information, performs filtering to restore the dynamic range to pixels belonging to the boundary region of the post-processing intermediate image, and outputs as an output image (S42). The process of step S42 has a function equivalent to the inverse operation of the boundary area preprocessing unit 205.

(B-3) Effects of the Second Embodiment As described above, according to the second embodiment, in addition to the effects of the first embodiment described above, the following effects can be obtained.

  When the function of the ROI coordinate normalization unit 303 is enabled in the ROI coordinate correction unit U3 by dividing the background region into the external region and the boundary region and applying a filter that limits each signal to different dynamic ranges In comparison, it is possible to prevent an increase in the code amount due to the edge due to the boundary between the background region and the ROI entering in the block while minimizing the increase in pixels that must be allocated the code amount equivalent to the ROI.

  Furthermore, the boundary between the ROI and the background area smoothly changes, and the problem that the quality of the background area is lost more than necessary is alleviated, or the block of the ROI adjacent to the background area or the ROI is adjacent When a block in the background region is encoded in the intra block mode, intra prediction is more effective and coding efficiency can be improved.

(C) Other Embodiments Although various modified embodiments are mentioned in the first and second embodiments described above, the present invention can also be applied to the following modified embodiments.

  (C-1) In the first and second embodiments described above, an example is shown in which the entire image is divided into two types, ROI and background area, in order to simplify the description. There may also be a level of importance. For example, a pedestrian detection algorithm and a face detection algorithm are used for setting an ROI, and an ROI detected by the face detection algorithm is a face area, and a pedestrian area detected by the pedestrian detection algorithm is a relatively important background area An area that is neither a face nor a pedestrian may be a background area that is relatively unimportant. In this case, different values may be allocated to different dynamic ranges according to the importance of the background area, for example, relatively important background areas are compressed with a wide dynamic range, and relatively less important background areas are compressed. May be compressed in a narrow dynamic range. The allocation may be performed, for example, based on the background area importance and the filter strength look-up table set or created in advance. Further, although an example in which filtering is not applied to the ROI is described above for the sake of simplicity of the description, the ROI may be filtered more weakly than the background region.

  (C-2) In the first and second embodiments described above, for example, there is shown an example in which the background area is degraded when the face area is set as the ROI to reduce the data amount, but the ROI like the face area The area selected as may be degraded. In this case, in the definition of words in the present specification, the non-face area is the ROI, and the face area is the background area. By setting the non-face area as the ROI, for example, the effect of being able to protect privacy can be obtained.

  (C-3) In the first and second embodiments described above, it is described that data exchange between functional blocks is performed in image units, but the meaning of data of pixel value signals is clarified In implementation etc., data may be delivered in pixel units.

  (C-4) In the first and second embodiments described above, the ROI is mainly described as a limited area of a part of the image space, but the entire image is the ROI or the entire image There may be an image in which the background area is the background area.

  (C-5) In the first and second embodiments described above, each function in the image encoding device 10 (10, 10A) and each function in the image decoding device X2, 20 are included in a single device. Although described as being implemented, these are only one implementation example, and even if each function is implemented by another device, each signal is shown in each configuration diagram (FIG. 1, FIG. 5). If the input / output is performed as described above, the effect of the present invention is obtained. That is, even if the image encoding device 10 (10, 10A) and the image decoding devices X2 and 20 are configured of one or more devices and the respective functions are distributed and implemented, the effects of the present invention still exist.

  (C-6) In the first and second embodiments described above, differences in processing for each color component are not particularly mentioned, but in each embodiment, the same dynamic range may be used for each color component. You can use different dynamic ranges.

  (C-7) In the first and second embodiments described above, the image processing systems 1 and 1A show an example in which the image decoding devices X2 and 20 are provided. Even when only the encoding device 10 (10, 10A) is provided, the present invention has a certain degree of effect.

  As described in Non-Patent Document 1, ROI encoding is performed by limiting the dynamic range by preprocessing. In the image processing systems 1 and 1A, preprocessing is performed by the image encoding device 10 (10, 10A). The image processing of the background area is restored to a certain extent by performing the corresponding post-processing in the image decoding device X2, 20. However, as also described in Non-Patent Document 1, an image that does not know that ROI coding is being performed by pre-processing in a conventional image decoding device (image coding device 10 (10, 10A) without post-processing is known Even when decoded by the decoding device), only the color tone differs between the ROI and the background area, and an image in which a human can decipher a certain meaning can be obtained. Then, even with this configuration, the problems targeted by the present invention such that the pixel values of the background region ooze out to the ROI and the pixel values of the ROI ooze out to the background region occur, and the problems are solved by introducing the present invention. It can be solved. Therefore, even when the image processing system 1 or 1A includes only the image encoding device 10 (10, 10A), the present invention has an effect to a certain extent.

1, 1A, Z: image processing system, 10, 10A, X1: image coding device, 20, X2: image decoding device, 102: ROI enlargement unit, 205: boundary region preprocessing unit, 209: boundary region for back Processing unit 301: ROI setting unit 303: ROI coordinate normalization unit 304: external region preprocessing unit 306: encoder unit 307: decoder unit 308: external region post-processing unit U1: preprocessing unit , U2 ... post-processing unit, U3 ... ROI coordinate correction unit, U4 ... pre-processing unit, U5 ... post-processing unit.

Claims (15)

In a coding device for coding an input image,
An attention area setting unit which sets an attention area corresponding to the input image and outputs attention area information;
An attention area coordinate correction unit that corrects the attention area information and outputs the corrected attention area information;
Based on the corrected attention area information, the pixels belonging to the background area in the input image are identified, and the pixels belonging to the identified background area are subjected to filtering for limiting the dynamic range, and output as an image before encoding A pre-processing unit that
An encoding unit that compresses the image before encoding using a predetermined image encoding method and outputs encoded data that has been compressed. The attention area coordinate correction unit includes an attention area enlargement unit that enlarges the attention area in the peripheral direction by N (N is an integer of 1 or more) pixels based on the attention area information.
The encoding device according to claim 1, wherein the attention area coordinate correction unit outputs enlarged attention area information expanded using the attention area expansion unit as the corrected attention area information. The attention area coordinate correction unit has an attention area coordinate normalization unit that normalizes the enlarged attention area information so that the boundary between the background area and the attention area is aligned with a block boundary.
The encoding apparatus according to claim 2, wherein the attention area coordinate correction unit outputs the enlarged attention area information normalized using the attention area coordinate normalization unit as the corrected attention area information. .   The region-of-interest coordinate normalization unit considers that all pixels included in the block belong to the region of interest if at least one of the pixels belonging to the block of a predetermined size is included in the region of interest. The encoding apparatus according to claim 3, wherein normalization is performed. The predetermined image coding method is as follows. H.264 / MPEG-4 AVC or H.264. 265 / MPEG-H HEVC,
The encoding apparatus according to any one of claims 2 to 4, wherein the N is 1 or 2 or 3. The input image is a boundary area which is a set of M (M is an integer of 1 or more) double pixels surrounding the target area among the pixels belonging to the background area, and the boundary among the pixels belonging to the background area And an external area which is a set of pixels other than the area,
The pre-processing unit
An external area preprocessing unit for specifying pixels belonging to the external area among the input image based on the corrected attention area information and performing filtering for limiting a dynamic range to the pixels belonging to the specified external area;
A boundary area preprocessing unit for specifying pixels belonging to the boundary area among the input image based on the corrected attention area information and performing filtering for limiting the dynamic range to the pixels belonging to the specified boundary area; The coding apparatus according to any one of claims 1 to 5, wherein the coding apparatus is provided.   7. The encoding apparatus according to claim 6, wherein a dynamic range of pixels belonging to the boundary area is limited to a dynamic range wider than a dynamic range for pixels belonging to the external area.   When the boundary area is double or more, the dynamic range of the Y (Y is an integer of 1 or more) second boundary area is larger than the dynamic range of the Y + 1th boundary area. The encoding apparatus according to claim 6 or 7, characterized in that: The predetermined image coding method is as follows. H.264 / MPEG-4 AVC or H.264. 265 / MPEG-H HEVC,
The encoding apparatus according to any one of claims 6 to 8, wherein the M is 1 or 2 or 3.   The said attention area coordinate correction | amendment part is not equipped with the said attention area expansion part, The said attention area information is input into the said attention area coordinate normalization part as said expansion attention area information. The encoding device according to any one.   10. The encoding apparatus according to any one of claims 6 to 9, wherein the encoding apparatus does not include the attention area coordinate correction unit, and the attention area information is directly input as the corrected attention area information to the preprocessing unit. The encoding device according to claim 1. In the picture processing system in which the encoding device and the decoding device face each other,
A picture processing system characterized in that the coding device according to any one of claims 1 to 11 is applied as the coding device. The decoding device is
A decoding unit that decodes the input coded data according to a method corresponding to the predetermined image coding method used by the coding unit, and outputs the decoded data as a decoded image;
A post-processing unit that identifies pixels belonging to the background region among the decoded image based on the region of interest information, applies filtering to restore the dynamic range to the pixels belonging to the identified background region, and outputs as an output image The picture processing system according to claim 12, characterized in that it comprises: In the picture processing system in which the encoding device and the decoding device face each other,
The encoding device according to any one of claims 6 to 11 is applied as the encoding device,
The decoding device is
A decoding unit that decodes the input coded data according to a method corresponding to the predetermined image coding method used by the coding unit, and outputs the decoded data as a decoded image;
An external region post-processing unit that identifies pixels belonging to the external region among the decoded image based on the corrected attention region information, and performs filtering for restoring a dynamic range to the pixels belonging to the identified external region. When,
A post-process for boundary area which specifies a pixel belonging to the boundary area among the decoded image based on the corrected attention area information, and performs filtering for restoring a dynamic range to the pixel belonging to the specified boundary area An image processing system comprising: A computer mounted on a coding device for coding an input image,
An attention area setting unit which sets an attention area corresponding to the input image and outputs attention area information;
An attention area coordinate correction unit that corrects the attention area information and outputs the corrected attention area information;
Based on the corrected attention area information, the pixels belonging to the background area in the input image are identified, and the pixels belonging to the identified background area are subjected to filtering for limiting the dynamic range, and output as an image before encoding A pre-processing unit that
An encoding program characterized in that the image before encoding is compressed using a predetermined image encoding method, and the encoding program functions as an encoding unit that outputs compressed encoded data.

Images