JP6294708B2 - Image processing apparatus and program - Google Patents

Description

  The present invention relates to an image processing apparatus and a program for encoding a low resolution image cut out from a high resolution image.

  2. Description of the Related Art Conventionally, there has been proposed an image processing apparatus that extracts a part of an image from a high-resolution image having a large number of pixels and outputs the image as a low-resolution image having a smaller number of pixels than the high-resolution image. For example, an apparatus that captures a high resolution image, encodes and transmits a low resolution screen cut out from the image is known (see Patent Document 1).

  In addition, since the development of equipment that handles ultra-high-definition video such as 8K Super Hi-Vision and 4K is progressing, a device that cuts out a small size like 2K Hi-Vision from an ultra-high-definition video with a large image size, It is used as a new video production tool (see Non-Patent Document 1). In such an apparatus, an image at an arbitrary position can be cut out from an original large-size image, and output in, for example, a high-vision size. It is also possible to vary the position to be cut out at an arbitrary speed in terms of time.

  On the other hand, MPEG-2 and AVC / H. 264, HEVC / H. A general compression encoding method such as H.265 performs motion estimation. Motion estimation predicts the magnitude and direction (motion vector) of the motion of an image in units such as blocks. Although the accuracy of prediction differs depending on the method, the prediction efficiency is increased by using 1/2 pixel accuracy or 1/4 pixel accuracy instead of 1 pixel accuracy. In this case, the values of ½ pixel and ¼ pixel between the original pixels are generated by interpolation after applying an interpolation filter or the like. Since the amount of motion of the subject in the video is usually not 1 pixel accuracy, the prediction efficiency is improved by setting it to 1/2 pixel accuracy or higher.

JP 2004-336805 A

“High-performance“ small SHV down converter ”developed for the first time! -Dramatically improved SHV "One Resource Multi-use" program production- ", [online], April 17, 2009, NHK, [March 28, 2013 search], Internet <URL: http : //www.nhk.or.jp/pr/marukaji/pdf_ver/231.pdf>

  However, as the prediction accuracy of motion estimation increases, the amount of motion vector information increases. When the amount of motion vector information increases, the amount of processing for interpolation and interpolation increases, and the memory for reference images required for motion compensation also increases. There is a problem that it is necessary. In other words, the extraction process from the high-resolution image can be performed at an arbitrary position and with an arbitrary accuracy. However, if the accuracy of the extraction position is smaller than one pixel, the motion vector information is used when compressing and encoding the extracted image. There is a problem that the amount of processing, the amount of processing, reference memory for motion compensation, and the like increase.

  An object of the present invention made in view of such circumstances is to provide an image processing apparatus and program capable of encoding a low-resolution image cut out from a high-resolution image in consideration of the amount of generated motion vector information. There is to do.

In order to solve the above problems, an image processing apparatus according to the present invention is an image processing apparatus that encodes a low-resolution image cut out from a high-resolution image according to a cut-out position for each frame, and the motion of the low-resolution image A motion estimation unit that generates a motion vector by performing estimation, outputs a motion compensated image compensated for motion using the motion vector, and outputs a motion vector information amount indicating an information amount of the motion vector; and the low resolution An encoding unit that encodes a difference image between an image and the motion compensated image and outputs encoded data; and an encoding control unit that controls movement accuracy of the cut-out position between frames based on the amount of motion vector information When, wherein the encoding control unit, the motion or amount vector information exceeds a threshold value, or the total amount of the motion vector information amount and the amount of information of the encoded data The ratio of the motion vector information amount the movement accuracy is characterized that you control to be lower when exceeding the threshold value.

  Furthermore, the image processing apparatus according to the present invention further includes an image quality evaluation unit that calculates a difference value between the low-resolution image and a locally decoded image of the low-resolution image, and the encoding control unit includes the difference value Control is performed so that the movement accuracy is increased when a threshold value is exceeded.

  Furthermore, the image processing apparatus according to the present invention further includes a cutout processing unit that generates the low resolution image from the high resolution image according to the cutout position determined with the movement accuracy.

  Further, in the image processing apparatus according to the present invention, the cutout processing unit outputs information on a moving speed of the cutout position between frames, and the motion estimation unit performs the motion estimation using the information on the moving speed. Generating the motion vector.

  In order to solve the above problems, a program according to the present invention causes a computer to function as the image processing apparatus.

  According to the present invention, it is possible to encode a low resolution image cut out from a high resolution image in consideration of the amount of motion vector information generated.

1 is a block diagram illustrating a configuration example of an image processing apparatus according to a first embodiment of the present invention. It is a figure which shows the low-resolution image cut out from a high-resolution image and a high-resolution image. 3 is a flowchart showing an operation of the image processing apparatus according to the first embodiment of the present invention. It is a block diagram which shows the structural example of the image processing apparatus which concerns on the 2nd Embodiment of this invention.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

(First embodiment)
FIG. 1 is a block diagram illustrating a configuration example of an image processing apparatus according to the first embodiment of the present invention. In the example illustrated in FIG. 1, the image processing apparatus 1 includes a clipping device 10 and an encoding device 20. In the present embodiment, the cutting device 10 and the encoding device 20 are separated from each other, but the cutting device 10 and the encoding device 20 may be integrated to form the image processing device 1. Further, the clipping device 10 may be provided outside the image processing device 1, and the image processing device 1 may include only the encoding device 20 that encodes the low-resolution image input from the clipping device 10.

  The cutout device 10 sets a cutout position of the high resolution image for each frame, generates a low resolution image cut out according to the cutout position, and outputs the low resolution image to the encoding device 20. For example, a high-resolution image such as 8K (7680 × 4320 pixels) or 4K (3840 × 2160 pixels) is cut into a high-vision size (1920 × 1080 pixels). The cutting device 10 includes a parameter setting unit 11 and a cutting processing unit 12.

  FIG. 2 is a diagram illustrating a high resolution image and a low resolution image cut out from the high resolution image. Hereinafter, the cutting device 10 will be described with reference to FIG. FIG. 2 shows an example in which a scene to be taken is extracted from a high-definition video of a football broadcast (players and spectators are omitted) taken from a bird's-eye view to obtain a high-definition video.

  The parameter setting unit 11 sets parameters including the cutting position for the high-resolution image A input to the cutting processing unit 12, the size of the cut image, and the movement accuracy of the cutting position between frames, and cuts the setting information. To the unit 12. The movement accuracy means a minimum pixel unit when moving the cutout position, and can be selected from, for example, 1 pixel, 1/2 pixel, and 1/4 pixel. Since there is no pixel with a precision finer than one pixel, when ½ pixel or ¼ pixel is selected, the original image is interpolated to generate a pixel between pixels, and then cut out. In order to perform the insertion process, the amount of processing increases, and a memory for storing video is also required separately.

  There are no particular restrictions on how to set the parameters. For example, the high-resolution image A is displayed on the display unit (not shown) of the cutting apparatus 10 and a rectangular frame B indicating the frame (cutting position) of the image to be cut out is superimposed and displayed, and the user interface is used. The rectangular frame B is moved between the frames with a predetermined movement accuracy to set the cutout position. The cutout processing unit 12 cuts out an image surrounded by the rectangular frame B and outputs it as a low resolution image C.

  The position of the rectangular frame B can be determined by moving it by operating a user interface such as a dial. For example, when each frame of a sports broadcast video is input as the high resolution image A, the user operates the user interface and moves the rectangular frame B as needed so that the image to be noticed is included. The size of the rectangular frame B is fixed to 1920 × 1080 pixels, for example, when cutting out to a high-vision size. When moving the cutout position, the rectangular frame B may be moved manually, or the rectangular frame B may be moved at a preset speed / direction. An initial value is set for the movement accuracy of the rectangular frame B, and can be changed by a control signal input from the encoding control unit 30 as described later.

  The cutout processing unit 12 cuts out an image from the high resolution image A according to the parameters set by the parameter setting unit 11 and outputs the cut out low resolution image C to the encoding device 20.

  The encoding device 20 is, for example, an AVC / H. H.264 and HEVC / H. Compression encoding is performed by a method compliant with H.265 or the like. In the example illustrated in FIG. 1, the encoding device 20 includes a subtraction unit 21, an orthogonal transformation unit 22, a quantization unit 23, an inverse quantization unit 24, an inverse orthogonal transformation unit 25, an addition unit 26, and a memory. 27, a motion estimation unit 28, an encoding unit 29, and an encoding control unit 30.

  The subtractor 21 generates a difference image by subtracting the low resolution image input from the clipping device 10 and the motion compensated image input from the motion estimator 28, and outputs the difference image to the orthogonal transform unit 22.

  The orthogonal transform unit 22 performs orthogonal transform (for example, DCT; Discrete Cosine Transform) processing on the difference image input from the subtraction unit 21 to generate an orthogonal transform coefficient and outputs it to the quantization unit 23. .

  The quantization unit 23 performs a quantization process on the orthogonal transform coefficient input from the orthogonal transform unit 22 so as to match the required bit rate, and the quantized orthogonal transform coefficient is encoded by the inverse quantization unit 24 and the encoding unit. To the unit 29.

  The inverse quantization unit 24 performs an inverse quantization process on the quantized orthogonal transform coefficient input from the quantization unit 23 and outputs the generated orthogonal transform coefficient to the inverse orthogonal transform unit 25.

  The inverse orthogonal transform unit 25 performs inverse orthogonal transform (for example, IDCT; Inverse Discrete Cosine Transform) processing on the orthogonal transform coefficient input from the inverse quantization unit 24 to generate a local decoded difference image, and an adder 26 Output to.

  The adding unit 26 adds the local decoded differential image input from the inverse orthogonal transform unit 25 and the motion compensated image input from the motion estimation unit 28 to generate a local decoded image, and outputs the local decoded image to the memory 27.

  The motion estimation unit 28 searches for a similar part by referring to a locally decoded image (reference image) stored in the memory 27 for each part obtained by dividing the low-resolution image into block shapes such as squares and rectangles. And outputs the motion vector information to the encoding unit 29. As a search method, an existing method such as a block matching method using an SSD (Sum of Squared Difference) method or an SAD (Sum of Absolute Intensity Difference) method, or a gradient method can be used. In addition, the motion estimation unit 28 outputs the information amount of the generated motion vector to the encoding control unit 30 as the motion vector information amount.

  In addition, the motion estimation unit 28 performs motion compensation on the locally decoded image stored in the memory 27 using the generated motion vector, generates a motion compensated image, and outputs the motion compensated image to the subtraction unit 21 and the addition unit 26.

  Since the motion of the subject in the image is not necessarily in units of pixels, the motion estimation unit 28 moves with finer pixel accuracy than the reference image, for example, with 1/2 pixel or 1/4 pixel accuracy, in order to increase the accuracy of motion estimation. A vector can be obtained. As a result, motion estimation closer to the actual motion becomes possible, and the coding efficiency is increased by improving the prediction efficiency. However, in this case, since the accuracy for expressing the motion vector is also increased, the amount of motion vector information increases. Further, in order to search for an accuracy of less than one pixel, a process for interpolating and enlarging the reference image with an accuracy smaller than one pixel is required.

  The encoding unit 29 performs encoding processing (for example, scan and variable length encoding processing) on the quantized orthogonal transform coefficient input from the quantization unit 23 to generate encoded data, and generates external data. Output to. The encoding unit 29 also performs the encoding process on the motion vector information input from the motion estimation unit 28 and outputs the obtained encoded data to the outside.

  In addition, the encoding unit 29 outputs the code amount generated by the encoding process to the encoding control unit 30.

  The encoding control unit 30 adjusts the degree of quantization according to the required bit rate of encoding and the like, and performs a process of keeping the generated information amount within the required bit rate. In normal encoding, the amount of generated information is controlled by controlling the quantization unit 23 to change the quantization width. According to the present invention, in addition to this, the image processing apparatus 1 also controls the amount of motion vector information by controlling the clipping device 10.

  In order to prevent an increase in the information amount of the motion vector, the encoding control unit 30 determines the cut-out position based on the motion vector information amount input from the motion estimation unit 28 and the code amount input from the encoding unit 29. Control the accuracy of movement. Assuming that the total amount of motion vector information and the amount of code is the total amount of information, the encoding control unit 30, for example, the total information amount, the motion vector information amount, and the ratio of the motion vector information amount to the total information amount (hereinafter referred to as “motion When at least one of “vector information etc.” exceeds a predetermined threshold, the parameter setting unit 11 is controlled so that the movement accuracy is lowered.

  When the parameter setting unit 11 receives a control signal for suppressing the amount of motion vector information from the encoding control unit 30, the parameter setting unit 11 imposes a limit on the movement accuracy of the cutout processing unit 12. For example, when the movement accuracy is 1/4 pixel, it is limited to 1/2 pixel accuracy or 1 pixel accuracy. In the processing of the clipping device 10, it is not necessary to generate an image corresponding to a finely accurate speed, so that interpolation processing and memory reduction are possible. Moreover, since the possibility that the accuracy of the motion vector estimated by the motion estimation unit 28 becomes coarse by increasing the moving accuracy of the cutout position is increased, the amount of motion vector information can be suppressed.

  Next, the operation of the image processing apparatus 1 will be described with reference to FIG. FIG. 3 is a flowchart showing the operation of the image processing apparatus 1.

  In the clipping device 10, first, the parameter setting unit 11 sets parameters including the size of the image to be clipped, the clipping position, and the movement accuracy of the clipping position between frames (step S101). Then, the cutout processing unit 12 uses the parameters set in step S101 to cut out a part of the image from the high resolution image to generate a low resolution image (step S102).

  When the low-resolution image is input, the encoding device 20 performs motion estimation processing and motion compensation processing by the motion estimation unit 28 (step S103). Then, the encoding control unit 30 determines the magnitude of the motion vector information amount based on the motion vector information amount generated in step S103. For example, it is determined whether or not the amount of motion vector information exceeds a threshold (step S104).

  If the amount of motion vector information is less than or equal to the threshold (step S104-No), the encoding process is continued and encoded data is output (step S105). On the other hand, when the amount of motion vector information exceeds the threshold (step S104—Yes), the encoding control unit 30 instructs the parameter setting unit 11 to set the movement accuracy to be lower than the current value.

  Note that the first threshold value and the second threshold value are provided, and the encoding control unit 30 sets the movement accuracy to the parameter setting unit 11 when the motion vector information amount exceeds the first threshold value. If the motion vector information amount is equal to or smaller than the second threshold value, the parameter setting unit 11 is instructed to set the movement accuracy higher than the current value. Also good.

  As described above, in the image processing apparatus 1 according to the first embodiment, the motion estimation unit 28 performs motion estimation of a low-resolution image to generate a motion vector, and motion compensation is performed using the motion vector. Outputs an image and outputs a motion vector information amount indicating the information amount of the motion vector, and encodes a difference image between the low resolution image and the motion compensated image by the encoding unit 29 to output encoded data. The encoding control unit 30 controls the movement accuracy of the cutout position between frames based on the motion vector information amount. With this configuration, when the amount of motion vector information is large, the motion vector information amount can be suppressed by reducing the moving accuracy of the extraction position, and the amount of processing, memory, and other resources can be reduced. It becomes.

(Second Embodiment)
Next, a second embodiment of the present invention will be described. FIG. 4 is a block diagram illustrating a configuration example of an image processing apparatus according to the second embodiment. In the example illustrated in FIG. 4, the image processing device 2 includes a clipping device 10 and an encoding device 20. The cutting device 10 includes a parameter setting unit 11 and a cutting processing unit 12. The encoding device 20 includes a subtraction unit 21, an orthogonal transformation unit 22, a quantization unit 23, an inverse quantization unit 24, an inverse orthogonal transformation unit 25, an addition unit 26, a memory 27, and a motion estimation unit 28. An encoding unit 29, an encoding control unit 30, and an image quality evaluation unit 31. The image processing apparatus 2 according to the second embodiment is different from the image processing apparatus 1 according to the first embodiment in that an image quality evaluation unit 31 is further provided. Since other configurations are the same as those of the first embodiment, the same reference numerals are given and description thereof is omitted as appropriate.

  The image quality evaluation unit 31 calculates a difference value between the low-resolution image input from the clipping device 10 and the local decoded image input from the memory 27 and outputs the difference value to the encoding control unit 30. The difference value can be calculated by various calculation methods such as a sum of absolute values of differences, a product of absolute values of differences, and a sum of squares of differences.

  In addition to the motion vector information described in the first embodiment, the encoding control unit 30 also takes into account the difference value calculated by the image quality evaluation unit 31 and controls the movement accuracy of the cutout position. For example, when the difference value exceeds a threshold value, image quality deterioration due to encoding is large, and therefore control is performed so that at least one of the quantization accuracy by the quantization unit 23 and the movement accuracy by the extraction processing unit 12 is increased.

  Further, the cutout processing unit 12 may output information on the movement speed and movement accuracy of the cutout position to the motion estimation unit 28. If the movement speed of the cutout position is known in advance, the global motion per frame can be known, so the motion estimation unit 28 can perform motion estimation using the global motion. If the movement accuracy of the cutout position is known in advance, the motion estimator 28 can simplify interpolation processing with more than one pixel accuracy required for normal motion estimation, memory preparation, and the like. Note that since the global motion is the motion of the entire low-resolution image generated by the clipping device 10, there is no correlation with the local motion that the original video has.

  As described above, the image processing apparatus 2 according to the second embodiment further includes the image quality evaluation unit 31 that calculates the difference value between the low resolution image and the locally decoded image of the low resolution image, and the encoding control unit 30. Controls so that at least one of the quantization accuracy and the movement accuracy increases when the difference value exceeds the threshold value. For this reason, it is possible to prevent deterioration due to encoding while suppressing the amount of motion vector information and the like.

  In addition, the cutout processing unit 12 outputs information on the moving speed of the cutout position between frames (global motion information) to the motion estimation unit 28, and the motion estimation unit 28 performs motion estimation using the global motion information. Thus, the motion vector can be estimated with a small amount of processing and with high accuracy.

  Note that a computer can be suitably used to cause the image processing apparatuses 1 and 2 to function as described above, and such a computer can store a program describing processing contents for realizing the functions of the image processing apparatuses 1 and 2. It can be realized by storing the program in a storage unit of a computer and reading and executing the program by the CPU of the computer. This program can be recorded on a computer-readable recording medium.

  Although the above embodiment has been described as a representative example, it will be apparent to those skilled in the art that many changes and substitutions can be made within the spirit and scope of the invention. Therefore, the present invention should not be construed as being limited by the above-described embodiments, and various modifications and changes can be made without departing from the scope of the claims. For example, a plurality of constituent blocks described in the embodiments can be combined into one, or one constituent block can be divided.

DESCRIPTION OF SYMBOLS 1, 2 Image processing apparatus 10 Extraction apparatus 11 Parameter setting part 12 Extraction processing part 20 Encoding apparatus 21 Subtraction part 22 Orthogonal transformation part 23 Quantization part 24 Inverse quantization part 25 Inverse orthogonal transformation part 26 Adder part 27 Memory 28 Motion estimation Unit 29 encoding unit 30 encoding control unit 31 image quality evaluation unit

Claims (5)

An image processing apparatus that encodes a low-resolution image cut out from a high-resolution image according to a cut-out position for each frame,
Motion estimation for generating a motion vector by performing motion estimation of the low-resolution image, outputting a motion compensated image compensated for motion using the motion vector, and outputting a motion vector information amount indicating an information amount of the motion vector And
An encoding unit that encodes a difference image between the low-resolution image and the motion-compensated image and outputs encoded data;
An encoding control unit that controls the movement accuracy of the cut-out position between frames based on the amount of motion vector information;
Equipped with a,
The coding control unit moves the movement vector information amount when the amount of motion vector information exceeds a threshold value, or when the ratio of the motion vector information amount to the total amount of the motion vector information amount and the encoded data information amount exceeds a threshold value. the image processing apparatus according to claim that you control such accuracy is lowered. An image quality evaluation unit that calculates a difference value between the low resolution image and a locally decoded image of the low resolution image;
The image processing apparatus according to claim 1, wherein the encoding control unit performs control so that the movement accuracy is increased when the difference value exceeds a threshold value. Wherein according to said cut position determined by the movement accuracy, characterized in that from the high resolution image, further comprising a clipping processor for generating the low-resolution image, the image processing apparatus according to claim 1 or 2. The cutout processing unit outputs information on the moving speed of the cutout position between frames,
The image processing apparatus according to claim 3 , wherein the motion estimation unit generates the motion vector by performing the motion estimation using the moving speed information. The program for functioning a computer as an image processing apparatus as described in any one of Claim 1 to 4 .

Images