WO2011129163A1

WO2011129163A1 - Intra prediction processing method and intra prediction processing program

Info

Publication number: WO2011129163A1
Application number: PCT/JP2011/055327
Authority: WO
Inventors: 望倉本
Original assignee: コニカミノルタホールディングス株式会社
Priority date: 2010-04-16
Filing date: 2011-03-08
Publication date: 2011-10-20
Also published as: JP5376049B2; JPWO2011129163A1

Abstract

Disclosed is an intra prediction processing method provided with a processing block setting step for setting a coding processing target block constituted by a plurality of pixels of an input image; a prediction mode candidate restricting step for restricting a predetermined number of prediction mode candidates to a fewer plurality of prediction mode candidates, on the basis of resolution information of the processing target block, wherein the prediction mode candidate is an interpolation pattern representing how to interpolate the processing target block from which adjacent block located in which direction so as to create a prediction image; a prediction mode determination step for determining a prediction mode having the minimum prediction coding cost as a prediction mode to be used for prediction coding, among the prediction mode candidates which are restricted in the prediction mode candidate restricting step, and for using the determined prediction mode for prediction coding processing; and a prediction coding step for coding the processing target block using the prediction image after the prediction image of the processing target block is created in the prediction mode determined in the prediction mode determination step.

Description

Intra prediction processing method and intra prediction processing program

The present invention relates to an intra prediction processing method and an intra prediction processing program for predicting a corresponding prediction block from an adjacent block according to a prediction mode for a processing target block composed of a plurality of pixels for use in predictive encoding of an input image.

In video compression processing and the like, prediction encoding techniques such as encoding by intra prediction processing for performing prediction within the same frame and inter prediction encoding for performing prediction between temporally continuous frames are known.

These are also adopted in international standards and standards for image coding systems. An H.264 / MPEG-4 AVC (hereinafter referred to as H.264) encoder is configured as shown in FIG.

H. of FIG. In the intra prediction processing unit 2 of the H.264 encoder 1, intra prediction processing for determining a prediction pattern from adjacent blocks is executed.

In the intra prediction processing method, a processing target block composed of a plurality of pixels is set in an input image, and a corresponding prediction block is predicted from an adjacent block, and a prediction encoding process is performed based on this prediction.

The closer the predicted block is to the processing target block, the smaller the cost of predictive coding. That is, it can be further compressed as the amount of information. The image of the processing target block is estimated to be close to the image of the adjacent block, and is predicted from the adjacent block.

Although details will be described later, a plurality of prediction modes are set depending on how the pixel is predicted from which adjacent block (a specific example is shown in FIG. 4).

The plurality of prediction modes mainly indicate in which direction the adjacent block is closer, but it is not known in advance which prediction mode (prediction direction) will improve the prediction accuracy.

Therefore, it is a common practice to search for a prediction mode with the highest prediction accuracy, that is, the lowest coding cost, by executing all the prediction modes.

However, since all the prediction modes must be executed and the processing must be advanced to the stage where the encoding cost is calculated and evaluated, the amount of calculation becomes enormous.

As described above, the intra prediction processing method is known as a process with a large amount of calculation, and therefore, reduction of the calculation amount for determining the prediction mode in the intra prediction processing method has been studied as an important technical problem.

For example, Patent Document 1 proposes a method of narrowing down the prediction mode with reference to the prediction mode of an adjacent block that has been encoded by intra prediction processing.

When a specific example is shown with reference to FIG. 4, when the adjacent block above the target block is mode M1 and the left adjacent block is mode M5, in addition to M1 and M5, M4 is an angle between them. , M6 (see the lower right in FIG. 4) are limited as candidates.

Also, in Patent Document 2, the number of prediction mode candidates is limited according to the spatial position of the target block or the position on the time axis, and the amount of calculation for evaluating the prediction mode is reduced.

According to Patent Document 2, a block that limits the number of prediction mode candidates in a certain block in an image is determined by external input. The number of modes to be limited is also given as an external input and is variable for each block.

In this way, based on the number of prediction mode candidates determined in each block, the candidate prediction mode is determined from the prediction modes of the encoded neighboring blocks with priorities.

JP 2006-128770 A JP 2006-148419 A

As described above, the intra prediction process for predictive coding has a large amount of calculation, and a method for reducing the process has been proposed.

Particularly, techniques such as those described in Patent Document 1 and Patent Document 2 have been studied in order to reduce the amount of calculation by narrowing down the number of prediction mode candidates.

However, in view of the accuracy of prediction by the narrowed prediction mode candidates, the conventional methods described in

Patent Documents

1 and 2 have the following problems.

First, the prediction mode itself of the encoded neighboring block that is referred to in order to limit prediction mode candidates is also the prediction mode determined in a limited manner as described above. For this reason, it is not known whether the optimum prediction mode has been determined.

Second, only encoded neighboring blocks can be referenced. Therefore, when there is a region that shows a tendency of a prediction mode similar to an unprocessed region and it does not exist in the encoded processing region, the prediction mode is limited using the prediction mode of the encoded processing region. Is not optimal.

Thirdly, these methods that refer to the prediction mode of the encoded neighboring block for each block for which intra prediction processing is performed are unsuitable for processing in real time. For example, when a constant feature is always included as in the super wide-angle system, the process of referring to neighboring blocks for each intra prediction processing block becomes redundant.

In other words, in the conventional method as described above, the method for limiting the prediction mode candidates depends on the processing contents in the encoded region or the characteristics specific to the input image to be processed. It could not be said that it could narrow down to the optimal prediction mode candidates.

The present invention has been made in view of the above technical problems.

The object of the present invention is to narrow down the prediction mode candidates to the optimal prediction mode easily and quickly without depending on the prediction mode of the encoded region and the characteristics specific to the processing target image in the predictive encoding process of the input image. To provide an intra prediction processing method and an intra prediction processing program capable of reducing the amount of calculation for mode determination.

In order to solve the above problems, the present invention has the following features.

The intra prediction processing method according to the first aspect includes: a processing block setting step for setting a processing processing block for encoding composed of a plurality of pixels of an input image; and a target block from which adjacent block in which direction. Prediction mode candidates for narrowing down a predetermined number of prediction mode candidates, which are interpolation patterns indicating whether to create a predicted image by interpolation, to a plurality of prediction mode candidates smaller than the predetermined number based on the resolution information of the processing target block From the prediction mode candidates limited in the limiting step and the prediction mode candidate limiting step, the prediction mode that minimizes the cost of prediction encoding is determined as the prediction mode used for prediction encoding, and the prediction encoding process is performed. In the prediction mode determination step to be used and the prediction mode determined in the prediction mode determination step, the prediction image of the processing target block is After form, the intra prediction processing method is characterized in that and a predictive coding step of coding the current block using the predictive image.

In the intra prediction processing method according to the second aspect, in the prediction mode candidate limiting step, for the plurality of predetermined prediction mode candidates, based on the direction component of the resolution information corresponding to the direction indicated by each prediction mode candidate. The intra prediction processing method according to the first aspect, wherein the priority of the prediction mode candidates is determined to narrow down to limited prediction mode candidates.

In the intra prediction processing method according to the third aspect, in the prediction mode candidate limiting step, an optical characteristic of an imaging device that captures the input image is used as the resolution information. An intra prediction processing method according to an aspect.

The intra prediction processing method according to the fourth aspect is the intra prediction processing method according to the third aspect, wherein the optical characteristic of the imaging device is MTF.

In the intra prediction processing method according to the fifth aspect, in the prediction mode candidate limiting step, the resolution information determined by the image processing content in the preprocessing of the input image is used. The intra prediction processing method which concerns on the aspect.

The intra prediction processing method which concerns on a 6th aspect hold | maintains the prediction mode candidate table which matched the process target block position and the said prediction mode candidate limited in the said prediction mode candidate limitation process regarding the imaging system of the said input image. A prediction mode candidate holding step, and in the prediction mode determination step, the limited prediction mode candidates are set based on information of the prediction mode candidate table held for the same imaging system. An intra prediction processing method according to any one of the first to third aspects.

An intra prediction processing program according to a seventh aspect includes: a processing block setting step for setting a processing processing block for encoding composed of a plurality of pixels of an input image in a computer; and the processing target block as a neighboring block in any direction From this, the predetermined number of prediction mode candidates, which are interpolation patterns representing how to create a predicted image by interpolation, are narrowed down to a plurality of prediction mode candidates smaller than the predetermined number based on the resolution information of the processing target block. The prediction mode candidate prediction step and the prediction mode candidate limited in the prediction mode candidate limitation step is determined as a prediction mode to be used for prediction encoding, and the prediction mode with the lowest prediction encoding cost is determined, and the prediction code In the prediction mode determination step used for the conversion processing and the prediction mode determined in the prediction mode determination step. After creating a prediction image of the block, the intra prediction processing program is characterized in that and a predictive coding step of coding the current block using the predictive image.

In the prediction mode candidate limiting step, the intra prediction processing program according to the eighth aspect is based on the direction component of the resolution information corresponding to the direction indicated by each prediction mode candidate for the plurality of predetermined prediction mode candidates. The intra prediction processing program according to the seventh aspect is characterized by narrowing down to limited prediction mode candidates by determining the priority of the prediction mode candidates.

An intra prediction processing program according to a ninth aspect uses, in the prediction mode candidate limiting step, an optical characteristic of an imaging device that captures the input image as the resolution information. An intra prediction processing program according to an aspect.

The intra prediction processing program according to the tenth aspect is the intra prediction processing program according to the ninth aspect, wherein the optical characteristic of the imaging device is MTF.

An intra prediction processing program according to an eleventh aspect uses the resolution information determined by the image processing content in the preprocessing of the input image in the prediction mode candidate limiting step. An intra prediction processing program according to the above aspect.

An intra prediction processing program according to a twelfth aspect relates to a prediction mode in which the processing target block position and the prediction mode candidates limited in the prediction mode candidate limiting step are associated with the computer in the input image imaging system. A prediction mode candidate holding step for holding a candidate table is executed, and in the prediction mode determination step, the limited prediction mode candidates are set based on information in the prediction mode candidate table held for the same imaging system. An intra prediction processing program according to any one of the seventh to tenth aspects.

According to the intra prediction processing method and the intra prediction processing program according to the present invention, in predictive encoding processing of an input image, based on resolution information at the position of a processing target block in the input image, a plurality of predetermined prediction mode candidates are used. Narrow down to limited prediction mode candidates, and predict prediction blocks corresponding to the processing target block from neighboring blocks.

As a result, it is possible to easily and quickly narrow down to the optimal prediction mode candidates without depending on the prediction mode of the encoded region and the characteristics unique to the processing target image, and to reduce the calculation amount for determining the prediction mode. it can.

FIG. 1 is a flowchart illustrating an example of a processing procedure of the intra prediction processing method according to the present embodiment. FIG. 2 is a block diagram illustrating a functional configuration example of an encoder that performs intra prediction processing. FIG. 3 is an explanatory view showing a target image example for explaining the intra prediction process. FIG. 4 is a pattern diagram of the luminance signal intra 4 × 4 prediction mode. FIG. 5 is a pattern diagram of a luminance signal intra 16 × 16 prediction mode. FIG. 6 is a pattern diagram of an intra 8 × 8 prediction mode for color difference signals. FIG. 7 is a flowchart illustrating an example of a prediction mode determination procedure in the intra prediction process. FIG. 8 is a graph showing a typical example of the relationship between MTF and image height. FIG. 9 is a diagram illustrating the frequencies in the sagittal direction and the tangential direction when the arbitrary angle α is the frequency k. FIG. 10 is a diagram for explaining the outline of the prediction mode candidate limiting step in the form using an optical system (lens) for imaging. FIG. 11 is a diagram for illustrating the position of the processing target block in the target image. FIG. 12 is a diagram illustrating the directionality of the intra 4 × 4 prediction mode at the point (i, j). FIG. 13 is a diagram for explaining the calculation of the MTF in the d ₀ direction. FIG. 14 is a diagram for explaining the calculation of the MTF in the d ₁ direction. FIG. 15 is a diagram for explaining the calculation of the MTF in the d ₅ direction. FIG. 16 is a diagram illustrating an example in which distortion is corrected by image processing. FIG. 17 shows an H.264 image processing that incorporates image processing. 2 is a diagram illustrating an example of an H.264 encoder. FIG.

(First embodiment)
An embodiment of an intra prediction processing method and an intra prediction processing program according to the present invention will be described below with reference to the drawings.

(Intra prediction processing procedure example)
FIG. 1 is a flowchart illustrating an example of a schematic processing procedure of the intra prediction processing method according to the present embodiment. An example of a schematic processing procedure of the intra prediction processing method according to the present embodiment will be described with reference to FIG.

The processing procedure of the intra prediction processing method described below is executed by applying an intra prediction processing program programmed to execute them to an information processing apparatus such as a computer.

In FIG. 1, step S1 sets a processing target block composed of a plurality of pixels in the target image as a block to be subjected to predictive encoding processing by intra prediction of the input image.

Step S2 calculates or acquires the direction component of resolution information (for example, MTF) at the set position of the processing target block in the input image. MTF is a Module Transfer Function, details of which will be described later.

Step S3 is a prediction mode candidate limiting step, and a prediction mode candidate of a processing target block is determined according to the direction component of the resolution information (hereinafter referred to as MTF) obtained in step S2 from a plurality of predetermined prediction mode candidates. Limit.

The prediction mode indicates a pattern for predicting a processing target block from an adjacent block, and a plurality of modes having different pattern directions are set as candidates. Details will be described later.

The limited number of prediction mode candidates may be one or plural. In the present embodiment, description will be made assuming that the number of candidates is limited.

Details of the prediction mode candidate limiting step based on the resolution information in step S3 will be described later with a specific example.

Steps S4 to S6 are prediction mode determination steps, and for the prediction mode candidates limited in step S3, the prediction mode candidate that minimizes the cost of predictive coding is determined as the prediction mode.

For this reason, in step S4 to step S6, the cost of predictive coding for each prediction mode candidate is evaluated.

Step S4 calculates a prediction block by referring to an adjacent block according to the prediction mode for each prediction mode candidate limited in step S3 with respect to the set processing target block.

Step S5 performs a cost calculation process of the processing target block using the prediction block calculated for each limited prediction mode candidate. This is for comparing the cost evaluation of prediction encoding for each prediction mode candidate.

In step S6, one prediction mode for the set processing target block is determined based on the cost evaluation result of the prediction encoding in step S5. That is, among the prediction mode candidates, the prediction mode candidate with the lowest coding cost is determined as the prediction mode.

Details of the prediction mode determination process from step S4 to step S6 will be described later with a specific example.

Step S7 is a predictive encoding step, and the predictive encoding process in the prediction mode determined in step S6 is performed on the set processing target block.

Note that the predictive encoding in the determined prediction mode may be executed up to the predictive encoding process at the time of cost evaluation in step S5, and the processing result may be referred to.

In step S8, all the processing target blocks are set for the processing target image, and it is determined whether or not the intra prediction processing according to the above procedure is finished.

If the determination is Yes in step S8, the intra prediction process for the target image ends.

If the determination in step S8 is No, the process returns to step S1, sets the next block to be processed, and repeats up to step S8.

The repetition from step S1 to step S8 continues until the determination in step S8 is Yes and the intra prediction process for the target image is completed.

In the intra-prediction processing method and the intra-prediction processing program for executing the same according to the present embodiment, in this way, the prediction mode determination process and the characteristic specific to the prediction mode of the encoded region and the processing target image are performed before the prediction mode determination step. A prediction mode candidate limiting step based on resolution information that does not depend on or the like is executed.

This makes it possible to reduce the amount of calculation for encoding cost evaluation for determining the prediction mode, and to narrow down to the optimal prediction mode candidates easily and quickly.

First, the details of the prediction mode determination step as the prior art of the intra prediction process will be described first, and then the calculation of the MTF and its direction component as optical characteristics will be described, and the prediction mode candidate limiting step based on the resolution and direction component An example of the process will be described.

(Prediction mode determination process)
Details of the prediction mode determination step from step S4 to step S6 described above will be described.

In the prediction mode determination step, a process for determining a prediction mode used for predictive encoding is performed from a plurality of prediction mode candidates set in the intra prediction process. In this process, an example of using a known technique (International Standard for Image Coding, H.264 / MPEG-4 AVC) will be described.

The intra prediction process is a predictive encoding method that is also applied to a conventional image encoding apparatus, and is a predictive encoding process that performs prediction within a frame.

In the input image, a processing target block consisting of a plurality of pixels is set, and a corresponding prediction block is predicted from an adjacent block, and a predictive encoding process is performed based on this prediction.

For example, H. The configuration of the H.264 / MPEG-4 AVC encoder has already been shown in FIG. H. of FIG. In the intra prediction processing unit 2 of the H.264 encoder 1, intra prediction processing for determining a prediction pattern from adjacent blocks is executed.

FIG. 3 is an explanatory diagram showing an example of a target image for explaining the intra prediction process. The intra prediction process performed in the intra prediction process part 2 is demonstrated using FIG.

When one frame (target image) 3 is extracted from a moving image (input image), the subjects in the image are usually distributed in a certain area, and pixel values (luminance and color) are similar in adjacent pixels. Often doing. Using this, the pixel value of the encoding target block 4 is estimated from the pixels of the adjacent block 5 in the vicinity, which is the intra prediction process.

H. In H.264, the upper, left, upper right, and upper left blocks included in the encoded area 6 are defined as adjacent blocks 5 with respect to the encoding target block 4.

Next, specific intra prediction processing will be described.

<About prediction mode>
A plurality of prediction modes are set as patterns for how to predict from which adjacent block.

H. H.264 defines an intra 4 × 4 (pixel) prediction mode using a luminance signal, an intra 16 × 16 (pixel) prediction mode, and an intra 8 × 8 (pixel) prediction mode using a color difference signal. .

In each prediction mode, a number called a mode representing an average value and a pixel prediction direction is defined.

For example, FIG. 4 shows a pattern diagram of the luminance signal intra 4 × 4 prediction mode. In the intra 4 × 4 prediction mode, as shown in FIG. 4, Mode 2 representing an average value (hereinafter referred to as ModeX = M _X ) and M ₀ , M ₁ , M ₃ , M ₄ ,. Nine types of prediction modes with M ₈ are defined.

Furthermore, in the luminance signal, an intra 16 × 16 prediction mode in which the pixel unit is a 16 × 16 pixel size is defined by four types of prediction modes as shown in FIG. FIG. 5 shows a pattern diagram of the luminance signal intra 16 × 16 prediction mode.

Similarly, an 8 × 8 pixel intra 8 × 8 prediction mode is defined in the color difference signal as shown in FIG. FIG. 6 shows a pattern diagram of the color difference signal intra 8 × 8 prediction mode.

By encoding these prediction mode numbers and the difference values from the results actually predicted in that mode, a much higher compression rate than that of the image itself is realized.

This is shown more specifically using the alphabet symbols in Fig. 4.

For example, when the M ₀ of the intra 4 × 4 prediction mode, if the adjacent pixels A ~ D is already encoded, the target block a ~ p of coding,
a, e, i, m = A (1)
b, f, j, n = B (2)
c, g, k, o = C (3)
d, h, l, p = D (4)
In the case of M _{1 in} the intra 4 × 4 prediction mode, if adjacent pixels I to L have been encoded,
a, b, c, d = I (5)
e, f, g, h = J (6)
i, j, k, l = K (7)
m, n, o, p = L (8)
It is predicted. For M _{2 in} intra 4 × 4 prediction mode:
a to p = (A + B + C + D + I + J + K + L + 4) >> 3
(When A to D and I to L are already encoded)
a to p = (I + J + K + L + 2) >> 2
(When any of A to D cannot be referenced and I to L are all encoded)
a to p = (A + B + C + D + 2) >> 2
(When A to D are all encoded and any of I to L cannot be referenced)
a to p = 128
(When A to D and I to L contain unreferenceable)
(9)
It is predicted. Similarly, M ₃ , M ₄ ,..., M ₈ are also predicted according to the directions shown in FIG.

<Determination of prediction mode and calculation amount>
H. Since the H.264 intra prediction process performs prediction in units of pixels, fine prediction is possible, and high-quality and high-compression encoding is realized. On the other hand, however, the amount of calculation processing required for prediction has increased dramatically.

For example, an example of processing for determining a prediction mode of a luminance signal macroblock of 16 × 16 pixels will be described with reference to FIG. FIG. 7 is a flowchart illustrating an example of a prediction mode determination procedure in the intra prediction process.

Only the important steps in Fig. 7 are described below.

Step S11: Prediction is performed in the intra 16 × 16 prediction mode. Since there are four types of prediction modes (see FIG. 5), the encoding cost calculation process is repeated four times to determine Mα _i that minimizes the cost.

Step S12: It is determined whether or not an edge is included in the macroblock. If no edge is included, the prediction mode of the macroblock is determined as Mα _i of the intra 16 × 16 prediction mode.

Step S13: When an edge is included in the macroblock, the macroblock is divided into 16 sub-macroblocks of 4 × 4 pixels, and nine types of intra 4 × 4 prediction modes (see FIG. 4) are provided for each macroblock. To consider. That is, the encoding cost calculation process is repeated up to 9 × 16 times.

Step S14: Every time the intra 4 × 4 prediction mode in a certain sub macroblock is determined, the current cumulative coding cost (ΣCost (Mβ _bj )) (Σ is b = 0 to 15) in the macroblock, and step S11 Is compared with the encoding cost of Mα _i obtained in the above.

If is smaller coding cost of m.alpha _i discontinue calculation at sub-macroblock, it determines the m.alpha _i as the prediction mode of the macroblock. If the cumulative coding cost of the sub macroblock is smaller, 16 prediction modes Mβ _bj (0 ≦ b ≦ 15) are determined for the macroblock.

As described above, it is necessary to evaluate the encoding cost for all prediction directions (prediction modes) in each block. In order to evaluate the encoding cost, it is necessary to proceed to the next step of the intra prediction process in order to determine how much code amount is required when encoding is performed in the prediction mode. Will be enormous.

Therefore, in order to reduce the amount of calculation processing, various methods have been conventionally proposed (for example, see Patent Documents 2 and 3).

As described above, these methods are intended to reduce the amount of calculation by limiting and narrowing down prediction mode candidates, but the method of limiting prediction mode candidates is the processing content in the encoded region. It cannot be said that the prediction mode candidates can be narrowed down to the optimum prediction mode easily and quickly by depending on the characteristics inherent to the input image to be processed.

In the present embodiment, a prediction mode candidate limiting step is provided prior to the prediction mode determination step, in which the prediction mode candidate limiting step is limited from a plurality of predetermined prediction mode candidates based on resolution information at the position of the processing target block in the input image. Processing to narrow down to prediction mode candidates.

This makes it easy and quick to rely on resolution information that does not depend on the input image, such as optical characteristics (MTF) at the time of imaging, i.e., does not depend on the prediction mode of the encoded region or the characteristics specific to the processing target image. The calculation amount for encoding cost evaluation for determining the prediction mode is reduced by narrowing down to the optimal prediction mode candidates.

In addition, because it is based on resolution information that does not depend on the input image, if limited prediction mode candidates are stored in a table, it is not necessary to perform evaluation for narrowing down each time, and prediction encoding processing in real time Is also possible.

In the following, the decrease in resolution (direction component) due to optical characteristics will be described first. Although MTF is taken as an example, other characteristics such as OTF and PSF may be used as optical characteristics (resolution information).

<MTF>
One index that represents the performance of an optical system in an imaging apparatus such as a camera is MTF (Module Transfer Function).

MTF expresses how faithfully the contrast of a subject can be reproduced as a spatial frequency characteristic. The spatial frequency indicates the number of patterns (such as a sine wave) included per 1 [mm].

The MTF generally has a fixed frequency (60 lines / mm, 100 lines / mm, etc.), and a radial direction (Sagittal) from the image center in a graph in which the horizontal axis indicates the distance (image height) from the image center. ) And a concentric direction (Tangential).

FIG. 8 is a graph showing a typical example of the relationship between MTF and image height.

In general, the MTF tends to decrease as the image height increases, and may take different values in the radial direction (Sagital) and the concentric direction (Tangential). The difference appears remarkably in a wide-angle optical system.

Further, as a method for measuring MTF, a method as disclosed in a patent document (Japanese Patent Laid-Open No. 2001-324413) is disclosed.

<Image processing>
On the other hand, the captured image may be subjected to image processing before intra prediction processing. In such a case, resolution information as a result of image processing may be used.

For example, in the patent document (Japanese Patent Laid-Open No. 2009-284421), an image obtained by distortion correction and viewpoint conversion of an image taken with a wide-angle on-vehicle camera is displayed.

In such an image, enlargement / reduction processing by image processing is performed on an image captured by the original imaging system, and the resolution is lowered particularly in a portion where the enlargement processing is performed.

The generated image resolution is constant in each area of the screen while applying the same image processing.

(Calculation of direction component of MTF)
Here, a case where the MTF of the imaging system is used as the resolution characteristic of the image will be described. In order to select prediction mode candidates according to the MTF direction component, a method for calculating horizontal and vertical MTF components from the design data of the optical system will be described first.

<Calculation from optical system design data>
The MTF in the vertical and horizontal directions at each coordinate on the image can be obtained by simulation using design data of the optical system.

X _k , the coordinates of N points of a spot diagram at a given point on the image (a group of points formed on the image by rays passing through N lattice points on the pupil plane from a point light source). Let y _k . Assuming that the angle between the Sagittal direction and the horizontal plane of the image is α and the frequency in that direction is w lines / mm, the frequency s in the Sagittal direction and the frequency t in the Tangential direction are shown in FIG. )become that way.

s = wcosα (10)
t = wsinα (11)
The horizontal and vertical values of MTF, MTF _h (i, j) and MTF _v (i, j) can be expressed as in Equation (12) and Equation (13), respectively, and can be calculated from the spot diagram.

<Calculation from MTF value for image height>
If the values of the horizontal and vertical components of the MTF are calculated, the component in an arbitrary direction can be obtained although the description is omitted.

<Calculation from MTF value for image height>
A method for obtaining a component in an arbitrary direction of the MTF when only the MTF values in the sagittal direction and the tangential direction as shown in FIG. 8 are obtained will be described later.

(Prediction mode candidate limited process)
Details of the prediction mode candidate limiting step in step S3 in the intra prediction processing procedure of FIG. 1 described above will be described.

<Limitation of prediction mode candidates based on direction component of resolution information>
In the prediction mode candidate limiting step, the prediction mode candidates of the processing target block are limited according to the direction component of the resolution information (MTF) obtained in step S2 from a plurality of predetermined prediction mode candidates.

This step in the present embodiment is for narrowing down prediction mode candidates using resolution information and reducing the amount of calculation processing of intra prediction processing in steps S4 to S6. Since the resolution information of the input image is used, there is no need to perform image analysis in advance or generate a pattern assuming the input image.

The outline of the prediction mode candidate limiting step in the present embodiment will be described with reference to FIG. 10, taking an example of using an optical system (lens) for imaging.

When an object to be imaged is imaged through a lens as an optical system, the resolving power varies depending on the distance from the center position of the lens. The resolution distribution is a parameter that varies depending on the lens.

Using this resolving power information, the amount of calculation processing is reduced so as not to deteriorate the image quality.

As shown in FIG. 10, if the resolving power of the lens is high (upper part of FIG. 10), processing is performed pixel by pixel, and prediction aiming at high image quality and high compression is performed.

On the other hand, if the resolving power of the lens is low in the first place (the lower part of FIG. 10), it is not necessary to perform processing for each pixel, which has a high calculation cost. Without considering the prediction mode representing the direction, if the prediction mode is limited to a prediction mode candidate having a low calculation cost, for example, M ₂ (see FIG. 4), the prediction mode evaluation process can be greatly reduced.

Incidentally, the resolving power of such a lens is information that varies depending on the distance (image height) from the center of the lens and the direction at that position. For this reason, each lens determines the resolving power in consideration of the position in the image and the direction of the prediction mode, and selects the prediction mode candidates in light of the directionality, thereby reducing the optimal amount of calculation processing. Do.

For example, for a plurality of predetermined prediction mode candidates at the set position of the processing target block, by selecting the prediction mode candidates based on the direction component of the resolution information corresponding to the directionality of each prediction mode candidate, Narrow down to limited prediction mode candidates.

Specific processing for limiting prediction mode candidates according to the direction component of resolution will be described below.

<Calculation of direction component from MTF value for image height>
FIG. 11 is a diagram for illustrating the position of the processing target block 4 in the target image 3.

Suppose that the intra prediction process of the processing target block 4 of 4 × 4 pixels with the pixel f (i, j) shown in FIG.

At this time, if the center of the image is (C _X , C _Y ), the angle θ formed by the image center and the target pixel, and the distance (image height) 1 between the image center and the target pixel are expressed as follows. .

θ = arctan (| j−C _Y | / | i−C _X |) (14)
l = | (i, j) − (C _X , C _Y ) | (15)
Furthermore, when the block with the pixel f (i, j) at the upper left is the processing target block 4, the directionality of the intra 4 × 4 prediction mode at the point (i, j) is represented as shown in FIG.

Here, it is assumed that the direction d _X is a direction representing M _X. Further, s and t shown in the figure represent the S (Sagittal) direction and the T (Tangential) direction, and the MTF at the point (i, j) is a function of the image height l in the S direction and the T direction, respectively. Defined as S (l) and T (l).

Using these, the MTF in the d _X (X = {0, 1, 3, 4,..., 8}) direction at the point (i, j) is defined as follows.

MTF (d _X ) = | S (l) cosδ | + | T (l) sinδ | (16)
Here, δ is as follows.

δ = | θ− (1/2) π | (when d ₀ ) (17)
δ = θ (when d ₁ ) (18)
δ = | θ− (3/4) π | (when d ₃ ) (19)
δ = | θ− (1/4) π | (when d ₄ ) (20)
δ = | θ− (3/8) π | (when d ₅ ) (21)
δ = | θ− (1/8) π | (when d ₆ ) (22)
δ = | θ− (5/8) π | (when d ₇ ) (23)
δ = | θ + (1/8) π | (when d ₈ ) (24)
For example, the MTF in the d ₀ direction can be captured as shown in FIG. Similarly, the directions d ₁ and d ₅ can also be expressed as shown in FIGS.

<Limitation of prediction mode candidates>
A processing example for selecting a prediction mode candidate having a corresponding directionality according to the d _X direction component of the resolution (MTF) obtained as described above will be described.

For example, the MTF in the obtained d _X (X = {0, 1, 3, 4,..., 8}) direction is compared with a predetermined threshold value H,
MTF (d _X ) <H (25)
When the prediction mode M _Y with (e.g., vertical) direction that corresponds to the d _X direction shall be deleted from the prediction mode candidate.

For example, when the MTF in the d ₀ direction falls below the threshold value H, the prediction mode M ₁ is deleted from the prediction mode candidates. When the MTF in the d ₃ direction is less than the threshold value H, the prediction mode M ₄ is deleted from the prediction mode candidates.

Here, the threshold value H is determined by the pixel pitch of the image sensor and the frequency of the MTF data.

According to the prediction mode candidate limiting method of the present embodiment, the number of prediction modes can be reduced based on the point image reproducibility of the lens, so that the amount of calculation processing can be reduced without causing deterioration in image quality. .

If all of the MTF (d _X ) (X = {0, 1, 3, 4,..., 8}) are equal to or less than the threshold value H, the prediction mode of the block can be determined as M _2. The amount of calculation processing can be greatly reduced.

In the prediction mode candidate limited processing described above, only the intra 4 × 4 prediction mode has been described, but the present invention can also be applied to other intra prediction modes such as the intra 16 × 16 prediction mode.

For example, in a 16 × 16 pixel macroblock, if the average value prediction M ₂ can be determined, it is not necessary to perform processing in units of 4 × 4 pixels as illustrated in FIG. Can be suppressed.

<Priority of prediction mode candidate>
Further, not only the number of prediction mode candidates but also the above-described MTF (d _x ) are arranged in descending order, the priority can be determined. That is, it is possible to preferentially delete a candidate from a mode having directionality corresponding to a direction with low resolution.

Depending on the processing system, there is a limit to the number of prediction mode processes, and there is a possibility that not all prediction mode candidates can be calculated. You may make it exclude a prediction mode candidate from a candidate according to the priority determination according to.

(Second Embodiment)
The above-described embodiments have been described using the MTF of the imaging system as the optical characteristics.

When image processing is performed before intra prediction processing, resolution directionality depending on the position in the image, such as image distortion caused by the image processing, may be used.

As an example, a second embodiment of the present invention will be described below.

There is a conventional technique for correcting an image photographed with a wide-angle lens or a lens that generates distortion using image processing. For example, as shown in FIG. 16, this is a technique for correcting a distorted image. In FIG. 16, (a) a distorted image before image processing and (b) a corrected image after image processing are represented as a grid-like image.

Since such image processing is subjected to some kind of interpolation, the resolution varies depending on the direction, as can be seen from the lattice shape in the figure.

Since this resolution information can be handled in the same manner as the above-described MTF, it can be used to select a prediction mode candidate and can be prioritized.

Note that, unlike the MTF, the second embodiment may dynamically fluctuate unlike static data. For example, in an in-vehicle camera viewpoint conversion application, image processing is performed as time passes.

FIG. 17 shows an example of an encoder incorporating this image processing in its configuration.

In the encoding process shown in FIG. 17, image processing 8 is first performed on the input signal acquired from the imaging device. The image processing 8 includes various processing such as processing for viewpoint conversion, zoom processing, pan processing, color correction processing, and the like.

If the parameter information generated at this time is input to the intra prediction processing unit 9 in the second embodiment, the resolution (direction component) is similarly calculated, and the intra prediction is performed according to the characteristics of the image processed in the image processing 8. Processing can be performed.

As described above, the present invention that can assign priorities to a plurality of prediction mode candidates can be applied both statically and dynamically.

Note that the present invention relates to the H.264 standard. The present invention can be applied not only to H.264 intra prediction processing but also to general intra prediction processing for predicting a processing target block from adjacent blocks. Prediction using a weighted average or prediction processing involving some conversion can also be applied.

<Preservation and use of results limited to prediction mode candidates>
Based on the direction component at the position of the processing target block of the resolution information in the input image, the result of narrowing down a plurality of predetermined prediction mode candidates to the limited prediction mode candidates is saved, so that the intra prediction process described above is saved. It can also be effectively used to reduce the amount of calculation.

For example, in the procedure of the intra prediction process of FIG. 1, a prediction mode candidate holding step is provided during or before or after the prediction mode determination step of steps S4 to S6 after the prediction mode candidate limiting step of step S3 ends. May be.

In the prediction mode candidate holding step, the result of the prediction mode candidate limiting step relating to the input image imaging system, that is, a prediction mode candidate table in which the processing target block position is associated with the limited prediction mode candidate is held.

Thus, for input images of the same imaging system, a step of setting prediction mode candidates based on the information of the prediction mode candidate table held as described above is provided, so that step S2 and step of FIG. The step of S3 can be omitted.

Thus, since the resolution information used for limiting the prediction mode candidates does not depend on the input image, it can be applied as it is when the prediction mode candidate limitation results obtained once are the same system.

As described above, according to the intra-prediction processing method and the intra-prediction processing program according to the present embodiment, when predictive encoding processing of an input image, a plurality of information are obtained based on resolution information at the position of the processing target block in the input image. The prediction mode candidates are narrowed down to the limited prediction mode candidates, and the prediction block corresponding to the processing target block is predicted from the adjacent block.

In addition, the above-mentioned embodiment is an illustration and restrictive at no points. The scope of the present invention is defined by the terms of the claims, rather than the description above, and is intended to include any modifications within the scope and meaning equivalent to the terms of the claims.

1H. H.264 Encoder 2 Intra Prediction Processing Unit 3 Frame (Intra Prediction Process Target Image)
4 processing target block 5 adjacent block 6 encoded region 7 unprocessed region 8 image processing unit 9 intra prediction processing unit (second embodiment)

Claims

A processing block setting step for setting a processing processing block of encoding composed of a plurality of pixels of the input image;
Based on the resolution information of the processing target block, a predetermined number of prediction mode candidates that are interpolation patterns representing how to interpolate the processing target block from adjacent blocks in which direction to create a prediction image. A prediction mode candidate limiting step for narrowing down to a plurality of prediction mode candidates smaller than a predetermined number;
From the prediction mode candidates limited in the prediction mode candidate limiting step, the prediction mode that minimizes the cost of prediction encoding is determined as the prediction mode used for prediction encoding, and the prediction mode used for prediction encoding processing is determined. Process,
A prediction encoding step of encoding the processing target block using the prediction image after creating a prediction image of the processing target block in the prediction mode determined in the prediction mode determination step;
An intra prediction processing method characterized by comprising:
In the prediction mode candidate limiting step,
For the plurality of predetermined prediction mode candidates, by determining the priority of the prediction mode candidate based on the direction component of the resolution information corresponding to the direction indicated by each prediction mode candidate, the prediction mode candidates are limited. The intra prediction processing method according to claim 1, wherein narrowing down is performed.
In the prediction mode candidate limiting step,
The intra prediction processing method according to claim 1, wherein an optical characteristic of an imaging apparatus that has captured the input image is used as the resolution information.
4. The intra prediction processing method according to claim 3, wherein the optical characteristic of the imaging device is MTF.
In the prediction mode candidate limiting step,
The intra prediction processing method according to claim 1, wherein the resolution information determined by image processing content in the preprocessing of the input image is used.
A prediction mode candidate holding step for holding a prediction mode candidate table in which the processing target block position and the prediction mode candidates limited in the prediction mode candidate limiting step are associated with the input image imaging system,
In the prediction mode determination step,
The intra prediction processing method according to claim 1, wherein the limited prediction mode candidates are set based on information in the prediction mode candidate table held for the same imaging system.
On the computer,
A processing block setting step for setting a processing processing block of encoding composed of a plurality of pixels of the input image;
Based on the resolution information of the processing target block, a predetermined number of prediction mode candidates that are interpolation patterns representing how to interpolate the processing target block from adjacent blocks in which direction to create a prediction image. A prediction mode candidate limiting step for narrowing down to a plurality of prediction mode candidates smaller than a predetermined number;
From the prediction mode candidates limited in the prediction mode candidate limiting step, the prediction mode that minimizes the cost of prediction encoding is determined as the prediction mode used for prediction encoding, and the prediction mode used for prediction encoding processing is determined. Process,
A prediction encoding step of encoding the processing target block using the prediction image after creating a prediction image of the processing target block in the prediction mode determined in the prediction mode determination step;
An intra prediction processing program characterized by comprising:
In the prediction mode candidate limiting step,
For the plurality of predetermined prediction mode candidates, by determining the priority of the prediction mode candidate based on the direction component of the resolution information corresponding to the direction indicated by each prediction mode candidate, the prediction mode candidates are limited. The intra prediction processing program according to claim 7, wherein narrowing down is performed.
In the prediction mode candidate limiting step,
The intra prediction processing program according to claim 7, wherein optical characteristics of an imaging device that captures the input image are used as the resolution information.
The intra prediction processing program according to claim 9, wherein the optical characteristic of the imaging device is MTF.
In the prediction mode candidate limiting step,
The intra prediction processing program according to claim 7, wherein the resolution information determined by image processing content in the preprocessing of the input image is used.
In the computer,
With respect to the input image imaging system, a prediction mode candidate holding step for holding a prediction mode candidate table in which a processing target block position and the prediction mode candidates limited in the prediction mode candidate limiting step are associated is executed.
In the prediction mode determination step,
The intra prediction processing program according to claim 7, wherein the limited prediction mode candidates are set based on information in the prediction mode candidate table held for the same imaging system.