JP4735375B2 - Image processing apparatus and moving image encoding method. - Google Patents

Description

The present invention relates to a technique for encoding a moving image.

  In the image compression coding process, it is known to use a learning function to obtain an optimal compression parameter used for the process. For example, in Patent Document 1, the observer performs objective image quality evaluation while changing the compression parameter, and learns the statistical properties of the image using a neural network by using the compression parameter of the image that is determined to have good image quality by this evaluation. Generating optimal compression parameters is disclosed.

Japanese Patent Laid-Open No. 7-184062

  By the way, when a video stream is distributed via a network or when a received television video signal is compressed and stored in a hard disk as needed, it is necessary to generate an encoded image in real time. Need to speed up. In such a case, both high image quality and high speed encoding processing are required. However, Patent Document 1 does not consider such a case. Further, since Patent Document 1 learns using a compression parameter changed by an observer, it cannot be learned in real time when the apparatus is actually used, and it is difficult to obtain a suitable parameter.

  The present invention has been made in view of the above problems, and an object thereof is to provide a technique suitable for obtaining high image quality while speeding up the encoding process.

  In order to achieve the above object, the present invention is determined by a first mode determination unit that determines an encoding mode for encoding an input image using a predetermined calculation model, and the first mode determination unit. A second mode determination unit is provided that learns statistical information about the correspondence between the coding mode and the information using the coding mode and information about the state of the image corresponding thereto as a teacher signal.

  In the configuration as described above, for example, at the time of initial use of the apparatus including the first and second mode determination units, the first determination unit is selected, and according to a predetermined calculation model (for example, RD-Optimization method). The encoding mode is determined. At this time, the second determination unit learns the statistics of the determination result in association with the image state at that time. For example, if the second determination unit is selected after a lapse of a predetermined time and the encoding mode is determined by this, an encoding mode suitable for images in various states is selected according to the learning result. Thus, it is possible to quickly determine without using a predetermined calculation model. At this time, it is preferable that the second discriminating unit selects and discriminates the coding mode having the highest likelihood corresponding to each image state obtained as a result of learning.

  The selection of the first and second mode determination units may respond to an operation on the switching unit for switching the encoding mode.

  According to the present invention, it is possible to obtain high image quality while speeding up the encoding process.

  Embodiments of the present invention will be described below with reference to the drawings.

  First, an example of an image processing apparatus to which the present invention can be applied will be described with reference to FIG. In the present embodiment, a television receiver that incorporates a recording medium such as an HDD (hard disk drive) or a semiconductor memory (for example, a flash memory) will be described as an example of an image processing apparatus. However, any apparatus having an encoding process may be used. It can be applied to anything. For example, the present invention can be similarly applied to a video camera, a mobile phone with a built-in camera function, an HDD recorder, a DVD recorder, and the like.

  In FIG. 12, an image capturing unit 1 is a tuner that receives a television signal, for example, and the captured image is supplied to the encoding unit 2. The encoding unit 2 performs an encoding process on the input image to perform compression encoding, and supplies the image to a recording unit 3 including a recording medium such as an HDD. Here, the encoding unit 2 performs an image encoding process using a predetermined encoding method such as MPEG 2 (Moving Picture Experts Group), MPEG-4, or H.264 / AVC (Advanced Video Coding) standard. . In this embodiment, encoding processing is performed according to the H.264 / AVC standard, which can encode moving images at a compression rate about twice that of MPEG-4, but is not limited thereto. It is not a thing. Of course, other encoding schemes can be used.

  The recording unit 3 records the image compressed and encoded by the encoding unit 2. The image read from the recording unit 3 is decoded by the decoding unit 4 and supplied to the signal processing unit 5 as a video signal. The signal processing unit 5 performs predetermined signal processing such as color correction, contrast correction, and gamma correction on the video signal, and supplies the video signal to the display unit 6 configured by, for example, a PDP (plasma display panel) or a liquid crystal panel. To do. The display unit 6 displays an image according to the supplied video signal. The control unit 7 is constituted by a CPU, for example, and supplies control signals to the image capturing unit 1, the encoding unit 2, the recording unit 3, the decoding unit 4, and the signal processing unit 5. Take control. The control unit 7 is electrically connected to a command input terminal to which a command from a user is input, and is configured to control the encoding unit 2 according to the user's command.

  Since such an image processing apparatus compresses and encodes, for example, a television signal in real time and records it on a recording medium such as an HDD as needed, it is necessary to speed up the encoding process in the encoding unit 2. In addition, it is desirable to perform the encoding process so that image quality deterioration of the image displayed on the display unit 6 is suppressed. Therefore, in this embodiment, the encoding process in the encoding unit 2, particularly the process for determining the encoding mode used when performing the encoding process is provided with a learning function to speed up the determination process. is there. Details of the encoding process according to the present embodiment will be described below. Here, the coding mode is an intra-screen (Intra) coding mode, an inter-screen skip (PSkip) coding mode, an inter-screen (Inter) coding mode, or a block type when performing block matching processing. And sizes (for example, 8 × 8, 8 × 4, 4 × 8, and 4 × 4 blocks), and combinations thereof.

  FIG. 1 shows a specific example of the encoding unit 2 shown in FIG. The stream of images from the image capturing unit 1 is input to the original image memory (101) and stored as original image data. The original image read from the original image memory (101) is supplied to the block dividing unit (102). The block division unit (102) divides the original image into a plurality of blocks having several types of preset sizes, and the motion search unit (103), the predicted image generation unit (104), and the first mode determination unit These are supplied to a learning mode determination unit (107) and a high-speed operation mode determination unit (108) which is a second mode determination unit. The motion search unit (103) uses the block image from the block dividing unit (102) and the reference image stored in the reference image memory (106), and has a high correlation with the block image (that is, the block image) The block in the reference image is searched. Then, a motion vector in the block image is calculated using the block image and a block in the reference image that has a high correlation. This motion vector is supplied to the predicted image generation unit (104) and the first switch unit (110). The predicted image generation unit (104) creates a predicted image when encoded in each encoding mode using the motion vector information from the motion search unit (103), and supplies the predicted image to the first switch unit (110). .

  In response to the control signal from the control unit 7, the first switch unit (110) is a block image from the block dividing unit (102), a motion vector from the motion search unit (103), and a predicted image generation unit ( The prediction image from 104) is switched to be supplied to either the learning mode determination unit (107) or the high-speed operation mode determination unit (108).

  The learning mode determination unit (107) generates a difference signal between the input predicted image and the block image according to a predetermined calculation model, and uses the difference signal and the input motion vector to encode the encoding cost of each mode. Is calculated to determine the optimum encoding mode. Then, the learning mode determination unit (107) obtains learning data including the determined encoding mode and information on the state of the image corresponding to the encoding mode (that is, information necessary for determining the encoding mode). The high-speed operation mode determination unit (108) outputs the teacher signal.

  The high-speed operation mode determination unit (108) performs a learning operation using statistics about the correspondence relationship between the information about the state of the image and the encoding mode, using the teacher signal that is the learning data. That is, the high-speed operation mode determination unit (108) learns a coding mode having a high likelihood determined for each image state from the statistical information obtained using the teacher signal. When an image in a certain image state is input to the high-speed operation mode determination unit (108), the highest likelihood encoding mode corresponding to the image state is selected and determined according to the learning result.

  The second switch unit (111) selects one of the encoding modes from the learning mode determination unit (107) and the high-speed operation mode determination unit (108) in response to the control signal from the control unit 7. It is configured as follows. That is, the second switch unit (111) is configured so that one of the encoding modes from the learning mode determination unit (107) and the high-speed operation mode determination unit (108) is supplied to the encoding processing unit (109). The switching operation is performed. The first switch unit (110) and the second switch unit (111) both select the learning mode determination unit (107) when using the learning mode determination unit (107), and select the high-speed operation mode. When the determination unit (108) is used, both operate so as to select the high-speed operation mode determination unit (108).

  The encoding processing unit (109) generates an encoded stream using the encoding mode from the second switch unit (111) and supplies it to the recording unit 3 shown in FIG. The encoded stream is also supplied to the decoding processing unit (105), where decoding processing is performed. The image decoded by the decoding processing unit (105) is stored in the reference image memory (106) as the reference image.

Next, the learning mode determination unit (107) will be described. The learning mode determination unit (107) according to the present embodiment uses an algorithm called an RD-Optimization scheme that is advantageous for obtaining a high compression rate, for example, as an arithmetic model for determining the encoding mode in units of blocks. It is used as a predetermined calculation model. This RD-Optimization method is introduced in Reference Document 1 below, for example.
[Reference 1] G. Sullivan and T. Wiegand: "Rate-Distortion Optimization for Video Compression", IEEE Signal Processing Magazine, vol. 15, no. 6, pp. 74-90, (November, 1998).
In the RD-Optimization method, the encoding mode is determined as follows. First, a prediction image of the corresponding macroblock is generated in all candidate modes, a difference signal from the original image is generated, DCT and quantization are performed, and temporary encoding is performed once. Next, the temporarily encoded data is decoded by performing IDCT (Inverse DCT: Inverse Discrete Cosine Transform) and inverse quantization. Then, the distortion amount and the generated code amount of the actual image are measured from the decoded data and optimized by the Lagrange undetermined multiplier method. As a result, an encoding mode in which image quality deterioration is minimized is determined. In general, it is known that this method can select an ideal mode.

  However, this method requires a large amount of time because the amount of calculation for provisional encoding and decoding is very large. For this reason, a method has also been proposed in which the speed is increased by predicting the image quality after encoding without performing encoding / decoding processing on the difference signal and performing mode selection. For example, JM (Joint Model), which is H.264 standardized reference software, has introduced a high speed method that does not use the RD-Optimization method. However, since the high-speed method introduced in JM approximates the coding cost used for mode determination mainly by a linear expression of prediction error, the prediction accuracy is not sufficient and the image quality is deteriorated. Furthermore, the amount of calculation is large because prediction images are generated for all candidate encoding modes.

  Therefore, in this embodiment, an RD-Optimization method suitable for maintaining high image quality is used as an arithmetic model in the learning mode determination unit (107). A specific example of the learning mode determination unit (107) is shown in FIG. In FIG. 2, a difference signal generation unit (201) uses a prediction image and an original image (block image) supplied via the first switch unit (110) to calculate a difference value for each pixel in the block. calculate. This difference value is supplied to the encoding processing unit (202) and the teacher signal generation unit (206). The encoding processing unit (202) performs DCT (Discrete Cosine Transformation) and quantization in block units with a specified size (for example, 8 × 8) on the input differential signal. The DCT coefficient and the quantization parameter obtained by this encoding process are supplied to the decoding processing unit (203).

  The decoding processing unit (203) performs inverse quantization and IDCT (Inverse DCT) on the encoded data, decodes it into a differential signal, and outputs it to the cost calculation unit (204). The cost calculation unit calculates the coding cost of the corresponding mode based on the decoded difference signal and the motion vector information input via the first switch unit (110), and sends it to the mode determination unit (205). Supply. The mode determination unit (205) selects and determines the encoding mode with the lowest encoding cost as the optimum mode, and outputs the result to the second switch unit (111) and the teacher signal generation unit (206).

  The teacher signal generation unit (206) is a learning used in the high-speed operation mode determination unit (108) based on the difference signal from the difference signal generation unit (201) and the encoding mode from the mode determination unit (205). A teacher signal as data is generated and supplied to the high-speed operation mode determination unit (108). That is, the high-speed operation mode determination unit (108) according to the present embodiment uses the RD-Optimization encoding mode determination process as a model, and inputs (difference between the block image as the image state and the predicted image and the motion vector). ) (Encoding mode) is learned.

Subsequently, the high-speed operation mode determination unit (108) according to the present embodiment will be described. FIG. 3 shows a basic configuration of the high-speed operation mode determination unit (108) according to the present embodiment. The high-speed operation mode determination unit (108) generates a difference signal that generates a difference signal between the predicted image and the original image. A generation unit (301) and determination units (302) to (306) connected in multiple stages are provided. The high-speed operation mode determination unit (108) according to the present embodiment operates as follows in each operation mode.
(1) Learning mode:
This mode is a mode in which learning is performed using the processing result of the coding mode discrimination according to the predetermined calculation model executed in the learning mode judgment unit (107) and the information of the image state corresponding thereto. At this time, the first switch unit (110) has selected the learning mode determination unit (107), the block image from the block division unit (102), the motion vector from the motion search unit (103), and The predicted image from the predicted image generation unit (104) is not supplied to the high-speed mode determination unit (108), but is supplied to the learning mode determination unit (107). The second switch unit (111) also selects the learning mode determination unit (107), and the encoding mode that is the calculation result in the learning mode determination unit (107) is the second switch unit (111). To the encoding processing unit (109).

As described above, in the learning mode, the block image, the motion vector, and the predicted image are not supplied to the high-speed operation mode determination unit (108), but the teacher signal is supplied from the learning mode determination unit (107). This teacher signal is input to each of a plurality of determination units (302) to (306) connected in multiple stages. Each of the determiners (302) to (306) takes statistics about the correspondence between the coding mode, which is the calculation result in the learning mode determination unit (107), and information on the image state. The statistical information is used to learn the relationship between the input (that is, the image state) and the output (that is, the determined coding mode) to the learning mode determination unit (107). The first stage determination unit (302) performs a rough coding mode determination, and if necessary, the second stage determination unit 1 (303), the second stage determination unit n (304) connected thereto, Further, detailed determination may be performed by the s-th stage determination unit 1 (305) and the s-th stage determination unit m (306). Here, n, s, and m are integers of 2 or more. That is, the second stage determination unit and the s stage determination unit may be configured by a plurality (n or m), respectively.
(2) High-speed operation mode:
This mode is a mode for discriminating the encoding mode at a high speed (faster than the arithmetic processing in the learning mode determination unit (107)) by using the learning result in each determination unit in the high-speed operation mode determination unit (108). It is. At this time, the first switch unit (110) has selected the high-speed operation mode determination unit (108), the block image from the block division unit (102), the motion vector from the motion search unit (103), and The predicted image from the predicted image generation unit (104) is supplied to the high-speed mode determination unit (of course, they are not supplied to the learning mode determination unit (107)). The second switch unit (111) also selects the high-speed operation mode determination unit (108), and the coding mode that is the determination result in the high-speed operation mode determination unit (108) is the second switch unit (111). To the encoding processing unit (109).

  The input block image and predicted image are input to the difference signal generation unit (301). The difference signal generation unit (301) calculates a difference between the original image and the predicted image for each pixel using the block image as an original image, and supplies the difference to the first stage determination unit (302). In the first stage determination unit (302), the difference signal and the motion vector information from the difference signal generation unit (301) are input as image states, and the corresponding encoding mode is used using the learned statistical information. To select and determine. The encoding mode selected here is the encoding mode with the highest likelihood determined by the predetermined calculation model for the input image information. That is, the high-speed operation mode determination unit (108) grasps the likelihood of the encoding mode corresponding to each state of the image by the learning. Then, when a certain image is input, if the most likely encoding mode corresponding to the image state is selected, the optimum encoding can be performed without performing a calculation by a predetermined calculation model such as the RD-Optimization method. You can get mode.

  The first stage determination unit (302) performs a rough determination of the encoding mode and then ends the mode determination process according to the determination result, or which of the second stage determination units (303) and (304) determines Determines whether to use the part for the next determination process. Such a determination process is performed in multiple stages. Then, in the s-stage determination unit (305) (306), which is the final stage, the optimum encoding mode is almost certainly determined, and the mode determination process is terminated. The optimum encoding mode determined by these determination units is supplied to the encoding processing unit (109) via the second switch (111) instead of the output of the learning mode determination unit (107).

  Any specific method may be used for these determination units as long as the learning function is used. For example, a nonlinear classifier using a neural network, a SVM (Support Vector Machine) based on a kernel method, a k-nearest neighbor method, or the like may be used. A linear discriminator using SVM, linear discriminant analysis, or the like may be used. Furthermore, the likelihood when encoding is performed in a candidate encoding mode may be calculated. Furthermore, the determination rule itself may be learned by a probabilistic means such as a Bayes net, a hidden Markov model, or a decision tree learning. A plurality of discriminators may be combined by means such as using boosting.

  Next, a specific example of the high-speed operation mode determination unit (108) according to the present invention will be described with reference to FIGS. This shows an example in which each determination unit (302) to (306) shown in FIG. 3 is configured by a neural network. This specific example is a 16 × 16 pixel macroblock unit, 4 × 4 or 16 × 16 block intra prediction method, 16 × 16 block inter-screen skip (PSkip) method, 8 × 8, An optimal encoding mode is selected from a total of seven encoding modes of the inter prediction scheme of 8 × 16, 16 × 8, or 16 × 16 blocks. Here, the inter-screen skip (PSkip) method is a method in which an image of a block in a reference image indicated by a motion vector is used as it is without using difference information.

  The high-speed operation mode determination unit (108) according to the present example includes a difference signal generation unit (401) that generates a difference signal between the predicted image and the original image, and an optimal encoding method among the Intra method, the Pskip method, and the Inter method. And a first stage determination unit comprising a neural network 1 (402) for selecting. Further, a second stage comprising two types of determination units, a neural network 2 (403) for determining an optimum block size in the intra method and a neural network 3 (404) for determining an optimum block size in the inter method. A determination unit is provided.

  In the learning mode, as in the example shown in FIG. 3, each neural network is input with a teacher signal from the learning mode determination unit (107). Statistic of correspondence with the computerization mode and learning.

  In the high-speed operation mode, the block image and the predicted image from the first switch unit (110) are input to the difference signal generation unit (401). In the difference signal generation unit (401), using the input block image as an original image, a difference from the predicted image is generated for each pixel and supplied to the neural network 1 (402). The neural network 1 (402) selects an optimal encoding mode corresponding to the input differential signal from the Intra method, the Pskip method, and the Inter method. As in the example of FIG. 3, the selection here is configured to select the highest likelihood among the Intra, Pskip, and Inter methods. When the PSkip mode is selected, the result is output to the second switch (111), and the encoding mode selection process is terminated. On the other hand, when the Intra method is selected, the determination process is continued in the neural network 2 (403) in order to determine the optimum block size for the encoding process. If the Inter method is selected, the determination process is continued in the neural network 3 (404) in order to determine the optimum block size for the encoding process. In the neural network 2 (403) and the neural network 3 (405), the optimum block size obtained by learning for each coding method (that is, the block size having the highest likelihood for each coding method). Select. The result is output to the second switch (111) and the mode selection process is terminated.

  Here, the neural network is a network in which a plurality of threshold logic units are arranged hierarchically from the input layer to the output layer. In a feed-forward network, coupling between units exists only between adjacent layers and is unidirectional from the input layer to the output layer. A weight is given to the combined units, and the input to the upper layer unit is the product sum of the values output by the lower layer unit group. When learning is performed, these weightings are adjusted so that a desired result is obtained in the output layer. Compared with JM's high-speed method, which uses a neural network to approximate the encoding mode selection rule in multiple dimensions, the prediction error is calculated using a linear expression, predicting mainly the coding cost is highly accurate. It becomes possible.

  FIG. 5 shows a specific example of the neural network 1 (402) shown in FIG. The neural network 1 (402) according to this example is a feed-forward network having five units in the input layer and three units in the output layer. Further, although the number of layers and the number of units are not particularly specified for the intermediate layer, for example, a single intermediate layer having seven units may be set. The neural network 1 includes, for example, QP (Quantization Parameter), spatial resolution, 4 × 4 block Intra mode (I4x4) as learning data included in the teacher signal from the learning mode determination unit (107). Prediction distortion (difference signal) when predicted in, prediction distortion when predicted in Pskip mode, and prediction distortion when predicted in 8 × 8 block Inter mode are input. Note that the predicted distortion is the sum of differences between an original image and a reference image for each pixel in a certain block, and may be hereinafter referred to as a difference signal.

  When such information and signals are input, the likelihood when encoding is performed in each mode of Intra mode, Pskip mode, and Inter mode is output. Among the likelihoods for these three encoding modes, the highest one is set as the optimal encoding method. Here, when the most likely encoding method in the input shown in FIG. 5 is, for example, the Inter mode, the Inter mode is selected when there is a similar input in the high-speed operation mode.

  FIG. 6 shows a specific example of the neural network 2 (403) shown in FIG. The neural network 2 (403) is a feedforward network having four units in the input layer and two units in the output layer. Further, although the number of layers and the number of units are not particularly specified for the intermediate layer, for example, a single intermediate layer having five units may be set. The neural network 2 includes, as learning data included in the teacher signal from the learning mode determination unit (107), for example, QP, spatial resolution, a difference signal when predicted in 4 × 4 block intra mode (I4x4), The difference signal when predicted in the 16 × 16 block intra mode (I16 × 16) is input. When these information and signals are input, the likelihood of encoding in I4x4 mode and I16x16 mode is output. Among the likelihoods for these two encoding modes, the highest one is set as the optimum encoding method. Here, when the coding method having the highest likelihood in the input shown in FIG. 6 is, for example, the I4x4 mode, the I4x4 mode is selected when there is a similar input in the high-speed operation mode.

  FIG. 7 shows a specific example of the neural network 3 (404) shown in FIG. The neural network 3 (404) is a feedforward network having 11 units in the input layer and 4 units in the output layer. Further, although the number of layers and the number of units are not particularly specified for the intermediate layer, for example, a single intermediate layer having 10 units may be set. In this example, a 16 × 16 macroblock is further divided into four 8 × 8 blocks. The neural network 3 includes QP, spatial resolution, vector difference information calculated in 8 × 8 block units, 8 × 8 block units as learning data included in the teacher signal from the learning mode determination unit (107). The calculated difference signal information is input. And when these information and signals are input, 8 × 8 block Inter mode (P8 × 8), 8 × 16 block Inter mode (P8 × 16), 16 × 8 block Inter mode (P16 × 8) The likelihood when encoding in the 16 × 16 block Inter mode (P16 × 16) is output. Here, when the most likely encoding method in the input shown in FIG. 7 is, for example, the P16 × 8 mode, the P16 × 8 mode is selected when there is a similar input in the high-speed operation mode. The

  Generally, most of the processing time for encoding a moving image is consumed by the motion search unit (103) for performing inter-screen prediction. Therefore, the calculation cost for generating the difference signal between the predicted image and the original image in the Inter prediction mode is very high. Therefore, if differential signals are generated for all block sizes in the Inter mode and input to the neural network 3 (404), a huge amount of calculation time is required. For this reason, in this example, speedup is realized by generating a prediction signal by performing motion search only for the Inter mode of 8 × 8 blocks. On the other hand, in the intra prediction mode, the calculation cost for generating the difference signal between the predicted image and the original image is not so high. Therefore, in the neural network 2 (403) according to the present example, the prediction accuracy is improved by generating and inputting the difference signal in all block sizes in the intra prediction mode. Thus, in this embodiment, the image quality and the processing speed are improved by adjusting the input parameters according to the size of the processing load for generating the prediction signal.

  FIG. 8 shows a processing flow of the high-speed operation mode determination processing unit (108) in the present embodiment described above. First, in step 802, a differential signal is generated in three types of encoding modes I4x4, Pskip, and P8 × 8 and input to the neural network 1. In step 803, it is determined whether the neural network 1 (402) has selected the Pskip mode, that is, whether the likelihood of the Pskip mode is the highest. As a result of the determination, if “yes”, it is determined that the optimal encoding mode of the corresponding macroblock is Pskip, the process proceeds to step 807, and the mode selection process ends. If the determination in step 803 is “no”, the process proceeds to step 804.

  In step 804, it is determined whether the neural network 1 (402) has selected the Intra mode, that is, whether the likelihood of the Intra mode is the highest. As a result of the determination, if “yes”, the process proceeds to Step 805. In step 805, a difference signal is created with another block size I16 × 16 in the Intra mode and input to the neural network 2 (403) together with the I8 × 8 difference signal. Then, the coding mode having the highest likelihood in the output layer of the neural network 2 (403) is selected as the optimum mode, and the process proceeds to step 807 to end the selection process.

  On the other hand, if the result of determination in step 804 is “no”, that is, if the neural network 1 has selected the Inter mode (that is, if the likelihood of the Inter mode is the highest), the process proceeds to step 806. In step 806, the P8 × 8 difference signal and motion vector information are input to the neural network 3 (404). Then, the coding mode having the highest likelihood in the output layer is selected as the optimum mode, and the process proceeds to step 807 to end the mode selection process.

The learning method using the neural network is not particularly limited in the present embodiment. However, for example, if a back propagation method (BP method) is used, a great effect can be seen. The BP method is described in detail in Chapter 3 of Reference Document 2 below, for example.
[Reference 2] Kenichiro Ishii, Nobuyoshi Ueda, Eisaku Maeda, Hiroshi Murase: “Intuitive Pattern Recognition”, Ohmsha, 1998.
In the above embodiment, the selection rule of the RD-Optimization method is learned. However, the selection rule to be learned may not be based on such objective judgment. For example, it may be based on subjective judgment such as image quality evaluation by a person.

  Furthermore, the generation order of the prediction signals used in the learning mode selection unit (107) and the high-speed operation mode determination unit (108) is not limited. For example, the mode may be selected by searching for a motion vector in units of macroblocks, or the mode may be selected after performing a vector search in batches in units of screens or slices.

  Further, in the high-speed operation mode determination unit (108), it is possible to further increase the speed by correcting the motion vector at the same time as selecting the encoding mode. For example, for the motion search unit (103), two types of a motion search unit for learning that searches to decimal precision over time and a high-speed motion search unit that only searches to integer precision are prepared. The difference information as a result of the above may be learned. Then, when the high-speed operation mode determination unit (108) determines that the inter-screen prediction method is optimal, the motion vector amount may be corrected based on the learned statistical information. Thereby, a motion search can be performed at high speed.

  According to the configuration of the present embodiment, it is possible to encode a moving image with high speed and high image quality. In addition, by repeating the learning periodically in the user's environment, it becomes possible to determine the encoding mode suitable for the use environment and perform the encoding, thereby further improving the image quality. For example, when the image processing apparatus according to the present embodiment is applied to a digital camera, it is possible to perform appropriate shooting according to the characteristics of the camera and the shooting environment by performing shooting in the learning mode for a while after the purchase of the digital camera and adjusting the parameters. The encoding mode can be learned. Then, if shooting is performed in the learning mode for a while and then shooting in the high-speed operation mode, the high-speed operation mode reflects the result of learning in the learning mode, and optimal encoding can be performed at high speed according to various scenes. Further, according to the present embodiment, if the use time of the image processing apparatus is increased, learning results for various images are accumulated accordingly, so that the time required for encoding is shortened as the use time is increased.

  Such mode switching may be performed manually by the user of the image processing apparatus. For example, a manual switch or a menu screen may be displayed on the digital camera, and an encoding mode may be selected and switched in response to an operation on the manual switch or menu screen. In response to the manually input command, the control unit 9 controls the first switch unit (110) and the second switch unit (111) described above. That is, when the user selects the learning mode, the control unit 9 controls the first switch unit (110) and the second switch unit (111) to select the learning mode determination unit (107). . On the other hand, when the user selects the high-speed operation mode, the control unit 9 performs control so that the first switch unit (110) and the second switch unit (111) select the high-speed operation mode determination unit (108). To do.

  Switching between the learning mode determination unit (107) and the high-speed operation mode determination unit (108) may be performed manually by a user instruction as described above, or may be performed automatically. When performing automatically, you may make it switch based on the operation time of learning mode, for example. For example, in the case of a digital camera to which the image processing apparatus according to the present embodiment is applied, the control unit 9 performs the first and first control so that the learning mode is automatically selected during initial use (initial operation) by the user. 2 switches. The control unit 9 has a counter function, and counts the operation time after the first use of the device, that is, the operation time in the learning mode. When the count value exceeds 72 hours, the control unit 9 controls the first and second switch units so as to automatically select the high-speed operation mode. That is, in this example, the 72-hour learning mode operation is a so-called “run-in operation”. After this “run-in operation” is completed, high-quality and high-speed encoding can be performed.

  Of course, both manual and automatic modes may be switched. Even after the above-described “run-in operation”, if the shooting scene changes, the learning mode may be manually selected each time.

  In the automatic mode switching, for example, since the learning mode determination unit (107) has a long processing time, the processing speed may be improved by reducing the frame rate in the learning mode. As a result, it is possible to switch between the two without reducing the processing speed. When both are operated in parallel, normally only one of them may be operating, and both of them may operate in parallel for some period or time.

  Furthermore, in this embodiment, it is possible to construct an encoding process specialized for the shooting environment. For this reason, when the shooting location and the nature of the video are limited to some extent, such as a video conference system or a surveillance system. The encoding process according to the present embodiment is particularly effective.

  Next, a second embodiment of the present invention will be described with reference to FIGS. In this embodiment, when the control unit 9 is constituted by a CPU, the learning mode and the high-speed operation mode are automatically switched according to the load of the CPU, that is, the usage rate of the CPU. In this embodiment, a CPU usage rate measurement unit (190) is provided in place of the control unit 9 or in the control unit 9, thereby controlling the first switch (110) and the second switch (111). This is different from the first embodiment. Other points are the same as in the first embodiment.

  The CPU usage rate measurement unit (190) according to the present embodiment measures, for example, whether there is a CPU resource vacancy, and if there is a vacancy, the learning mode determination unit in parallel with the high-speed mode determination unit (108) The first and second switches are controlled so as to operate (107). That is, in this embodiment, the high operation speed mode determination unit (108) is always operating, and the learning mode determination unit (107) partially operates and the result is reflected in the high speed operation mode determination unit (108). I try to let them.

  With this configuration, there is a feature that the encoding efficiency is improved as the number of uses of the image processing apparatus to which the present embodiment is applied increases. FIG. 10 shows the relationship between the usage time and the distortion amount of the image processing apparatus according to the present embodiment when encoding is performed at a constant bit rate. FIG. 11 shows the relationship between the usage time and the bit rate of the image processing apparatus according to this embodiment. In each figure, the learning mode determination unit (107) is selected and operated in three sections from time t1 to time t2, from time t3 to time t4, and from time t5 to time t6. As is apparent from this figure, during the period in which the learning mode determination unit (107) operates in parallel with the high-speed operation mode determination unit (108), the encoding efficiency is gradually improved. That is, it can be understood from FIG. 10 that the image quality is improved by performing learning, and the generated code amount is decreased by performing learning from FIG.

  As described above, according to the present embodiment, the encoding process is speeded up according to the number of uses, and the image quality deterioration due to the encoding is reduced and the image quality is improved.

  The present invention can be applied to an image processing apparatus having a function of encoding a moving image, such as a hard disk recorder, a mobile phone, a digital camera, a surveillance system, a video conference system, and the like.

The figure which shows the example of 1 structure of the encoding part which concerns on 1st Example. The figure which shows one specific example of the mode determination part 107 for learning The figure which shows an example of a basic structure of the high-speed operation mode determination part 108 The figure which shows one specific example of the high-speed operation mode determination part 108 The figure which shows one specific example of the neural network 1 (402) shown by FIG. The figure which shows one specific example of the neural network 2 (403) shown by FIG. The figure which shows one specific example of the neural network 3 (404) shown by FIG. The figure which shows the flow of a process of the encoding mode determination in the high-speed operation mode determination part 108. The figure which shows the example of 1 structure of 2nd Example based on this invention. The figure which shows the effect of 2nd Example The figure which shows the effect of 2nd Example The figure which shows the example of 1 structure of the image processing apparatus with which this invention is applied.

Explanation of symbols

DESCRIPTION OF SYMBOLS 101 ... Original image memory, 102 ... Block division part, 103 ... Motion search part, 104 ... Predictive image generation part, 105 ... Decoding processing part, 106 ... Reference image memory, 107 ... Learning mode determination part, 108 ... High-speed operation mode Judgment unit, 109... Encoding processing unit, 110... First switch, 111.

Claims (1)

Coding mode defined by a prediction mode of any one of three prediction methods of inter-screen skip mode, intra mode, and inter mode, and a predetermined block size applied when intra mode or inter mode is selected. And an image processing apparatus for encoding in the selected encoding mode,
It has a learning mode and a high-speed operation mode,
In the learning mode,
Encode the block image in the learning original image in each encoding mode,
Find the coding cost in each coding mode,
The first determiner used in the high-speed mode based on the correspondence relationship between the spatial resolution, QP, the coding mode with the smallest code amount, and the difference signal according to the coding mode with the smallest code amount. And a second determiner and a third determiner,
In the high speed operation mode,
When coding a block image of an image to be processed at high speed,
The difference signal is obtained for each of the inter-screen skip mode, the intra mode applied to one block size of the specified block size, and the inter mode applied to one block size of the specified block size. In the case of the spatial resolution, QP, and inter mode of the image to be processed at high speed, a difference vector signal is further obtained and input to the first determiner. Output the likelihood of each mode, select the prediction method of the coding mode with the highest likelihood,
Next, an image processing apparatus characterized by performing any one of the following steps (a) to (c).
(A) When the inter-screen skip mode is selected as the prediction method by the first determiner, prediction and encoding are performed in the inter-screen skip mode.
(B) When an intra mode is selected as a prediction method by the first determiner, a difference signal for the intra mode applied to another block size of the specified block size is obtained. While inputting into the second determiner, the spatial resolution and QP of the high-speed processed image are input to the second determiner, so that the likelihood for each coding mode is output, and the most likely of these is output. A block size of a coding mode having a higher value is selected, and prediction and coding are performed in an intra mode to which the selected block size is applied.
(C) When inter mode is selected as the prediction method in the first determiner, the difference signal and difference vector information for the inter mode applied to another block size of the specified block size Each of the inter modes applied to all block sizes by inputting the spatial resolution, QP, and difference vector information of the high-speed processed image to the third determiner. The likelihood is output, the block size of the coding mode with the highest likelihood is selected, and prediction and coding are performed in the inter mode to which the selected block size is applied.

Images