JP5328854B2 - Motion vector detection apparatus and motion vector detection method - Google Patents

Abstract

A motion vector detection apparatus is configured to calculate a temporal distance between a frame to be coded and each of a plurality of reference candidate frames referred to by the frame to be coded. The motion vector detection apparatus searches for a candidate motion vector between the frame to be coded and each the plurality of reference candidate frames and detects a motion vector for the frame to be coded from the candidate motion vectors. In searching for and detecting a candidate motion vector, the motion vector detection apparatus changes an amount of the calculation performed during the detection of a candidate motion vector according to the calculated temporal distance between the frame to be coded and the reference candidate frame, and a coding type of the reference candidate frame.

Description

  The present invention relates to a motion vector detection device and a motion vector detection method, and is particularly suitable for use in detecting a motion vector between screens.

  In recent years, the digitization of information related to multimedia has been rapidly progressing. Along with this, there is an increasing demand for higher image quality of video information. For example, broadcasting media is being transferred from a conventional SD (Standard Definition) of 720 × 480 pixels to an HD (High Definition) of 1920 × 1080 pixels. However, such a demand for higher image quality of video information simultaneously causes an increase in digital data. Therefore, there is a demand for compression encoding technology and decoding technology that exceed the performance of the prior art.

  In response to these requirements, standardization of an encoding scheme using inter-picture prediction using correlation between pictures is being promoted by activities of ITU-T SG16 and ISO / IEC JTC1 / SC29 / WG11. Among such encoding schemes, as an encoding scheme that is said to be capable of encoding with the highest efficiency, H.264 is currently available. H.264 / MPEG-4 PART10 (AVC). In the following description, this encoding method is referred to as H.264. H.264.

  H. In H.264, reference images for detecting motion vectors between screens can be selected relatively freely. Further, H.C. In H.264, it is possible to detect a motion vector in a smaller unit than in the prior art by detecting a motion vector by dividing an image to be encoded into units of a macroblock or less. As a result, the amount of generated code can be further reduced.

  H. As a technique using H.264, Patent Document 1 has a plurality of frame memories, and a plurality of images stored in the plurality of frame memories are referred to when an image to be encoded is encoded. A configuration to select from is disclosed.

  Conventional encoding schemes such as MPEG-1, MPEG-2, and MPEG-4 have a forward prediction function for predicting a future image from a past image and a future image as a function for motion prediction. And a backward prediction function for predicting past images. Here, to predict a past image from a future image is to predict an image skipped from encoding from the current image. In the following description, conventional encoding methods such as MPEG-1, MPEG-2, and MPEG-4 are collectively referred to as MPEG encoding methods.

  In many cases, the correlation between images is considered to be higher as the image is closer to the encoding target in time. Therefore, in forward prediction and backward prediction in the MPEG encoding method, an I picture or P picture located in the vicinity of an image to be encoded is usually used as a reference image.

  However, in a video camera equipped with a codec (encoder / decoder) of the MPEG encoding method, when a moving image is shot, the camera moves quickly such as pan / tilt, or an image just like an image immediately after a cut change. The change between them may be abrupt. In such a case, even if the images are temporally close, the correlation between the images is low. Therefore, there is a problem that the advantage of motion compensation prediction cannot be utilized.

  A technology that is expected to improve this problem is H.264. This is a prediction method adopted in H.264. H. In H.264, predictive coding is performed not only on neighboring images but also on images at positions that are separated in time, and if improvement in coding efficiency is expected over neighboring images, images at separated positions are used. Can be used as a reference image.

JP 2005-184694 A

  As described above, H.P. In H.264, even when the motion of the camera that captured the moving image is fast or when a cut change occurs, an image that minimizes the error between the input image and the already encoded image can be freely used as a reference image. You can choose. Thereby, it is possible to improve the precision of motion compensation prediction.

  However, if an operation that selects an image that minimizes the error from the input image is performed on all the already encoded images, the amount of calculation increases in proportion to the number of reference candidate images. There is a problem that it takes time.

  In the case of a mobile device such as a video camera, when the calculation load increases, the consumption of the battery to be driven increases. Therefore, there is a problem that the time that can be taken is shortened.

  The present invention has been made in view of such problems, and an object of the present invention is to suppress an increase in the amount of calculation when detecting the motion vector while improving the detection accuracy of the motion vector.

In order to achieve the above object, a motion vector detection device according to the present invention is a motion vector detection device that detects a motion vector between screens, and is referred to by an encoding target image and the encoding target image. A calculation unit that calculates a temporal distance between each of a plurality of candidate images that are reference image candidates, and a motion vector is searched between the encoding target image and each of the plurality of candidate images. A motion vector detecting means for detecting a motion vector based on the search result, and when searching for a motion vector between the encoding target image and each of the plurality of candidate images, the motion vector detecting means, varying the amount of operation of said operation means is performed in response to the value of the peak signal to noise ratio of the temporal distance between the respective candidate images for each candidate image computed (PSNR) Characterized in that it.

The motion vector detection method according to the present invention is a motion vector detection method for detecting a motion vector between screens, and includes a plurality of encoding target images and reference image candidates referred to by the encoding target image. A calculation step of calculating a temporal distance between each of the candidate images, a motion vector between the encoding target image and each of the plurality of candidate images, and based on the search result A motion vector detection step for detecting a motion vector, and when performing a motion vector search between the encoding target image and each of the plurality of candidate images, an operation executed in the motion vector detection step amount, and characterized in that it is changed according to the value of the peak signal to noise ratio of the temporal distance between the respective candidate images for each candidate image calculated by said calculation step (PSNR) That.

  ADVANTAGE OF THE INVENTION According to this invention, the increase in the amount of calculations at the time of detecting a motion vector can be suppressed, improving the detection accuracy of a motion vector.

It is a block diagram which shows the structure of the video camera apparatus which concerns on the 1st Embodiment of this invention. It is a block diagram which shows the structure of the signal processing part which concerns on the 1st Embodiment of this invention. It is a block diagram which shows the structure of the motion vector detector which concerns on the 1st Embodiment of this invention. It is the figure which showed an example of the relationship between the encoding object frame and reference candidate frame which concern on the 1st Embodiment of this invention. It is a flowchart explaining an example of operation | movement of the motion vector detector which concerns on the 1st Embodiment of this invention. It is a block diagram which shows the structure of the motion vector detector which concerns on the 2nd Embodiment of this invention. It is a flowchart explaining an example of operation | movement of the motion vector detector which concerns on the 2nd Embodiment of this invention. It is a figure which shows an example of the reduced image which concerns on the 2nd Embodiment of this invention. It is a block diagram which shows the structure of the motion vector detector which concerns on the 3rd Embodiment of this invention. It is a flowchart explaining an example of operation | movement of the motion vector detector which concerns on the 3rd Embodiment of this invention. It is the figure which showed an example of the relationship between the encoding object frame and reference candidate frame which concern on the 3rd Embodiment of this invention. It is a block diagram which shows the structure of the signal processing part which concerns on the 4th Embodiment of this invention. It is a block diagram which shows the structure of the motion vector detector which concerns on the 4th Embodiment of this invention. It is a flowchart explaining an example of operation | movement of the motion vector detector which concerns on the 4th Embodiment of this invention.

(First embodiment)
Hereinafter, a first embodiment of the present invention will be described with reference to the drawings.

  FIG. 1 is a diagram illustrating an example of a configuration of a video camera device 10 according to the present embodiment. In the present embodiment, H.264. The video camera apparatus 10 that performs encoding by H.264 will be described as an example.

  In FIG. 1, the imaging unit 11 generates digital captured image data using, for example, an optical system including a lens, a photoelectric conversion element, an A / D conversion circuit, and the like, and the generated captured image data is not illustrated. Stored in the image memory. The signal processing unit 12 performs various signal processing and conversion on the captured image data read from the image memory so as to have a desired format for display and recording. Details of the signal processing unit 12 will be described later with reference to FIG.

  The recording unit 13 is for recording image data or the like on a recording medium, or reading image data or the like from the recording medium. For example, a semiconductor memory can be used as the recording medium.

  The system control unit 14 is for performing control of the entire video camera device 10 and various calculations. The system control unit 14 includes, for example, a CPU, a ROM, and a RAM. The CPU executes a program stored in the ROM using the RAM, thereby performing control of the entire video camera device 10 and various calculations. .

  The display unit 15 receives a signal processed by the signal processing unit 12 or a signal calculated by the system control unit 14 and displays an image based on the input signal on a display device such as an LCD (Liquid Crystal Display).

  The operation unit 16 is for a user to give an operation instruction to the video camera device 10 such as a main switch for turning on / off the power of the video camera device 10 and a switch for instructing start / end of shooting (recording). With various switches.

  Note that signal input / output among the signal processing unit 12, the recording unit 13, the system control unit 14, the display unit 15, and the operation unit 16 is performed via the bus 17.

  FIG. 2 is a block diagram illustrating a configuration of the signal processing unit 12 according to the present embodiment. In the following description, an encoding target image and a reference image (reference candidate image) used for prediction will be described using frame images (simply referred to as frames) as examples. An example of detecting a motion vector between screens, that is, between frames will be described.

  In FIG. 2, the selector 12 a selects the output destination of the captured image data read from the image memory in the imaging unit 11 in accordance with each encoding mode of intraframe encoding / interframe encoding. The intra predictor 12b receives the captured image data from the selector 12a, and outputs H.264 to the input captured image data. An intra prediction process using the H.264 encoding method is executed.

  The subtractor 12c subtracts the predicted image data output from the motion compensator 121 from the captured image data output from the selector 12a to calculate motion prediction error data. The converter 12d performs orthogonal transform on the data output from the subtractor 12c and the intra predictor 12b, and outputs the orthogonal transform coefficient obtained by the orthogonal transform to the quantizer 12e. The quantizer 12e quantizes the orthogonal transform coefficients and outputs all quantized orthogonal transform coefficients to the scan processor 12f and the inverse quantizer 12h.

  The scan processor 12f performs a scan process such as a zigzag scan on the quantized orthogonal transform coefficient in accordance with the encoding mode. The entropy encoder 12g entropy-encodes the output from the scan processor 12f, and outputs the entropy-encoded data (encoded data) to the bus interface (I / F) 12n. The encoded data output to the bus I / F 12n is supplied to the recording unit 13 via the bus 17 and is recorded by the recording unit 13. The encoded data recorded by the recording unit 13 may be further moved to a recording medium such as a hard disk or an optical disk for recording.

  The inverse quantizer 12h receives the quantized orthogonal transform coefficient from the quantizer 12e and performs inverse quantization. The inverse transformer 12i performs inverse orthogonal transform on the orthogonal transform coefficient inversely quantized by the inverse quantizer 12h, and decodes motion prediction error data obtained by the subtractor 12c. The adder 12j adds the prediction error data output from the inverse converter 12i and the predicted image data output from the motion compensator 121 to generate a restored image (local decoded image). The generated restored image data is output to the bus I / F 12n. The data output to the bus I / F 12n is recorded in the frame unit in the frame memory provided in the recording unit 13 as reference image data. In the following description, reference image data recorded in a frame memory provided in the recording unit 13 is referred to as a reference candidate frame as necessary.

  The motion vector detector 12k calculates an optimal motion vector based on the encoding target frame and the plurality of reference candidate frames. The motion vector detector 12k of this embodiment inputs the number of the encoding target frame to be encoded and the number of the reference candidate frame from the system control unit 14 via the bus I / F 12n, and the input number Is used to determine the search accuracy. Details of the motion vector detector 12k will be described later with reference to FIG.

  The motion compensator 121 generates predicted image data using the motion vector calculated by the motion vector detector 12k and the reference candidate frame having the smallest prediction error. The motion encoder 12m encodes the motion vector calculated by the motion vector detector 12k and outputs the encoded motion vector to the bus I / F 12n. The encoded motion vector output to the bus I / F 12n is recorded in the recording unit 13 in association with the encoded data.

  In the present embodiment, any of forward prediction referring to past frames, bidirectional prediction referring to past and future frames, and backward prediction referring to future frames can be used. Further, a device other than that shown in FIG. 2 may be provided in the signal processing unit 12.

  Next, details of the motion vector detector 12k will be described. FIG. 3 is a block diagram showing the configuration of the motion vector detector 12k according to this embodiment. The motion vector detector 12k in FIG. The present invention is suitable for application to an encoding device of H.264 or a related encoding method, and such an encoding device is suitable for use in a video camera device or the like.

  In FIG. 3, the motion vector detector 12k includes an encoding target frame storage unit 100, a reference candidate frame storage unit 101, a search accuracy determination unit 102, and a motion vector calculation unit 103.

  The encoding target frame storage unit 100 stores the encoding target frame 300 that is used for motion vector search and is about to be encoded. A plurality of reference candidate frames 301 are stored in the reference candidate frame storage unit 101.

  FIG. 4 is a diagram conceptually illustrating an example of the relationship between the encoding target frame and the reference candidate frame.

  In the example of FIG. 4, there are three selectable reference candidate frames 301 a to 301 c as reference frame candidates for the encoding target frame 300. In terms of frame numbers, a case where the reference candidate frame number 303 is “0” to “2” and the encoding target frame number 302 is “3” is shown as an example. In addition, 3t, 2t, and t shown in FIG. 4 represent temporal distances (time intervals) from the encoding target frame 300 to the reference candidate frames 301a, 301b, and 301c, respectively.

  FIG. 5 is a flowchart for explaining an example of the operation of the motion vector detector 12k according to the present embodiment. An example of the operation of the motion vector detector 12k shown in FIG. 3 will be described with reference to the flowchart of FIG.

  The search accuracy determination unit 102 waits until the encoding target frame number 302 and the reference candidate frame number 303 are specified from the system control unit 14 via the bus I / F 12n (step S101).

  In step S101, when the encoding target frame number 302 and the reference candidate frame number 303 are designated, the search accuracy determination unit 102 determines the temporal relationship between the encoding target frame 300 and the reference candidate frames 301a to 301c. A simple distance is calculated (step S102).

  Then, the search accuracy determination unit 102 changes the motion vector search accuracy according to the temporal distance between the encoding target frame 300 and the reference candidate frames 301a to 301c (step S103).

  Specifically, in the example shown in FIG. 4, the time between the reference candidate frame 301c whose reference candidate frame number 303 is “2” and the encoding target frame 300 whose encoding target frame number 302 is “3”. Since the typical distance is as short as “t”, there is a high probability that an optimal motion vector will be found. Therefore, the search accuracy determination unit 102 instructs the motion vector calculation unit 103 to perform a detailed search with one pixel accuracy in both the vertical direction and the horizontal direction, for example.

  On the other hand, the temporal distance between the reference candidate frame 301a whose reference candidate frame number 303 is “0” and the encoding target frame 300 whose encoding target frame number 302 is “3” is “3t”, which is far away. ing. For this reason, it is considered that the probability that an optimal motion vector is found in the reference candidate frame 301a is lower than the probability that an optimal motion vector is found in the reference candidate frame 301c. Therefore, for example, the search accuracy determination unit 102 instructs the motion vector calculation unit 103 to perform a coarse search with one pixel skip in both the vertical and horizontal directions. In addition, the search accuracy determination unit 102 instructs the motion vector calculation unit 103 so that the search range is the same as that of the reference candidate frame 301c, for example.

  In this way, when searching for a motion vector in the reference candidate frame 301a, the search range of the motion vector is made the same as in the case of searching for a motion vector in the reference candidate frame 301c. Furthermore, the motion vector search accuracy is halved in both the vertical and horizontal directions when searching for a motion vector in the reference candidate frame 301c. For this reason, when a motion vector is searched for in the reference candidate frame 301a, the amount of calculation can be reduced to ¼ compared to a case where a motion vector is searched for in the reference candidate frame 301c.

  The temporal distance between the reference candidate frame 301b with the reference candidate frame number 303 of “1” and the encoding target frame 300 with the encoding target frame number 302 of “3” is “2t”. As described above, the temporal distance between the reference candidate frame 301b and the encoding target frame 300 is slightly different from that between the reference candidate frame 301a and the encoding target frame 300. Therefore, the probability that the optimal motion vector is found in the reference candidate frame 301b is higher than the probability that the optimal motion vector is found in the reference candidate frame 301a, but is lower than the probability that the optimal motion vector is found in the reference candidate frame 301c. Conceivable.

  Therefore, the search accuracy determination unit 102 instructs the motion vector calculation unit 103 to set the search accuracy in the vertical direction for each pixel and the search accuracy in the horizontal direction to skip one pixel, for example. In addition, the search accuracy determination unit 102 instructs the motion vector calculation unit 103 so that the search range is the same as that of the reference candidate frame 301c, for example.

  Thus, when searching for the reference candidate frame 301b, the search range of the motion vector and the vertical search accuracy are made the same as when searching for the reference candidate frame 301c, and the horizontal search accuracy is referred to. The candidate frame 301c is halved when searching. For this reason, when the motion vector is searched for in the reference candidate frame 301b, the amount of calculation can be reduced to ½ compared to the case where the search is performed in the reference candidate frame 301c.

When the motion vector search accuracy is determined by the search accuracy determination unit 102 as described above, the motion vector calculation unit 103 determines the motion vector 304 (step S104). Specifically, the motion vector calculation unit 103 performs a reference candidate frame 301 stored in the reference candidate frame storage unit 101 for each macroblock included in the encoding target frame 300 stored in the encoding target frame storage unit 100. Search and estimate the motion vector. Here, for example, if a motion vector of a macroblock having a size of N × N (N is a natural number) is searched in a range wider by ± P pixels than the macroblock (p is a natural number), the search range is (1) It is expressed by a formula.
Search range = (N + 2p) × (N + 2p) (1)
The motion vector calculation unit 103 calculates a correlation coefficient at (2p + 1) two positions that can be motion vector candidates, and then determines a position indicating the maximum correlation degree as a motion vector.

  In order to estimate the motion vector having the maximum degree of correlation, the motion vector calculation unit 103 uses an evaluation function such as MSE (Mean Square Error), MAE (Mean Absolute Error), or MAD (Mean Absolute Difference). For example, MSE is represented by the following formula (2), and MAE is represented by the following formula (3).

  Here, Sref indicates a reference frame, and Scur, k indicates the k-th macroblock in the current frame for which a motion vector is searched. (I, j) indicates the spatial position of the reference frame in the kth macroblock of the current frame for which a motion vector is being searched.

  X and Y are the number of horizontal pixels and the number of vertical pixels in the motion vector search range. g and h are coefficients indicating the horizontal search accuracy and the vertical search accuracy specified by the search accuracy determination unit 102 (coefficients indicating how many pixels are calculated).

X and y are represented by the following formulas (4) and (5), respectively.
x = g × u (4)
y = h × v (5)
Furthermore, x, y, g, and h are natural numbers that satisfy the following expressions (6) to (9).
0 ≦ x ≦ X (6)
1 ≦ g ≦ X (7)
0 ≦ y ≦ Y (8)
1 ≦ h ≦ Y (9)
U and V are expressed by the following formulas (10) and (11), respectively.
U = X− | i | (10)
V = Y− | j | (11)
The evaluation functions represented by the formulas (2) and (3) are based on pixel differences. Accordingly, the motion vector calculation unit 103 selects a motion vector having the smallest MAE value or MSE value (when it becomes an LRV value) as a final motion vector in the current macroblock. It should be noted that functions other than those shown in FIG. 3 may be provided in the motion vector detector 12k.

  As described above, in this embodiment, the motion vector search accuracy is changed according to the temporal distance between the encoding target frame 300 and the reference candidate frames 301a to 301c. For example, since the temporal distance between the encoding target frame 300 and the reference candidate frame 301c is “t”, which is short, the probability that an optimal motion vector is found is high. Therefore, the search accuracy determination unit 102 performs a detailed search with 1 pixel accuracy in both vertical and horizontal directions, for example. On the other hand, since the temporal distance between the encoding target frame 300 and the reference candidate frame 301a is “3t”, which is far away, the probability of finding an optimal motion vector is low. Therefore, for example, the search accuracy determination unit 102 performs the search with the same search range as that of the reference candidate frame 301c and with a high accuracy of skipping one pixel both vertically and horizontally.

  As described above, when encoding an image separated in time, the motion vector search accuracy is changed according to the probability of finding the optimal motion vector. Therefore, the motion vector search range can be improved even for images that are separated in time, and the motion vector detection accuracy can be improved, and the motion vector search accuracy can be changed to detect a motion vector. The amount of computation can be reduced. Thereby, for example, it is possible to prevent as much as possible that the consumption of the battery to be driven increases and the photographing time is shortened.

  In the present embodiment, the search accuracy is set to one pixel in both the vertical direction and the horizontal direction, the case where the vertical direction and the horizontal direction are skipped by one pixel, and the vertical direction according to the temporal distance. The case where one pixel and the horizontal direction are skipped by one pixel is shown as an example. However, the search accuracy is not limited to these. For example, two pixels may be skipped in both the vertical direction and the horizontal direction. In the present embodiment, the case where the number of reference candidate frames is three has been described as an example, but the number of reference candidate frames is not limited to this. For example, the number of reference candidate frames may be increased. When the number of reference candidate frames is increased, the search accuracy can be further changed stepwise.

(Second Embodiment)
Next, a second embodiment of the present invention will be described. In the first embodiment described above, the motion vector search accuracy is changed according to the temporal distance between the encoding target frame 300 and the reference candidate frames 301a to 301c. On the other hand, in the present embodiment, the reduction ratios of the encoding target frame 300 and the reference candidate frames 301a to 301c are set according to the temporal distance between the encoding target frame 300 and the reference candidate frames 301a to 301c. I changed it. As described above, the present embodiment and the first embodiment are mainly different in the processing method using the encoding target frame 300 and the reference candidate frames 301a to 301c. Therefore, in the following description, the same parts as those of the first embodiment are denoted by the same reference numerals as those shown in FIGS.

  FIG. 6 is a block diagram showing an example of the configuration of the motion vector detector 12k according to this embodiment. FIG. 7 is a flowchart for explaining an example of the operation of the motion vector detector 12k according to this embodiment. The operation of the motion vector detector 12k shown in FIG. 6 will be described with reference to FIGS.

  The reduction rate determination unit 402 waits until the encoding target frame number 302 and the reference candidate frame number 303 are specified from the system control unit 14 via the bus I / F 12n (step S401). In step S401, when the encoding target frame number 302 and the reference candidate frame number 303 are designated, the reduction rate determination unit 402 determines the temporal relationship between the encoding target frame 300 and the reference candidate frames 301a to 301c. A simple distance is calculated (step S402).

  Then, the reduction rate determination unit 402 determines the image of the encoding target frame 300 and the images of the reference candidate frames 301a to 301c according to the temporal distance between the encoding target frame 300 and the reference candidate frames 301a to 301c. The reduction ratio is determined (step S403).

  The reduction rate determination unit 402 decreases the reduction rate as the temporal distance between the encoding target frame 300 and the reference candidate frames 301a to 301c is shorter, and the encoding target frame 300 and the reference candidate frames 301a to 301c are reduced. The reduction ratio is increased as the time distance between them is longer.

  Specifically, in the example shown in FIG. 4, the time between the reference candidate frame 301c whose reference candidate frame number 303 is “2” and the encoding target frame 300 whose encoding target frame number 302 is “3”. Since the typical distance is as short as “t”, there is a high probability that an optimal motion vector will be found. Therefore, the reduction rate determination unit 402 does not reduce the image, but uses the image of the encoding target frame 300 and the images of the reference candidate frames 301a to 301c as they are to search for a motion vector. Instruct 407.

  On the other hand, the temporal distance between the reference candidate frame 301a whose reference candidate frame number 303 is “0” and the encoding target frame 300 whose encoding target frame number 302 is “3” is “3t”, which is far away. ing. For this reason, it is considered that the probability that an optimal motion vector is found in the reference candidate frame 301a is lower than the probability that an optimal motion vector is found in the reference candidate frame 301c. Therefore, the reduction rate determination unit 402, for example, a reduced frame creation unit so as to reduce the image of the encoding target frame 300 and the images of the reference candidate frames 301a to 301c by 1/2 times in both the vertical direction and the horizontal direction. An instruction is issued to 404. The reduced frame creating unit 404 is image reducing means.

  In addition, the reduction rate determination unit 402 instructs the motion vector calculation unit 407 so that the search range is the same as that of the reference candidate frame 301c, for example. Accordingly, the number of pixels in the vertical direction and the horizontal direction are both halved. Therefore, when searching for a motion vector in the reference candidate frame 301a, the amount of calculation can be reduced to ¼ compared to searching for a motion vector in the reference candidate frame 301c.

  The temporal distance between the reference candidate frame 301b with the reference candidate frame number 303 of “1” and the encoding target frame 300 with the encoding target frame number 302 of “3” is “2t”. As described above, the temporal distance between the reference candidate frame 301b and the encoding target frame 300 is slightly different from that between the reference candidate frame 301a and the encoding target frame 300. Therefore, the probability that the optimal motion vector is found in the reference candidate frame 301b is higher than the probability that the optimal motion vector is found in the reference candidate frame 301a, but is lower than the probability that the optimal motion vector is found in the reference candidate frame 301c. Conceivable.

  Therefore, for example, the reduction rate determination unit 402 reduces the image of the encoding target frame 300 and the images of the reference candidate frames 301a to 301c so that the image is not reduced in the vertical direction but reduced to 1/2 times in the horizontal direction. An instruction is issued to the frame creation unit 404. In addition, the reduction rate determination unit 402 instructs the motion vector calculation unit 407 so that the search range is the same as that of the reference candidate frame 301c, for example. Therefore, the number of pixels in the horizontal direction is halved. For this reason, when a motion vector is searched for in the reference candidate frame 301b, the amount of calculation can be reduced to ½ compared to a case where a motion vector is searched for in the reference candidate frame 301c.

  As described above, when the reduction ratios of the image of the encoding target frame 300 and the reference candidate frames 301a to 301c are determined, the process proceeds to step S404. Then, the reduced frame creation unit 404 performs a reduction process on the image of the encoding target frame 300 and each of the reference candidate frames 301a to 301c according to the reduction rate determined by the reduction rate determination unit 402, and a plurality of reduced images. Is generated (step S404).

  FIG. 8 is a diagram illustrating an example of a reduced image of the encoding target frame 300 and reduced images of the reference candidate frames 301a to 301c according to the present embodiment.

  For example, when the reduced image 802a is generated by reducing the original image 801 shown in FIG. 8 by a factor of 1/2 both vertically and horizontally, the pixel values of two adjacent vertical and horizontal pixels are added and divided by 4. A reduced image 802a is generated. For example, the pixel A ′ in the reduced image 802 a can be generated by adding the pixel values of the pixels A, B, E, and F of the original image 801 and dividing the result by 4 ((A + B + E + F) / 4). In addition, the pixel B ′ in the reduced image 802 a can be generated by adding the pixel values of the pixels C, D, G, and H of the original image 801 and dividing the result by 4 ((C + D + G + H) / 4).

  On the other hand, when the reduced image 802b is generated by reducing the original image 801 shown in FIG. 8 by ½ times in the horizontal direction, the pixel values of two pixels adjacent in the horizontal direction are added and divided by two. A reduced image 802b is generated. For example, the pixel A ″ in the reduced image 802b can be generated by adding the pixel values of the pixels A and B in the original image 801 and dividing by 2 ((A + B) / 2). Also, the pixel in the reduced image 802b. B ″ can be generated by adding the pixel values of the pixels C and D of the original image 801 and dividing by 2 ((C + D) / 2).

  As described above, when the reduced frame creation unit 404 creates the reduced image of the encoding target frame 300 and the reduced images of the reference candidate frames 301a to 301c, the reduced image of the encoding target frame 300 is reduced. It is stored in the encoding target frame storage unit 405. Further, the reduced images of the reference candidate frames 301a to 301c are stored in the reduced reference candidate frame storage unit 406.

  Next, the motion vector calculation unit 407 determines a motion vector 408 (step S405). Here, when the reduction rate determination unit 402 instructs the motion vector calculation unit 407 not to reduce the image, the motion vector calculation unit 407 uses the motion vector calculation unit 103 of the first embodiment without using the reduced image. Perform the same operation.

  On the other hand, when the reduction ratio determining unit 402 instructs to reduce the image, the motion vector calculation unit 407 first receives a macroblock of the reduced image of the encoding target frame 300 from the reduction encoding target frame storage unit 405. Is read. Next, the motion vector calculation unit 407 searches for the motion vector within the range of the reduced images of the reference candidate frames 301a to 301c read from the reduced reference candidate frame storage unit 406 for the read macroblock. Based on this, a motion vector is estimated. The method for estimating the motion vector having the maximum correlation is the same as the method described in the first embodiment.

  It should be noted that functions other than those shown in FIG. 6 may be provided in the motion vector detector.

  As described above, in the present embodiment, the reduction rate of the encoding target frame 300 and the reduction of the reference candidate frames 301a to 301c according to the temporal distance between the encoding target frame 300 and the reference candidate frames 301a to 301c. The rate was changed. For example, since the temporal distance between the encoding target frame 300 and the reference candidate frame 301c is “t”, which is short, the image of the encoding target frame 300 and the image of the reference candidate frame 301c are not reduced. On the other hand, since the temporal distance between the encoding target frame 300 and the reference candidate frame 301a is “3t”, which is far away, the probability of finding an optimal motion vector is low. Therefore, a reduced image is generated by reducing the image of the encoding target frame 300 and the image of the reference candidate frame 301c by a factor of 1/2 both vertically and horizontally, and a motion vector is determined using the generated reduced image.

  As described above, when encoding an image that is temporally separated, the reduction rate of the encoding target frame 300 and the reduction of the reference candidate frames 301a to 301c are determined according to the probability of finding the optimal motion vector. Change the rate. Therefore, as in the first embodiment, the accuracy of motion vector detection can be improved, and an increase in the amount of computation when detecting a motion vector can be prevented as much as possible. Thereby, for example, it is possible to prevent as much as possible that the consumption of the battery to be driven increases and the photographing time is shortened.

  In the present embodiment, the image of the encoding target frame 300 and the images of the reference candidate frames 301a to 301c are reduced to 1/2 times both vertically and horizontally according to the temporal distance, and the image is reduced vertically. An example in which only the horizontal direction is reduced by a factor of 1/2 is shown as an example. However, the image reduction ratio of the encoding target frame 300 and the image reduction ratios of the reference candidate frames 301a to 301c are not limited to this. For example, the image of the encoding target frame 300 and the images of the reference candidate frames 301a to 301c may be reduced to 1/3 times or 1/4 times both vertically and horizontally. In the present embodiment, the case where the number of reference candidate frames is three has been described as an example, but the number of reference candidate frames is not limited to this. For example, the number of reference candidate frames may be increased. When the number of reference candidate frames is increased, the reduction ratio can be further changed in stages.

(Third embodiment)
Next, a third embodiment of the present invention will be described. In the first embodiment described above, the motion vector search accuracy is changed only in accordance with the temporal distance between the encoding target frame 300 and the reference candidate frames 301a to 301c. On the other hand, the present embodiment has the same configuration as that of the first embodiment, but in addition to the temporal distance between the encoding target frame 300 and the reference candidate frames 301a to 301c, the encoding of the reference candidate frame is performed. The motion vector search accuracy is changed according to the type. Accordingly, in the following description, portions other than the search accuracy determination unit 102 that performs an operation different from that of the first embodiment are denoted by the same reference numerals as in FIGS. To do. H. In an encoding method such as H.264, encoding in units of slices composed of one or more macroblocks smaller than a picture is possible. Therefore, the encoding type is replaced with “slice type” in addition to “picture type”. May be interpreted. Hereinafter, a slice type will be described as an example.

  FIG. 9 is a block diagram showing an example of the configuration of the motion vector detector 12k according to this embodiment. FIG. 10 is a flowchart for explaining an example of the operation of the motion vector detector 12k according to the present embodiment. The operation of the motion vector detector 12k shown in FIG. 9 will be described with reference to FIG.

  The search accuracy determination unit 102 waits until the encoding target frame number 302, the reference candidate frame number 303, and the slice type 901 of the reference candidate frame are specified from the system control unit 14 via the bus I / F 12n ( Step S1001).

  In step S1001, the encoding target frame number 302, the reference candidate frame number 303, and the slice type 901 of the reference candidate frame are designated. Then, the search accuracy determination unit 102 calculates a temporal distance td between the encoding target frame 300 and the reference candidate frame 301 (step S1002). Based on the calculated td, the search accuracy determination unit 102 increases the search accuracy in steps according to the temporal distance between the encoding target frame 300 and the reference candidate frame 301, as in the first embodiment. Change. Further, the search accuracy determination unit 102 changes the search accuracy according to the slice type 901 of the reference candidate frame.

  In other words, the search accuracy determination unit 102 determines whether the slice type 901 of the reference candidate frame is an I slice (step S1003). If the reference candidate frame slice type 901 is an I slice (YES in step S1003), the search accuracy determination unit 102 performs a process of subtracting 2t from td (step S1004).

  An I slice generally has a large amount of code to be assigned, and is likely to be a reference candidate frame with good image quality. Therefore, the search accuracy determination unit 102 increases the search accuracy by subtracting 2t in this way, even if the temporal distance is long for an I slice. If the slice type 901 of the reference candidate frame is not an I slice (NO in step S1003), the process proceeds to step S1005.

  Subsequently, the search accuracy determination unit 102 determines whether or not the slice type 901 of the reference candidate frame is a P slice (step S1005). If the reference candidate frame slice type 901 is a P slice (YES in step S1005), the search accuracy determination unit 102 performs a process of subtracting t from td (step S1006). Although the image quality of the P slice is inferior to that of the I slice, the image quality is often better than that of the B slice. Therefore, the search accuracy determination unit 102 does not increase the search accuracy as much as the I slice, even if the temporal distance is long if it is a P slice, but increases the search accuracy over the B slice. The search accuracy is increased by subtracting.

  If the slice type of the reference candidate frame is not a P slice (NO in step S1005), the process proceeds to step S1007. That is, when the reference candidate frame slice type 901 is a B slice other than I and P, the search accuracy determination unit 102 does not perform a weighting process that increases the search accuracy, and the temporal distance between frames. The search accuracy is determined only according to.

  In the conventional MPEG, H.264 is used. A B picture that is almost equivalent to the H.264 B slice cannot be set as a reference frame. In H.264, B slices can also be set as reference frames, and therefore can be added to reference candidate frames.

  In accordance with td obtained as described above, the search accuracy determination unit 102 changes the motion vector search accuracy (step S1007).

  FIG. 11 is a diagram illustrating an example of the relationship between the encoding target frame and the reference candidate frame according to the present embodiment. In the case of the example illustrated in FIG. 11, the temporal distance td between the encoding target frame 300 whose encoding target frame number 302 is “3” and the reference candidate frame 301 c whose reference candidate frame number 303 is “2” is “ t ”. Further, the temporal distance td from the reference candidate frame 301b whose reference candidate frame number 303 is “1” is “2t”. Further, the temporal distance td from the reference candidate frame 301a having the reference candidate frame number 303 of “0” is “3t”.

  However, since the reference candidate frame 301a is an I slice, 2t is subtracted and changed to td = t, and since the reference candidate frame 301b is a P slice, t is subtracted and changed to td = t. Therefore, in the example illustrated in FIG. 11, all of the reference candidate frame 301a, the reference candidate frame 301b, and the reference candidate frame 301c are td = t.

  That is, in this embodiment, in addition to the effects of the first embodiment, if the reference candidate frame is an I slice, the search accuracy is made as high as that of the immediately preceding frame even if the frame is considerably separated in time. Can do. If the reference candidate frame is a P slice, the search accuracy can be made as high as that of the immediately preceding frame even if the reference candidate frame is a little apart in time. At that time, the total number of searches is made the same as in the first embodiment.

  When the search accuracy of the motion vector is determined by the search accuracy determination unit 102, the motion vector calculation unit 103 determines the motion vector 304 (step S1008). Note that the specific processing for obtaining the motion vector is the same as that in the first embodiment, and a description thereof will be omitted.

  As described above, in the present embodiment, the slice type 901 of the reference candidate frames 301a to 301c is considered in addition to the temporal distance between the encoding target frame 300 and the reference candidate frames 301 (301a to 301c). Then, the motion vector search accuracy is changed according to the probability of finding the optimal motion vector.

  Therefore, the motion vector search range can be improved even for images that are separated in time, and the motion vector detection accuracy can be improved, and the motion vector search accuracy can be changed to detect a motion vector. The amount of computation can be reduced. Furthermore, by considering the slice type (picture type) of the reference candidate frame, the motion vector search accuracy can be further improved. Thereby, for example, it is possible to prevent as much as possible that the consumption of the battery to be driven increases and the photographing time is shortened.

  In this embodiment, “2t” is subtracted from td as a value to be subtracted from td, and “t” is subtracted from P slice. However, these values are merely examples, and the numerical values are not limited. For example, “t” may be set to be decreased at the time of I slice, and “0.5 t” may be decreased at the time of P slice.

  In this embodiment, the search accuracy of the motion vector is changed. However, as in the second embodiment, the reduction rate of the encoding target frame 300 and the reduction rates of the reference candidate frames 301a to 301c are set as reference candidate frames. It may be changed according to the slice type (picture type).

(Fourth embodiment)
Next, a fourth embodiment of the present invention will be described. In the first embodiment described above, the motion vector search accuracy is changed only in accordance with the temporal distance between the encoding target frame 300 and the reference candidate frames 301a to 301c. In contrast, the present embodiment has the same configuration as that of the first embodiment, but includes a peak signal-to-noise ratio (PSNR) calculation unit 12o in the signal processing unit 12. Then, in addition to the temporal distance between the encoding target frame 300 and the reference candidate frames 301a to 301c, the motion vector search accuracy is changed according to the PSNR of the reference candidate frame. Accordingly, in the following description, parts other than the search accuracy determination unit 102 and the PSNR calculation unit 12o that perform operations different from those of the first embodiment are denoted by the same reference numerals as those shown in FIGS. Detailed description is omitted.

  FIG. 12 is a block diagram illustrating an example of the configuration of the signal processing unit 12 according to the present embodiment. In the present embodiment, only the operation of the PSNR calculation unit 12o having a configuration different from that of the first embodiment will be described in the signal processing unit 12, and the description of other components that perform the same operation as the first embodiment will be given. Omitted.

  In FIG. 12, the PSNR calculation unit 12o compares the unencoded image data coming from the image pickup unit 11 with the restored image data coming from the adder 12j, and compares a pair of peak signals that serve as an index of image degradation. The value of the noise ratio (PSNR) is calculated from the following equation (12).

  Here, N and M represent the number of vertical and horizontal pixels of the image, respectively. P (i, j) represents the pixel value at the position (i, j) in the image data before encoding, and p ′ (i, j) represents the pixel value at the position (i, j) in the restored image data. Represent. T represents the number of gradations of an image minus 1 (T = 255 for an 8-bit / pixel image).

  The PSNR value obtained by the PSNR calculation unit 12o is transmitted to the motion vector detector 12k in FIG. 12, and is used to change the motion vector search accuracy.

  FIG. 13 is a block diagram showing the configuration of the motion vector detector 12k according to this embodiment. FIG. 14 is a flowchart illustrating an example of the operation of the motion vector detector 12k according to this embodiment. The operation of the motion vector detector 12k shown in FIG. 13 will be described with reference to FIG.

  The search accuracy determination unit 102 waits until the encoding target frame number 302, the reference candidate frame number 303, and the PSNR value 1301 of the reference candidate frame are specified from the system control unit 14 via the bus I / F 12n. (Step S1401).

  In step S1401, the encoding target frame number 302, the reference candidate frame number 303, and the PSNR 1301 of the reference candidate frame are designated. Then, the search accuracy determination unit 102 calculates a temporal distance td between the encoding target frame 300 and the reference candidate frame 301 (step S1402). Based on the calculated td, the search accuracy determination unit 102 increases the search accuracy in steps according to the temporal distance between the encoding target frame 300 and the reference candidate frame 301, as in the first embodiment. Change. Further, the search accuracy determination unit 102 changes the search accuracy according to the PSNR 1301 of the reference candidate frame.

  That is, the search accuracy determination unit 102 determines whether or not the PSNR 1301 of the reference candidate frame is larger than the predetermined threshold Th1 (step S1403). When the PSNR 1301 of the reference candidate frame is larger than the predetermined threshold Th1, that is, when PSNR> Th1 is satisfied (YES in step S1403), the search accuracy determination unit 102 performs a process of subtracting t from td (step S1404).

  In general, when the PSNR is high, there is little deterioration of the reference candidate frame, and there is a high possibility that it is suitable as a reference frame. Therefore, if PSNR> Th1 is satisfied, the search accuracy determination unit 102 increases the search accuracy by subtracting t in this way even when the temporal distance between frames is long. .

  The threshold value Th1 can be set to a fixed value such as 30 (dB), which is said to be a practical image quality level in SD (Standard Definition), for example. If the case where the PSNR of most restored images is less than 30 (dB) is taken into consideration, the average value of the PSNR of the restored images may be used as a variable threshold obtained while being updated as needed. If the PSNR 1301 of the reference candidate frame does not satisfy PSNR> Th1 (NO in step S1403), the process proceeds to step S1405.

  Subsequently, the search accuracy determination unit 102 determines whether the PSNR 1301 of the reference candidate frame is smaller than a predetermined threshold Th2 (where Th2 <Th1) (step S1405). When the PSNR 1301 of the reference candidate frame is smaller than the predetermined threshold Th2, that is, when PSNR <Th2 is satisfied (YES in step S1405), the search accuracy determination unit 102 performs a process of adding t to td (step S1406). The setting of the threshold Th2 is the same as that for the threshold Th1.

  In general, when the PSNR is low, the deterioration of the reference candidate frame is large, and it is highly possible that the reference frame is inappropriate. Therefore, if PSNR <Th2 is satisfied, the search accuracy determination unit 102 decreases the search accuracy by adding t in this way even when the temporal distance between frames is short.

  If the PSNR 1301 of the reference candidate frame does not satisfy PSNR <Th2 (NO in step S1405), the process proceeds to step S1407. That is, when the PSNR is in an intermediate state that is neither high nor low, the search accuracy determination unit 102 determines the search accuracy only according to the temporal distance between frames.

  In accordance with td obtained as described above, the search accuracy determination unit 102 changes the motion vector search accuracy (step S1407).

  Thus, in this embodiment, in addition to the effects of the first embodiment, if the PSNR of the reference candidate frame is a large value, the search accuracy can be increased even for frames that are separated in time. If the PSNR of the reference candidate frame is a small value, the search accuracy can be lowered even for frames that are close in time.

  When the search accuracy of the motion vector is determined by the search accuracy determination unit 102, the motion vector calculation unit 103 determines the motion vector 304 (step S1408). Note that the specific processing for obtaining the motion vector is the same as that in the first embodiment, and a description thereof will be omitted.

  As described above, in the present embodiment, in addition to the temporal distance between the encoding target frame 300 and the reference candidate frames 301 (301a to 301c), the PSNR value 1301 of the reference candidate frames 301a to 301c is also considered. Then, the motion vector search accuracy is changed according to the probability of finding the optimal motion vector.

  Therefore, the motion vector search range can be improved even for images that are separated in time, and the motion vector detection accuracy can be improved, and the motion vector search accuracy can be changed to detect a motion vector. The amount of computation can be reduced. Furthermore, by considering the PSNR value of the reference candidate frame, the motion vector search accuracy can be further improved and the calculation can be reduced more efficiently. Thereby, for example, it is possible to prevent as much as possible that the consumption of the battery to be driven increases and the photographing time is shortened.

  In the present embodiment, “t” is subtracted as a value to be subtracted from td when the PSNR is high. However, these values are merely examples and the numerical values are not limited. For example, you may set so that "0.5t" may be reduced.

  Furthermore, in the present embodiment, td is corrected according to the PSNR value and the search accuracy is determined (changed), but as another method, the PSNR value itself may be corrected according to td. In this case, the search accuracy may be determined based on the corrected PSNR.

  That is, when td is large and the temporal distance to the reference candidate frame is large, the search accuracy determination unit 102 performs correction so as to subtract a large value from the PSNR. When td is small and the temporal distance to the reference candidate frame is small, the search accuracy determination unit 102 performs, for example, no correction on the PSNR. Then, the search accuracy determination unit 102 determines to increase the search accuracy when the corrected PSNR is large and to decrease the search accuracy when the PSNR after correction is small.

  In the present embodiment, the search accuracy of the motion vector is changed. However, as in the second embodiment, the image reduction rate of the encoding target frame 300 and the image reduction rates of the reference candidate frames 301a to 301c. May be changed according to the PSNR of the reference candidate frame.

(Other embodiments of the present invention)
In order to operate various devices to realize the functions of the above-described embodiments, program codes of software for realizing the functions of the above-described embodiments are provided to an apparatus or a computer in the system connected to the various devices. You may supply. What was implemented by operating said various devices according to the program stored in the computer (CPU or MPU) of the system or apparatus is also included in the category of the present invention.

  In this case, the program code of the software itself realizes the functions of the above-described embodiment. The program code itself and means for supplying the program code to a computer, for example, a recording medium storing the program code constitute the present invention. As a recording medium for storing the program code, for example, a flexible disk, a hard disk, an optical disk, a magneto-optical disk, a CD-ROM, a magnetic tape, a nonvolatile memory card, a ROM, or the like can be used.

  Further, the functions of the above-described embodiments are not only realized by executing the program code supplied by the computer. It goes without saying that the program code is also included in the embodiment of the present invention even when the function of the above-described embodiment is realized in cooperation with an operating system or other application software running on the computer. Yes.

  Further, after the supplied program code is stored in the memory provided in the function expansion board of the computer, the CPU provided in the function expansion board performs part or all of the actual processing based on the instruction of the program code. Needless to say, the present invention includes the case where the functions of the above-described embodiments are realized by the processing.

  Further, after the supplied program code is stored in the memory provided in the function expansion unit connected to the computer, the CPU or the like provided in the function expansion unit performs part or all of the actual processing based on the instruction of the program code. Do. Needless to say, the present invention includes the case where the functions of the above-described embodiments are realized by the processing.

  Note that each of the above-described embodiments is merely a specific example for carrying out the present invention, and the technical scope of the present invention should not be construed as being limited thereto. . That is, the present invention can be implemented in various forms without departing from the technical idea or the main features thereof.

DESCRIPTION OF SYMBOLS 11 Image pick-up part 12 Signal processing part 13 Recording part 14 System control part 15 Display part 16 Operation part 17 Bus 12k Motion vector detector 100 Encoding object frame preservation | save part 101 Reference candidate frame preservation | save part 102 Search accuracy determination part 103 Motion vector calculation part 300 encoding target frame 301 reference candidate frame 302 encoding target frame number 303 reference candidate frame number 304 motion vector 402 reduction rate determination unit 404 reduced frame creation unit 405 reduced encoding target frame storage unit 406 reduced reference candidate frame storage unit 407 motion Vector calculation unit 408 motion vector 901 reference candidate frame 1301 PSNR of reference candidate frame

Claims (7)

A motion vector detection device for detecting a motion vector between screens,
A computing means for computing a temporal distance between the encoding target image and each of a plurality of candidate images that are candidates for a reference image referred to by the encoding target image;
A motion vector detecting means for searching for a motion vector between the encoding target image and each of the plurality of candidate images, and detecting a motion vector based on the search result;
When searching for a motion vector between the encoding target image and each of the plurality of candidate images, the motion vector detection means includes a temporal distance to each candidate image calculated by the calculation means, A motion vector detection apparatus characterized in that the amount of calculation to be executed is changed according to a peak signal-to-noise ratio (PSNR) value of a candidate image. The motion vector search accuracy between the encoding target image and each of the plurality of candidate images is determined by calculating the temporal distance with respect to each candidate image calculated by the calculation means and the PSNR value of each candidate image. And determining means for determining according to
2. The motion vector detection unit performs a motion vector search between the encoding target image and each of the plurality of candidate images with a search accuracy determined by the determination unit. The motion vector detection device described in 1.   The determination means increases the search accuracy of the motion vector as the temporal distance to each candidate image calculated by the calculation means is shorter, and the temporal determination for each candidate image calculated by the calculation means The motion vector detection apparatus according to claim 2, wherein the search accuracy of the motion vector is lowered as the distance is longer. 4. The motion vector detection apparatus according to claim 3, wherein the determination unit increases the search accuracy of the motion vector when the PSNR value of the candidate image is larger than a predetermined threshold Th1 . 4. The motion vector detection device according to claim 3, wherein the determination unit reduces the search accuracy of the motion vector when the PSNR value of the candidate image is smaller than a predetermined threshold Th2. The determination means increases the search accuracy of the motion vector when the PSNR value of the candidate image is larger than a predetermined threshold Th1, and when the PSNR value of the candidate image is smaller than the predetermined threshold Th2. The motion vector detection apparatus according to claim 3, wherein search accuracy of the motion vector is lowered. A motion vector detection method for detecting a motion vector between screens,
A calculation step of calculating a temporal distance between the encoding target image and each of a plurality of candidate images that are candidates for the reference image referred to by the encoding target image;
A motion vector detection step of searching for a motion vector between the encoding target image and each of the plurality of candidate images, and detecting a motion vector based on the search result;
When searching for a motion vector between the encoding target image and each of the plurality of candidate images, the amount of calculation executed in the motion vector detection step is for each candidate image calculated in the calculation step. A motion vector detection method, wherein the method is changed according to a temporal distance and a peak signal-to-noise ratio (PSNR) value of each candidate image.