JP2010103715A - Motion vector detection device, motion vector detecting method, and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To improve accuracy in the case of detecting a candidate vector by using the evaluation information of a motion vector obtained from moving image data. <P>SOLUTION: This motion vector detection device is provided with a processing part 12 for detecting motion vectors with high frequency and generating evaluation information, an extracting part 13 for extracting the motion vectors on the basis of the evaluation information, and a motion vector determining part 14 for determining motion vectors among extracted candidates. An evaluation information generation processing part 12 determines whether a motion vector candidate calculated from a correlation information between a pixel of interest of one frame on a time base and a reference pixel within the search range of the other frame has a high correlation to the vectors with high frequency. When the correlation is high in the determination, a pixel of interest and/or a reference pixel corresponding to the motion vector candidate are selected from pixels in the frames, and the evaluation information of a motion vector which evaluates a possibility that the reference pixel is a motion candidate for the pixel of interest is formed on the basis of pixels other than the selected pixels. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention detects a motion vector from moving image data, and is applied to a case where image processing such as high efficiency encoding is performed. Related to the program.

  2. Description of the Related Art Conventionally, in the field of moving image processing, image processing has been performed efficiently using motion information, that is, the motion direction and size of an object in temporally different images. For example, a motion right detection result is used for motion compensation interframe coding in high-efficiency coding of an image or motion parameter control in a television noise reduction apparatus using an interframe time domain filter. A block matching method is known as a method for obtaining motion. In the block matching method, an area in which a block has moved is searched in block units in which an image of one frame is a block of a predetermined pixel unit. This motion vector detection process by block matching is the most popular general process as an image process using a motion vector, which has been put to practical use in the MPEG system or the like.

  However, since the block matching method is processing in units of blocks, it cannot be said that motion detection of an image in each frame is necessarily detected with high accuracy. For this reason, the present applicant has proposed the motion vector detection process described in Patent Document 1. In this motion vector detection process, an evaluation value related to the movement of each pixel position is detected from an image signal, the detected evaluation value is held as an evaluation value table, and a plurality of candidate vectors in one screen are obtained from the data of the evaluation value table. Extract the vectors. And the correlation of the pixel between the flame | frames matched with a candidate vector is determined for every pixel of the whole screen for the extracted several candidate vector. As a result, a candidate vector connecting pixels determined to have the highest correlation is determined as a motion vector for the pixel. Details of this processing will be described in an embodiment described later.

  FIG. 27 is a diagram showing a configuration of the previously proposed evaluation value table forming unit when a motion vector is determined using this evaluation value table. Referring to FIG. 27, the image signal obtained at the input terminal 1 is supplied to the correlation calculation unit 2. The correlation calculation unit 2 includes a reference point memory 2a, an attention point memory 2b, and an absolute value calculation unit 2c. The image signal obtained at the input terminal 1 is first stored in the reference point memory 2a, the stored data in the reference point memory 2a is moved to the attention point memory 2b, and one frame is generated between the reference point memory 2a and the attention point memory 2b. The pixel signals having the difference are stored. Then, the pixel value of the target pixel in the image signal stored in the target point memory 2b and the pixel value of the pixel position selected as the reference pixel in the image signal stored in the reference point memory 2a are read. The absolute value detector 2c detects the difference between the two signals. Data of the absolute value of the detected difference is supplied to the correlation determination unit 3. The correlation determination unit 3 includes a comparison unit 3a, compares it with a set threshold value, and compares it with the threshold value to obtain an evaluation value. As an evaluation value, a correlation value can be used as an example. For example, when the difference is equal to or less than a threshold value, it is assumed that the correlation is high.

  The evaluation value obtained by the correlation determination unit 3 is supplied to the evaluation value table calculation unit 4, the evaluation value integration unit 4a integrates the evaluation value, and the integration result is stored in the evaluation value table memory 4b. Then, the stored data of the evaluation value table memory 4b is supplied as evaluation value table data from the output terminal 5 to the subsequent circuit.

FIG. 28 is a diagram showing an outline of a processing state in which a motion vector is determined using the conventional evaluation value table shown in FIG. As shown in FIG. 28A, first, a pixel position serving as a reference for determining a motion vector in the previous frame F0 which is image data one frame before the current frame (current frame) F1 is set as a target pixel d0. . When the target pixel d0 is determined, a search area SA in a predetermined range around the pixel position of the target pixel d0 is set in the current frame F1. When the search area SA is set, an evaluation value is calculated using each pixel in the search area SA as a reference pixel d1 and registered in the evaluation table. Then, from the values registered in the evaluation value table, the reference pixel having the highest evaluation value in the search area SA is obtained as the pixel position of the current frame moved from the target pixel of the previous frame. Thus, by obtaining the reference pixel having the highest evaluation value, the motion vector is determined from the amount of motion between the reference pixel having the highest evaluation value and the target pixel, as shown in FIG.
The processing shown in FIGS. 27 and 28 is performed by using each pixel in one frame as a target pixel, so that evaluation values of motion vectors that may exist between the previous frame and the current frame are the same for all pixels. Integrated evaluation value table data is obtained. By sending the evaluation value table data to the subsequent processing unit, it is possible to detect a motion vector based on the evaluation value table data.
JP 2005-175869 A

  When a motion vector is detected based on the evaluation value table data, the determination of the optimal motion vector depends on the performance of the evaluation value table. In the conventional method shown in FIG. 28, the correlation determination between the target pixel and the motion candidate destination pixel in the search range in the future frame (current frame), specifically, if the difference absolute value of the luminance value is below a certain threshold value. The frequency is counted in the evaluation value table as a motion candidate vector.

However, in this conventional method, the information in the evaluation value table depends on the number of pixels that have the same movement in the screen.
Specifically, for example, consider a screen in which a relatively small object in the screen moves differently from the background in a situation where the background in the screen is moving greatly in a certain direction. One example of such a screen occurs when, for example, a certain sporting event is photographed and an image of a player chasing a player or a ball moving on the field is taken.
In the screen in such a situation, most of the motion vectors accumulated in the evaluation value table are motion vectors based on the background motion, and there are few detections of motion vectors of objects that move differently from the background. It will be buried in the value table.

Therefore, it is difficult to select a motion vector of a relatively small object as a candidate even if determination of a candidate vector is performed in a later processing system using this type of evaluation value table in the past. There was a problem that other movements other than the movement were not extracted. For this reason, there is a possibility that the motion vector cannot be correctly assigned to a motion different from the large motion on the screen. In particular, when there are a plurality of movements different from the movement of the background in the image, the evaluation values resulting from the respective movements are buried, making it difficult to detect the respective motion vectors.
Although the background has been described as an example of a large movement on the screen here, if the background is moving in the same direction over a relatively wide range even if it is not the background on the screen, the same problem will occur. There is. For example, the movement of the foreground object that occupies a large screen is the same as the background.

  The present invention has been made in view of such a point, and an object thereof is to improve accuracy in detecting a motion vector using evaluation information such as an evaluation value table. Another object of the present invention is to detect a plurality of movements even when there are a plurality of movements.

The present invention is applied when detecting a motion vector from moving image data.
As its processing configuration, a high-frequency motion vector detection process for detecting a high-frequency motion vector, a pixel selection process, a process for generating evaluation information, a process for extracting a motion vector based on the evaluation information, and an extraction process are extracted. An allocation process for determining a motion vector from the candidates is performed.
In the pixel selection process, a motion vector candidate obtained from pixel value correlation information between a target pixel of one frame on the time axis and reference pixels in the search area of the other frame of moving image data is a high-frequency motion vector. Determine if the correlation is high. When the correlation is high in the determination, the target pixel and / or reference pixel corresponding to the motion vector candidate is selected from the pixels in the frame.
In the process of generating the evaluation information, the evaluation information of the motion vector that evaluates the possibility that the reference pixel is the motion candidate of the target pixel is formed based on the pixels other than the pixel selected by the pixel selection process.

  According to the present invention, when creating the evaluation information, the evaluation information related to the pixel of the motion vector having a high correlation with the vector that appears frequently is selected and removed. Therefore, even if there is a large movement in one direction in the screen, such as a background movement, as an evaluation value indicating the possibility of becoming a motion vector candidate, the movement in the screen different from that is integrated. Can be discriminated from the evaluation information. Therefore, by using the obtained evaluation information, all movements in the screen can be detected correctly.

  According to the present invention, when creating the evaluation information, since the evaluation information related to the pixel of the motion vector having a high correlation with the vector that appears frequently is selected and removed, the fine motion existing in each frame is evaluated. Can be discriminated from. Therefore, when assigning a motion vector to each pixel in the motion vector assignment process, it is highly possible that a correct motion vector is included in a candidate even if the pixel is a part of the screen that moves differently from the majority of the screen, such as the background. Thus, the motion vector allocation accuracy is improved. For example, even when there are a plurality of movements of a relatively small object in one screen, an evaluation value for each movement can be appropriately obtained, and a plurality of movements can be calculated simultaneously.

Examples of embodiments of the present invention will be described in the following order.
1. Overview of overall configuration for detecting motion vectors: FIG.
2. Overview of overall processing for detecting motion vectors: FIG.
3. Configuration Example of First Embodiment (Example 1): FIG.
4). Processing Example of First Embodiment (Example 1 in Consideration of Pixel of Interest and High Frequency Motion): FIG.
5). Explanation of relationship between pixel of interest and high-frequency motion: FIG.
6). Example of image and evaluation value table in FIG. 4 example: FIGS.
7). Configuration Example of First Embodiment (Example 2): FIG.
8). Processing Example of the First Embodiment (Example 2 in Consideration of Pixel of Interest and High Frequency Motion): FIG.
9. Another processing example of the first embodiment (example 2 in which attention pixel and reference pixel and high-frequency motion are considered): FIG.
10. Explanation of relationship between reference pixel and high-frequency motion: FIG.
11. Example of Evaluation Value Table in FIG. 12 Example: FIG.
12 Modification of the first embodiment 13. Configuration example of the second embodiment: FIG.
14 Processing Example of Second Embodiment: FIG.
15. Explanation of the spatial inclination pattern: FIGS.
16. Example of motion vector extraction unit: FIGS. 22 and 23
17. Example of motion vector assigning unit: FIGS.
18. Description of modification common to each embodiment

[1. Overview of overall configuration for motion vector detection]
A first embodiment of the present invention will be described with reference to FIGS.
In the present embodiment, a motion vector detection apparatus that detects a motion vector from moving image data is used. As the detection process, evaluation information is formed from pixel value correlation information, and the motion vector is calculated from the evaluation information. Is determined. In the following description, what stores motion vector evaluation information is referred to as an evaluation value table. However, this evaluation value table does not necessarily have to be configured as table-like storage information. Any information may be used. For example, the evaluation information in the form of a histogram may be included as information in which the evaluation value is converted into a histogram.

  FIG. 1 is a diagram illustrating an overall configuration of a motion vector detection device. The image signal obtained at the image signal input terminal 11 is supplied to the evaluation value table forming unit 12 to form an evaluation value table. The image signal is, for example, a digital video signal from which an individual luminance value is obtained at each pixel in each frame. The evaluation value table forming unit 12 creates an evaluation value table having the same size as the search area. An evaluation value table having the same size as the search area is a table in which motion vectors indicating which positions in the search area of the next frame have moved from the pixel position corresponding to the center position of the search area are integrated. It becomes.

The evaluation value table data created by the evaluation value table forming unit 12 is supplied to the motion vector extraction unit 13, and a plurality of motion vectors are extracted as candidate vectors from the evaluation value table. Here, a plurality of candidate vectors are extracted based on the peaks that appear in the evaluation value table. The plurality of candidate vectors extracted by the motion vector extraction unit 13 is supplied to the motion vector allocation unit 14. The motion vector assignment unit 14 determines, for each pixel on the entire screen, the correlation between the pixels associated with the candidate vectors by region matching or the like for each of the plurality of candidate vectors extracted by the candidate vector extraction unit 13. . Then, the candidate vector connecting the pixel or block having the highest correlation is set as the motion vector corresponding to the pixel. The process of obtaining these motion vectors is executed under the control of the control unit (controller) 16.
The set motion vector data is output from the motion vector output terminal 15. At this time, if necessary, the image signal obtained at the input terminal 11 may be added and output. The output motion vector data is used for high-efficiency encoding of image data, for example. Alternatively, it is used for high image quality processing when an image is displayed on a television receiver. Furthermore, the motion vector detected by the processing of this example may be used for other image processing.

[2. Overview of overall processing to detect motion vectors]
The flowchart in FIG. 2 shows an example of processing until the motion vector is determined. First, an evaluation value table, which is motion vector evaluation information, is formed from the input image signal (step S11), and a plurality of candidate vectors are extracted from the formed evaluation value table (step S12). Then, the motion vector having the highest correlation is determined from the extracted candidate vectors (step S13). The process of the flowchart of FIG. 2 is executed in each frame. Up to this point, the configuration is a general configuration as a motion vector detection configuration using the evaluation value table.

  In the present embodiment, the evaluation value table forming process in the evaluation value table forming unit 12 is configured as shown in FIG. In the example of FIG. 3, when the evaluation value table is formed, the frequency of each motion vector is determined, and the evaluation information related to the pixel serving as the target pixel of the motion vector having a high correlation with the high-frequency motion vector is integrated. Sorting and removing are excluded. Note that the target pixel is a pixel at the position of a reference point (reference point: target point) for determining a motion vector. As the target pixel, for example, all the pixels in one frame are selected in order as the target pixel. The reference pixel is a pixel at the position of a point (reference point) that may move from the target pixel. The reference pixel is a pixel in the vicinity of the pixel position of the target pixel (that is, in the search area) in another frame after or before the target pixel.

Before describing the configuration of FIG. 3, which is a feature of the present embodiment, the relationship between the target pixel and the reference pixel will be described with reference to FIG. 5.
As shown in FIG. 5, a pixel position serving as a reference for determining a motion vector in a previous frame F10 that is image data one frame before the current frame (current frame) F11 is set as a target pixel d10. When the target pixel d10 is determined, a search area SA in a predetermined range around the pixel position of the target pixel d10 is set in the current frame F11. When the search area SA is set, an evaluation value is calculated using each pixel in the search area SA as a reference pixel d11.

[3. Configuration Example of First Embodiment (Example 1)]
As shown in FIG. 5, the target pixel and the reference pixel are set, and data of the evaluation value table is generated by the configuration of FIG.
The configuration of the example of FIG. 3 will be described. The image signal obtained at the input terminal 11 is supplied to the correlation calculation unit 20 in the evaluation value table forming unit 12. The correlation calculation unit 20 includes a reference point memory 21, an attention point memory 22, and an absolute value calculation unit 23. In the image signal obtained at the input terminal 11, the pixel value of the frame used as the reference pixel is stored in the reference point memory 21. A process of moving the frame signal stored in the reference point memory to the attention point memory 22 in the next frame period is performed. In this example, the reference pixel is an example of a signal one frame before.

  Then, the pixel value of the target pixel stored in the target point memory 22 and the pixel value of the reference pixel stored in the reference point memory 21 are supplied to the absolute value calculation unit 23, and the absolute value of the difference between the two signals is obtained. To detect. The difference here is a difference in luminance value of the pixel signal. The absolute value of the detected difference is supplied to the correlation determination unit 30.

Here, in the absolute value calculation unit 23, the position of the target pixel is the pixel position among the two pixel positions used to calculate the absolute value of the difference, that is, the pixel position of the target pixel and the pixel position of the reference pixel. It is determined whether the pixel position instructed from the selection processing unit 40 is excluded from the evaluation value integration. When the pixel position instructed by the pixel selection processing unit 40 matches with the determination, output of the absolute value of the difference to the correlation determination unit 30 which is a subsequent processing unit is limited. In addition, the absolute difference values (that is, all the absolute difference values) that are not subjected to such restriction are supplied to the correlation determination unit 45 in the pixel selection processing unit 40.
The correlation determination unit 30 includes a comparison unit 31 and compares it with a set threshold value to obtain an evaluation value. As the evaluation value, for example, a binary value is assumed in which the correlation is high when the difference is equal to or less than the threshold and the correlation is low when the difference is exceeded. The evaluation value obtained by the correlation determination unit 30 is supplied to the evaluation value table calculation unit 50.

  The evaluation values supplied to the evaluation value table calculation unit 50 are integrated by the evaluation value integration unit 51 in the evaluation value table calculation unit 50, and the integration result is stored in the evaluation value table memory 52. The evaluation value stored in the evaluation value table memory 52 is an evaluation value that has been subjected to pixel selection based on an instruction from the pixel selection processing unit 40. The data stored in the evaluation value table memory 52 thus obtained is supplied as evaluation value table data from the output terminal 12a to the subsequent circuit (motion vector extraction unit 13 in FIG. 1).

Next, the configuration of the pixel selection processing unit 40 will be described. The pixel selection processing unit 40 supplies all (that is, not selected) difference absolute values output from the absolute value calculation unit 23 to the correlation determination unit 45, and compares the difference absolute value with a threshold value. The processing in the correlation determination unit 45 is the same as the processing in the correlation determination unit 30. The determination result in the correlation determination unit 45 is supplied to the high-frequency motion calculation unit 41. Based on the correlation determination result, a motion having a high frequency is calculated among candidate vectors detected in one frame. For example, the motion vector having the highest frequency is calculated as the motion having the highest frequency. Alternatively, instead of creating an evaluation value table, it is also possible to have a counter for each vector position and calculate up to the vector with the highest count value.
In the present embodiment, the motion vector having the highest frequency is described as being extracted. However, a high-frequency motion vector having a predetermined order such as a motion vector having the highest frequency may be extracted. Processing for extracting a plurality of motion vectors such as the second one will be described later.

The motion vector data having a high frequency calculated by the high-frequency motion calculation unit 41 is supplied to the high-frequency motion determination unit 42. The high-frequency motion determination unit 42 looks at the reference pixel data stored in the reference point memory 21 and the target pixel data stored in the target point memory 22, and the motion vector that is a candidate when viewed from the target pixel is It is determined whether or not it is the same as the high-frequency motion vector. Here, the same means that the distance and direction connecting the target pixel and the reference pixel of the candidate motion vector are highly correlated with the distance and direction connecting the start point and end point of the most frequent motion vector.
If the candidate motion vector is the same as the high-frequency motion vector, the pixel position within one frame of the target pixel at that time is instructed to the selected target point memory 43, and the corresponding pixel position is integrated with the evaluation value. Remember to be excluded from. Therefore, when the processing for all the target pixels in one frame is completed, the selected target point memory 43 stores the pixel positions of the target pixel in one frame of the target pixel having a motion vector highly correlated with the high-frequency motion vector. Information is stored. In FIG. 3, the sorting reference point memory 44 is shown. An example using the sorting reference point memory 44 will be described later.
The information stored in the selected attention point memory 43 is supplied to the absolute value calculation unit 23 and used for selecting the output of the difference absolute value.

Note that the frame between which the high-frequency motion calculation unit 41 calculates the high-frequency motion includes a frame in which the pixel of interest currently being processed exists from the frame one frame before the frame where the pixel of interest is currently set (the previous frame). Frame). That is, the frame between which the high-frequency motion is calculated is a frame one frame before the frame in which the motion vector candidate is being detected. Then, the calculation result of the high-frequency motion calculation unit 41 is supplied to the high-frequency motion determination unit 42 when the next frame is processed.
In this way, it is easy to detect a high-frequency motion vector one frame before in the configuration of FIG. 3, but there is a reference pixel from a frame (previous frame) in which the pixel of interest currently being processed exists. Motion to the frame (current frame) may be detected. In the case of the movement from the previous frame to the current frame, a memory (not shown) for temporary storage for adjusting the timing by the correlation calculation unit 20 or the like is required.

  From the output terminal 12a, the high-frequency motion vector data calculated by the high-frequency motion calculation unit 41 is also output and supplied to the motion vector extraction unit 13. The motion vector extraction unit 13 extracts candidate vectors using the high-frequency motion vector calculated by the high-frequency motion calculation unit 41 and the motion vector evaluation information indicated by the evaluation value table data. An example of processing in the motion vector extraction unit 13 will be described later.

[4. Processing Example of First Embodiment (Example 1)]
The flowchart of FIG. 4 shows the processing operation in the configuration of FIG.
The flowchart in FIG. 4 shows an example of the process up to the determination of whether or not to add to the evaluation value table in order, and does not necessarily match the signal flow in the configuration in FIG. Absent. If each process shown in the flowchart of FIG. 4 is performed, each process may be executed in a different order from the flowchart of FIG. The same applies to other flowcharts described later in that they may be processed in different orders.

  First, the high-frequency motion calculation unit 41 calculates a high-frequency motion vector (step S21). As the high-frequency motion vector calculated here, there are a case where there is only one most frequent motion as described above, and a case where the frequency vector is predetermined in order of frequency. In the flowchart of FIG. 3, high frequency motions up to the nth (n is a positive integer) are obtained and shown, and when n = 1, the most frequent motion is used, and n = 2 or more values such as n = 2. At some point, we will use the high-frequency movement of the rank up to that value. Further, this high-frequency motion vector is detected between frames in which high-frequency motion is calculated, between frames one frame before the frame where the motion vector candidate is being detected.

  When the high-frequency motion vector is calculated in step S21, the high-frequency motion determination unit 42 determines whether or not the current target pixel is related to the n-th high-frequency motion, and uses the selected target point memory 43. Then, pixel selection is performed (step S22). The relation to the high-frequency motion here is performed by determining whether or not the motion vector that is a candidate when viewed from the point of interest is the same as the vector of the high-frequency motion. If it is the same as the vector of the high-frequency motion, all the evaluation values related to the target pixel, that is, the evaluation values of the motion vector starting from the target pixel are excluded from the integration of the evaluation values.

  Then, after the selection process based on the high-frequency motion, the comparison unit 31 determines whether or not there is a correlation between the pixel of interest and the reference pixel (step S23). The determination of whether or not there is a correlation between the target pixel and the reference pixel is performed by determining whether or not the difference between the two pixels is equal to or less than a threshold value by the comparison unit 31.

If it is determined in step S23 that the difference between the two pixels is equal to or smaller than the threshold value, the count value for the motion vector to the reference pixel viewed from the target image at that time is sent to the evaluation value table integrating unit 51, and the evaluation value The table is integrated (step S24).
If it is determined in step S23 that there is no correlation between the target pixel and the reference pixel, addition of evaluation values of the target pixel and the reference pixel to the evaluation value table is prohibited (step S25).
The process of the flowchart of FIG. 4 is performed for all the target pixels in one frame.

[5. Explanation of relationship between pixel of interest and high-frequency movement]
Here, an example of a state related to the high-frequency motion and the target pixel will be described with reference to FIG.
FIG. 6 illustrates the previous frame F10 at the timing (t) at which the target pixel exists, the current frame F11 at the timing (t + 1) at which the reference pixel is set, and the timing (t-1) one frame before the previous frame F10. Three frames of the frame F9 are shown. In this example, detection of high-frequency motion (most frequent motion) is performed one frame before.

  It is assumed that the high-frequency motion between the frame F9 before the timing (t−1) and the previous frame F10 at the timing (t) is the motion vector m ′. At this time, it is assumed that the motion vector m determined by the target pixel of the previous frame F10 at the timing (t) and the reference pixel of the current frame F11 is highly likely to be a motion destination candidate viewed from the target pixel. The motion vector m ′ of the high-frequency motion and the candidate vector m from the target pixel d10 are the same motion vector. Here, the same is the case where the motion direction of the motion vector is exactly the same as the number of pixels moved in the horizontal direction and the number of pixels moved in the vertical direction.

  In such a case, a selective removal process is performed to prohibit the integration of all evaluation values related to the target pixel, that is, the evaluation values of all vectors starting from the target pixel of interest into the evaluation value table. When a vector that is highly likely to be a motion destination candidate viewed from the target pixel is a vector different from the high-frequency motion vector m ′, the evaluation value related to the target pixel is integrated into the evaluation value table. Do not limit.

[6. Example of evaluation value table]
Next, an example of an actual image motion state and an evaluation value table obtained based on the motion state will be described.
FIG. 7 is an example of an image of three frames: a frame F9 before the timing (t−1), a previous frame F10 at the timing (t), and a current frame F11 at the timing (t + 1).
In this example, it is assumed that in any of the three frames, there is a movement Ma in which the entire background in the screen is lowered downward at a substantially constant speed. In this state, it is assumed that the small object X in the screen moves one frame at a time with an obliquely downward movement m1 different from the movement Ma.

The evaluation value table without selection obtained from the correlation between the target pixel on the previous frame F10 and the reference pixel on the current frame F11 in the image having the movement shown in FIG. 7 is in the state shown in FIG. . That is, the evaluation value table shown in FIG. 8 is an evaluation value table that is not subjected to the selective removal process of the present embodiment, and corresponds to the evaluation value table obtained in the conventional example.
This evaluation value table shows an integrated value of evaluation values at the position of each motion vector in two dimensions in the horizontal direction Vx and the vertical direction Vy from the central coordinate position (0, 0). The vertical axis is the integrated value of evaluation values. In other words, the frequency of 100,000 means that the corresponding motion vector becomes a candidate, and all pixels in one frame are judged as pixels of interest, and have been 100,000 times. As can be seen from FIG. 8, in the case of the evaluation value table without the selection and removal process, there is a large peak in the position of the motion vector corresponding to the background motion, and other motions in the image cannot be determined from the evaluation value table. It is in a state.

FIG. 9 is an example in which an evaluation value table is created by performing the processing of the present embodiment.
FIG. 9A is an evaluation value table based on the high-frequency motion calculated by the high-frequency motion calculation unit 41. This evaluation value table may be obtained either between the previous frame F10 and the current frame F11 shown in FIG. 7 or between the previous frame F9 and the previous frame F10. In the case of continuous moving images, there is no significant difference between the high-frequency motions in any case.
The evaluation value table calculated by the high-frequency motion calculation unit 41 shown in FIG. 9A is the same as the evaluation value table obtained by the conventional process without selection shown in FIG.
Based on the evaluation value table integration unit in FIG. 9A, a high-frequency motion (most frequent motion) is detected. In this example, one downward background movement is detected as a high-frequency movement.

  The evaluation value table obtained by the evaluation value table calculation unit 52 by performing the process of selecting and removing the evaluation value related to the target pixel correlated with the motion vector of the high-frequency motion detected as described above is shown in FIG. It is shown in b). 9B is different from the evaluation value table of FIG. 9A in the scale of the integrated value on the vertical axis, and the frequency is about 1000 at the peak position.

In the evaluation value table of FIG. 9B, the movement of the small object X shown in FIG. 7 appears as one of the peaks. Therefore, a motion vector corresponding to the motion of the small object X can be sent as a candidate vector to the subsequent processing unit from the peak of the integrated value.
In FIG. 9B, the peak position of the integrated value also exists at a position different from the movement position of the small object X, but this other peak position is a movement corresponding to the background movement.

  As described above, according to the present embodiment, even when there is a movement of a large area in the screen such as a background movement, a small movement in the screen can be detected. Therefore, when determining the motion vector to be assigned to each pixel from the candidate vectors in the subsequent processing unit, the vector assigned to the small motion object is likely to be a correct motion vector, and accurate motion vector detection is possible. It becomes. In the example of FIG. 9B, an example of a small object that moves differently from the background is shown. However, even when there are a plurality of small objects, the movement of the plurality of small objects is improved similarly. Can be detected.

[7. Configuration Example of First Embodiment (Example 2)]
In the processing configurations of FIGS. 3 and 4, the evaluation value is selectively removed by limiting the output to the subsequent processing unit of the absolute difference value obtained by the correlation calculation unit 20. On the other hand, an example will be described in which a gate unit for limiting the output of the correlation determination unit 30 is provided, and evaluation values are selectively removed by the gate unit.
FIG. 10 is a diagram showing a configuration example in that case. In FIG. 10, the same parts as those in FIG.

  Referring to the configuration of FIG. 10, the evaluation value determined by the comparison unit 31 in the correlation determination unit 30 as compared with the threshold value is used to supply the temporary storage memory 61 in the image selection processing unit 60. Accumulate once. All evaluation values in one frame are accumulated in the temporary storage memory 61 and then read out and supplied to the gate unit 62. The gate unit 62 restricts the output when the evaluation value of the pixel is related to the target pixel related to the high-frequency motion determined by the high-frequency motion determination unit 64. When the evaluation value is not related to the pixel of interest related to the high-frequency motion, the evaluation value integration unit 51 of the evaluation value table calculation unit 50 passes through the gate unit 62 as it is, and the integration result is evaluated. It is stored in the table memory 52.

  In addition, the evaluation value output from the comparison unit 31 is supplied to the high-frequency motion calculation unit 63 in the image selection processing unit 60. The high-frequency motion calculation unit 63 integrates the evaluation values within one frame, and calculates a motion with a high frequency among candidate vectors detected within one frame. As the motion with high frequency here, the motion vector with the highest frequency is calculated. Alternatively, high-frequency motion vectors up to a predetermined order, such as motion vectors having the highest frequency up to the second, may be extracted.

The motion vector data having a high frequency calculated by the high-frequency motion calculation unit 63 is supplied to the high-frequency motion determination unit 64. The high-frequency motion determination unit 64 looks at the reference pixel data stored in the reference point memory 21 and the target pixel data stored in the target point memory 22, and the motion vector that is a candidate when viewed from the target point is It is determined whether or not it is the same as the high-frequency motion vector.
If the candidate motion vector is the same as the high-frequency motion vector, the gate unit 62 restricts passage of the evaluation value related to the target pixel at that time. There is a temporary storage memory 61 in front of the gate unit 62, and the output of the temporary storage memory 61 is supplied to the gate unit 62 after the determination by the high-frequency motion determination unit 64 is completed. The gate unit 62 does not output the evaluation value related to the target pixel that needs to be limited, and supplies the evaluation value integration unit 51 with the evaluation value that does not need to be limited. In order to determine whether the evaluation value needs to be restricted, a flag is set in the present embodiment, as will be described later with reference to the flowchart of FIG. The processing after the evaluation value integrating unit 51 is the same as the example of FIG.

[8. Processing Example of First Embodiment (Example 2 in Considering Pixel of Interest and High Frequency Motion)]
Next, a processing example with the configuration of FIG. 10 will be described with reference to the flowchart of FIG.
First, the high-frequency motion calculation unit 63 calculates a high-frequency motion vector (step S51). As the high-frequency motion vector calculated here, there are a case where there is only one most frequent motion as described above, and a case where the frequency vector is predetermined in order of frequency. In the flowchart of FIG. 11, high frequency motions up to the nth (n is a positive integer) are obtained and shown, and when n = 1, the most frequent motion is used, and n = 2 or more values such as n = 2. At some point, we will use the high-frequency movement of the rank up to that value.

  When the high-frequency motion vector is calculated in step S51, the position of the target pixel is set in the currently processed frame (step S52), and the flag is set to 0 (step S53). Thereafter, a reference pixel is set in the search area SA (see FIG. 5) centering on the pixel of interest (step S54). Then, it is determined whether or not the target pixel is at the same pixel position as the target pixel of the motion vector related to the n-th high-frequency motion (step S55). (Step S56).

Then, it is determined whether or not there is a correlation between the target pixel and the reference pixel (step S57). If there is a correlation, that is, if the pixel value is equal to or less than the threshold value, addition to the temporary storage memory 61 is performed ( Step S58). When the pixel value exceeds the threshold value, it is not stored in the temporary storage memory 61.
Thereafter, it is determined whether or not the processing for all the reference pixels in the search area SA has been completed (step S59), and if the processing for all the reference pixels has not been completed, the process returns to step S54 to search for the search area SA. The reference pixel is set at another position in the and the same process is repeated.
If it is determined in step S59 that all the reference pixels have been processed, the data stored in the temporary storage memory 61 is supplied to the gate unit 62, and the passage is restricted based on the flag. That is, it is determined whether or not the flag is 1 (step S60). When the flag is 1, the passage is restricted, and when the flag is 0, the flag is stored in the evaluation value table memory 52 ( Step S61).

Thereafter, it is determined whether or not the processing for all the target pixels of one frame has been completed (step S62). If not, the process returns to the setting of the target pixel position in step S52, and the newly set The process for the pixel of interest is repeated. When the process for all the target pixels in one frame is completed, the process in this frame is terminated and the process proceeds to the next frame.
As shown in the flowchart of FIG. 11, even when the processing configuration is such that the processing is performed in order by setting the flag, as in the case of the processing configurations of FIGS. 3 and 4, the pixel of interest related to the high-frequency motion is limited. Can be done.

[9. Other processing example of first embodiment]
In the processing example of the flowchart in FIG. 4, the evaluation value related to the pixel of interest correlated with the detected high-frequency motion is selected and removed. However, the evaluation value regarding the reference pixel correlated with the detected high-frequency motion is also selected. It may be removed by sorting.
The process of selecting and removing the evaluation value related to the reference pixel having a correlation with the high-frequency motion for the reference pixel is calculated by the high-frequency motion calculating unit 41 in the pixel selecting unit 40 shown in FIG. Based on the result, the high-frequency motion determination unit 42 performs the determination. Based on the determination, information on the pixel position of the corresponding reference pixel is instructed to the selection reference point memory 44 as shown in FIG. 3, and the corresponding pixel position is excluded from the evaluation value integration. Remember. The information stored in the selection reference point memory 44 is used to limit the output of the absolute value of the difference from the absolute value calculation unit 23 to the correlation determination unit 30. The position information of the target pixel stored in the selected target point memory 43 is also used in the same manner as in the example of FIG. 3 and used to limit the output of the absolute value of the difference from the absolute value calculation unit 23 to the correlation determination unit 30. Is done.
The flowchart in FIG. 12 shows a processing example in this case. Also in the flowchart of FIG. 12, the processes in the flowchart may be executed in a different order from the flowchart of FIG.

  First, the high-frequency motion calculation unit 41 calculates a high-frequency motion vector (step S31). As the high-frequency motion vector calculated here, there are a case where there is only one most frequent motion and a case where it is a predetermined number in the descending order of frequency. In the flowchart of FIG. 12, high frequency motions up to the nth (n is a positive integer) are obtained and shown, and when n = 1, the most frequent motion is used, and a value of 2 or more such as n = 2. At some point, we will use the high-frequency movement of the rank up to that value.

When the high-frequency motion vector is calculated in step S31, the high-frequency motion determination unit 42 determines whether or not the current pixel of interest is related to the n-th high-frequency motion, and uses the selected target point memory 43. Then, pixel selection is performed (step S32). This pixel selection is the same as step S22 in the flowchart of FIG. 4 described above.
Next, the high-frequency motion determination unit 42 determines whether or not the current reference pixel is related to the n-th high-frequency motion, and performs pixel selection using the selection reference point memory 44 (step S33). ).

  Then, after the selection process based on the high-frequency movement, the comparison unit 31 determines whether or not there is a correlation between the target pixel and the reference pixel (step S34). The determination of whether or not there is a correlation between the target pixel and the reference pixel is performed by determining whether or not the difference between the two pixels is equal to or less than a threshold value by the comparison unit 31.

If it is determined in step S34 that the difference between the two pixels is equal to or smaller than the threshold value, the count value for the motion vector to the reference pixel viewed from the target image at that time is sent to the evaluation value table integrating unit 51, and the evaluation value The table is integrated (step S35).
If it is determined in step S34 that there is no correlation between the target pixel and the reference pixel, addition of evaluation values of the target pixel and the reference pixel to the evaluation value table is prohibited (step S36).
The process of the flowchart of FIG. 4 is performed for all the target pixels in one frame.

[10. Explanation of relationship between reference pixel and high-frequency motion]
Here, an example of a state related to the high-frequency motion and the reference pixel will be described with reference to FIG.
FIG. 13 shows two frames, the current frame F11 at the timing (t + 1) when the reference pixel is set and the previous frame F10 at the timing (t).

Here, for the reference pixel set in the current frame F11, the target pixel of the motion vector of the motion vector for the reference pixel is determined. Then, when a motion in the opposite direction to the motion of viewing the target pixel from the reference pixel is a motion vector candidate, it is determined whether or not the motion vector matches the high-frequency motion in the previous frame F10. When this determination is made in step S33 of the flowchart of FIG. 12 and coincides with the high-frequency motion, the integration of all evaluation values into the evaluation value table with the pixel position of the corresponding reference pixel as the reference pixel is limited. The
In the case of the example of the flowchart of FIG. 12, in addition to the selective removal of the evaluation values related to the reference pixels related to the high-frequency motion shown in FIG. 13, the processing related to the high-frequency motion already shown in FIG. The evaluation value relating to the target pixel is also selectively removed.

[11. Example of evaluation value table in another processing example]
FIG. 14 is an example of an evaluation value table obtained by executing the processing shown in the flowchart of FIG. The evaluation value table in this case is also an example in the case where there is a movement of the small object X different from the background movement that moves downward as shown in FIG.
As can be seen from the comparison of the evaluation value table of FIG. 14 with the evaluation value table of FIG. 9B, not only the target pixel but also the reference pixel related to the high-frequency motion is selected and removed. This is an evaluation value table that effectively removes background motion. Therefore, by performing the processing of the flowchart of FIG. 12, compared with the example of the flowchart of FIG. 4, there is an effect that the movement of the small object can be detected from the evaluation information better. This also has an effect that, for example, when there are a plurality of small objects on the screen, the movement of each small object can be detected better.

[12. Modification of First Embodiment]
In the example of the flowchart in FIG. 12, the high-frequency motion is considered in both the target pixel and the reference pixel. However, only the reference pixel may have a processing configuration for selectively removing evaluation values related to the high-frequency motion. . That is, the relationship between the reference pixel and the high-frequency motion may be determined in step S33 after the high-frequency motion is detected in step S31 without performing the processing in step S32 in the flowchart of FIG.
Even in the case of limiting only by judgment on the reference pixel side, there is an effect that the movement of a small object in the screen can be detected well.

As described in the description of the flowchart of FIG. 4 and the like, the high-frequency motion to be determined at the time of pixel selection is a process of detecting only the most frequent motion with the highest motion in one frame, and the frequency is Any of the cases in which processing is performed in order from the highest to the predetermined one may be used.
In the example of the evaluation value table shown in FIG. 9 or FIG. 14, only the most frequent movement is detected and the evaluation value is selectively removed. However, depending on the content of the image, a plurality of high-frequency movements may be displayed. In some cases, it is better to detect and sort out.
For example, high-frequency motion from the highest frequency to the second or third, such as the third, is detected and related to the pixel (target pixel and / or reference pixel) correlated with the detected high-frequency motion. The evaluation value may be selected and removed.
It is good also as a fixed value beforehand how many high frequency motions are detected and removed (that is, the value of n in the flowchart of FIG. 4). Alternatively, for example, as shown in FIG. 9A, all the frequencies of the high frequency motion detected in the evaluation value table may be determined to be the high frequency motion.
Also, if the frequency of the most frequently detected motion between frames is below a certain threshold and it is determined that there is no motion in a large area within one screen, the high frequency motion described so far The sorting and removal based on the above may not be executed.
Further, these conditions may be changed depending on the contents of the image to be processed (for example, whether the image is intensely moving).

[13. Configuration example of second embodiment]
Next, a second embodiment of the present invention will be described with reference to FIGS.
Also in the present embodiment, a motion vector detection device that detects a motion vector from moving image data is formed. An evaluation value table is formed from pixel value correlation information, and a motion vector is determined from the data of the evaluation value table. This is the same as in the first embodiment.
The overall configuration and overall processing of the motion vector detection device are the same as the processing configuration of FIG. 1 and the flowchart of FIG. 2 described in the first embodiment. The definition of the target pixel (target point) and the reference pixel (reference point) is the same as the definition described in the first embodiment.

In the present embodiment, the evaluation value table forming unit 12 of the motion vector detecting device shown in FIG. 1 is configured as shown in FIG. In the evaluation value table forming unit 12 of FIG. 15, the same parts as those of the evaluation value table forming unit 12 of FIG. 3 described in the first embodiment are denoted by the same reference numerals.
In the example of the present embodiment illustrated in FIG. 14, as the evaluation value table forming process in the evaluation value table forming unit 12, in addition to performing selective removal of evaluation values corresponding to high-frequency motion, the target pixel and the reference pixel The evaluation value accumulation limitation using the spatial inclination pattern is performed.

  15 will be described. The correlation calculation unit 20 and the correlation determination unit 30 are the same as those shown in FIG. That is, the pixel value of the frame used as the reference pixel in the image signal obtained at the input terminal 11 is stored in the reference point memory 21. A process of moving the frame signal stored in the reference point memory to the attention point memory 22 in the next frame period is performed.

Then, the pixel value of the target pixel stored in the target point memory 22 and the pixel value of the reference pixel stored in the reference point memory 21 are supplied to the absolute value calculation unit 23, and the absolute value of the difference between the two signals is obtained. To detect. The difference here is a difference in luminance value of the pixel signal. The detected absolute value difference data is supplied to the correlation determination unit 30.
Here, in the absolute value calculation unit 23, the two pixel positions used for calculating the absolute value of the difference, that is, the pixel position of the target pixel and the pixel position of the reference pixel are determined by the selection target point memory of the pixel selection processing unit 70. 73 and the pixel position designated by the selection reference point memory 74 is determined. The pixel position instructed from the selection reference point memory 74 here is a pixel position excluded from the evaluation value integration. If it is determined that the pixel position is excluded from the evaluation value integration instructed from the pixel selection processing unit 70, the output of the absolute value of the difference to the correlation determination unit 30 which is a subsequent processing unit is limited. . Further, the absolute difference value (that is, all the absolute difference values) without such restriction is supplied to the correlation determination unit 76 in the pixel selection processing unit 70.
In the case of the example of the present embodiment, the selected attention point memory 73 matches the specified spatial inclination pattern in addition to the data of the position of the attention pixel for pixel selection described in the example of FIG. Data indicating that the target pixel has been selected is also stored. Similarly, in the sorting reference point memory 74, in addition to data on the position of the target pixel for pixel selection, data indicating that the target pixel matches the designated spatial inclination pattern is also stored. The designated spatial inclination pattern will be described later.

  The correlation determination unit 30 includes a comparison unit 31 and compares it with a set threshold value to obtain an evaluation value. As the evaluation value, for example, a binary value is assumed in which the correlation is high when the difference is equal to or less than the threshold and the correlation is low when the difference is exceeded. The evaluation value obtained by the correlation determination unit 30 is supplied to the evaluation value table calculation unit 50.

  The evaluation values supplied to the evaluation value table calculation unit 50 are integrated by the evaluation value integration unit 51 in the evaluation value table calculation unit 50, and the integration result is stored in the evaluation value table memory 52. The evaluation value stored in the evaluation value table memory 52 is an evaluation value that has been subjected to pixel selection based on an instruction from the pixel selection processing unit 40. The data stored in the evaluation value table memory 52 thus obtained is supplied as evaluation value table data from the output terminal 12a to the subsequent circuit (motion vector extraction unit 13 in FIG. 1).

  Next, the configuration of the pixel selection processing unit 70 will be described. The pixel selection processing unit 70 supplies all (that is, not selected) difference absolute values output from the absolute value calculation unit 23 to the correlation determination unit 76, and compares the difference absolute values with a threshold value. The process in the correlation determination unit 76 is the same as the process in the correlation determination unit 30. The determination result in the correlation determination unit 76 is supplied to the high-frequency motion calculation unit 71. Based on the correlation determination result, a motion having a high frequency is calculated among candidate vectors detected in one frame. For example, the motion vector having the highest frequency is calculated as the motion having the highest frequency.

Data of a motion vector having a high frequency calculated by the high-frequency motion calculation unit 71 is supplied to the high-frequency motion determination unit 72. The high-frequency motion determination unit 72 looks at the reference pixel data stored in the reference point memory 21 and the target pixel data stored in the target point memory 22, and the motion vector that is a candidate when viewed from the target pixel is It is determined whether or not it is the same as the high-frequency motion vector.
If the candidate motion vector is the same as the high-frequency motion vector, the pixel position within one frame of the target pixel at that time is instructed to the selected target point memory 73, and the corresponding pixel position is integrated with the evaluation value. Remember to be excluded from.
When the candidate motion vector is the same as the high-frequency motion vector, the pixel position within one frame of the reference pixel at that time is instructed to the selection reference point memory 74, and the corresponding pixel position is evaluated. Remember that it is excluded from integration.

In addition, the pixel selection processing unit 70 includes a spatial inclination pattern determination unit 75.
The spatial inclination pattern determination unit 75 calculates a spatial inclination pattern of the reference pixel and the surrounding eight pixels from the data obtained in the reference point memory 21. Details of the spatial inclination state will be described later. The calculated spatial inclination pattern is compared with a designated spatial inclination pattern stored in advance, and it is determined whether or not it matches the designated pattern. A specific example of the designated pattern and details of comparison of the pattern will be described later.

  When the coincidence between the spatial gradient pattern designated by the spatial gradient pattern determination unit 75 and the spatial gradient pattern between the target pixel and its surroundings (or the spatial gradient pattern between the reference pixel and its surroundings) is detected, This is stored in the selected attention point memory 73. Further, when a coincidence between the spatial gradient pattern designated by the spatial gradient pattern determination unit 75 and the spatial gradient pattern between the reference pixel and its surroundings is detected, this is stored in the selected reference point memory 74.

  In the case of this embodiment, two types of pixel selection based on high-frequency motion and restriction based on a spatial gradient pattern are performed. Therefore, in the present embodiment, the evaluation values to be integrated are narrowed down under more severe conditions than in the case of the first embodiment.

[14. Processing Example of Second Embodiment]
Next, the processing state in the configuration of FIG. 15 will be described with reference to the flowchart of FIG.
Also in the flowchart of FIG. 16, the processes in the flowchart may be executed in a different order from the flowchart of FIG.

  First, the high-frequency motion calculation unit 71 calculates a high-frequency motion vector (step S41). The high-frequency motion vector calculated here may be either the case of only the most frequent motion or the predetermined frequency in the descending order of frequency. In the flowchart of FIG. 16, the high-frequency motion up to the nth (n is a positive integer) is obtained and shown, and when n = 1, the most frequent motion is used, and a value of 2 or more such as n = 2. At some point, we will use the high-frequency movement of the rank up to that value.

  When the high-frequency motion vector is calculated in step S41, pixel selection is performed based on information regarding whether each of the target pixel and the reference pixel is related to the n-th high-frequency motion (step S42). This step S42 collectively shows the processing of step S32 and the processing of step S33 in the flowchart of FIG.

  Next, each spatial inclination pattern is calculated from the adjacent difference between the target pixel and the reference pixel (step S43). Then, it is determined whether or not the calculated spatial inclination pattern on the target pixel side and the spatial inclination pattern on the reference pixel side match a predetermined spatial inclination pattern determined in advance (step S44). If the spatial inclination patterns match, the process proceeds to step S45. If the spatial inclination patterns do not match, the process proceeds to step S47.

  In step S45, it is determined whether or not there is a correlation between the target pixel and the reference pixel. The determination of whether or not there is a correlation between the target pixel and the reference pixel is performed by determining whether or not the difference between the two pixels is equal to or less than a threshold value by the comparison unit 31.

If it is determined in step S46 that the difference between the two pixels is equal to or less than the threshold value, the evaluation value at that time is sent to the evaluation value table integration unit 51 and integrated in the evaluation value table (step S46).
If it is determined in step S44 that it does not match the predetermined spatial inclination pattern, or if it is determined in step S45 that there is no correlation between the target pixel and the reference pixel, the corresponding evaluation value is supplied to the evaluation value table calculation unit 50. Not added to the evaluation value table.
The process of the flowchart of FIG. 16 is performed for all target pixels and reference pixels in one frame.

[15. Explanation of spatial inclination pattern]
Next, the spatial inclination pattern of the target pixel and the reference pixel applied in the present embodiment will be described with reference to FIGS.
First, referring to FIG. 17, the peripheral pixel adjacent to the target pixel and the reference pixel is shown. As shown in FIG. 17A, when a pixel of interest d10 is set in the previous frame F10 and a certain reference pixel d11 in the search area SA of the next current frame F11 is viewed from the pixel of interest d10, each pixel In d10 and d11, there are 8 pixels as adjacent pixels.
FIG. 17B shows the peripheral pixels adjacent to the target pixel and the reference pixel.
Here, the spatial inclination pattern of the pixel of interest refers to the difference between the pixel of interest and the neighboring eight neighboring pixels, and the difference is within a certain range and exceeds the certain range in the − direction or the + direction. Divide into cases. The difference here is, for example, a difference in luminance value of the corresponding pixel. When the difference exceeds a certain range in the − direction or the + direction, the sign − or + has a spatial inclination, and when the difference is within the certain range, the spatial inclination state is 0 (that is, no spatial inclination). . A spatial inclination pattern is obtained by patterning the presence or absence of the spatial inclination with the surrounding eight adjacent pixels.

18 to 21 show examples of spatial inclination patterns.
18 is a spatial gradient pattern defined as mode 1, FIG. 19 is a sky 9 gradient pattern defined as mode 2, FIG. 20 is a spatial gradient pattern defined as mode 3, and FIG. 13 is defined as mode 4. It is a spatial inclination pattern.

The mode 1 pattern shown in FIG. 18 is a 9 pixel pattern of 8 adjacent pixels and 1 central pixel, as shown in the upper left, and the relationship between the pixel of interest or reference pixel in the center and the adjacent 8 pixels. Are all other than 0 (i.e., the sign + or-). FIG. 18 corresponds to the case where the target pixel and the reference pixel all have different luminance values from the adjacent pixels.
When this mode 1 is set at the time of comparison in step S44 of the flowchart of FIG. 16, if even one of the surrounding 8 pixels has no spatial inclination, the evaluation value is prevented from passing through the gate unit 61. , Excluded from evaluation value accumulation.

The mode 2 pattern shown in FIG. 19 compares the spatial inclination codes of the target pixel and the reference pixel when there is a spatial inclination in at least 7 pixels among the surrounding 8 pixels. Only when the coincidence is detected, the evaluation value when the reference pixel is viewed from the corresponding target pixel is passed through the gate unit 61. Then, the passed evaluation value is integrated in the evaluation value table.
When the mode 2 pattern is used, it is not necessary to compare the spatial inclination code with the surrounding 8 pixels, and it is only necessary to compare the 7 pixels determined to have the spatial inclination. Of course, it is good also as comparing a code | symbol about eight surrounding pixels.

In the mode 3 pattern shown in FIG. 20, in addition to the conditions of mode 1 and mode 2, a region of 4 × 2 × 2 pixels including the target pixel or reference pixel among the surrounding 8 pixels has no spatial inclination. In addition to the candidates, the spatial gradient codes of the target pixel and the reference pixel are compared. Then, only when a match is detected in the comparison, the evaluation value when the reference pixel is viewed from the corresponding target pixel is passed through the gate unit 61 and integrated.
Even in this case, it is only necessary to compare the signs only in the adjacent pixel direction where there is a spatial inclination.

In the pattern of mode 4 shown in FIG. 21, in addition to the conditions of modes 1 to 3, there are three continuous pixels in the horizontal, vertical, and diagonal directions including the target pixel or reference pixel among the surrounding eight pixels, and there is no spatial inclination. Exclude it. In this way, the spatial gradient codes of the target pixel and the reference pixel are compared. Only when a match is detected by the comparison, the evaluation values when the reference pixel is viewed from the corresponding target pixel are integrated.
Even in this case, it is only necessary to compare the signs only in the adjacent pixel direction where there is a spatial inclination.

  At the time of the processing in step S44 of the flowchart of FIG. 16 described above, one of mode 1, mode 2, mode 3, and mode 4 is set. By setting mode 1, the strictest selection is performed, and the selection criteria are relaxed in order of mode 2, mode 3, and mode 4.

In this way, as described in the present embodiment, the limitation of the integration to the evaluation value table based on the determination of the high-frequency motion and the limitation of the integration to the evaluation value table based on the spatial inclination pattern are performed together. As a result, a better evaluation value table can be obtained. In other words, the integrated value of the evaluation value table becomes a more integrated integrated value than in the case of the first embodiment, and detection is performed when there is a relatively small movement different from the large area movement in the image such as the background. Can be detected better.
In addition, as the limitation of the integration of the evaluation value table based on the spatial tilt, the processing using the spatial tilt pattern described so far is an example, and other limiting processing using the spatial tilt may be performed. For example, the spatial inclination codes themselves obtained for obtaining the spatial inclination patterns may be compared, and the restriction may be performed according to the coincidence or mismatch between the target pixel side and the reference pixel side of the code. In that case, the spatial gradient codes may be compared only in a limited direction such as a moving direction.

[16. Example of motion vector extraction unit]
Next, an example of the configuration and operation of the motion vector extraction unit 13 in the motion vector detection device shown in the configuration of FIG. 1 will be described with reference to FIGS.
FIG. 22 shows an example of the motion vector extraction unit 13 of FIG.
In the motion vector extraction unit 13, the evaluation value table data and the high-frequency motion data are supplied to the input terminal 13a. This evaluation value table data is, for example, motion vector evaluation value table data obtained by any of the processing configurations of the first to second embodiments already described. This is the data obtained by integrating the motion vectors that may become.
For example, the data is supplied from the evaluation value table memory 52 in the evaluation value table calculation unit 50 in FIG. 3 and is supplied to the evaluation value table data conversion unit 111.

In the evaluation value table data conversion unit 111, a vector group obtained by adding the high-frequency motion used when forming the evaluation value table to the candidate vector indicated by the supplied evaluation value table data is set as the final candidate vector. The candidate vector is converted into data such as a frequency value or a differential value thereof. The converted data is processed by the frequency order sort processing unit 112 to rearrange candidate vectors in order of frequency within one frame. The evaluation value table data of candidate vectors rearranged in order of frequency is supplied to the candidate vector evaluation unit 113. Here, candidate vectors up to a predetermined order among the candidate vectors rearranged in the order of frequency are supplied to the candidate vector evaluation unit 113.
For example, from the most frequently used candidate vectors existing in one frame, the tenth most frequently used candidate vector is extracted and supplied to the candidate vector evaluating unit 113.

  The candidate vector evaluation unit 113 performs a process of evaluating each of the supplied candidate vectors having a high frequency according to a predetermined condition. For example, even if the frequency value is a candidate vector within a predetermined high order, the evaluation process is determined by excluding candidate vectors that are not so high because the frequency value is not more than a predetermined threshold. Evaluate on condition.

In this way, based on the evaluation result of each candidate vector obtained by the candidate vector evaluation unit 113, the candidate vector reliability determination unit 114 selects a candidate vector having high reliability among the candidate vectors, The candidate vector data with high reliability is output from the output terminal 13b.
The reliability data of the candidate vector output from the output terminal 13b is supplied to the motion vector determination unit 14 in FIG.

FIG. 23 is a flowchart showing an example of processing operation when the motion vector extraction unit 13 shown in FIG. 22 extracts candidate vectors from the evaluation value table data.
First, a process of sorting the candidate vectors indicated by the evaluation value table data in order of frequency is performed (step S111). From the evaluation value table rearranged in the order of the frequencies, up to a predetermined number of candidate vectors are extracted in order from the highest frequency. As this predetermined number, for example, the one with the determined order such as the tenth is extracted from the highest frequency (step S112).
Thereafter, it is evaluated whether or not the plurality of extracted candidate vectors are appropriate as candidate vectors, and the candidate vectors are narrowed down as necessary (step S113). For example, the degree of frequency value of each extracted candidate vector is determined, and an evaluation process is performed to lower the evaluation value for a candidate vector having a frequency value equal to or less than a threshold value. Various processes can be considered for the candidate vector evaluation process, and the evaluation process affects the accuracy of extracting candidate vectors.

  Based on the evaluation processing result, the reliability of each candidate vector is determined, and only candidate vectors with high reliability, that is, candidate vectors that are highly likely to be assigned to the image, are sent to the vector assigning unit 14 (FIG. 1) in the subsequent stage. Supply (step S114).

[17. Configuration and operation example of motion vector assignment unit]
Next, an example of the configuration and operation of the motion vector assignment unit 14 in the motion vector detection device shown in the configuration of FIG. 1 will be described with reference to FIGS.
FIG. 24 shows a configuration example of the motion vector determination unit 14 of FIG.
Motion vector candidate data and an image signal for the candidate vector are supplied to the input terminal 14a of the motion vector assignment unit 14. The image signal is supplied to the reference point memory 211 which is a frame memory and stored for one frame. Then, the image signal stored in the reference point memory 211 is moved to the attention point memory 212 for each frame period. Therefore, the image signal stored in the reference point memory 211 and the image signal stored in the attention point memory 212 are always signals shifted by one frame period.

  Then, from the image signal stored in the attention point memory 212, the pixel signal in the steady region centered on the attention pixel is read out to the data reading unit 213. The steady region here is determined from the data stored in the attention point memory 212 by the steady region determination unit 214. The steady region determined by the steady region determination unit 214 is a region where pixel values at the same level as the target pixel are continuous around the target pixel. Therefore, the size varies depending on the state of the image signal of each frame, and the stationary region is always set adaptively. In the present specification, the term “steady region” refers to an adaptively set region in which the same pixel values (in this case, luminance values) are continuous. Judgment with the same pixel value is acceptable, for example, when the pixel value of the pixel of interest and the pixel value of the difference of a predetermined level or less are judged to be the same pixel value, and a slight change in pixel value is acceptable. Like that. Depending on the image state, there may be no steady region.

  Then, from the image signal stored in the reference point memory 211, the pixel signal in the area centered on the reference pixel is read out to the data reading unit 213. The area to be read out at this time is an area having the same size as the steady area on the target pixel side. The pixel position of the target pixel and the reference pixel (the target pixel and the reference pixel) read by the data reading unit 213 is determined by the data reading unit 213 from the candidate vector data supplied from the motion vector extraction unit 13 (FIG. 1). Is done.

Then, the pixel signal in the stationary region centered on the target pixel and the pixel signal in the same range as the stationary region centered on the reference pixel, read by the data readout unit 213, are supplied to the evaluation value calculation unit 215. Then, the difference between the pixel signals in both regions is detected, and the correlation between both regions is determined. In this way, the evaluation value calculation unit 215 determines a pixel signal in the same area as the stationary area of all the reference pixels connected to the target pixel currently being evaluated and the candidate vector, and the stationary area centered on the target pixel. Compare with the pixel signal.
Then, the evaluation value calculation unit 215 selects a reference pixel having an area most similar to the pixel signal of the steady area centered on the target pixel as a result of the comparison. As the comparison here, as will be described later, the sum of absolute differences obtained by adding the absolute values of the differences between the luminance values of the pixels in each region is compared, and the one having the smallest difference absolute value sum is selected.
The candidate vector data connecting the reference pixel selected by the evaluation value calculation unit 215 and the target pixel is sent to the vector determination unit 216. The vector determination unit 216 performs determination processing for assigning the corresponding candidate vector to the motion vector from the target pixel, and outputs the determined candidate vector from the output terminal 215.

The flowchart of FIG. 25 is a flowchart showing an example of the vector assignment processing operation of FIG.
To explain sequentially according to FIG. 25, first, candidate vectors are read based on the data of the evaluation value table (step S121). The coordinate position of the target pixel for the read candidate vector is determined, the pixel value of the pixel (target pixel) at that position is determined, and pixel values that can be regarded as the same pixel value around the target pixel are consecutive. A region to be performed (steady region) is determined. The determined pixel value of the steady region is read (step S122).
Further, the coordinate position of the reference pixel for all the read candidate vectors is determined, and an area having the same size as the stationary area is individually set around the pixel (reference pixel) at the determined position. The pixel value in each area is read from the reference point memory 61 (step S123).
Then, an absolute value sum of differences between pixel levels (pixel values) in each region is calculated (step S124).
Then, the pixel in the region set for the reference pixel is compared with the pixel in the region set for the target pixel to obtain a difference value, and the absolute value of the difference value is added in all the regions to obtain the absolute value. A difference sum is obtained, and a reference pixel region where the absolute value difference sum is minimized is searched. In this process, when a reference pixel region that minimizes the sum of absolute values is determined, a candidate vector that connects the reference pixel in the center of the determined region and the target pixel is assigned as a motion vector for the target pixel. (Step S125).

FIG. 26 is a diagram showing an outline of the processing state in the configuration of FIG. 24 and the flowchart of FIG.
In this example, the target pixel d20 exists in the frame F20 (target frame), and a plurality of candidate vectors V21, V22 exist between the next frame F21 (reference frame) on the time axis. To do. In the frame F21, there are reference pixels d21 and d22 connected to the target pixel d20 by candidate vectors V21 and V22.

Assuming such a state of FIG. 26, in step S122 of FIG. 25, a pixel value at the same level as the pixel value (in this case, the luminance value) of the target pixel d20 is set at the center of the target pixel d20 in the frame F20. A steady region A20 continuous around is set. The pixel values of all the pixels in the steady region A20 are determined.
In step S123 of FIG. 13, regions A21 and A22 are set in the frame F21 with the center of the reference pixels d21 and d22. Regions A21 and A22 are regions having the same area (the same number of pixels) as the steady region A20 set in the immediately preceding frame F20. The positional relationship between the center reference pixels d21 and d22 and the regions A21 and A22 and the region arrangement direction are also the same as the relationship between the target pixel d20 and the steady region A20. For each of the areas A21 and A22 set in this way, the pixel values of all the pixels in each area are determined. Then, the pixel in the determined reference pixel side area and the pixel at the same position in the target pixel side area are compared to obtain a pixel value difference, and the absolute value of the difference in the entire area is obtained. The sum is obtained and the sum of absolute differences is calculated for each reference pixel region.

Then, the absolute value sum of the differences obtained using the region A21 and the region A20 and the absolute value sum of the differences obtained using the region A22 and the region A20 are compared with which is the smallest. In this comparison, for example, if it is determined that the absolute value difference sum obtained using the steady region A21 is smaller, a candidate vector V21 connecting the reference pixel d21 at the center of the region A21 and the target pixel d20 is selected. The selected candidate vector V21 is assigned as a motion vector of the target pixel d20.
In FIG. 26, two candidate vectors have been described for the sake of simplicity, but in reality, more candidate vectors may exist for one target pixel. In FIG. 26, only one pixel of interest is shown for simplicity of explanation, but in the case of the processing in this embodiment, a plurality of representative pixels or all pixels in one frame are included. Such a pixel of interest.

By performing the process of determining the vector to be selected from the candidate vectors in this way, the pixel state around the target pixel and the pixel state around the reference pixel are selected, The motion vector to be assigned to each pixel can be favorably selected.
In particular, using the data of the evaluation value table in which the target pixel or the reference pixel is selectively removed based on the correlation with the high-frequency motion, which is the processing configuration of the present embodiment described above, the motion vector as shown in FIG. By performing the determination process, a better motion vector can be assigned.

[18. Description of Modifications Common to Each Embodiment]
In each of the above-described embodiments, the pixel-of-interest selection process has not been specifically described, but various processes can be applied to the pixel-of-interest selection process. For example, all pixels in one frame may be selected as the target pixel in order, and a motion vector may be detected for each pixel. Alternatively, the present invention may be applied to a case where a representative pixel in one frame is selected as a target pixel, and a motion vector for the selected pixel is detected.
For the reference pixel selection processing for the target pixel, the search area SA illustrated in FIG. 6 and the like is an example, and a configuration in which search areas of various sizes are set for the target pixel can be applied.

  Further, in each of the above-described embodiments, the example in which the motion vector detection device is configured has been described. However, the motion vector detection device may be incorporated into various image processing devices. For example, it can be incorporated into an encoding device that performs high-efficiency encoding, and encoding can be performed using motion vector data. Alternatively, it may be incorporated in an image display device that performs display using input (received) image data or an image recording device that performs recording, and motion vector data may be used to improve image quality.

  Furthermore, each component that performs motion vector detection according to the present invention may be programmed, and the program may be implemented in various information processing apparatuses such as a computer apparatus that performs various data processing. By doing in this way, when performing the process which detects a motion vector from the image signal input into the information processing apparatus, it becomes possible to perform the same process.

It is a block diagram which shows the apparatus structural example by embodiment of this invention. It is a flowchart which shows the example of the whole process by embodiment of this invention. It is a block diagram which shows the example of evaluation value table formation (example 1 which considered high frequency motion) by the 1st Embodiment of this invention. It is the flowchart which showed the process example (Example 1 which considered the attention pixel and high frequency motion) by the 1st Embodiment of this invention. It is explanatory drawing which showed the relationship between a reference pixel and an attention pixel. FIG. 5 is an explanatory diagram illustrating an overview of processing in consideration of the relationship between a pixel of interest and the most frequent motion in the example of FIGS. 3 and 4. It is explanatory drawing which shows the example of the image which has a motion different from a background motion. It is explanatory drawing which shows the example of the evaluation value table without the selection in a pixel selection part. It is a characteristic view which shows the example of the evaluation value table (a) of the most frequent movement, and the evaluation value table (b) except the most frequent movement. It is a block diagram which shows the evaluation value table formation example (example 2 which considered high frequency motion) by the 1st Embodiment of this invention. It is the flowchart which showed the process example (Example 2 which considered the attention pixel and high frequency motion) by the 1st Embodiment of this invention. It is the flowchart which showed another processing example (example which considered the attention pixel, the reference pixel, and high frequency motion) by the 1st Embodiment of this invention. It is explanatory drawing which shows the outline | summary of the process which considered the relationship between the reference pixel and the most frequent motion by the example of FIG. FIG. 13 is a characteristic diagram illustrating an example of an evaluation value table excluding the most frequent motion for a target pixel and a reference pixel according to the example of FIG. 12. It is a block diagram which shows the evaluation value table formation example (example which considered the high frequency motion and the space inclination pattern) by the 2nd Embodiment of this invention. It is the flowchart which showed the example of a process by the 2nd Embodiment of this invention. It is explanatory drawing which shows the outline | summary of the space inclination pattern by the example of FIG. It is explanatory drawing which shows the example (example of mode1) of a space inclination pattern. It is explanatory drawing which shows the example (example of mode2) of a space inclination pattern. It is explanatory drawing which shows the example (example of mode3) of a space inclination pattern. It is explanatory drawing which shows the example (example of mode4) of a space inclination pattern. It is a block diagram which shows the structural example of the motion vector extraction part by embodiment of this invention. It is the flowchart which showed the process by the example of FIG. It is a block diagram which shows the structural example of the motion vector determination part by embodiment of this invention. It is the flowchart which showed the process by the example of FIG. It is explanatory drawing which shows the example of the motion vector determination process state by the example of FIG. It is a block diagram which shows an example of the conventional evaluation value table data generation process structure. It is explanatory drawing which shows the outline | summary of the conventional evaluation value table data generation process example.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 1 ... Image signal input terminal, 2 ... Correlation calculation part, 2a ... Reference point memory, 2b ... Interesting point memory, 2c ... Absolute value calculation part, 3 ... Correlation determination part, 3a ... Comparison part, 4 ... Evaluation value table calculation part 4a ... evaluation value integration unit, 4b ... evaluation value table memory, 5 ... evaluation value table data output terminal, 11 ... image signal input terminal, 12 ... evaluation value table forming unit, 12a ... evaluation value table data output terminal, 13 ... Motion vector extraction unit, 14 ... motion vector allocation unit, 15 ... motion vector output terminal, 16 ... control unit (controller), 20 ... correlation calculation unit, 21 ... reference point memory, 22 ... attention point memory, 23 ... absolute value calculation , 30 ... correlation determination unit, 31 ... comparison unit, 40 ... pixel selection processing unit, 41 ... high frequency motion calculation unit, 42 ... high frequency motion determination unit, 43 ... selection attention point memory, 44 ... selection reference point memory DESCRIPTION OF SYMBOLS 50 ... Evaluation value table calculation part, 51 ... Evaluation value accumulation part, 52 ... Evaluation value table memory, 60 ... Pixel selection process part, 61 ... Temporary memory, 62 ... Gate part, 63 ... High frequency motion calculation part, 64 ... High frequency motion determination unit, 70 ... Pixel selection processing unit, 71 ... High frequency motion calculation unit, 72 ... High frequency motion determination unit, 73 ... Selection attention point memory, 74 ... Selection reference point memory, 75 ... Spatial inclination pattern determination unit 111 ... Evaluation value table data conversion unit, 112 ... Frequency order sort processing unit, 113 ... Candidate vector evaluation unit, 114 ... Candidate vector reliability determination unit, 211 ... Reference point memory, 212 ... Attention point memory, 213 ... Data reading 214, stationary region determination unit, 215 ... evaluation value calculation unit, 216 ... vector determination unit

Claims (10)

A high-frequency motion vector detection unit that detects a high-frequency motion vector from moving image data composed of a plurality of frames;
A motion vector candidate obtained from pixel value correlation information between a target pixel of one frame on the time axis and a reference pixel in the search area of the other frame of the moving image data has a high correlation with the high-frequency motion vector. A pixel selecting unit that selects a pixel of interest and / or a reference pixel corresponding to the motion vector candidate from pixels in the frame;
An evaluation information forming unit that forms motion vector evaluation information that evaluates the possibility that the reference pixel is a motion candidate of the target pixel based on pixels other than the selected pixels;
Based on the motion vector detected from the evaluation information formed by the evaluation information forming unit and the information on the high-frequency motion vector detected by the high-frequency motion vector detection unit, the motion vector for each pixel in the frame of the moving image data A motion vector extraction unit for extracting candidates;
A motion vector detection apparatus comprising: a motion vector allocation unit that determines a motion vector to be allocated to each pixel in a frame from candidate motion vectors extracted by the motion vector extraction unit. The high-frequency motion vector detection unit obtains a high-frequency motion vector from a motion vector detected from a correlation between the one frame before the one frame in which the target pixel is set by the evaluation information forming unit and the one frame. The motion vector detection device according to claim 1, which detects the motion vector. The high-frequency motion vector detection unit detects a high-frequency motion vector from a motion vector detected from a correlation between the one frame in which a pixel of interest is set by the evaluation information forming unit and the other frame. 2. The motion vector detection device according to 1. The high-frequency motion vector detection unit detects the most frequent motion vector, and evaluates information related to a pixel serving as a target pixel and / or a reference pixel of a motion vector having a high correlation with the detected most frequent motion vector, The motion vector detection apparatus according to any one of claims 1 to 3, wherein the pixel selection unit performs selection and removal excluded from the information to be integrated. The high-frequency motion vector detection unit detects from the most frequent motion vector to a predetermined most frequent motion vector, and a pixel of interest of a motion vector highly correlated with the detected most frequent motion vector and / or The motion vector detection device according to any one of claims 1 to 3, wherein the pixel selection unit performs selection and removal that excludes evaluation information related to a pixel serving as a reference pixel from the information to be integrated. The evaluation information forming unit determines a spatial inclination state between the target pixel and its adjacent pixel in the one frame, determines a spatial inclination state between the reference pixel of the other frame and its adjacent pixel, The motion vector detection device according to any one of claims 1 to 3, wherein a motion vector from a corresponding target pixel to a reference pixel is added to the evaluation information when the slope codes match. The motion vector detection device according to claim 6, wherein the spatial inclination state is determined by a pattern based on a spatial inclination between a target pixel or a reference pixel and neighboring pixels around each pixel. The motion vector allocating unit adaptively sets a steady region according to a state in which the same pixel value continues for the target pixel and the reference pixel of the candidate motion vector extracted by the motion vector extracting unit, and the setting The motion vector detection device according to any one of claims 1 to 3, wherein a motion vector of each pixel is assigned from the one frame to the other frame based on a result of comparing the correlations between the steady regions. A high-frequency motion vector detection process for detecting a high-frequency motion vector from moving image data composed of a plurality of frames;
A motion vector candidate obtained from pixel value correlation information between a target pixel of one frame on the time axis and a reference pixel in the search area of the other frame of the moving image data has a high correlation with the high-frequency motion vector. A pixel selection process for selecting a target pixel and / or a reference pixel corresponding to the motion vector candidate from the pixels in the frame;
An evaluation information forming process for forming evaluation information of a motion vector that evaluates the possibility that a reference pixel is a motion candidate of the target pixel, based on pixels other than the pixels selected in the pixel selection process;
Extraction of motion vector candidates for each pixel in a frame of moving image data based on the evaluation information formed by the evaluation information forming process and the information of the motion vector having a high frequency detected by the high-frequency motion vector detection process Motion vector candidate extraction processing;
A motion vector detection method for performing a motion vector assignment process for determining a motion vector to be assigned to each pixel in a frame from candidate motion vectors extracted by the motion vector extraction process. A high-frequency motion vector detection process for detecting a high-frequency motion vector from moving image data composed of a plurality of frames;
A motion vector candidate obtained from pixel value correlation information between a target pixel of one frame on the time axis and a reference pixel in the search area of the other frame of the moving image data has a high correlation with the high-frequency motion vector. A pixel selection process for selecting a target pixel and / or a reference pixel corresponding to the motion vector candidate from the pixels in the frame;
An evaluation information forming process for forming evaluation information of a motion vector that evaluates the possibility that a reference pixel is a motion candidate of the target pixel, based on pixels other than the pixels selected in the pixel selection process;
Extraction of motion vector candidates for each pixel in a frame of moving image data based on the evaluation information formed by the evaluation information forming process and the information of the motion vector having a high frequency detected by the high-frequency motion vector detection process Motion vector candidate extraction processing;
A motion vector assignment process for determining a motion vector to be assigned to each pixel in a frame from candidate motion vectors extracted by the motion vector extraction process.
A program that is implemented and executed on an information processing device.

Images