JP2012239093A - Image processing device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing device, capable of frame interpolation into a high-speed panning moving image, using a circuit and a circuit scale substantially equivalent to the conventional device.SOLUTION: The image processing device includes: a memory control circuit for receiving pixel data of a present frame successively in order of lines to store into a line memory, and for reading out pixel data of a past frame successively in order of lines from a frame memory to store into the line memory; and an interpolation frame generator circuit for receiving a motion vector of a reduced vertical direction component, having a vertical direction component reduced by a first moving amount M1, defined according to the vertical direction component of panning speed, and for generating pixel data of an interpolation frame, based on the present frame pixel data and the past frame pixel data associated by the motion vector. The memory control circuit moves timing to store the past frame image data into the line memory from standard timing when the first moving amount M1 is zero, by a second moving amount M2 corresponding to the first moving amount M1.

Description

  The present invention relates to an image processing apparatus that generates an output image with a frame rate doubled by inserting an interpolation frame between frames of an input image.

  There is a technique called frame rate conversion (FRC) that doubles the frame rate of an input image to smooth the video. With this technique, for example, an input image of 60 frames per second is converted into an image of 120 frames per second. This is realized by inserting an interpolation frame between adjacent frames of the input image (original image) based on the motion vector of the image.

  To detect a motion vector and generate an interpolated frame, at least two original images (for example, a current frame and a past frame) are necessary, and these are stored in, for example, a frame memory (for example, an external DRAM). In general, since the frame memory is slow, image information in a certain range centering on the pixel to be processed is read in advance into a high-speed line memory (for example, SRAM) and is used as a processing target.

  Hereinafter, an example of a conventional image processing apparatus that performs FRC will be described.

  FIG. 19 is a block diagram illustrating an example of a configuration of a conventional image processing apparatus. The image processing apparatus 40 shown in FIG. 1 includes a frame memory 42, a memory controller 44, a line memory 46, a motion vector detection circuit 48, and an interpolation frame generation circuit 50. The line memory 46 includes a current frame line memory that stores current frame pixel data and a past frame line memory that stores past frame pixel data.

  The frame memory 42 stores past frame pixel data of a past frame (a frame immediately before the current frame) immediately before the current frame (input image) under the control of the memory controller 44.

  The current frame pixel data is received in the order of the lines under the control of the memory controller 44, stored in the frame memory 42, and stored in the current frame line memory. The past frame pixel data stored in the frame memory 42 is read out from the frame memory 42 in the order of the lines, and the past frame pixel data corresponding to the line of the current frame stored in the current frame line memory is stored in the past. Stored in the frame line memory.

  As shown in FIG. 16, the input timing of the line of the current frame is the same as the timing at which the line of the past frame corresponding to the line of the current frame is read from the frame memory 42. In other words, the current frame line range stored in the current frame line memory is the same as the past frame line range (the past frame line range read from the frame memory 42) stored in the past frame line memory. It is.

  When pixel data of a predetermined number of storage lines is stored in the current frame line memory and the past frame line memory, the current frame pixel stored in the current frame line memory, which is image information, by the motion vector detection circuit 48 Based on the data and reference past frame pixel data stored in the past frame line memory, a motion vector V between the past frame and the current frame is detected.

  Further, in the motion vector detection circuit 48, as shown in FIG. 16, the allocation processing for allocating the detected motion vector V to the pixel of the interpolation frame, or the motion vector of the pixel of the interpolation frame for which the motion vector could not be allocated is compensated. A vector correction process such as an interpolation process is performed to generate a corrected vector Vc. The correction vector is output from the motion vector detection circuit 48 as a motion vector.

  Finally, the interpolation frame generation circuit 50 stores the motion vector Vc subjected to vector correction processing by the motion vector generation circuit 48, the current frame pixel data stored in the current frame line memory, and the past frame line memory. Interpolated frame pixel data of an interpolated frame between the past frame and the current frame is generated based on the past frame pixel data.

  FIG. 18 is a conceptual diagram showing the phase and position of an interpolation frame generated by a conventional image processing apparatus. In the figure, the vertical axis represents the position of a pixel generated in the interpolation frame (spatial interpolation position, that is, the position of the line), and the horizontal axis represents the phase of the interpolation frame (temporal interpolation position). As shown in the figure, in the conventional image processing apparatus, the motion vector between the pixel of the past frame and the corresponding pixel of the current frame is V, the phase from the past frame of the interpolation frame is α, the current frame of the interpolation frame If the phase from is 1−α, the pixel of the interpolation frame is generated at the position of αV with reference to the position of the pixel of the past frame.

  Here, there are Patent Documents 1 and 2 as prior art documents of the present invention.

  Patent Document 1 relates to an image processing apparatus. In this image processing apparatus, as shown in FIG. 4 of the same document, the vector detection unit uses the input image (frame t + 1) and the image of the frame t read from the frame memory 51, and the focus of the frame t A motion vector is detected between the block and the target block of frame t + 1 and stored in the detection vector memory 53. The vector assigning unit 54 assigns the motion vector obtained in the input 24P (24 frames per second progressive) original image frame to the interpolated frame pixels of 60P (60 frames per second) to be interpolated. The motion vector of the pixel that did not exist is supplemented by the allocation compensation unit 57 and stored in the allocation vector memory 55 (see paragraphs 0036 to 0039). The image interpolation unit 58 interpolates and generates the pixel value of the interpolation frame using the motion vector of the allocation vector memory 55 and the pixel values of the frames t and t + 1 (see paragraph 0040).

  Patent Document 2 relates to a motion vector detection device. This document describes that a shift amount of a motion vector search range in the next frame is determined from a motion vector histogram for one frame (see summary). In this document, by shifting the search range, the range in which the motion vector detection unit 103 of FIG. 1 of the same document reads reference pixel data from the frame memory is shifted.

JP 2006-101267 A Japanese Patent Laid-Open No. 11-205799

  In the conventional image processing apparatus 40, the range of the past frame image that can be referred to by the size of the line memory 46 (the number of stored lines), that is, the motion vector detectable range (how fast the motion can be detected) and interpolation are possible. The range (how fast the object can be interpolated) is determined. Therefore, for example, if the motion vector detectable range and the interpolation possible range are to be expanded in order to enable processing of an image that moves at high speed in the vertical direction ("panning"), the line memory 46 is increased. There is a problem that the circuit scale increases.

  An object of the present invention is to provide an image processing apparatus that can perform high-speed frame interpolation of a panning moving image with a circuit and circuit scale substantially the same as those of a conventional one.

In order to achieve the above object, the present invention provides an image processing apparatus that generates an interpolation frame between a plurality of frames each composed of a plurality of horizontal lines each including a plurality of pixels.
Current frame pixel data representing pixel values of pixels of a plurality of lines constituting the current frame is received in the order of the lines, stored in a current frame line memory having a first predetermined storage line number, and a frame Past frame pixel data representing pixel values of pixels of a plurality of lines constituting the past frame immediately before the current frame stored in the memory is read out from the frame memory in the order of the lines, and is stored in a second predetermined storage. A memory control circuit for storing in a past frame line memory having the number of lines;
A current frame pixel data stored in the current frame line memory and a past frame pixel stored in the past frame line memory, which receives a motion vector indicating the motion of the pixel of the interpolation frame and is associated with the motion vector. An interpolation frame generation circuit that generates interpolation frame pixel data representing pixel values of pixels of each line of the interpolation frame based on the data,
The interpolation frame generation circuit reduces the vertical component as the motion vector by the first movement amount M1 determined according to the vertical component of the pan speed, which is the overall motion speed of the interpolation frame. A direction vector reduced motion vector is received as the motion vector;
The memory control circuit is configured to determine the past frame in the past frame line memory, which is determined relative to the timing of storing the current frame pixel data of each line constituting the current frame in the current frame line memory. From the standard timing when the first movement amount M1 is zero, the second movement amount M2 corresponding to the first movement amount M1 is stored. The present invention provides an image processing apparatus characterized by only moving.

Here, the memory control circuit is
Current frame pixel data of a plurality of lines constituting the current frame is stored in the current frame line memory for a predetermined time for each line from the first start timing, and a plurality of data constituting the past frame is stored. The past frame pixel data of the current line is stored in the past frame line memory taking the predetermined time for each line from the second start timing determined relatively to the first start timing. And
It is preferable that the second start timing is moved by the second movement amount M2 from the standard start timing when the first movement amount M1 is zero.

In addition, the interpolation frame generation circuit may calculate the interpolation frame pixel data of a plurality of lines constituting the interpolation frame for each line from a third start timing that is relatively determined with respect to the first start timing. It is generated in a certain amount of time, and
Interpolation frame generation circuit control for moving the third start timing by a movement amount M3 determined according to the second movement amount M2 from a standard start timing when the first movement amount M1 is zero. It is preferable to provide a part.

  In addition, the interpolation frame is an interpolation frame at a time position relatively separated from the past frame by α (0 <α <1) and 1-α from the current frame, and the interpolation frame generation circuit control unit includes: The movement amount M3 is preferably moved so that M3 = (1−α) M2.

  The panning speed is calculated using the panning speed of the past frame obtained based on the motion vector distribution between the pixels of the previous frame and the pixels of the past frame. It is preferable to further include a speed calculation circuit.

  Further, by using the current frame pixel data stored in the current frame line memory and the past frame pixel data stored in the past frame line memory, the vertical component is reduced by the first movement amount M1. A motion vector detection circuit that detects a motion vector between a pixel in the past frame and a pixel in the current frame and supplies the motion vector to the interpolation frame generation circuit as the motion vector with the vertical component reduced. preferable.

  According to the present invention, it is possible to perform frame interpolation even for high-speed panning video without significantly changing the circuit and without greatly increasing the circuit scale as compared with the conventional image processing apparatus. become.

It is a block diagram of one embodiment showing composition of an image processing device of the present invention. (A) is an image of the past frame, (B) is an image of the current frame, (C) is an image obtained by superimposing the current frame and the past frame, and (D) is an image of the past frame with respect to the image of the current frame. The image is moved downward. It is an example of a graph showing the relationship between pan speed and apparent movement. It is an example of a graph showing the relationship between pan speed and reference range movement amount. It is an example timing chart showing the input timing of the line of the present frame and the read timing from the frame memory of the past frame. FIG. 6 is an example timing chart showing storage states of current and past frame line memories in the timing chart shown in FIG. 5. (A) And (B) is a conceptual diagram which shows the production | generation procedure of the interpolation frame in the conventional image processing apparatus. (A)-(D) are the conceptual diagrams which show the production | generation procedure of the interpolation frame in the image processing apparatus of this embodiment. It is a timing chart of an example showing the processing timing of motion vector detection and interpolation frame generation when the readout timing of the line of the past frame is delayed. It is a timing chart of an example showing the processing timing of motion vector detection and interpolation frame generation when the readout timing of the line of the past frame is advanced. It is a conceptual diagram showing the state where the line of the past frame memorize | stored in the line memory for past frames is moved below 2 lines with respect to the line of the present frame memorize | stored in the line memory for present frames. FIG. 12 is a conceptual diagram illustrating a relationship between a line of a current frame and a line of a past frame when a motion vector is detected in the state illustrated in FIG. 11. FIG. 12 is a conceptual diagram illustrating a relationship between a current frame line and a past frame line when an interpolation frame is generated in the state illustrated in FIG. 11. It is a conceptual diagram showing the production | generation process of an interpolation frame when there exists a downward motion between the past frame and the present frame. It is a conceptual diagram showing the production | generation process of an interpolation frame when there exists an upward motion between a past frame and the present frame. It is a conceptual diagram showing the production | generation process of the interpolation frame in the conventional image processing apparatus. It is a conceptual diagram showing the phase and position of the interpolation frame produced | generated with the image processing apparatus of this invention. It is a conceptual diagram showing the phase and position of the interpolation frame produced | generated with the conventional image processing apparatus. It is a block diagram of an example showing the structure of the conventional image processing apparatus.

  Hereinafter, an image processing apparatus according to the present invention will be described in detail based on preferred embodiments shown in the accompanying drawings.

  First, the outline of the image processing apparatus of the present invention will be described.

  For example, consider a case in which motion vector detection and interpolation frame generation are performed using two images, the past frame image shown in FIG. 2A and the current frame image shown in FIG.

  As shown in FIG. 5C, when these two images are overlapped (in order to facilitate the distinction between them, the image of the past frame is indicated by a dotted line), the interval between the past frame and the current frame is shown. It can be seen that the car (object) is moving downward. In the conventional image processing apparatus 40, when the vertical movement (speed) of the car is very large, the motion vector V cannot be detected unless the line memory 46 is provided. That is, in the case of vertical movement, the interpolation performance is limited by the capacity of the line memory 46 (number of stored lines).

  In the image processing apparatus of the present invention, when it is detected that the input image is a panning moving image that moves downward, as shown in FIG. Is moved downward by the first movement amount M1 to reduce the apparent motion vector V ′. In this example, since the car is moving downward, the image of the past frame is moved downward so that the movement looks small when the two images are compared.

  As described above, the image processing apparatus according to the present invention, when the vertical movement of the object is large, moves the current frame image and the past frame image relatively closer to each other so as to make an apparent motion vector. The motion vector is detected with the same circuit and the same circuit scale (equivalent line memory capacity) as in the case of moving images that pan at high speed by reducing Can do.

  Hereinafter, embodiments of the image processing apparatus of the present invention will be described.

  FIG. 1 is a block diagram of an embodiment showing a configuration of an image processing apparatus of the present invention. The image processing apparatus 10 shown in the figure generates an interpolation frame between a plurality of frames each composed of a plurality of lines in the horizontal direction (line direction) each including a plurality of pixels. The image processing apparatus 10 includes a frame memory 12, a memory controller 14, a line memory 16, a motion vector detection circuit 18, an interpolation frame generation circuit 20, a reference range determination circuit 22, a timing controller 24, and a correction circuit 26. And a pan detection circuit 28 and a screen edge processing circuit 30.

  The frame memory 12 stores past frame pixel data representing pixel values (luminance values, lightness values, saturation values, etc.) of pixels of a plurality of lines constituting a past frame, which will be described later, under the control of the memory controller 14.

  Under the control of the memory controller 14, for example, the line memory 16 deletes the data of the oldest line in the received order and stores the data of the newly received line, thereby storing pixel data of a certain number of storage lines. To maintain. The line memory 16 includes a current frame line memory for storing current frame pixel data having a first predetermined storage line number and a past frame for storing past frame pixel data having a second predetermined storage line number. Line memory.

  Normally, the first storage line number of the current frame line memory is the same as the second storage line number of the past frame line memory. However, it is not essential that the number of storage lines is the same, and the number of storage lines may be different.

  For each frame, the memory controller 14 stores current frame pixel data representing pixel values of pixels of a plurality of lines constituting the current frame (input image) in the frame memory 12. The current frame pixel data stored in the frame memory 12 in each frame is a future frame immediately after (the frame immediately after the current frame), and a past frame immediately before the current frame (a frame immediately before the current frame). Are read out from the frame memory 12 as past frame pixel data representing pixel values of pixels of a plurality of lines.

  Further, the memory controller 14 receives the current frame pixel data in the order of the lines, stores the current frame pixel data of the latest first storage line number in the current frame line memory, and stores the current frame pixel data in the frame memory 12. The past frame pixel data is read from the frame memory 12 in the order of the lines, and the past frame pixel data of the second storage line number corresponding to the current frame pixel data of the line stored in the line memory for the current frame is stored in the past frame. Store in the line memory.

  In addition, the memory controller 14 stores in the past frame line memory, which is determined relative to the timing of storing the pixel data of each line in the current frame line memory (in other words, the input timing of the line of the current frame). The first movement amount M1 is zero when the pixel data of the line of the past frame corresponding to the line of the current frame is stored (in other words, the timing of reading the pixel data of the line of the past frame from the frame memory 12). In this case, the operation of the line memory 16 is controlled so as to move by the second movement amount M2 corresponding to the first movement amount M1. The first movement amount M1 is a relative movement amount between the past frame and the current frame, as shown in FIG. The second movement amount M2 is a movement amount (time) at which the pixel data of the line of the past frame is read from the frame memory 12 and stored in the line memory for the past frame, and the line of the past frame read from the frame memory 12 is the first movement amount M2. This corresponds to the movement amount at the timing of moving by one movement amount M1.

  For example, the memory controller 14 stores the current frame pixel data in the current frame line memory taking a predetermined time for each line from the first start timing which is the standard timing described above. In addition, the past frame pixel data is predetermined for each line from the second start timing, which is determined relative to the first start timing and is moved by the second movement amount M2 from the standard timing. It takes time and stores it in the past frame line memory.

  Here, although described as “relatively determined”, the memory controller 14 normally sets the timing for storing the current frame pixel data to be constant and delays the timing for storing the past frame pixel data. However, for example, if the current frame pixel data is also stored once in the frame memory 12 and then read out and stored in the line memory 16, the current frame pixel data is stored at a constant timing for storing the past frame pixel data. It is also possible to make the storage timing earlier (slower).

  In addition, it has been described that “move by the second movement amount M2 corresponding to the first movement amount M1 from the standard timing when the first movement amount M1 is zero”. If not, the current frame pixel data and the past frame pixel data are stored at the same timing. However, this is not essential. For example, when the first predetermined storage line number is different from the second predetermined storage line number, even if no movement is performed, the current frame pixel data It is also possible to adopt a configuration in which storage and storage of past frame pixel data are performed at different timings.

  In addition, although it has been described as “requires a predetermined time”, reading and writing of each line requires time corresponding to the number of pixels of each line and a blanking period between the lines. This time is constant even if the start timing moves.

  Subsequently, the motion vector detection circuit 18 uses the current frame pixel data stored in the current frame line memory and the past frame pixel data stored in the past frame line memory to determine the overall motion of the interpolation frame. A motion vector whose vertical direction component has been reduced is detected in which the vertical direction component (direction orthogonal to the line direction) is reduced by a first movement amount M1 determined according to the vertical direction component of the panning speed. In order for the motion vector detection circuit 18 to detect a motion vector, the pixel data of the pixels of the corresponding line must be stored in the current frame line memory and the past frame line memory. Therefore, when the timing at which pixel data of the past frame is stored in the frame memory for the past frame is moved by the second movement amount M2, the motion vector detection circuit 18 performs motion vector detection according to the sign of M2. Move the timing to do.

  Here, although described as “pan speed that is the overall motion speed of the interpolated frame”, the pan detection circuit 28 to be described later is actually based on the current frame pixel data and the past frame pixel data. The pan speed between the frame and the current frame cannot be determined directly. Therefore, the pan detection circuit 28 approximately obtains the pan speed between the past frame and the current frame using, for example, a motion vector between the frame immediately before the past frame and the past frame.

  The motion vector detection circuit 18 is not an essential component. For example, the configuration may be such that the motion vector is received from outside the image processing apparatus 10 without providing the motion vector detection circuit 18.

  As described above, in order for the motion vector detection circuit 18 to detect a motion vector, pixel data of a pixel of the current frame and pixel data of a pixel of a past frame associated with the pixel by the motion vector are included. , Need to be stored in the current frame line memory and the past frame line memory, respectively. When past frame pixel data is stored in the past frame line memory at the timing moved by the second movement amount M2, the pixel data of the corresponding current frame or past frame line is displayed at the upper end and lower end of the screen. There is a high possibility that the motion vector detection circuit 18 cannot detect the motion vector because it is not stored in the memory. In view of this, the image processing circuit 10 limits the period during which the motion vector detection circuit 18 detects the motion vector or the range of lines to be subjected to motion vector detection based on the second movement amount M2.

  In this way, the motion vector cannot be detected at the upper and lower ends of the screen because there is an area where there is no line corresponding to the current frame or the past frame on the screen panning in the vertical direction. Is not due to move. Rather, by using the second movement amount M2 corresponding to the first movement amount M1 determined according to the vertical component of the pan speed, the range in which the motion vector detection circuit 18 detects the motion vector is limited. It is possible to prevent detection of an erroneous motion vector.

  Then, the screen edge processing circuit 30 refers to, for example, a motion vector in a temporally or spatially close area for a pixel of a line that is not subject to motion vector detection by the motion vector detection circuit 18. A motion vector is generated by performing correction processing such as assigning a panning speed. The screen edge processing circuit adds the motion vector thus generated to the motion vector detected by the motion vector detection circuit 18. The screen edge processing circuit also generates a corrected vector by performing an allocation process that allocates a motion vector to an interpolated frame pixel and a correction process that compensates for the motion vector of an interpolated frame pixel that could not be allocated a motion vector. You may do.

  The interpolation frame generation circuit 20 receives the corrected vector from the screen edge processing circuit 30, and stores the current frame pixel data stored in the current frame line memory and the past frame line memory, which are associated with the motion vector. Based on the past frame pixel data, interpolation frame pixel data representing pixel values of the pixels of each line of the interpolation frame is generated.

  The interpolation frame generation circuit 20 moves the generation timing of the interpolation frame pixel data from the standard timing by the third movement amount M3 determined according to the second movement amount M2. For example, the interpolation frame pixel data of a plurality of lines constituting the interpolation frame requires a predetermined time for each line from the third start timing that is determined relative to the first start timing that is the standard timing. To generate.

  For example, if the interpolation frame is α (0 <α <1) from the past frame, that is, an interpolation frame at a time position (phase) relatively separated from the current frame by 1-α, the third movement amount M3 moves so that M3 = (1-α) M2.

  Thereby, by moving the timing of storing the pixel data of the line of the past frame in the line memory, the position of the interpolation frame generated based on the pixel data stored in the line memory is moved. The reference range can be returned to the original correct position that has not been moved.

  Here, although described as “a motion vector indicating the motion of the pixel of the interpolation frame”, the motion vector generated by the motion vector detection circuit 18 or the motion vector received from the outside at least approximates the motion of the pixel of the interpolation frame. Or a motion vector obtained between the past frame and the current frame, or a motion vector obtained between the past frame and the current frame as in Patent Document 1. Alternatively, the pixel may be reassigned to the interpolation frame pixels (and further compensated). Further, it is not essential to detect a motion vector for each pixel, and a motion vector may be detected for each block composed of a plurality of pixels.

  Subsequently, based on the second movement amount M2, the correction circuit 26 detects the vertical direction component-reduced motion vector obtained by reducing the vertical component by the first movement amount M1 detected by the motion vector detection circuit 18. The motion vector between the pixels of the past frame and the pixels of the current frame, in which the vertical movement amount is not reduced by the first movement amount M1, that is, the motion vector detected by the conventional motion vector detection circuit 48 The original motion vector between the past frame and the current frame is calculated.

  The pan detection circuit (pan speed calculation circuit) 28 detects the pan speed between the past frame and the current frame from the original motion vector calculated by the correction circuit 26. Panning according to the present invention refers to panning within the screen of the moving image being shot, in addition to the case where the entire moving image screen moves in one direction by moving the shooting device while moving the shooting device in one direction. This includes the case where most of the object to be imaged, for example, the object to be imaged of more than half of the screen moves in one direction.

  Here, the pan speed between the past frame and the current frame detected by the pan detection circuit 28 is, for example, for detecting a motion vector between the current frame and a frame immediately after the current frame (future frame). Can be used.

  The motion vector detection circuit 18 cannot detect a motion vector between the past frame and the current frame by using the pan speed between the past frame and the current frame. That is, since the panning speed is obtained from the distribution of motion vectors over the entire screen, the panning speed obtained from the motion vector between the past frame being detected by the motion vector detection circuit 18 and the current frame is set as the same past frame. It cannot be used for motion vector detection with the current frame. Therefore, the pan detection circuit 28 determines the pan speed used when the motion vector detection circuit 18 detects a motion vector between the past frame and the current frame, for example, the pixel of the frame immediately before the past frame and the past frame. Calculation (prediction) is performed using the pan speed of the past frame obtained based on the distribution of motion vectors between the pixels.

  Subsequently, the reference range determination circuit 22 is based on the pan speed detected by the pan detection circuit 28, and is a first frame that determines a line range of a past frame to be referred to in motion vector detection and interpolation frame generation, that is, a reference range. The movement amount M1 is determined and output as reference range information. For example, the reference range determination circuit 22 is configured so that the object of the past frame substantially overlaps the object of the current frame and falls within the motion vector detectable range or the interpolable range determined by the first and second storage line numbers of the line memory 16. First, the first movement amount M1 is determined. Most simply, the first movement amount M1 is equal to the vertical component of the pan speed between the past frame and the current frame.

  Finally, the timing controller (interpolation frame generation circuit control unit) 24 corresponds to the first movement amount M1 based on the first movement amount M1 as the reference range information determined by the reference range determination circuit 22. A second movement amount M2 and a third movement amount M3 corresponding to the second movement amount M2 are calculated. These movement amounts M2 and M3 are sent to the memory controller 14, the motion vector detection circuit 18, the screen edge processing circuit 30, and the interpolation frame generation circuit 20, and are used to control the operation timing of these circuits.

  Here, the reference range determining circuit 22, the memory controller 14, and the timing controller 24 constitute a memory control circuit of the present invention.

Next, specific actions in each part will be described.
First, a panning moving image determination method and a panning speed detection method will be described.

  In the pan detection circuit 28, first, it is determined whether or not the input image is a panning moving image using the original motion vector described above. For example, by counting the frequency of motion vectors and creating a histogram, it is possible to know the trend of the motion vectors of the entire screen. For example, when the frequency is biased toward a small number of similar motion vectors, or when the motion vector distribution is biased in a specific direction, it can be determined that the video is a panning video.

  The pan detection circuit 28 detects the pan speed when it is determined that the input image is a pan moving image. For example, the pan speed can be obtained from a motion vector having a peak value of the histogram.

In the case of a panning moving image with acceleration, it is desirable to calculate the panning speed of the future frame by examining the change of the panning speed of the past several frames in order to accurately predict the panning speed of the next frame. The pan speed s2 of the future frame can be obtained by the following formula (1), for example, where the pan speed of the past frame is s0, the pan speed of the current frame is s1, and the acceleration is a1.
s2 = s1 + a1 = s1 + (s1-s0) (1)

  Next, a method for determining the amount of movement of the reference range of the line in the past frame will be described.

  When the pan detection circuit 28 determines that the video is a pan moving image, the reference range determination circuit 22 determines the pan speed based on the relationship between the vertical component of the pan speed and the motion vector detectable range (or interpolable range). The first movement amount M1 is determined so as to be well within the detectable range. The panning speed is a vector value having a direction and a size. The movement amount M1 is a scalar amount in the vertical direction (downward or upward) and is represented by the number of lines. Accordingly, in determining the first movement amount M1, only the vertical component of the pan speed is taken into consideration.

  The timing controller 24 calculates a second movement amount M2 corresponding to the first movement amount M1 and a third movement amount M3 corresponding to the second movement amount M2.

  However, it is desirable to limit the amount of movement of the reference range so that it can cope with a sudden scene change. Further, it is desirable not to move the reference range when the pan speed is sufficiently small with respect to the detectable range.

  FIG. 3 is a graph illustrating an example of the relationship between the pan speed and the reference range movement amount. As described above, the pan speed is a vector value having a direction and a magnitude, but only the vertical direction component is considered in determining the movement amount. In the following, for simplicity, it is assumed that the pan speed has only a vertical component, and the downward direction is treated as a positive scalar value and the upward direction is treated as a positive scalar value. In the figure, the horizontal axis represents the pan speed v [line / frame], and the vertical axis represents the reference range movement amount m [line]. The speed A [line / frame] is a threshold value for determining whether or not to move the reference range according to the pan speed, and the reference range movement amount B [line] is the limit value of the movement amount, and the speed C [line / frame]. ] Means the pan speed when the movement amount is limited. In terms of circuit, the values of A, B, and C are variable.

  The timing controller 24 uses the relationship shown in FIG. 4 to determine the second moving amount M2 from the panning speed detected by the pan detecting circuit 28, that is, the first moving amount M1 determined by the reference range determining circuit 22. The third movement amount M3 is uniquely calculated. The relationship between the pan speed (or the first movement amount M1) and the reference range movement amount shown in FIG. 3 can be realized, for example, in the form of a calculation formula or a lookup table (LUT).

  As shown in the graph of FIG. 3, the apparent movement v 'when moving the reference range is as shown in the graph of FIG. In the conventional image processing apparatus 40, as indicated by a dotted line, the pan speed v and the apparent movement v 'are equal. In the image processing apparatus 10 according to the present embodiment, as shown by the solid line, the apparent movement v ′ increases gradually as the pan speed v increases. For example, when the panning speed is C, the reference range movement amount is B per frame, so that the apparent movement v ′ is C−B.

  Next, a method for moving the reference range of the line in the past frame will be described.

  The memory controller 14 reads the past frame from the frame memory 12 based on the second movement amount M2 calculated by the timing controller 24 and stores the current frame in the past frame line memory. The reference range of the past frame is moved by making it late or early with respect to the timing stored in the frame.

  For example, when it is detected that panning is downward and it is found from the graph of FIG. 3 that the reference range of the line of the past frame only needs to be moved downward by two lines, FIG. As described above, the read timing of the line of the past frame is delayed by two lines with respect to the input timing of the line of the current frame. As a result, the reading of the lines Line ′ # 1, Line ′ # 2, Line ′ # 3,... In the past frame is started from the position of the input timing of the line Line # 3 in the current frame.

  In other words, the movement of the spatial position in the vertical direction in the image is realized only by the movement of the timing in the reading of the frame memory 12 by the control of the timing controller 24 and the memory controller and the roller 14.

  When the readout timing of the lines of the past frame is moved downward by two lines, the range of pixel data developed on the line memory 16 is as shown in FIG. 6 (when the number of storage lines in the line memory 16 is 10 lines) ). In the example shown in the figure, the current frame line Line # 3, Line # 4,..., Line # 12 is stored in the current frame line memory, and correspondingly shifted by two lines in the past line line memory. This represents a state in which the lines Line ′ # 1, Line ′ # 2,..., Line ′ # 10 of the past frame are stored.

  As described above, when the motion vector detection is performed using the pixel data developed on the line memory 16 with the line of the past frame moved by two lines with respect to the line of the current frame, the detected zero vector is detected. Actually means that the vector has a magnitude of 2 in the downward direction. This can be considered as a state where the range of the downward component of the detectable motion vector is expanded by two lines.

  Next, detection of a motion vector when the reference range of a line in the past frame is moved will be described.

  The motion vector detection by the motion vector detection circuit 18 is stored in the current frame line memory in the same manner as the motion vector detection circuit 48 of the conventional image processing device 40 regardless of whether the reference range is moved or not. This is performed using the current frame pixel data and the past frame pixel data stored in the past frame line memory.

  However, as described above, when the reference range is moved, the motion vector detection circuit 18 does not calculate a motion vector at the screen edge (the upper edge and the lower edge of the screen). For this region, the screen edge processing circuit 30 adds a motion vector, for example, by referring to a motion vector in a nearby region or assigning a detected pan speed.

  Subsequently, generation of an interpolation frame when the reference range of the line of the past frame is moved will be described.

  When the reference range is moved, an area where pixel data of the past frame or the current frame associated with the motion vector does not exist in the line memory is generated at the end of the screen. For this area, the interpolation frame generation circuit 20 interpolates, for example, an area where pixel data of the past frame does not exist with pixel data of the current frame, and an area where no pixel data of the current frame exists interpolates with pixel data of the past frame. Is done.

  However, when the reference range is moved, the spatial position of the pixel generated in the interpolation frame is also moved. Therefore, the interpolation frame generation circuit 20 needs to correct the movement of the position of the pixel of the interpolation frame for the amount of movement of the reference range. Hereinafter, for the sake of simplicity, it is assumed that the motion vector also has only a vertical component.

  7A and 7B are conceptual diagrams showing a procedure for generating an interpolation frame in a conventional image processing apparatus. The figure shows an example in which an interpolation frame is generated at an intermediate position between the current frame and the past frame (the center position between both). In this case, after detecting the motion vector V (FIG. (A)), the interpolation frame generation circuit 50 detects the past frame pixels and the current frame pixels associated with the motion vector V (FIG. (A)). , That is, a pixel of the interpolation frame is generated at a position of V / 2 with reference to the pixel of the past frame ((B) in the figure).

  FIGS. 8A to 8D are conceptual diagrams illustrating an interpolation frame generation procedure in the image processing apparatus according to the present embodiment. When the reference range of the line of the past frame is moved by the movement amount M ((A) in the same figure), the motion vector detection circuit 18 detects a motion vector VM obtained by subtracting the movement amount M from the original motion vector V. (FIG. 5B). When the interpolation frame generation circuit 20 generates an interpolation frame using this motion vector VM, M + (VM) / 2 = with reference to the pixel position of the past frame associated with the motion vector VM. A pixel of an interpolation frame is generated at a position of (V / 2 + M / 2) (when an interpolation frame is generated at an intermediate position as in FIG. 7C and FIG. 7). Since the spatial position of the pixel of the interpolation frame has been moved by the amount of movement of the reference range in FIG. 8A, finally, the pixel of the interpolation frame is moved by −M / 2 (to move the reference range). On the other hand, it is moved in the reverse direction by M / 2) to correct to the position of V / 2 ((D) in the figure).

  In the correction shown in FIG. 4D, generally, the correction amount is calculated from the movement amount M shown in FIG. 2A and the phase α of the interpolation frame (temporal interpolation position from the past frame). Move the entire image of the interpolation frame. When the phase of the interpolation frame to be generated is α (0 ≦ α ≦ 1, α = 0.5 in the case of generating an interpolation frame at an intermediate position), it is made up of pixels generated in the interpolation frame-(1-α ) Just move M.

  The process of moving the entire image of the interpolation frame can be realized by moving the timing at which the interpolation process is started by the interpolation frame generation circuit 20, that is, the read timing of the pixel data from the line memory 16. That is, as in the case of detecting a motion vector, the movement of the spatial position in the vertical direction of the image can be realized only by the movement of the timing of reading of the line memory 16.

  FIG. 9 is a timing chart showing the processing timing of motion vector detection and interpolation frame generation when the reading timing of the past frame line is delayed, and FIG. 10 is when the reading timing of the past frame line is advanced. . In these timing charts, Line # i and Line ′ # i (1 ≦ i ≦ N) are the i-th line of the current frame and the past frame, V # i is the motion vector of the i-th line of the interpolation frame, and C # i indicates the i-th line of the interpolation frame.

  In practice, detection of motion vectors and generation of interpolated frames are started after a predetermined number of storage lines of pixel data are stored in the past frame line memory. For this reason, the generation timing of the motion vector V and the corrected motion vector Vc and the generation timing of the interpolation frame are delayed compared to the input timing of the current frame line and the reading timing of the previous frame line. 9 and 10, this timing delay is not shown. In addition, motion vector detection and interpolation frame generation are performed in units of pixels. Alternatively, it is possible to generate a motion vector for each block composed of a plurality of pixels.

  As shown in the timing chart of FIG. 9, when the line read timing of the past frame is delayed by two lines with respect to the input timing of the line of the current frame, the first two lines Line # of the current frame are displayed at the upper end of the screen. The line of the past frame corresponding to 1, Line # 2 does not exist, and the line of the current frame corresponding to the last two lines Line ′ # N−1, Line ′ # N of the past frame does not exist at the lower end.

  The actual motion vector detection is not performed using only the pixel data of the line of the current frame and the pixel data of the line of the past frame input at the same timing, but the pixels of a plurality of lines stored in the line memory. Use data. For this reason, depending on the vertical component of the motion vector remaining after the reduction of the vertical component by the first movement amount M1 even in the region where the corresponding current frame or past frame line does not exist at the upper and lower ends of the screen, In some cases, vector detection is possible. Even considering this, there is a high possibility that the motion vector cannot be detected. Therefore, in the upper and lower end regions, as described above, the screen end processing circuit 30 refers to the motion vector in the temporally or spatially close region or assigns the detected pan speed. Thus, correction for adding the motion vector of this region is performed.

  The interpolation frame generation circuit 20 generates an interpolation frame based on the corrected motion vector. At this time, by delaying the interpolation processing start timing, that is, the pixel data reading timing from the line memory 16 by one line, the reading timing of the previous frame line is delayed by two lines with respect to the input timing of the current frame line. The amount of movement of the interpolation frame position is corrected.

  This will be described with reference to FIGS. In these figures, the number of storage lines in the line memory 16 is N.

  As shown in FIG. 11, the lines Line ′ # 1, Line ′ # 2,..., Line ′ # N of the past frame are lower than the lines Line # 1, Line # 2,. Suppose that it has moved by two lines in the direction. Then, it is assumed that the vertical component of the motion vector is equal to the movement amount M, that is, the vertical component of the motion vector after the reduction of the vertical component is zero.

  In this case, as shown in FIG. 12, the lines Line # 3, Line # 4,..., Line # N-1, Line # N of the current frame and the lines Line ′ # 1, Line ′ # 2,. , Line ′ # N-3 and Line ′ # N-2 respectively correspond to the line of the past frame corresponding to the line Line # 1 and Line # 2 of the current frame and the line Line ′ # N− of the past frame. There is no line of the current frame corresponding to 1, Line ′ # N.

  Therefore, when the vertical direction component of the motion vector after the reduction of the vertical direction component is 0, the motion vector detection circuit 18 uses the lines Line ′ # 1, Line ′ # 2,. Motion vectors V # 2, V # 3,..., V # N− between Line ′ # N-2 and the current frame lines Line # 3, Line # 4,. 2, V # N−1 can be detected. However, the motion vectors V # 1 and V # N necessary to generate the interpolation frame lines C # 1 and C # N cannot be directly detected.

  Therefore, the screen edge processing circuit 30 performs correction for adding the motion vectors V # 1 and V # N with reference to surrounding motion vectors or the predicted pan speed, for example. Then, the interpolation frame generation circuit 20 generates an interpolation frame using the corrected motion vector. As illustrated in FIG. 13, lines C # 1 and C # N of the interpolation frame are generated with reference to Line # 2 of the current frame and Line ′ # N−1 of the past frame, respectively. It can be seen that the generated interpolation frame has the expected result.

  On the other hand, as shown in the timing chart of FIG. 10, when the read timing of the line of the past frame is advanced by two lines with respect to the input timing of the line of the current frame, the first frame of the past frame is displayed at the upper end of the screen. There is no line of the current frame corresponding to the two lines Line ′ # 1 and Line ′ # 2, and the lines of the past frame corresponding to the last two lines Line # N−1 and Line # N of the current frame are present at the lower end. not exist.

  Similarly, since the motion vector detection circuit 18 cannot detect a motion vector in this area, the screen edge processing circuit 30 performs correction for adding the motion vector in this area.

  The interpolation frame generation circuit 20 generates an interpolation frame based on the corrected motion vector. At this time, by interpolating the interpolation processing start timing, that is, the timing for reading out the pixel data from the line memory 16 by one line, the reading timing of the line in the past frame is equivalent to two lines with respect to the input timing of the line in the current frame. The amount of movement of the interpolated frame position that has been accelerated is corrected.

  Next, the operation of the image processing apparatus 10 will be described.

  The motion vector detection circuit 18 detects a motion vector V-M2. However, if the pan detection circuit 28 detects the pan speed using the motion vector V-M2, the original correct pan speed cannot be detected.

  Therefore, first, the correction circuit 26 corrects the motion vector V-M2 detected by the motion vector detection circuit 18 to the original correct motion vector V.

  Subsequently, based on the motion vector V corrected by the correction circuit 26, the pan detection circuit 28 detects whether or not the video is a panning video. If the panning video is a panning video, the panning speed is further detected. The pan speed obtained here is a predicted value of the pan speed between the past frame and the current frame.

  Then, the first movement amount M1 is determined by the reference range determination circuit 22 from the pan speed detected by the pan detection circuit 28, and the second movement amount M2 corresponding to the first movement amount M1 is determined by the timing controller 24. Then, a third movement amount M3 corresponding to the second movement amount M2 is calculated.

  The frame memory 12 stores the past frame pixel data of the past frame immediately before the current frame (input image) under the control of the memory controller 14.

  The current frame pixel data is received in the order of the lines under the control of the memory controller 14, and the current frame pixel data is stored in the current frame line memory. The past frame pixel data stored in the frame memory 12 is read from the frame memory 12 in the line order, and the past frame pixel data corresponding to the line of the current frame stored in the current frame line memory is stored in the past frame. Stored in the line memory.

  FIG. 14 is a conceptual diagram showing an interpolation frame generation process when there is a downward movement between a past frame and a current frame. The figure shows the input timing of the current frame pixel data of each line of the current frame as the reference timing, the read timing of the past frame pixel data of the corresponding line of the past frame, the detection timing of the motion vector by the motion vector detection circuit 18, The timing at which the screen edge processing circuit adds a motion vector and the generation timing of interpolation frame pixel data for the corresponding line of the interpolation frame are shown. As shown in FIG. 14, the line of the past frame is read from the frame memory 12 at a timing delayed by the second movement amount M2 with respect to the input timing of the line of the current frame. In other words, the past frame line memory stores the past frame of the line in the range moved backward (down) by the second movement amount M2 with respect to the range of the line of the current frame stored in the current frame line memory. Pixel data is stored.

  When pixel data of a predetermined number of storage lines is stored in the current frame line memory and the past frame line memory, the current frame pixel stored in the current frame line memory, which is image information, by the motion vector detection circuit 18 Based on the data and past frame pixel data stored in the past frame line memory, a motion vector between the past frame and the current frame is detected.

  Since the motion vector detected by the motion vector detection circuit 18 is read from the frame memory 12 of the line of the past frame by the second movement amount M2 with respect to the input timing of the line of the current frame, The vertical component is reduced by the second movement amount M2. As in the case of FIG. 16, in practice, the generation timing of the motion vector V and the corrected motion vector Vc and the generation timing of the interpolation frame are the same as the input timing of the current frame line and the reading timing of the previous frame line. It is late compared. In FIG. 14, this delay is not shown.

  Further, the screen edge processing circuit 30 adds the motion vector of the line at the upper and lower ends of the screen to the motion vector V-M2 detected by the motion vector detection circuit 18 based on the third movement amount M3. And a corrected motion vector Vc-M2 is generated. Furthermore, it is also possible to perform processing such as allocation processing for allocating the detected motion vector to the pixels of the interpolation frame and interpolation processing for compensating for the motion vector of the pixel of the interpolation frame for which no motion vector could be allocated.

  Finally, based on the corrected vector Vc-M2, the current frame pixel data stored in the current frame line memory, and the past frame pixel data stored in the past frame line memory by the interpolation frame generation circuit 20. Interpolated frame pixel data of an interpolated frame between the past frame and the current frame is generated. The addition of the motion vectors at the upper and lower ends of the screen by the screen edge processing circuit 30 is performed within the same frame period as the motion vector detection by the motion vector detection circuit 18, that is, within the period in which pixel data of the current frame is input. It is also possible. Alternatively, it is also possible to carry out within the next frame period, that is, within the period in which the pixel data of the next frame of the current frame is input. In the latter case, the interpolation frame generation circuit 20 also needs to generate an interpolation frame in the next frame period. In the former case, it is possible to generate an interpolated frame within the same frame period as the motion vector detection, or to perform it in the next frame period.

  When the interpolated frame is generated within the next frame period, the interpolated frame is generated separately from the line memory for storing the current frame and past frame pixel data used by the motion vector detection circuit 18 for motion vector detection. It is preferable to provide a line memory for storing pixel data of a current frame and a past frame for use by the circuit 20 in generating an interpolation frame. In this case, as described above, if the motion vector is not detected for each pixel but for each block made up of a plurality of pixels, the line memory used for motion vector detection includes some pixels. Only pixel data need be stored. Thus, the capacity of the line memory used for motion vector detection can be reduced compared to the line memory storing pixel data of each pixel for use in interpolation frame detection.

  FIG. 17 is a conceptual diagram showing the phase and position of an interpolation frame generated by the image processing apparatus of the present invention. As shown in FIG. 18, assuming that the motion vector between the pixels of the past frame and the pixel of the current frame is V and the phase of the interpolation frame is α, in the conventional image processing apparatus, the pixel of the interpolation frame is the motion vector V Is generated at the position of αV with reference to the position of the pixel of the past frame associated with. On the other hand, in the image processing apparatus 10, as shown in FIG. 17, the image of the interpolation frame is moved by moving the read timing of the past frame pixel data from the frame memory 12 by the second movement amount M2. It is generated at the position of α (V−M2) with reference to the position of the pixel in the subsequent past frame. Therefore, compared with the case of FIG. 18, the pixel of the interpolation frame moves by α (V−M2) + M2−αV = (1−α) M2.

  In response to this, in the interpolation frame generation circuit 20, the start timing of the interpolation processing (timing to start reading out the pixel data used for the interpolation processing from the line memory 16) is the third movement amount M3 = (1-α). Delayed by M2. Thereby, the position of the pixel of the interpolation frame moved by moving the reference range of the line of the past frame by the second movement amount M2 can be returned to the original correct position.

  On the other hand, FIG. 15 is a conceptual diagram illustrating an interpolation frame generation process when there is an upward movement between a past frame and a current frame.

  The operation when there is an upward movement is that the line of the past frame is read from the frame memory 12 at a timing that is advanced by the second movement amount M2 with respect to the input timing of the line of the current frame, and interpolation processing This is the same as the case where there is a downward movement, except that the start timing of the pixel data, that is, the timing to start reading out the pixel data from the line memory 16 used for the interpolation processing is advanced by (1-α) M2. is there.

  Compared to the conventional image processing apparatus 40, the image processing apparatus 10 can perform frame interpolation even for high-speed panning videos without greatly changing the circuit and without greatly increasing the circuit scale. It becomes possible.

The present invention is basically as described above.
Although the present invention has been described in detail above, the present invention is not limited to the above-described embodiment, and it is needless to say that various improvements and modifications may be made without departing from the gist of the present invention.

10, 40 Image processing device 12, 42 Frame memory 14, 44 Memory controller 16, 46 Line memory 18, 48 Motion vector detection circuit 20, 50 Interpolation frame generation circuit 22 Reference range determination circuit 24 Timing controller 26 Correction circuit 28 Pan detection circuit 30 Screen edge processing circuit

Claims (6)

An image processing apparatus that generates an interpolated frame between a plurality of frames each composed of a plurality of horizontal lines each including a plurality of pixels,
Current frame pixel data representing pixel values of pixels of a plurality of lines constituting the current frame is received in the order of the lines, stored in a current frame line memory having a first predetermined storage line number, and a frame Past frame pixel data representing pixel values of pixels of a plurality of lines constituting the past frame immediately before the current frame stored in the memory is read out from the frame memory in the order of the lines, and is stored in a second predetermined storage. A memory control circuit for storing in a past frame line memory having the number of lines;
A current frame pixel data stored in the current frame line memory and a past frame pixel stored in the past frame line memory, which receives a motion vector indicating the motion of the pixel of the interpolation frame and is associated with the motion vector. An interpolation frame generation circuit that generates interpolation frame pixel data representing pixel values of pixels of each line of the interpolation frame based on the data,
The interpolation frame generation circuit reduces the vertical component as the motion vector by the first movement amount M1 determined according to the vertical component of the pan speed, which is the overall motion speed of the interpolation frame. A direction vector reduced motion vector is received as the motion vector;
The memory control circuit is configured to determine the past frame in the past frame line memory, which is determined relative to the timing of storing the current frame pixel data of each line constituting the current frame in the current frame line memory. From the standard timing when the first movement amount M1 is zero, the second movement amount M2 corresponding to the first movement amount M1 is stored. An image processing apparatus characterized by only moving. The memory control circuit includes:
Current frame pixel data of a plurality of lines constituting the current frame is stored in the current frame line memory for a predetermined time for each line from the first start timing, and a plurality of data constituting the past frame is stored. The past frame pixel data of the current line is stored in the past frame line memory taking the predetermined time for each line from the second start timing determined relatively to the first start timing. And
The image processing apparatus according to claim 1, wherein the second start timing is moved by the second movement amount M2 from a standard start timing when the first movement amount M1 is zero. The interpolation frame generation circuit outputs the interpolation frame pixel data of a plurality of lines constituting the interpolation frame for each line from a third start timing that is determined relative to the first start timing. It takes time to generate, and
Interpolation frame generation circuit control for moving the third start timing by a movement amount M3 determined according to the second movement amount M2 from a standard start timing when the first movement amount M1 is zero. The image processing apparatus according to claim 2, further comprising a unit.   The interpolated frame is an interpolated frame at a time position relatively separated from the past frame by α (0 <α <1) and 1-α from the current frame, and the interpolated frame generation circuit control unit moves the moving frame. 4. The image processing apparatus according to claim 3, wherein the amount M3 is moved so that M3 = (1−α) M2.   A pan speed calculation that calculates the pan speed using the pan speed of the past frame obtained based on a motion vector distribution between a pixel of the frame immediately before the past frame and a pixel of the past frame. The image processing apparatus according to claim 1, further comprising a circuit.   Using the current frame pixel data stored in the current frame line memory and the past frame pixel data stored in the past frame line memory, the vertical component is reduced by the first movement amount M1, A motion vector detection circuit that detects a motion vector between a pixel in the past frame and a pixel in the current frame and supplies the motion vector to the interpolation frame generation circuit as the motion vector with the reduced vertical component; The image processing apparatus according to claim 1.

Images