JP2010154300A - Image processing device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an FRC (frame rate converter) capable of obtaining a satisfactory viewing and listening feeling by mutually complementing the defect of a vector type FRC and a frequency dispersion type FRC. <P>SOLUTION: An image processing device comprises: a vector interpolation image creating means 4 for creating a vector interpolation image 5; an addition interpolation image creating means 6 for creating the addition interpolation image 7; an interpolation method determining means 8 for extracting the features of the image input to its own and determining a mixing rate of the vector interpolation image 5 and the addition interpolation image 7 every area in the interpolation image on the basis of the features; an interpolation image mixing means 10 for mixing the vector interpolation image 5 and the addition interpolation image 7 every area on the basis of the mixing rate determined by the interpolation method determining means 8 to create the mixed interpolation image 11; a gradated image creating means 12 for gradating the mixed interpolation image 11; and an emphasized image creating means 14 for putting emphasis on an input image 2, an input image 3, or on both of the input image 2 and the input image 3. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to an improvement in image quality in an image display device such as a liquid crystal display device.

  Known devices for displaying moving images include cathode ray tubes and liquid crystal displays. In these display devices, pixels arranged on the display device emit light to display an image. When displaying a moving image, it is necessary to periodically update the image. In this case, the light emission amount of the pixel is updated to the light emission amount of the next image at each image update timing.

  A cathode ray tube scans an electron beam to emit light from a phosphor to perform display. In this method, the light emission of the pixel is in an impulse shape. Such a display device is called an impulse-type display device.

  In a liquid crystal display, display is performed by applying a voltage to the liquid crystal and controlling the light transmittance. Since the applied voltage is held until the next update, the pixels emit light with the same luminance until the next image display timing. Such a display device is called a hold-type display device.

  According to Patent Document 1, Patent Document 2, and the like, a problem has been pointed out that in a hold-type display device, a blur phenomenon occurs due to an observer's eyes following the movement of a moving object when displaying a moving image.

  In order to solve this problem, a technique has been proposed in which a blurring phenomenon is reduced by generating a new image between temporally consecutive images and increasing the update frequency of the displayed image. Such a technique is called FRC (Frame Rate Converter). There are several methods for FRC. The following description describes two FRC techniques related to the present invention.

  The first FRC technique is a method using a motion vector represented by Patent Document 1. In the following description, this FRC technique is called a vector type FRC. FIG. 13 is a plan view showing the basic concept of the vector type FRC.

  Assume that there is an image 101 and an image 102 that is the next image in time with respect to the image 101. When these images are continuously displayed, it appears that the circle 103 in each image moves in an obliquely upward direction (upper right direction). From these images, it is detected in which direction the circle 103 has moved by image processing. The detected motion is indicated by a vector 104. Such information is called a motion vector.

  If the motion vector 104 is known, an image 106 that is an intermediate image between the image 101 and the image 102 can be obtained by drawing a circle 105 at a position where the length of the motion vector is halved.

  The unit of time in FIG. 13 is a display time of one frame. For example, since Japanese TV broadcasting is broadcast at 59.94 Hz, the display time of one frame is 1 / 59.94 seconds = about 16.7 milliseconds.

  In practice, the vector type FRC often performs processing in units of blocks. This is shown in FIG. FIG. 14A shows a state where the image 201 is divided into 13 × 10 = total 130 rectangular blocks. The image 201 is the same as the image 101 in FIG. 13, but is divided into minute rectangular blocks (rectangular regions).

  Now, pay attention to the rectangular block 202 here. Where the rectangular block 202 moves in the next image is searched by pattern matching. An image 203 in FIG. 14B is the same as the image 102 in FIG. 13, but the image 201 is synthesized.

  As a result of pattern matching, it can be seen that the rectangular block 202 is substantially equal to the rectangular block 204. Here, the motion vector 205 for the rectangular block 202 is obtained.

  Next, attention is focused on FIG. The contents of the rectangular block 202 are copied from the position of the rectangular block 202 to the rectangular block 206 at a position moved by the motion vector 207 obtained by halving the size of the motion vector 205. An intermediate image can be obtained by repeating the same process for other blocks. This is a method of obtaining an intermediate image using vector type FRC.

  FIG. 15 is a block diagram showing the configuration of the vector type FRC. The input image 302 is a frame image that is temporally next to the input image 301. The input image 301 and the input image 302 are input to the vector interpolation image generation unit 303. The vector interpolated image generation unit 303 includes a motion vector detection unit 304 and a block copy unit 305.

  The input image 301 and the input image 302 input to the vector interpolation image generation unit 303 are input to a motion vector detection unit 304 that detects a motion vector. That is, the motion vector detection unit 304 performs an operation of detecting the motion vector 205 shown in FIG. 14B, for example.

  After the motion vector detection unit 304 detects the motion vector, the input image 301 and the motion vector are input to the block copy unit 305. The block copy means 305 performs block unit copying based on the motion vector, and generates a vector interpolated image 306 that is an intermediate image by vector type FRC. That is, the block copy unit 305 performs an operation for obtaining an intermediate image, for example, as shown in FIG.

  Thereafter, when displayed on the display device, the input image 301 is output as the output image 307, the vector interpolation image 306 is output as the output image 308, and the input image 302 is output as the output image 309 in this order.

  The second FRC technique is the technique of Patent Document 2. In the following description, this FRC technique is referred to as a frequency dispersion type FRC. FIG. 16 is a plan view showing a basic concept of the frequency dispersion type FRC.

  Assume that there is an input image 401 and an input image 402 that is the next image in time of the input image 401. An image in which the input image 401 is emphasized is output as an output image 404, and an image in which the input image 402 is emphasized is output as an output image 406.

  Also, an addition interpolated image 403 obtained by adding the input image 401 and the input image 402 is created, and an image obtained by blurring this is output as an output image 405. Each output image is displayed in the order of an output image 404, an output image 405, and an output image 406. According to Patent Document 2, such a display can reduce blurring due to human visual characteristics.

  FIG. 17 is a block diagram showing a configuration of a frequency dispersion type FRC. The input image 501 and the input image 502 are input to an addition interpolation image generation unit 503 that performs addition for each pixel and processed, whereby an addition interpolation image 504 is output.

  The added interpolated image 504 is blurred by the blurred image generating unit 505 to become an output image 508. The input image 501 and the input image 502 are input to an enhanced image generating unit 506 that performs image enhancement, and an output image 507 and an output image 509 are obtained.

  Thereafter, when the image is displayed by the display device, the output image 507, the output image 508, and the output image 509 are output in this order.

Note that the unit of time in FIG. 16 is a display time of one frame as in FIG.
Patent No. 3295437 (patent on April 5, 2002) International Publication WO 2007/052441 (published on September 10, 2007)

  In the vector type FRC, there is a possibility that a high-quality intermediate image cannot be generated and the viewing feeling is greatly impaired. The cause is a motion vector detection error. The following factors tend to cause motion vector detection errors.

  As a first factor, there is a case where noise on the screen is large. It may be affected by camera noise or compression noise. As a second factor, there is a case where a plurality of objects are moving on the screen and they cross each other. As a third factor, there is a case where movement is large. The search range of motion vectors is often limited to speed up the processing. A motion larger than the search range cannot be detected. As a fourth factor, there is a case where the movement is other than parallel movement such as enlargement / reduction / rotation. When a motion vector detection error occurs, a phenomenon such as blurring of an image of a moving object occurs.

  In the frequency dispersion type FRC, in the enhancement process, the pixel value (pixel luminance value) of the image after enhancement exceeds the displayable range of the display device, and the image after enhancement as expected may not be obtained. . This is shown in FIG. FIG. 18 is a graph showing changes in luminance in the horizontal direction of an image. In the graph of FIG. 18, the vertical axis of the graph indicates the luminance, and the horizontal axis of the graph indicates the pixel position x.

  Assume that there is an edge 601 in the image as shown in FIG. In FIG. 18B, an image in which the edge 601 is emphasized is shown. The edge 601 is emphasized and becomes an edge emphasized like 602.

  However, as shown in FIG. 18C, consider a case where there are an upper limit 603 and a lower limit 604 in luminance values that can be expressed by the display device. For example, in the lower part, the luminance change should be essentially the same as that of the dotted line portion 605, but the dotted line portion 605 is below the lower limit 604, and such luminance cannot be expressed, and thus is rounded as indicated by reference numeral 606. End up. When such a situation occurs, a theoretically emphasized image cannot be obtained, and blurring of the image occurs around the pixel where the rounding occurs.

  The present invention has been made in view of the above problems, and an object of the present invention is to provide an FRC capable of obtaining a good viewing feeling by mutually complementing the drawbacks of the vector type FRC and the frequency dispersion type FRC. It is.

  In order to solve the above-described problem, an image processing apparatus according to the present invention is an image processing apparatus, and includes a first image that is a frame image input to the image processing apparatus, and a temporal relationship between the first image and the first image. An interpolated image that is an intermediate image between the first image and the second image is generated from the second image that is the next frame image, and further, the first image and the second image are generated. In the image processing apparatus that performs image processing on the image, a motion vector that is information indicating the motion of the image is detected from the first image and the second image, and a vector interpolated image is obtained based on the motion vector. Vector interpolation image generation means for generating, and addition interpolation for generating an addition interpolation image by adding and normalizing pixel values of the respective pixels at the same position of the first image and the second image Image generation means and input to itself An interpolation method determining means for extracting a feature of the image and determining a mixing ratio of the vector interpolation image and the addition interpolation image for each area in the interpolation image based on the feature; and the interpolation Based on the mixing ratio determined by the method determining means, the vector interpolated image and the additive interpolated image are mixed for each area to generate a mixed interpolated image, and the mixed image A blurred image generating means for blurring the interpolated image, and an emphasized image generating means for enhancing the first image, the second image, or both the first image and the second image. It is characterized by providing.

  According to the invention, an appropriate FRC (frame rate conversion) method is determined for each area by the interpolation method determination means, and the vector interpolation image and the addition interpolation image are the interpolation image mixing means. Is mixed at an appropriate mixing ratio. Therefore, it is possible to provide an FRC that can obtain a favorable FRC result and can obtain a good viewing feeling by mutually complementing the defects of the vector type FRC and the frequency dispersion type FRC.

  A feature of the configuration of the image processing apparatus is that the blurred image generation unit performs a blurring process on the mixed interpolated image. The mixed interpolated image includes an area where the vector interpolated image is employed. The blur processing is also performed for this area.

  In the conventional vector type FRC, processing such as blurring and emphasis is not performed on the output of the FRC. However, according to Patent Document 2, there is also proposed a blur reduction method by blurring / emphasizing a vector interpolation image that is an output of a vector type FRC instead of an addition interpolation image.

  For this reason, in the present invention, the first image and the second image emphasized as necessary by performing a blurring process on the entire mixed interpolated image after mixing. Thus, the interpolated image mixing means need not be used, the processing amount of the image processing apparatus is reduced, and the configuration of the image processing apparatus is simplified.

  In the image processing device, the interpolation method determination means includes a high-luminance area and a luminance of both the first image and the second image, the first image, or the second image. A low area may be discriminated, and the ratio of the vector-interpolated image may be made higher in the mixing ratio of the high luminance area and the mixing ratio of the low luminance area.

  The high luminance area and the low luminance area have a high possibility that the pixel value of the image after the enhancement, that is, the luminance value of the pixel exceeds the displayable range of the display device in the image enhancement by the frequency dispersion type FRC. It is an area.

  In such an area, since the image by the frequency dispersion type FRC is blurred, the vector type FRC is more likely to obtain a better result than the frequency dispersion type FRC. Therefore, in the high luminance area and the low luminance area, the ratio of the vector-interpolated image generated by the vector type FRC is set to be the mixing ratio.

  In the image processing device, the interpolation method determination unit includes the first image, the second image, and the first image temporally previous to the first image, the first image, or the second image. A roll telop may be discriminated from the image, and the ratio of the vector-interpolated image may be increased as the mixing ratio of the area where the roll telop is detected.

  In the area where the roll telop is detected, there are many portions where the luminance is extremely high and low, and the luminance difference between pixels tends to be large. Therefore, in image enhancement by frequency dispersion type FRC, there is a high possibility that the pixel value of the image after enhancement exceeds the displayable range of the display device. Therefore, in the area where the roll telop is detected, the vector type FRC is more likely to obtain a better result than the frequency dispersion type FRC. Accordingly, in the area where the roll telop is detected, the ratio of the vector-interpolated image generated by the vector type FRC is set to the mixing ratio.

  In the image processing device, the interpolation method determination unit includes the first image, the second image, and the first image temporally previous to the first image, the first image, or the second image. A still telop may be discriminated from the image, and the ratio of the added interpolated image may be increased in the mixing ratio of the area where the still telop is detected.

  In the vector type FRC, when a stationary telop is superimposed on an image whose background is panned, a motion vector detection error is likely to occur, and there is a high possibility that the image will be blurred. In the frequency dispersion type FRC, no motion vector is detected, and such a problem does not occur. Therefore, in the area where the stationary telop is detected, the mixing ratio is set so that the ratio of the added interpolated image becomes higher.

  In the image processing apparatus, the interpolation method determination unit receives information on the reliability of the motion vector detected for each area from the vector interpolation image generation unit, and in an area where the reliability of the motion vector is low. You may make it the said mixing rate make the ratio of the said addition interpolation image high.

  In the area where the reliability of the motion vector is low, there is a high possibility that a motion vector detection error has occurred, and there is a high possibility that the image will be blurred. In the frequency dispersion type FRC, no motion vector is detected, and such a problem does not occur. Therefore, in the area where the reliability of the motion vector is low, the mixing ratio is set so that the ratio of the added interpolated image becomes high.

  In the image processing apparatus, the interpolation method determination unit may smooth the difference when the mixing ratio of a certain area and the mixing ratio of the surrounding area are large.

  The vector type FRC and the frequency dispersion type FRC are slightly different in the effect of reducing the blur of the output image. Therefore, when the difference between the mixing ratio of a certain area and the surrounding area is large, that is, when the mixing ratio changes abruptly, the change point of the mixing ratio can be visually recognized. End up. Specifically, a phenomenon such as a line appearing in the image may occur, and the viewing experience may be greatly deteriorated. Therefore, the change point is made inconspicuous by making the change of the mixing ratio moderate.

  In the image processing apparatus, the interpolation method determination unit may make the change in the mixing rate moderate when the mixing rate of the current frame image greatly changes with respect to the previous frame image.

  The vector type FRC and the frequency dispersion type FRC are slightly different in the effect of reducing the blur of the output image. Therefore, when the mixing rate of the current frame image changes greatly with respect to the previous frame image, that is, when the mixing rate changes rapidly with time, the change point of the mixing rate can be visually recognized. . Specifically, a phenomenon that the image is flushed occurs, and the viewing feeling may be greatly deteriorated. Therefore, the change point is made inconspicuous by making the change of the mixing ratio moderate.

  As described above, the image processing apparatus of the present invention detects a motion vector, which is information indicating the motion of an image, from the first image and the second image, and generates a vector-interpolated image based on the motion vector. A vector interpolated image generating unit that adds the pixel values of the respective pixels at the same position of the first image and the second image, and generates an added interpolated image by normalizing And generating means for extracting a feature of the image input to itself and, based on the feature, an interpolation for determining a mixing ratio of the vector interpolation image and the addition interpolation image for each area in the interpolation image. Based on the mixing ratio determined by the method determining means and the interpolation method determining means, the vector interpolated image and the additive interpolated image are mixed for each area to generate a mixed interpolated image. The interpolated image mixing means and the mixed interpolated image A blur image generation unit and an enhanced image generation unit that emphasizes the first image, the second image, or both the first image and the second image. is there.

  Therefore, there is an effect of providing an FRC that can obtain a good viewing feeling by mutually complementing the drawbacks of the vector type FRC and the frequency dispersion type FRC.

  An embodiment of the present invention will be described below with reference to Examples 1 to 4 and FIGS.

  FIG. 1 is a block diagram of an image processing apparatus 1 according to the present embodiment. The input image 2 and the input image 3 are input to the image processing apparatus 1 in this order.

  The input image 2 and the input image 3 are input to the vector interpolation image generation unit 4. The vector interpolated image generating means 4 generates a vector interpolated image 5 that is an intermediate image between the input image 2 and the input image 3 by vector type FRC.

  Further, the input image 2 and the input image 3 are input to the addition interpolation image generation unit 6. The added interpolated image generating means 6 adds the pixels at the same position in the input image 2 and the input image 3 to generate a normalized added interpolated image 7.

  Further, the input image 2 and the input image 3 are input to the interpolation method determination unit 8. The interpolation method determination means 8 extracts the features of the image from both the input image 2 and the input image 3, the input image 2, or the input image 3, and regarding each of the plurality of areas (regions) obtained by dividing the image, Whether the vector type FRC processing or the frequency dispersion type FRC processing is good is determined based on the information on the above characteristics, and the mixing ratio 9 of the vector interpolation image and the addition interpolation image in each area is determined. Ask. The method for obtaining the mixing ratio is defined in Example 1 and later.

  In the interpolated image mixing unit 10, the vector interpolated image 5 and the addition interpolated image 7 are mixed according to the mixing rate 9 obtained from the interpolation method determining unit 8 to generate a mixed interpolated image 11.

  The blurred image generation unit 12 blurs the mixed interpolated image 11 by a process such as a spatial filter. An image output from the blurred image generation unit 12 is an output image 13.

  The emphasized image generating unit 14 emphasizes both the input image 2 and the input image 3, the input image 2, or the input image 3. The image output from the emphasized image generation unit 14 is an output image 15 for the input image 2 and an output image 16 for the input image 3. The display order on the display device is the order of the output image 15, the output image 13, and the output image 16.

  In the configuration of the image processing apparatus 1, an appropriate FRC method is determined for each area by the interpolation method determination unit 8, and the vector interpolation image 5 and the addition interpolation image 7 are appropriately converted by the interpolation image mixing unit 10. To provide an FRC that can obtain a good FRC result because it is mixed at a mixing ratio, and can obtain a good viewing feeling by mutually complementing the drawbacks of a vector type FRC and a frequency dispersion type FRC. Is possible.

  Further, the characteristic of the configuration of the image processing apparatus 1 is that the blurred image generation unit 12 performs blurring processing on the mixed interpolated image 11. The mixed interpolated image 11 includes an area where the vector interpolated image 5 is employed. The blur processing is also performed for this area.

  In the conventional vector type FRC, processing such as blurring and emphasis is not performed on the output of the FRC. However, according to Patent Document 2, there is also proposed a blur reduction method by blurring / emphasizing a vector interpolation image that is an output of a vector type FRC instead of an addition interpolation image.

  Therefore, in the present invention, the interpolated image mixing means is applied to the output images 15 and 16 emphasized as necessary by adopting a configuration in which the entire mixed interpolated image 11 after mixing is blurred and processed. 10 is not required, the processing amount of the image processing apparatus 1 is reduced, and the configuration of the image processing apparatus 1 is simplified.

  For reference, FIG. 2 shows image processing in which a vector type FRC and a frequency dispersion type FRC are simply synthesized and mixed by an interpolated image mixing unit with respect to each image after blurring processing and enhancement processing are performed. FIG. 4 is a block diagram of the device 17. Compared with the image processing apparatus 1 of FIG. 1, the configuration is complicated.

  Specifically, the interpolated image mixing unit 22 of the image processing device 17 performs the following processing. First, the input image 2 and the emphasized image 20 are mixed and an output image 23 is output. Next, the vector interpolated image 5 and the addition blurred image 19 are mixed and an output image 24 is output. The added blurred image 19 is an image generated by the blurred image generating means 18 blurring the added interpolated image 7. Finally, the input image 3 and the emphasized image 21 are mixed and an output image 25 is output. The display order on the display device is the order of the output image 23, the output image 24, and the output image 25.

  Returning to FIG. 1, the interpolation method determination means 8 determines an area having a high luminance and an area having a low luminance for both the input image 2 and the input image 3, the input image 2, or the input image 3. The ratio of the vector interpolated image 5 may be increased by increasing the mixing ratio with respect to the area.

  In the high luminance area and the low luminance area, in the image enhancement by the frequency dispersion type FRC, there is a high possibility that the pixel value (pixel luminance value) of the image after the enhancement exceeds the displayable range of the display device. It is an area.

  In such an area, since the image by the frequency dispersion type FRC is blurred, the vector type FRC is more likely to obtain a better result than the frequency dispersion type FRC. Therefore, in the high luminance area and the low luminance area, the mixing ratio is increased to increase the ratio of the vector interpolation image 5 generated by the vector type FRC.

  The interpolation method determination means 8 detects a roll telop area from the input image 2, the input image 3, the image before the input image 2 in time, the input image 2, or the input image 3, and the roll telop area is detected. The mixing ratio may be increased and the ratio of the vector interpolation image 5 may be increased.

  In the area where the roll telop is detected, there are many portions where the luminance is extremely high and low, and the luminance difference between pixels tends to be large. Therefore, in image enhancement by frequency dispersion type FRC, there is a high possibility that the pixel value of the image after enhancement exceeds the displayable range of the display device. Therefore, in the area where the roll telop is detected, the vector type FRC is more likely to obtain a better result than the frequency dispersion type FRC. Therefore, in the area where the roll telop is detected, the mixing ratio is increased to increase the ratio of the vector interpolated image generated by the vector type FRC.

  Further, the interpolation method determining means 8 detects a still telop from the input image 2, the input image 3, and the image temporally previous to the input image 2, the input image 2 or the input image 3, and the still telop is detected. Alternatively, the mixing ratio for the area may be lowered and the ratio of the added interpolated image 7 may be increased.

  In the vector type FRC, when a stationary telop is superimposed on an image whose background is panned, a motion vector detection error is likely to occur, and there is a high possibility that the image will be blurred. In the frequency dispersion type FRC, no motion vector is detected, and such a problem does not occur. Therefore, in the area where the stationary telop is detected, the mixing rate is lowered and the mixing rate of the added interpolated image 7 is increased.

  Further, the interpolation method determination unit 8 receives information on the reliability of the motion vector detected for each area from the vector interpolation image generation unit 4, and reduces the mixing rate in an area where the reliability of the motion vector is low. However, the ratio of the added interpolated image 7 may be increased.

  In the area where the reliability of the motion vector is low, there is a high possibility that a motion vector detection error has occurred, and there is a high possibility that the image will be blurred. In the frequency dispersion type FRC, no motion vector is detected, and such a problem does not occur. Therefore, in the area where the reliability of the motion vector is low, the mixing ratio is lowered and the ratio of the added interpolated image 7 is increased.

  Furthermore, when the difference between the mixing ratio of a certain area and the mixing ratio of the surrounding area is large, the interpolation method determination means 8 may smooth them. That is, the interpolation method determination unit 8 may smoothly adjust the change in the mixing rate so that the mixing rate does not change rapidly in space.

  The vector type FRC and the frequency dispersion type FRC are slightly different in the effect of reducing the blur of the output image. Therefore, when the difference between the mixing ratio of a certain area and the surrounding area is large, that is, when the mixing ratio changes abruptly, the change point of the mixing ratio can be visually recognized. End up. Specifically, a phenomenon such as a line appearing in the image may occur, and the viewing experience may be greatly deteriorated. Therefore, the change point is made inconspicuous by making the change of the mixing ratio moderate.

  Furthermore, the interpolation method determination means 8 may make the change in the mixing rate moderate when the mixing rate of the current frame image changes greatly with respect to the previous frame image. That is, the interpolation method determination unit 8 may smoothly adjust the change in the mixing rate so that the mixing rate does not change rapidly with time.

  The vector type FRC and the frequency dispersion type FRC are slightly different in the effect of reducing the blur of the output image. Therefore, when the mixing rate of the current frame image changes greatly with respect to the previous frame image, that is, when the mixing rate changes rapidly with time, the change point of the mixing rate can be visually recognized. . Specifically, a phenomenon that the image is flushed occurs, and the viewing feeling may be greatly deteriorated. Therefore, the change point is made inconspicuous by making the change of the mixing ratio moderate.

[Example 1]
FIG. 3 is a block diagram of the image processing apparatus 26 according to the first embodiment. An input image 27 is periodically sent to the image processing device 26 at a cycle of about 60 Hz. The input image 27 is first stored in the frame buffer 28.

  When the input image 27 is stored in the frame buffer 28, the contents of the frame buffer 28 include an image enhancement device (enhanced image generation unit) 29, a vector type FRC device (vector interpolation image generation unit) 30, and an image addition device (in addition) Insertion image generation means) 31 and interpolation method determination device (interpolation method determination means) 32 read, and each device performs its own processing.

  The frame buffer 33 stores a frame image immediately before the image stored in the frame buffer 28. The vector type FRC device 30, the image addition device 31, and the interpolation method determination device 32 read the image of the frame buffer 33 in parallel with the reading from the frame buffer 28. The frame buffer 33 stores an image one frame before the input image 27 currently being processed.

  The image enhancement device 29 performs image enhancement processing using a spatial filter on the image stored in the frame buffer 28, and stores the processed enhanced image in the frame buffer 34.

  The vector type FRC device 30 is a device that obtains an intermediate image between two input images by a vector type FRC. The vector type FRC device 30 obtains a motion vector in the image from the image read from the frame buffer 28 and the image read from the frame buffer 33, and generates a vector interpolated image, that is, the intermediate image based on the motion vector. Then, the vector interpolated image is stored in the frame buffer 35.

  The image adder 31 adds the image read from the frame buffer 28 and the image read from the frame buffer 33 together. That is, the pixel values of the pixels at the same position are added and divided by 2 (normalized) to obtain the pixel value of the added interpolated image to be output. The added interpolated image is stored in the frame buffer 36.

  The interpolation method determination device 32 obtains a mixing ratio that is an integer value having a range of 0 to 16. The mixing ratio is a ratio at which the added interpolated image output from the image adding device 31 and the vector interpolated image output from the vector type FRC device 30 are mixed.

  When the mixing ratio is 0, it means that an addition interpolated image is adopted. As the mixing ratio increases, the ratio of the vector interpolation image increases. It means to adopt an inset image. The mixing ratio is obtained at a rate of one for each pixel and is held in the interpolation method determination device 32.

  Here, the flowchart which shows the process of the interpolation method determination apparatus 32 is shown in FIG. In the following description, the operation of the interpolation method determination device 32 will be described with reference to FIG.

  First, the interpolation method determination device 32 reads the input image 37 from the frame buffer 28 (step S1). Next, the interpolation method determination device 32 reads the input image 38 from the frame buffer 33 (step S2).

  Next, the interpolation method determination device 32 investigates the luminance of each pixel of the input image 37 read from the frame buffer 28 and the input image 38 read from the frame buffer 33. The luminance has a value from 0 to 255. Temporary mixing ratios are calculated according to the luminance values so as to correspond in Table 1 shown below (steps S3 and S4). When the mixing rate is different between the image of the frame buffer 28 and the image of the frame buffer 33, the larger value is adopted as the mixing rate (steps S5 to S7). The calculated mixing ratio is temporarily stored.

  Although the above-described mixing ratio may be used as it is, the portion with high luminance and the portion with low luminance may be static telops. In the vector type FRC, a motion vector detection error is likely to occur around a stationary telop, and image blurring is likely to occur. For this reason, processing is performed to reduce the mixing ratio of the portion that seems to be a static telop.

  Various known techniques can be used as a method for detecting a stationary telop. For example, the input images 37 and 38 are divided into 8 × 8 rectangular areas (step S8), and the following checks are performed on each area.

  First, in each pixel in the area, it is checked whether the sum of the difference between the pixel value at the time of the input image 37 and the pixel value at the time of the input image 38 is equal to or less than a threshold value (S9). Secondly, it is checked whether the total value of the mixing ratios existing in the area is equal to or greater than a threshold (S10). Third, it is checked whether the sum of squares of the difference between the pixel value of each pixel in the area and the pixel value of the adjacent pixel is greater than or equal to a threshold value (S11).

  Since an area that satisfies all of the above conditions may be a static telop, 12 is subtracted from the mixture ratio of the pixels corresponding to the area (S12). If the mixing ratio becomes a negative value as a result of subtracting 12, the mixing ratio is rounded to zero.

  Returning to FIG. 3, the mixing rate obtained by the interpolation method determination device 32 as described above is sent to the image mixing device (interpolated image mixing means) 39. The image mixing device 39 reads the vector interpolated image from the frame buffer 35 and the addition interpolated image from the frame buffer 36, and generates a mixed interpolated image according to the mixing rate output from the interpolation method determining device 32. The mixed interpolated image is generated according to the mixing ratio obtained for each pixel. The generated mixed interpolated image is recorded in the frame buffer 40. The pixel value of each pixel in the mixed interpolated image is obtained by the following equation (1).

Pixel value of mixed interpolation image = [(mixing rate) × pixel value of vector interpolation image + {16− (mixing rate)} × pixel value of addition interpolation image] ÷ 16 (1)
The blurred image generation device 41 reads the mixed interpolated image from the frame buffer 40, generates a blurred image using a spatial filter, and stores the blurred image in the frame buffer 42.

  The image output device 43 first reads an emphasized image from the frame buffer 34 and outputs it to the liquid crystal panel 44. After 1/120 second, the blurred image is read from the frame buffer 42 and output to the liquid crystal panel 44.

  When the above processing is completed, the image stored in the frame buffer 28 is transferred to the frame buffer 33. After 1/60 seconds from the previous image input, a new image is input to the image processing device 26, stored in the frame buffer 28, and the same processing is repeated.

[Example 2]
The configuration of the image processing device 45 of the second embodiment is the same as the configuration of the image processing device 26 of the first embodiment. However, the operation of the interpolation method determination device 32 is different, and the vector FRC device 30 has a roll telop area detection function.

  In the second embodiment, the vector type FRC device 30 detects a roll telop area. Various known techniques can be used for detecting the telop area. For example, an area where motion vectors having a specific speed (vector length) are concentrated is detected as a roll telop area, and the roll telop area often scrolls in the horizontal direction. Judge whether there are many motion vectors with the same speed (vector length) for each band area, and roll telop in the horizontal band area with many motion vectors with the same speed (vector length) It may be detected as a region.

  The detection result of the roll telop area by the vector type FRC device 30 is sent to the interpolation method determination device 32. In the interpolation method determination device 32. The mixing ratio is obtained based on the detection result of the roll telop area. The mixing ratio is an integer value having a range of 0 to 16, and has the same meaning as the mixing ratio described in the first embodiment.

  The concept of the operation of the interpolation method determination device 32 according to the second embodiment will be described with reference to the flowchart of FIG. 5A and the diagram showing the allocation of the mixing ratio to the screens of FIGS. 5B to 5D. explain.

  The vector type FRC device 30 determines the roll telop area 47 of the input image 46 from the motion vector distribution or the like (step S21). The mixing ratio 16 is assigned to the area 47 determined as the roll telop area, and the mixing ratio 0 is assigned to the other non-roll telop areas 48 (steps S22 to S24). Finally, LPF (LowPassFilter) is applied to the mixing ratio (step S25). As a result, blur occurs at the boundary portion 49 of the mixing ratio, and the boundary between the roll telop area 47 and the non-roll telop area 48 becomes gentle. The mixing ratio calculated in this way is sent to the image mixing device 39. The subsequent processing of the image processing device 45 is the same as that of the image processing device 26 of the first embodiment.

Example 3
The configuration of the image processing apparatus 50 according to the third embodiment is the same as that of the image processing apparatus 26 according to the first embodiment. However, the operation of the interpolation method determination device 32 is different, and the vector type FRC device 30 has a function of calculating and outputting the reliability of the motion vector.

  The reliability of the motion vector can be calculated by various methods. Here, DFD (Displaced Frame Difference) is used.

  FIG. 6 is a diagram showing a DFD calculation process based on FIG. 2 showing the concept of the vector type FRC. The input image 51 is input to the image processing device 50 at time t, the next input image 52 is input to the image processing device 50 at time t + 1, and the block 53 of the input image 51 goes to block 54 in the input image 52. It is assumed that the motion vector 55 is detected when moving.

  At this time, the enlarged block 53 is a block enlarged view 56, and the block 54 is enlarged is a block enlarged view 57. FIG.

  In the enlarged block diagrams 56 and 57, a small rectangle indicated by reference numeral 58 or reference numeral 59 indicates one pixel. The DFD calculation unit 60 takes the absolute value of the difference between the pixel values of the pixels at the same position in the two blocks, for example, the pixel 58 and the pixel 59. The block ABS in the DFD calculation unit 60 takes the absolute value of the input pixel value.

  Similarly, the DFD calculation unit 60 calculates the difference between each pixel in the block and obtains the sum of their absolute values. This is DFD.

  When the DFD is large, there is a high possibility that the movement is not accurately taken. That is, when the DFD is large, it is considered that the reliability of the motion vector 55 is low, and the DFD can be used as one index indicating the reliability of the motion vector 55. In this case, it can be said that the smaller the DFD, the more reliable.

  The DFD obtained for each block in the screen in this way is sent to the interpolation method determination device 32. The interpolation method determination device 32 obtains the mixing ratio based on the reliability of the motion vector. The mixing ratio is an integer value having a range of 0 to 16, and has the same meaning as the mixing ratio described in the first embodiment.

  Next, the operation of the interpolation method determination device 32 in the third embodiment will be described. FIG. 7 is a flowchart illustrating the process of the interpolation method determination device 32 according to the third embodiment, and FIG. 8 is a diagram illustrating the process of determining the mixing ratio according to the third embodiment. The operation of the interpolation method determination device 32 in the third embodiment will be described with reference to these drawings.

  In FIG. 8, a small rectangle represents one block, and the mixing ratio is lower as the block color is blacker, and the mixing ratio is higher as the block color is whiter. The blackest block 61 represents a block with a mixing ratio of 0, and the whitest block 62 represents a block with a mixing ratio of 16. In the third embodiment, it is assumed that the mixing rate has one mixing rate for each block.

  The interpolation method determination device 32 includes a mixing rate buffer 63 that stores a mixing rate for one screen. When the image processing device 50 is powered on, the interpolation method determination device 32 initializes the mixing rate of the mixing rate buffer 63 with 16 (step S31).

  The processing in step S31 in FIG. 7 corresponds to the screen 64 in FIG. After step S32, the processing of steps S33 to S44 is repeated for each input image. When an image is input and vector type FRC is executed, a DFD is calculated for each block (step S33).

  Next, it is determined whether the DFD obtained for each block in steps S35 to S37 after step S34 is higher or lower than a threshold value (step S35), and the provisional mixing ratio is obtained from the result.

  When the DFD is higher than the threshold value, it is considered that the reliability of the motion vector is low, so the temporary mixing ratio is set to 0 in order to employ the frequency dispersion type FRC (step S36). If the DFD is lower than the threshold value, the temporary mixing ratio is set to 16 in order to adopt the vector type FRC (step S37).

  The processing in steps S36 and S37 in FIG. 7 corresponds to the screen 65 in FIG. When the temporary mixing ratio is obtained for all the blocks, next, the boundary is blurred by applying LPF to the temporary mixing ratio (step S38). The processing in step S38 in FIG. 7 corresponds to the screen 66 in FIG. By the processing in step S38, the boundary portion of the mixing ratio is less noticeable.

  In step S39 and subsequent steps, the processes in steps S40 to S44 are repeated for each pixel. In step S40, the mixing ratio of the previous frame image stored in the mixing ratio buffer 63 is compared with the temporary mixing ratio, and the lower mixing ratio is adopted as the new mixing ratio (steps S41 and S42). The processing in steps S40 to S42 in FIG. 7 corresponds to the screen 67 in FIG.

  The new mixing rate is output to the image mixing device 39 (step S43), and a mixed inner layer image is created based on the value of the new mixing rate.

  Finally, the new mixing ratio is incremented by 1 in all areas (step S44) and stored in the mixing ratio buffer 63. By incrementing the new mixing ratio by 1 in the entire region, it is possible to suppress the rapid change in the mixing ratio in the time direction only in one direction, and the occurrence of flushing can be significantly suppressed. The processing in step S44 in FIG. 7 corresponds to the screen 68 in FIG. Also for the subsequent frame images, the processing of steps S33 to S44 shown in the flowchart of FIG. 7 is repeated (screens 69 to 72). The subsequent processing of the image processing apparatus 50 is the same as that of the image processing apparatus 26 of the first embodiment.

Example 4
FIG. 9 is a block diagram of the image processing apparatus 73 according to the fourth embodiment. The mixed interpolated image generation device 74 is a combination of the members 28 to 31, 33 to 36, and 39 to 43 shown in FIG. 3, and includes three interpolation method determination devices.

  The interpolation method determination device 75 is the method of the first embodiment, the interpolation method determination device 76 is the method of the second embodiment, and the interpolation method determination device 77 is the interpolation method determination device of the method of the third embodiment. These interpolation method determination devices 75 to 77 do not operate at the same time, and any one of them operates. Which one operates is determined by an interpolation method determination signal 79 output from the control CPU 78 to the mixed interpolation image generation device 74.

  The control CPU 78 receives a user operation from the remote control or the button 80. The user can select an image quality such as “standard”, “sports”, or “dynamic”, and the control CPU 78 switches image processing characteristics according to the image quality selected by the user. The characteristics include the interpolation method in addition to the contrast, brightness, and hue of the image.

  The image adjustment device 81 is a device that adjusts the contrast, brightness, and hue of the input image 82. When the user selects image quality, the control CPU 78 outputs an image adjustment signal 83 to the image adjustment device 81, and operates the contrast, brightness, hue, etc. of the image.

  When the user selects the image quality, the control CPU 78 outputs the image adjustment signal 83 to the image adjustment device 81, and at the same time, switches the interpolation method determination devices 75 to 77 according to the selected image quality as shown below.

  When the user selects “standard”, the control CPU 78 outputs an interpolation method determination signal 79 so as to operate the interpolation method determination device 77.

  When the user selects “sports”, the control CPU 78 outputs an interpolation method determination signal 79 so as to operate the interpolation method determination device 76. This is because in the case of a general sports video, the motion in the screen is large, and in the vector type FRC, a motion vector detection error is likely to occur. 76 is operated.

  When the user selects “dynamic”, the control CPU 78 outputs an interpolation method determination signal 79 so as to operate the interpolation method determination device 75 and simultaneously outputs an image adjustment signal 83 to the image adjustment device 81. To do. As a result, the contrast of the input image 82 is enlarged to make a flashy picture. At this time, since the high luminance portion and the low luminance portion are more likely to appear than other image quality, the interpolation method determination device 75 that performs processing on the high luminance portion and the low luminance portion is operated.

[Process to blur the image]
In this embodiment, to blur an image means to smooth and smooth the image by applying a low pass filter (LPF) to the image. As processing for blurring an image in the present embodiment, for example, processing shown in FIG. 10 is performed.

In FIG. 10, attention is paid to a certain pixel (pixel) 91 constituting the image 90. When the x coordinate of the pixel 91 of interest is n and the y coordinate is m, the pixel value of the pixel 91 is represented by P (n, m). Now, assuming that the pixel value has changed from P (n, m) to Q (n, m) after performing the process of blurring the image with the LPF having the table 92, the pixel value Q (n, m) is It is calculated by the following equation (2). When the calculation according to the equation (2) is performed on all the pixels constituting the image, the high-frequency component of the image is smoothed and the image looks blurred.
Q (n, m) = {P (n-1, m-1) + P (n + 1, m-1) + P (n-1, m + 1) + P (n + 1, m + 1) + 2 * P (n, m-1) + 2 × P (n−1, m) + 2 × P (n + 1, m) + 2 × P (n, m + 1) + 4 × P (n, m)} / 16 (2)
[Process to emphasize images]
In this embodiment, emphasizing an image means emphasizing the contour of the image as opposed to smoothing.

Similar to the process of blurring the image, attention is paid to a certain pixel 91 constituting the image 90 in FIG. When the x coordinate of the pixel 91 of interest is n and the y coordinate is m, the pixel value of the pixel 91 is represented by P (n, m). Now, assuming that the pixel value after performing the process of blurring the image with the LPF is Q (n, m), and the pixel value after performing the emphasis process is R (n, m), the pixel value R ( n, m) is expressed by the following equation (3).
R (n, m) = P (n, m)-{Q {n, m} -P (n, m)} (3)
Alternatively, the image can be emphasized by replacing the table 92 of FIG. 10 with the table 93 of FIG.

  FIG. 12 shows a pixel value 95 when the process of blurring the pixel value 94 of the image with the edge is indicated by reference numeral 95, and when the process of emphasizing the pixel value 94 of the image with the edge is performed. This is a graph showing the pixel value of 96.

  Note that each of the processes shown in [Process for blurring an image] and [Process for emphasizing an image] is merely an example, and the process of blurring an image and the process of emphasizing an image may be performed by other methods. Good.

  Note that the unit of time in the present embodiment is a display time of one frame. For example, since Japanese TV broadcasting is broadcast at 59.94 Hz, the display time of one frame is 1 / 59.94 seconds = about 16.7 milliseconds.

  The image processing apparatus of the present invention can provide an FRC that can obtain a good viewing feeling by mutually complementing the drawbacks of the vector type FRC and the frequency dispersion type FRC, and thus can be suitably used for a liquid crystal display or the like. .

1 is a block diagram of an image processing apparatus according to an embodiment of the present invention. It is a block diagram of the image processing apparatus which synthesize | combines a vector type | mold FRC and a frequency dispersion type | mold FRC simply, and performs the mixing by an interpolation image mixing means with respect to each image after performing a blurring process and an emphasis process. 1 is a block diagram of an image processing apparatus according to an embodiment of the present invention. It is a flowchart which shows the process of the interpolation system determination apparatus which concerns on the Example of this invention. (A) is a flowchart which shows the process of the interpolation system determination apparatus which concerns on the other Example of this invention, (b)-(d) is a figure which shows the allocation of the mixture rate to the screen of (b). It is a figure which shows the calculation process of the displacement flame | frame difference which concerns on another Example of this invention. It is a flowchart which shows the process of the interpolation system determination apparatus which concerns on another Example of this invention. It is a figure which shows the determination process of the mixing rate which concerns on another Example of this invention. It is a block diagram of the image processing apparatus which concerns on another Example of this invention. It is a figure which shows the process which blurs an image based on embodiment of this invention. It is a figure which shows the table of LPF when emphasizing an image. Pixel value when the pixel value of the image with the edge, the pixel value when the processing for blurring the pixel value of the image with the edge is performed, and the processing for emphasizing the pixel value of the image with the edge are performed It is a graph which shows a value. It is a figure which shows the basic concept of vector type | mold FRC. (A)-(c) is a figure which shows performing a process per block in vector type | mold FRC. It is a block diagram which shows the structure of vector type | mold FRC. It is a figure which shows the basic concept of a frequency dispersion type | mold FRC. It is a block diagram which shows the structure of a frequency dispersion type | mold FRC. (A) is a graph showing an image having an edge, (b) is a graph showing an image in which the edge is emphasized, and (c) is a graph showing an image in which luminance is rounded at the edge.

Explanation of symbols

1, 17, 26, 45, 50, 73 Image processing apparatus 2 Input image (first image)
3 Input image (second image)
4 Vector Interpolated Image Generation Unit 5 Vector Interpolated Image 6 Addition Interpolated Image Generation Unit 7 Addition Interpolated Image 8 Interpolation Method Determination Unit 9,16 Mixing Rate 10 Interpolated Image Mixing Unit (Interpolated Image Mixing Unit)
11 Mixed Interpolated Image 12 Blur Image Generation Unit (Bokeh Image Generation Unit)
13, 15, 16 Output image 14 Enhanced image generation means 18 Blurred image generation means 19 Addition blurred image 20, 21 Enhanced image 22 Interpolated image mixing means 23-25 Output images 27, 37, 38, 46, 51, 52, 82 Input images 28, 33 to 36, 40, 42 Frame buffer 29 Image enhancement device 30 Vector type FRC device 31 Image addition device 32, 75 to 77 Interpolation method determination device 39 Image mixing device 41 Blurred image generation device 43 Image output device 44 Liquid crystal panel 47 Roll telop area 48 Non-roll telop area 49 Boundary part 53, 54, 61, 62 Block 55 Motion vector 56, 57 Block enlarged view 58, 59 Pixel 60 DFD calculation part 63 Mixing rate buffer 64-72 Screen 74 In mixing Inserted image generating device 78 Control CPU
79 Interpolation method determination signal 80 Remote control or button 81 Image adjustment device 83 Image adjustment signal 90 Image 91 Pixel 92, 93 Table 94-96 Pixel value t, T Time x Position

Claims (7)

An image processing apparatus,
From the first image that is a frame image input to the image processing device and the second image that is the next frame image in time of the first image, the first image and the second image In an image processing apparatus that generates an interpolated image that is an intermediate image to an image, and further performs image processing on the first image and the second image,
A vector interpolated image generating means for detecting a motion vector that is information indicating the motion of the image from the first image and the second image, and generating a vector interpolated image based on the motion vector;
An added interpolated image generating means for generating an added interpolated image by adding and normalizing pixel values of respective pixels at the same position of the first image and the second image;
An interpolation method determination unit that extracts a feature of an image input to itself and determines a mixing ratio of the vector interpolation image and the addition interpolation image for each area in the interpolation image based on the feature. When,
An interpolated image mixing unit configured to mix the vector interpolated image and the added interpolated image for each area based on the mixing ratio determined by the interpolating method determining unit and generate a mixed interpolated image; ,
A blurred image generating means for blurring the mixed interpolated image;
An image processing apparatus comprising: the first image, the second image, or an enhanced image generating unit that emphasizes both the first image and the second image.   The interpolation method determining means determines a high luminance area and a low luminance area for both the first image and the second image, the first image, or the second image, 2. The image processing apparatus according to claim 1, wherein the ratio of the vector-interpolated image is made higher for the mixing ratio of the high-luminance area and the mixing ratio of the low-luminance area.   The interpolation method determining means determines a roll telop from the first image, the second image, and the temporally previous image of the first image, the first image, or the second image. The image processing apparatus according to claim 1, wherein the ratio of the vector-interpolated image is set so that the mixing ratio of the area where the roll telop is detected is high.   The interpolation method determining means determines a still telop from the first image, the second image, and the image temporally previous to the first image, the first image, or the second image. The image processing apparatus according to claim 1, wherein a ratio of the added interpolated image is set to a high mixing ratio of areas where the still telop is detected.   The interpolation method determination means receives information on the reliability of the motion vector detected for each area from the vector interpolation image generation means, and adds the mixing ratio in an area where the reliability of the motion vector is low. The image processing apparatus according to claim 1, wherein a ratio of the interpolated image is increased.   The image processing apparatus according to claim 1, wherein when the difference between the mixing ratio of a certain area and the mixing ratio of a surrounding area is large, the interpolation method determination unit smoothes them.   The image processing apparatus according to claim 1, wherein the interpolation method determination unit makes the change of the mixing rate moderate when the mixing rate of the current frame changes greatly with respect to the previous frame.