JP4761102B2 - Image display apparatus and method, and program - Google Patents

Description

  The present invention relates to an image display device and method, and a program, and more particularly, to an image display device and method, and a program for converting the resolution of an input image for each predetermined area based on the amount of movement.

  Conventionally, in a display device, as a method for enabling an observer to recognize an image at a resolution exceeding the resolution of the displayed image, a plurality of liquid crystal display elements are used and optically shifted using a half mirror or the like. A first method for combining (see Patent Document 1), a second method for performing a pixel shifting operation called wobbling that optically vibrates the optical axis of light from a liquid crystal display element (see Patent Document 2), and the like are proposed. Has been. The perception of the image by the observer at a resolution exceeding the resolution of the displayed image is appropriately referred to as a super-resolution effect.

JP-A-6-18836 JP-A-6-324320

  However, in the display device according to the first method, it is necessary to use a plurality of display elements and an optical element for combining light output from the plurality of display elements in order to realize a high resolution effect. There is a problem that the cost of the apparatus becomes high and the structure becomes complicated and it is difficult to reduce the size.

  Further, the display device according to the second conventional method requires an optical element that optically shifts the optical axis in order to realize a high resolution effect. This has the problem that the driving method becomes complicated in addition to the cost increase due to the optical element for wobbling.

  The present invention has been made in view of such a situation, and is capable of easily obtaining the super-resolution effect without changing the structure of the display element.

An image input step for inputting an image signal having a first resolution, a resolution conversion step for converting the input image signal into an image signal having a second resolution lower than the first resolution, and a resolution A first image display method including an image display control step of displaying an image corresponding to the converted image signal on a display unit at a predetermined frame rate, wherein the resolution conversion step is a step of converting the image signal input in the image input step; A vision in which the display means for detecting the amount of movement of at least a part of the area and displaying it at a predetermined frame rate with a spatial thinning amount determined based on the detected amount of movement has a resolution exceeding the second resolution. It is characterized by performing spatial thinning processing that provides an effect.

  A horizontal thinning amount in the horizontal direction can be determined based on the horizontal movement amount of the region, and a vertical spatial thinning amount can be determined based on the vertical movement amount of the region.

  When the frame rate is high, the spatial thinning amount in the spatial thinning process can be increased.

The spatial decimation processing any frame, all aliasing components of the spatial decimation processed signal in a predetermined number of frames adjacent back and forth the frame temporally can be determined spatial thinning amount so cancel each other Can.

  In the resolution conversion step, spatial thinning processing can be performed on a signal obtained by performing spatial filtering on the input image signal.

In the resolution conversion step, the image signal input in the image input step can be divided into a plurality of regions, and the movement amount can be detected for each of the divided regions.

An image input step for inputting an image signal having a first resolution, a resolution conversion step for converting the input image signal into an image signal having a second resolution lower than the first resolution, and a resolution A second image display method including an image display control step of displaying the converted image signal on a display unit at a predetermined frame rate, wherein the resolution conversion step includes the image signal input in the image input step in a plurality of regions; After dividing, detecting the amount of movement for each area, setting the amount of spatial thinning for each area based on the detected amount of movement, and applying spatial filtering to each area of the input image signal in, inter space which gives a visual effect, such as having a resolution of a display means for displaying at a predetermined frame rate for each region based on the spatial decimation amount set exceeds a second resolution Can processing can be performed.

Image input means for inputting an image signal having a first resolution and resolution conversion for converting the image signal input to the image input means into an image signal having a second resolution lower than the first resolution according to the present invention. In the first image display apparatus, the resolution conversion unit is input by the image input unit. The first image display device includes: a display unit configured to display the image signal converted in resolution by the resolution conversion unit on the display unit at a predetermined frame rate. A moving amount detector for detecting a moving amount of at least a part of the image signal, and a display means for displaying at a predetermined frame rate with a spatial thinning amount determined based on the detected moving amount; And a resolution conversion processing unit that performs a spatial thinning process that gives a visual effect having a resolution exceeding the resolution.

The horizontal spatial thinning amount is determined based on the horizontal movement amount of the region, and the vertical spatial thinning amount can be determined based on the vertical movement amount of the region.
When the frame rate is high, the amount of space decimation in the space decimation process can be increased.
In the spatial decimation process for an arbitrary frame, the spatial decimation amount is determined so that all the aliasing components of the signal subjected to the spatial decimation process in a predetermined number of frames that are temporally adjacent to the corresponding frame cancel each other out. Can be.

The resolution conversion processing unit can perform spatial thinning processing on a signal obtained by subjecting the input image signal to spatial filtering.

The movement amount detection unit can divide the image signal input by the image input unit into a plurality of regions and detect the movement amount for each of the divided regions.

Image input means for inputting an image signal having a first resolution and resolution conversion for converting the image signal input to the image input means into an image signal having a second resolution lower than the first resolution according to the present invention. And a second image display device having an image display control means for causing the display means to display the image signal converted in resolution by the resolution conversion means at a predetermined frame rate. The resolution conversion means is input to the image input means. A moving amount detection unit that divides the image signal into a plurality of regions and detects a moving amount for each region, and sets a spatial thinning amount of each region based on the detected moving amount, and the input image signal after performing a spatial filter processing for each region, such as having a resolution of a display means for displaying at a predetermined frame rate for each region based on the spatial decimation amount set exceeds a second resolution And having a resolution conversion processing unit that performs spatial decimation processing to give sense effect.

An image input control step for controlling input of an image signal having a first resolution to a computer, and conversion of the input image signal into an image signal having a second resolution lower than the first resolution A program for executing processing including a resolution conversion control step for controlling the image and an image display control step for controlling display of an image corresponding to the resolution-converted image signal at a predetermined frame rate is provided in the resolution conversion control step. Detects the amount of movement of at least a part of the region of the image signal input in the processing of the image input control step, and displays the amount of spatial thinning determined based on the detected amount of movement at a predetermined frame rate. Spatial thinning processing can be performed which gives a visual effect such that the display means has a resolution exceeding the second resolution.

In the present invention, an image signal having the first resolution is input, the input image signal is converted to an image signal having a second resolution lower than the first resolution, and the resolution-converted image signal is supported. The image to be displayed is displayed on the display means at a predetermined frame rate, and at the time of resolution conversion, the amount of movement of at least a part of the input image signal is detected, and spatial thinning determined based on the detected amount of movement A spatial thinning process is performed in which the visual effect is such that the display means displaying at a predetermined frame rate has a resolution exceeding the second resolution.

In the present invention, an image signal having the first resolution is input, the input image signal is converted into an image signal having a second resolution lower than the first resolution, and the resolution-converted image signal is Displayed on the display means at a predetermined frame rate, and at the time of resolution conversion, the input image signal is divided into a plurality of regions, and the movement amount for each region is detected. Based on the detected movement amount, Display means for setting a spatial decimation amount and performing a spatial filter process on each region of the input image signal and then displaying each region at a predetermined frame rate based on the set spatial decimation amount Spatial thinning processing is performed that provides a visual effect that has a resolution that exceeds the second resolution.

  According to the present invention, the super-resolution effect can be easily obtained by resolution conversion such as spatial thinning based on the moving amount of the image.

  Embodiments of the present invention will be described below. The correspondence relationship between the invention described in this specification and the embodiments of the invention is exemplified as follows. This description is intended to confirm that the embodiments supporting the invention described in this specification are described in this specification. Therefore, although there is an embodiment which is described in the embodiment of the invention but is not described here as corresponding to the invention, it means that the embodiment is not It does not mean that it does not correspond to the invention. Conversely, even if an embodiment is described herein as corresponding to an invention, that means that the embodiment does not correspond to an invention other than the invention. Absent.

  Further, this description does not mean all the inventions described in this specification. In other words, this description is for the invention described in the present specification, which is not claimed in this application, that is, for the invention that will be applied for in the future or that will appear and be added by amendment. It does not deny existence.

An image input step for inputting an image signal having a first resolution, a resolution conversion step for converting the input image signal into an image signal having a second resolution lower than the first resolution, and a resolution A first image display method including an image display control step of displaying an image corresponding to the converted image signal on a display unit at a predetermined frame rate, wherein the resolution conversion step is a step of converting the image signal input in the image input step; Display means for detecting the amount of movement of at least a part of the region (for example, the processing in step S3 in FIG. 23) and displaying the amount of spatial thinning determined based on the detected amount of movement at a predetermined frame rate Performing spatial thinning processing (for example, processing in steps S4 to S7 in FIG. 23) that gives a visual effect having a resolution exceeding the second resolution. And butterflies.

  The resolution conversion step may include a spatial thinning process (for example, a process of step S7) that gives a visual effect such that the display unit has a resolution exceeding the second resolution.

  The amount of space thinning in the space thinning processing can be determined based on the amount of movement of the region (for example, the processing of steps SS4 and 5 in FIG. 23).

  The horizontal spatial thinning amount can be determined based on the horizontal movement amount of the region, and the vertical spatial thinning amount can be determined based on the vertical movement amount of the region (for example, step of FIG. 23). Processing of S4 and S5).

  When the frame rate is high, the spatial thinning amount in the spatial thinning process can be increased (for example, the processes in steps S4 and S5 in FIG. 23).

The spatial decimation processing any frame, all aliasing components of the spatial decimation processed signal in a predetermined number of frames adjacent back and forth the frame temporally can be determined spatial thinning amount so cancel each other Can. (For example, the process of step S5 of FIG. 23).

  In the resolution conversion step, spatial thinning processing can be performed on a signal obtained by performing spatial filtering on the input image signal (for example, processing in steps S6 and S7 in FIG. 23).

In the resolution conversion step, the image signal input in the image input step can be divided into a plurality of regions, and the movement amount can be detected for each of the divided regions (for example, the process in step S3 in FIG. 23).

An image input step for inputting an image signal having a first resolution, a resolution conversion step for converting the input image signal into an image signal having a second resolution lower than the first resolution, and a resolution A second image display method including an image display control step of displaying the converted image signal on a display unit at a predetermined frame rate, wherein the resolution conversion step includes the image signal input in the image input step in a plurality of regions; The movement amount for each area is detected by division (for example, the process of step S3 in FIG. 23), and the spatial thinning amount of each area is set based on the detected movement amount (for example, step S4 in FIG. 23). , S5), and after performing spatial filter processing on each area of the input image signal (for example, processing in step S6 in FIG. 23), the set spatial thinning amount Based display means for displaying at a predetermined frame rate for each region by the spatial decimation processing to provide a visual effect that has a resolution exceeding the second resolution (for example, step S7 in FIG. 23) can be performed.

The image input means (for example, the image input unit 11 in FIG. 18) for inputting an image signal having the first resolution and the image signal input to the image input means of the present invention are set to a second lower than the first resolution. Resolution conversion means (for example, the image conversion section 12 in FIG. 18) that converts the image signal having the resolution of the image, and image display control means for causing the display means to display the image signal that has undergone resolution conversion by the resolution conversion means at a predetermined frame rate. (For example, the image display unit 13 in FIG. 18), the first image display apparatus includes a resolution conversion unit that detects a movement amount of at least a partial area of the image signal input by the image input unit. detector (e.g., the movement amount detecting unit 21 of FIG. 18) exceeds the, at spatial thinning amount determined based on the detected movement amount, a display means for displaying at a predetermined frame rate and the second resolution solutions Resolution conversion processing unit that performs spatial decimation processing to provide a visual effect such as those having degrees (e.g., the number of pixel conversion section 22 of FIG. 18) and having a.

  The resolution conversion unit further includes a storage unit (for example, the storage unit 44A in FIG. 19) for storing a table of spatial thinning amounts corresponding to the movement amount, and the spatial thinning amount is determined by referring to the table. You can make it.

The resolution conversion processing unit performs spatial thinning processing on a signal obtained by performing spatial filtering on the input image signal (for example, the spatial thinning processing unit 42 in FIG. 19).

The movement amount detection unit can divide the image signal input by the image input unit into a plurality of regions and detect the movement amount for each of the divided regions (for example, the block matching circuit 34 in FIG. 19).

The image input means (for example, the image input unit 11 in FIG. 18) for inputting an image signal having the first resolution and the image signal input to the image input means of the present invention are set to a second lower than the first resolution. Resolution conversion means (for example, the image conversion section 12 in FIG. 18) that converts the image signal having the resolution of the image, and image display control means for causing the display means to display the image signal that has undergone resolution conversion by the resolution conversion means at a predetermined frame rate. (For example, the image display unit 13 in FIG. 18), the resolution conversion unit divides the image signal input to the image input unit into a plurality of regions, and sets the movement amount for each region. Based on the detected movement amount detection unit (for example, the movement amount detection unit 21 in FIG. 18), the spatial thinning amount of each region is set based on the detected movement amount (for example, the control unit 44 in FIG. 19), and input Image After performing a spatial filter processing for each area (for example, spatial filtering unit 41 of FIG. 19), display means for displaying at a predetermined frame rate for each region based on the spatial decimation amount set first And a resolution conversion processing unit (for example, the spatial thinning processing unit 42 in FIG. 19) that performs spatial thinning processing that gives a visual effect that has a resolution exceeding two resolutions.

  An image display device to which the present invention is applied converts an input image into a display image having a resolution lower than the resolution and displays the converted image on a display device. At that time, resolution conversion processing using predetermined visual characteristics is performed. This realizes a perceptual effect (super-resolution effect) in which the observer perceives an image at a resolution exceeding the resolution.

  Here, the visual characteristics and resolution conversion processing will be described.

  Human vision has a function of perceiving light when the sum of received light stimuli reaches a certain threshold (hereinafter referred to as a temporal integration function). That is, the perception of light follows the sum of the light integrated over time regardless of the distribution state of the light stimulus within the presentation time. Further, the stimulus (threshold) that can perceive light becomes smaller as the presentation time of the stimulus becomes longer, and becomes larger as the presentation time becomes shorter.

This relationship is known as the Block's law, and the following equation holds: Where I is the intensity of the stimulus as a threshold, T is the stimulus presentation time, and k is a constant.
I × T = k

  This relationship can be expressed as shown in FIG. 1, where the horizontal axis is the stimulus presentation time T and the vertical axis is the threshold (intensity I). This curve is known as the threshold presentation time curve. According to the threshold value presentation time curve of FIG. 1, when light of intensity Ia is presented in impulse for time Ta, light of intensity Ib 1 / n of Ia continues for time Tb that is n times Ta. The person feels the same brightness when presented.

  It should be noted that the rule of the block is valid up to a time when the stimulus is presented (time TL in the example of FIG. 1) (although it becomes a straight line to the right until time TL), the threshold value of the stimulus is exceeded when time TL is exceeded. It depends only on the intensity (it does not change with the presentation time, and as a result, the threshold presentation time curve shows a characteristic like a broken line). The maximum stimulus presentation time TL for which the block law holds is called the critical presentation time. Although this time TL varies depending on the stimulus presentation conditions such as the intensity of the background light, there is a report that it is approximately 25 ms to 100 ms.

  Details of the block rule are described in, for example, “Visual Information Processing Handbook, Japanese Visual Society, pp.219-220”.

  The human vision also has a function (hereinafter referred to as a sensory memory function) that, when a stimulus is perceived, the stimulus is stored for a certain period of time after the presentation of the stimulus ends. There are many reports that this time is 10 ms to 200 ms. This function is also called iconic memory or visual persistence, and is described in, for example, “Visual Information Handbook, Japanese Visual Society, pp.229-230”.

  Next, resolution conversion processing for realizing the super-resolution effect will be described. Note that the super-resolution effect in the present invention uses a visual characteristic that an observer perceives a sum of a plurality of images within a certain time. This is considered to be caused by the complicated relationship between the temporal integration function and the sensory memory function. In the following description, this is based on the temporal integration function for convenience.

  For example, when a subject moving in parallel in the horizontal direction is photographed at a predetermined frame rate (hereinafter referred to as an input image frame rate) and a predetermined sampling rate (hereinafter referred to as an input image sampling rate), as shown in FIG. 2A. An input frame Fa in which the subject image Wa moves rightward (X-axis direction) toward the drawing at a speed v (pixel / frame) is obtained. FIG. 2A shows four consecutive input frames Fa1 to Fa4.

  The input frame Fa obtained in this way is sampled in the X-axis direction (moving direction of the subject image Wa) at a sampling rate (hereinafter referred to as a display image sampling rate) that is 1 / m of the input image sampling rate. (Thinning shall be performed with the thinning amount m). In the case of FIG. 2A, since the input frame Fa is thinned by the thinning amount 4, the number of pixels in the X-axis direction is reduced to 1/4 (rough in the X-axis direction) as shown in FIG. Display frame Fb is obtained. The display frame Fb includes an image obtained by thinning the subject image Wa of the input frame Fa with a thinning amount 4 (hereinafter referred to as a display subject image Wb).

  The display frame Fb thus obtained is displayed at a predetermined frame rate (hereinafter referred to as display image frame rate). As a result, the observer perceives integrated images of a plurality of display frames Fb displayed within the integration time in the temporal integration function described above.

  Here, it is assumed that the observer's line of sight follows the display subject image Wb on the display frame Fb displayed in this way. In this case, since the viewpoint of the observer is always located at the center of the display subject image Wb, the display subject image Wb on the observer's retina is almost stationary. The coordinate axes Vx and Vy shown in FIG. 2B indicate coordinates on the retina, and the coordinate axes X and Y indicate coordinates on the frame (both are shown on the display frame Fb1 in the figure, but the display frame Fb2 The illustration of Fb4 to Fb4 is omitted). As for the coordinate axes Vx and Vy, since a reverse image of the real image is formed on the retina, the direction of the coordinate system is opposite to that of the coordinate axes X and Y.

  In addition, as shown by the dotted line in FIG. 3, the display frame Fb is sampled at a certain position on the frame (in this example, a position at an interval of 4 pixels). Therefore, when the movement amount does not match the multiple of the sampling interval, the position of the sampled subject image Wa is shifted by v for each frame, so that each display subject image Wb of the display frame Fb is the sampling position of the subject image Wa. It is formed in a part different by the amount of deviation.

  For example, when the moving speed v of the subject image Wa is 1 (pixel / frame), the amount of movement (1 pixel) between frames does not match a multiple of the sampling interval (4 pixels), and the sampled subject image Wa is sampled. Is shifted by one pixel in the X-axis direction. Therefore, in this case, each display subject image Wb of the display frame Fb is formed from a different portion of the subject image Wa.

  In this way, when the display subject image Wb is formed from a portion that differs from the subject image Wa by the deviation of the sampling position, the display subject image Wb is integrated over a plurality of frames in the visual system. An image in which the pixels are denser than Wb (an image having a higher resolution than the resolution of the display subject image Wb (hereinafter referred to as super-resolution)) is perceived.

  For example, when the integration time in the visual characteristic corresponds to the display time of the four display frames Fb in FIG. 2B and the four display subject images Wb in the display frames Fa1 to Fa4 are integrated, as shown in FIG. 2C. In addition, an image having a resolution of about four times the resolution of the display subject image Wb, that is, about the same resolution as the subject image Wa is perceived.

  The super-resolution effect is realized by this principle. However, when the thinning process is performed, a aliasing component is generated, which becomes aliasing distortion and image quality deteriorates. Therefore, in the present invention, as described below, a device for removing the folding component is devised.

・・・（１）

・・・（２） Expression (1) represents a signal f _s (x) obtained by discretizing a one-dimensional original signal f (x) with an interval X. In equation (1), δ (x) is a delta function. Equation (2) represents the Fourier transform F _s (ω) of the discretized signal f _s (x). In equation (2), F (ω) is the Fourier transform of the original signal f (x), and ω _s represents the sampling angular frequency.

... (1)

... (2)

・・・（３） Expression (3) represents the Fourier transform F _sφ (ω) of the signal f _sφ (x) _obtained by discretizing the original signal f (x) by an interval X while shifting the original signal f (x) in the real space.

... (3)

Equation (3) indicates that the fundamental wave of k = 0 is the same as the original signal, and the n-th harmonic of k = n is out of phase by 2πnφ. As described above, assuming that the subject image Wa is moving in parallel at a certain moving speed v and the sampling is sampled at 1 / m in the moving direction, the original signal is a band of m times the Nyquist frequency of the display frame Fb. have. Therefore, the sampling signal f _sφ (x) thinned and sampled at 1 / m has an aliasing component, and in Equation (3), k = 0 is the original signal component, and k = 1, 2 _,. (m-1) is a folding component.

FIG. 4 shows the Fourier transform F _sφ (ω) when the thinning amount m = 2. At this time, the band of the original signal is twice the Nyquist frequency, and the first harmonic return component exists in the sampling signal f _sφ (x) sampled by 1 / m. As can be seen from this figure, the sampling signal f _sφ (x) has the Fourier transform F (ω) component of the original signal f (x) as it is, and the first harmonic F (ω−ω _s ) at k = 1. And F (ω + ω _s ) are folded back by −2πφ and 2πφ, respectively.

When the thinning sampling interval is 1 / m, the sampling signal f _sφ (x) thinned by 1 / m includes 1st to (m−1) -order folding components, and each phase is 2πkφ. It will be shifted. Since the sampling signal f _sφ (x) is a signal _{obtained by} sampling the original signal f (x) shifted by φ by 1 / m, it is considered to correspond to any one display frame Fb in FIG. 2B.

・・・（４） Here, consider the signals of the display frames Fb that are different in time in FIG. 2B. When the subject (original signal f (x)) is translated at the speed v, the phase of the sample points is shifted for each frame as shown in FIG. Therefore, the sampling point shift amount φ in the equation (3) is a function of the time t, and depends on the speed v (pixel / frame) and the thinning-out amount m (pixel) as in the equation (4). Become. In Expression (4), T represents a time interval and is the reciprocal of the frame rate.

... (4)

Expression (4) represents that the shift amount φ ₀ becomes 0 when t = 0, and the shift amount increases by v / m as t = T, 2T, 3T. When equation (4) is applied to equation (3), the phase of the aliasing component at each time is obtained. FIG. 5 shows the phases of the primary folding components at time t = 0, T, 2T, 3T,. 6 shows the phase of the second-order aliasing component, FIG. 7 shows the phase of the third-order aliasing component, and FIG. 8 shows the phase of the fourth-order aliasing component at time t = 0, T, 2T, 3T,. Represents each.

  In this way, the k-th order folding component rotates at equal intervals (2πkφT intervals) as time advances, that is, the frame, and returns to phase 0 at time t = (m / v) T. Further, the phase rotation interval doubles as the order of the aliasing component increases.

  In this way, the phase of the next aliasing component of k (= 1, 2,..., (M−1)) generated by the thinning process (thinning sampling) with the thinning amount m rotates at 2πkφT. Depending on the direction and the number of images to be integrated (the number of folded components to be combined), the folded components may cancel each other. In other words, since φt depends on the moving speed v and the thinning amount m as shown in the equation (4), the aliasing components may be canceled out depending on the moving speed v and the thinning amount m and the number of images to be integrated. is there.

  For example, when v = 1, when m = 4 is thinned, the image of the display frame Fb has 0 (= 2π × 1 × [(1/4) × 0 / T] as shown in FIG. ), Π / 2 (= 2π × 1 × [(1/4) × (T / T)], π (= 2π × 1 × [(1/4) × 2T / T], 3 / 2π (= 2π .Times.1.times. [(1/4) .times.3T / T]), ..., there is a primary folding component whose phase changes (the phase changes at intervals of .pi. / 2). The folding components after 4T are not shown, and the same applies to FIGS.

  In the image of the display frame Fb, as shown in FIG. 10, 0 (= 2π × 2 × [1/4 × 0 / T]), π (= 2π × 2 × [1/4 × T / T] ), 2π (= 2π × 2 × [(1/4) × (2T / T)]), 3π (= 2π × 2 × [(1/4) × 3T / T]),. A second-order aliasing component that changes (the phase changes at an interval of π), and 0 (= 2π × 3 × [(1/4) × (0 / T)], 3π / 2 ( = 2π × 3 × [(1/4) × (T / T)], 3π (= 2π × 3 × [(1/4) × (2T × T)]), 9π / 2 (= 2π × 3 ×) [(1/4) × (3T / T)]), ..., there is a third-order aliasing component whose phase changes (the phase changes at intervals of 3π / 2).

  In this case, since the vectors of the primary to tertiary folding components at t = 0, T, 2T, and 3T are directed to cancel each other, as shown in FIGS. When four display frames Fb are integrated, they are all canceled out.

・・・（５）

・・・（６）

・・・（７） If the condition for canceling the k-th order aliasing component is expressed by an expression, the expression (5) is obtained. If the expression (5) is expanded by Euler's formula, the expressions (6) and (7) are obtained.

... (5)

... (6)

... (7)

  That is, in the present invention, the aliasing component is removed by determining the thinning-out amount m so that aliasing components cancel each other are generated according to the moving speed v of the subject image Wa.

Here, considering the case where the discretized signal f _sφ (x) is reduced to 1 / m by a band-limited digital filter, the original signal f (x) shifted by φ is a band at the Nyquist frequency so that no aliasing occurs. Limited. Therefore, for example, when m = 2, the Fourier space is as shown in FIG. 12, and each frame image corresponding to the signal reduced to 1 / m is a low-resolution image that does not include the aliasing component. Therefore, in this case, the fundamental wave of the reduced signal is a signal different from the original signal, and it is not possible to express a frequency component equal to or higher than the Nyquist frequency no matter how the addition processing is performed on images of a plurality of frames. The super resolution effect cannot be obtained. Therefore, in order to obtain the super-resolution effect, it is important not to limit the band of the original signal, and it is optimal to thin out and sample the original image having a wideband spatial frequency component.

  In the above, for the sake of simplicity, the case where the original signal is a one-dimensional signal has been described as an example, but the same applies to a two-dimensional image. Although the movement of the subject image Wa in the X-axis direction has been described as an example with reference to FIG. 2, the same applies to the movement in the Y-axis direction.

  Next, conditions for aliasing components that cancel each other (conditions for aliasing components of an image that can obtain a super-resolution effect) will be described.

  The condition for obtaining the super-resolution effect is that Expression (5) is satisfied, that is, Expression (6) and Expression (7) are satisfied. As shown in FIG. 13, if the k-th order folding component at time t is a vector Zk (t), the sum of the vectors Zk (t) in the integration range of the visual system is zero. The establishment of this condition depends on the integration time, but it is known that this integration time varies depending on the observation environment, and it is difficult to measure it accurately. It is difficult to limit.

On the other hand, for example, a subject image Wa moving in the X-axis direction or the Y-axis direction at a predetermined moving speed v is sampled at a predetermined thinning amount m and displayed at a predetermined frame rate, and the displayed display subject image Wb is displayed. From an experiment to confirm whether the observer actually perceived the image at the super resolution, if the frame rate is high, that is, if the number of images to be integrated is large, the super resolution effect can be obtained even if the thinning amount m increases. I know that. At this time, the condition for obtaining the super-resolution effect depends on the moving speed v, and it is considered that the relationship is approximately represented by Expression (8).
2πn + α ≦ 2πkφT ≦ 2π (n + 1) −α (8)

  As described above, the phase of each aliasing component rotates at intervals of 2πkφT, but Equation (8) indicates that the super-resolution effect cannot be obtained when the phase rotation interval of each subsequent aliasing component is close to a multiple of 2π. Represents. As shown in FIG. 14, the fact that the phase rotation interval is close to a multiple of 2π means that the phase of the aliasing component hardly changes even when the time t changes, and the aliasing component remains without being canceled. It is because it ends.

  For example, when the conditions for establishing Equation (8) are examined for the primary to tertiary folding components generated when m = 4, the shaded moving speed v (pixel / frame) in FIG. In the range, equation (8) does not hold and the super-resolution effect cannot be obtained.

  For this primary folded component, for example, when v = 4, the phase rotation interval of the folded component = 2π × 1 × (4/4) (2πkφT), and the phase rotation interval of the folded component is 1 × 2π. Therefore, in a certain range centered on the speed v = 4 (a range of speed where the phase rotation interval is a range of 2α centered on a multiple of 2π), the folding component of the linear expression is not canceled. That is, when v = 4n (n = 0, 1, 2, 3,...), The phase rotation interval is n times 2π. Therefore, in a certain range centered on v = 4n, the linear expression The folded component is not negated.

  For the second-order aliasing component, for example, when v = 2, the phase rotation interval = 2π × 2 × (2/4) (1 × 2π), and when v = 4, the phase rotation interval = 2π × Since 2 × (4/4) (twice 2π), a constant range centered on the velocity v = 2, 4 (a range of velocity where the phase rotation interval is a range of 2α centered on a multiple of 2π) , The folding component of the quadratic expression is not canceled out. That is, when v = 2n, the phase rotation interval is n times 2π. Therefore, the folding component of the quadratic expression is not canceled within a certain range centered on v = 2n.

  For the third-order aliasing component, for example, when v = 4/3, the phase rotation interval = 2π × 3 × (4/3) / 4 (1 × 2π), and when v = 8/3, the phase Rotation interval = 2π × 3 × (8/3) / 4 (2 × 2π), and when v = 4, phase rotation interval = 2π × 3 × 4/4 (3 × 2π), In a certain range centered on the speed v = 4/3, 8/3, 4 (the range of speed where the phase rotation interval is 2α centered on a multiple of 2π), the folding component of the cubic equation is canceled out. Disappear. That is, when v = (4/3) n, the phase rotation interval is n times 2π, so that the third-order aliasing component is canceled within a certain range centered on v = (4/3) n. Absent.

Since the phase rotation interval 2πkφT = 0 when the speed v = 0, each of the primary to tertiary folding components is in a certain range (0 to v _α1 , 0 to v _α2 , 0 to v _α3 ) no longer cancels.

  The primary and secondary aliasing components that exist when m = 3 (FIG. 16) and the primary aliasing component that exists when m = 2 (FIG. 17) are also described above by taking m = 4 as an example. In addition, at the speed at which the phase rotation interval is in the range of 2α centered on a multiple of 2π, the aliasing components of the following equations are not canceled out.

Further, as shown in FIG. 13, as the order of the aliasing component increases, the phase rotation interval at each order increases to 2 times and 3 times. If the phase rotation interval is θ, the moving speed v of the subject image Wa is small, and when the phase rotation interval θ is smaller than α, equation (8) does not hold and the super-resolution effect cannot be obtained. When the moving speed v of the subject image Wa increases and the phase rotation interval reaches α, super resolution is obtained. From this, it is considered that α is a critical point (phase rotation interval) at which the super-resolution effect is obtained. This α varies depending on the display image frame rate, and tends to decrease as the display image frame rate increases. If the moving speed of the subject image Wa at the critical point is v _α , Equation (9) is obtained, and if it is transformed, Equation (10) is obtained. Therefore, when the display image frame rate increases and α decreases, the velocity v _α (v _α1 , v _α2 , or v _{α3 in} the example of FIG. 15) decreases, and as a result, even if the movement amount is small, the super-resolution is achieved. An effect is obtained.

Also, it can be seen from equation (10) that v _{α at} the critical point depends on the thinning amount m and the order k of the aliasing component, and the critical point speed v _α increases as the thinning amount m increases. Since the velocity v _{α at the} critical point decreases as the order k increases (in the example of FIG. 15, v _α2 is smaller than v _α1 and v _α3 is smaller than v _α2 ), the higher-order folding component It can be seen that the region where the super-resolution effect is not obtained becomes narrow.

In summary, the following can be said about the super-resolution effect in the visual system.
The critical point α at which the super-resolution effect can be obtained becomes small in high frame rate display.
When the thinning amount is m, the 1st to m−1 order folding components need to satisfy the formula (8).
When the thinning amount m decreases, the speed v _α of the subject image Wa at the critical point decreases (when the thinning amount m is small, the super-resolution effect can be obtained even if the movement amount is small).

  From the above, it can be seen that the super-resolution effect can be realized by performing the thinning process according to the moving speed (size and direction) of the subject.

  Increasing the display image frame rate in this way is advantageous for obtaining the super-resolution effect, but increasing the display image frame rate can also improve image quality degradation such as motion blur and jerkiness. It will be advantageous.

  Next, a configuration example of the image display device 1 to which the present invention is applied will be described with reference to FIG.

  The image input unit 11 inputs an image composed of a predetermined frame rate and a predetermined spatial sampling rate (hereinafter simply referred to as a sampling rate), and converts it into an internal data format. For example, when the frame rate of the display device of the image display unit 13 is F and the number of display pixels is (x × y) pixels, an image having a frame rate of F and the number of pixels (ix × jy) is input. This is converted into an input frame Fa in the internal data format. i, j, x, and y are each a positive number.

  The image input unit 11 supplies the converted input frame Fa ((ix × jy) pixels) to the image conversion unit 12.

  Note that, when an analog signal is input, the image input unit 11 converts the analog signal into a digital signal according to an internal data format by a built-in analog-digital converter (not shown).

  The image input unit 11 is a television signal standardized by Society of Motion Picture and Television Engineers (SMPTE), an analog video signal standardized by VESA (Video Electronics Standards Association), or a DVI standardized by DDWG (Digital Display Working Group). It can also handle image signals such as (Digital Visual Interface) signals.

  Further, as described above, since the super-resolution effect is more easily obtained at the high frame rate, when the frame rate of the input frame Fa is high, the image input unit 11 can cope with the high frame rate. Has been made.

  The image conversion unit 12 has a display frame Fb ((x × y) pixel having a resolution with which the image display unit 13 can display the input frame Fa ((ix × jy) pixel) supplied from the image input unit 11. ). At that time, the image conversion unit 12 executes the resolution conversion process using the above-described super-resolution effect for each block of a predetermined size, so that an observer who has seen the image of the display frame Fb exceeds the resolution. Realize the visual effect (super-resolution effect) of perceiving at the resolution (super-resolution). The image conversion unit 12 supplies the generated display frame Fb to the image display unit 13.

  The image display unit 13 is configured by a display device having the number of (x × y) pixels, and displays the image supplied from the image conversion unit 12 at a predetermined frame rate. As described above, since the super-resolution effect is more easily obtained at the high frame rate, the image display unit 13 can display an image at the high frame rate.

  Next, the configuration of the image conversion unit 12 will be described.

  The movement amount detection unit 21 of the image conversion unit 12 is input from the input frame Fa ((ix × jy) pixels) (current frame Fa) supplied from the image input unit 11 and one frame time or several frames before. The motion vector of the block image ((ip × jq) pixels) of the current frame Fa is detected by block matching processing with the input frame Fa (previous frame Fa). p and q are both positive integers. In this case, the total number of blocks of the input frame Fa is (x / p × y / q).

  The movement amount detection unit 21 supplies the input frame Fa to the pixel number conversion unit 22 together with the detected motion vector in units of blocks.

  The pixel number conversion unit 22 applies to the block image ((ip × jq) pixels) of the input frame Fa supplied from the movement amount detection unit 21 for each frame in the X-axis direction and the Y-axis direction obtained from the motion vector. Filtering or sampling (thinning-out processing) in the X-axis direction and Y-axis direction according to the moving amount v (pixel) (moving speed v (pixel / frame)). The pixel number conversion unit 22 collects the block images ((p × q) pixels) of the display frame Fb obtained as a result of the processing by the total number of blocks (x / p × y / q), and displays one display frame Fb. ((X × y) pixels) is generated and output to the image display unit 13.

  Next, the configuration of the movement amount detection unit 21 and the pixel number conversion unit 22 of the image conversion unit 12 will be described with reference to FIG.

  The frame memory 31 of the movement amount detection unit 21 stores the input frame Fa (current frame Fa) supplied from the image input unit 11. The delay circuit 32 delays the previous frame Fa input one frame time or several frame times before the current frame Fa so that the phase thereof matches the phase of the current frame Fa. The delay circuit 32 supplies the delayed previous frame Fa to the frame memory 33. The frame memory 33 stores the previous frame Fa from the delay circuit 32.

  The block matching circuit 34 performs block matching in units of blocks of (ip × jq) pixels between the current frame Fa stored in the frame memory 31 and the previous frame Fa stored in the frame memory 33. A motion vector of a block image ((ip × jq) pixels) of the current frame Fa is detected. The block matching circuit 34 supplies the block image itself of the current frame Fa to the spatial filter processing unit 41 of the pixel number conversion unit 22 and the detected motion vector of the block image to the control unit 44 of the pixel number conversion unit 22. Supply.

  The configuration of the movement amount detection unit 21 is an example, and other configurations may be adopted as long as the motion vector of each block of the input frame Fa can be detected.

  A block image ((ip × jq) pixels) (see the left side of FIG. 20) of the input frame Fa from the block matching circuit 34 of the movement amount detection unit 21 is input to the spatial filter processing unit 41 of the pixel number conversion unit 22. The The spatial filter processing unit 41 is a digital filter that limits the bandwidth of the spatial resolution, reduces the aliasing component, and converts it into a desired number of pixels given from the control unit 44.

  Specifically, the spatial filter processing unit 41 thins out the block image ((ip × jq) pixels) of the input frame Fa with the thinning amounts mx and my in the subsequent spatial thinning processing unit 42 according to the control by the control unit 44. As a result, spatial filtering is performed so that a block image ((p × q) pixels) of the display frame Fb is obtained, and the block image is converted into an image of (mxp × myp) pixels (see the left side in FIG. 20).

  The spatial filter processing unit 41 supplies the converted (mxp × myq) pixel image to the spatial thinning processing unit 42. Note that mx and my are both positive integers.

  The spatial thinning processing unit 42 performs thinning sampling on the original image without converting the bandwidth of the spatial resolution, and converts the original image into a desired number of pixels given from the control unit 44.

  Specifically, for the image of (mxp × myq) pixels (see the right side of FIG. 20) supplied from the spatial filter processing unit 41, the thinning amount mx in the X-axis direction notified from the control unit 44, and the Y-axis In accordance with the direction thinning amount my, thinning is performed in the X-axis direction and the Y-axis direction to generate an image of (p × q) pixels (block image of the display frame Fb) (see the right side in FIG. 20). The spatial thinning processing unit 42 supplies the generated (p × q) pixel image to the frame memory 43.

  As in the example of FIG. 20, when mx <i, my <j, the thinning amount is mx and my notified from the control unit 44, but when mx> i, my> j, thinning is performed. The quantity is i or j.

  Returning to FIG. 19, the frame memory 43 stores the block image ((p × q) pixels) supplied by the spatial thinning processing unit 42 and generates one display frame Fb ((x × y) pixels). .

  A motion vector of the block image from the movement amount detection unit 21 is input to the control unit 44. The control unit 44 calculates the thinning amounts (maximum values) mx and my satisfying the equation (8) from the moving speed v in the X-axis direction and the Y-axis direction obtained from the motion vector. Detect (detect a thinning amount that satisfies a condition for obtaining the super-resolution effect).

  Note that α in the equation (8) varies depending on the number of images integrated within the integration time in the visual system as described above, and it is difficult to determine the value as appropriate. Therefore, in the present invention, for example, a subject image Wa moving in the X-axis direction or the Y-axis direction at a predetermined moving speed v is sampled and displayed with a predetermined thinning amount m, and the displayed display subject image Wb is observed. An experiment is performed in advance to confirm whether the person can actually perceive at the super resolution, and the thinning amount m is determined from the relationship between the moving speed v and the thinning amount m when there is a super resolution effect as a result of the experiment. It shall be.

  FIG. 21 shows an experimental result when the above-described experiment is performed on the subject image Wa moving in the horizontal direction.

  In FIG. 21, when the thinning-out amount m = 4, the moving speed within a certain range centered on the speed v = 4/3, 2, 8/3, 4 and when the thinning-out amount m = 3, the speed v = 3. It is shown that the super-resolution effect cannot be obtained at a moving speed within a certain range centered at / 2,3, and at a moving speed within a certain range centered at speed m = 2,4 at a thinning amount m = 2. Has been. Also, when the moving speed v is within a predetermined range P0 starting from 0 (below the speed indicated by the speed v0 in the figure), the super-resolution effect can be obtained at any of m = 2, 3 and 4. It is shown that it is not possible.

  This corresponds to the range of the moving speed v in which the equation (8) is not satisfied as described with reference to FIGS.

  For example, if m = 4 in FIG. 21 is associated with FIG. 15, it is as shown in FIG. Regarding the primary folding component, the folding component of the primary expression is not canceled within a certain range centered on v = 4n. Regarding the secondary folding component, the folding component of the quadratic expression is not canceled within a certain range centered on v = 2n. As for the third-order folding component, the third-order folding component is not canceled within a certain range centered on v = (4/3) n. Even in a certain range near v = 0, the first-order to third-order aliasing components are not canceled out.

  In other words, if any one of the first-order to third-order aliasing components is not canceled, the super-resolution effect cannot be obtained by thinning out at m = 4 within the range of the moving speed v. In FIG. 22, ranges in which such a super-resolution effect cannot be obtained are indicated by ranges U4a to U4d. Further, in FIG. 22, ranges where the super-resolution effect can be obtained when m = 4 are indicated by ranges P4a to P4d.

  Also in FIG. 21, ranges where the super-resolution effect is obtained when m = 4 are indicated by ranges P4a to P4d. Similarly, in FIG. 21, ranges where the super-resolution effect is obtained when m = 3 are shown as ranges P3a to P3c, and ranges where the super-resolution effect is obtained when m = 2 are shown as ranges P2a, This is indicated by P2b.

  Therefore, in this example, for example, when the moving speed v in the X-axis direction of the block image of the input frame Fa is, for example, the speed indicated by va in FIG. It is assumed that the effect is 2 (pixel), and the X-axis direction of the block image is thinned by the thinning amount 2. Further, for example, when the moving speed v in the X-axis direction of the block image of the input frame Fa is a speed indicated by vb faster than va in the range of 0 to 4/3, for example, a super-resolution effect can be obtained m = 2. The maximum 4 of 3 and 4 is set, and the X-axis direction of the block image is thinned by the thinning amount 4.

  When the moving speed v in the X-axis direction of the block image of the input frame Fa is within the range P0 (when the super-resolution effect cannot be obtained), the thinning amount m is set to 1 and the number of pixels All conversions are performed by spatial filtering. In other words, in the present invention, for a region where the moving speed v of the subject image Wa is high, the thinning process is performed so as to include the aliasing component to enable the super resolution effect, and the moving speed is low (range P0) (super resolution effect). In the region where (a) cannot be obtained, band limitation is performed by spatial filter processing so as not to return.

  If the frame rate is high, the range in which the super-resolution effect cannot be obtained is narrowed as described above (the range of speed at which the super-resolution effect can be obtained becomes large), and thinning can be performed with a larger thinning amount. .

  Summarizing the selection of the thinning amount, the table in FIG. 21 is obtained. That is, the control unit 44 determines the thinning amount based on this table (the same table for the Y-axis direction).

  For example, the control unit 44 stores a table as shown in FIG. 21 in the storage unit 44A, and detects the thinning amount m (mx, my) corresponding to the moving speed v in the table. The control unit 44 controls the spatial filter processing unit 41 to convert the block image ((ip × jq) pixels) of the input frame Fa into (mxp × myq) pixels (decimation by spatial filter processing for band limitation). In addition, the spatial thinning processing unit 42 is controlled to perform thinning with the thinning amounts mx and my on the (mxp × myq) pixel image supplied from the spatial filter processing unit 41 (bandwidth). Let me thin out without restriction).

  When the moving speed v is within the range P0, the control unit 44 cannot obtain the super-resolution effect in any of m = 2, 3 and 4. In this case, the pixel conversion is performed for all the pixel conversions with the thinning amount set to 1. The filtering is performed by the processing unit 41.

  Next, the operation of the image conversion unit 12 will be described with reference to the flowchart of FIG.

  In step S1, the block matching circuit 34 of the movement amount detection unit 21 of the image conversion unit 12 initializes a block number (for example, 1), and in step S2, detects a motion vector for all blocks of the input frame Fa. Determine whether or not.

  If it is determined in step S2 that motion vectors have not been detected for all blocks, the process proceeds to step S3, where the block matching circuit 34 of the movement amount detection unit 21 performs block matching processing on the block image indicated by the block number. And the motion vector of the block image is detected. The total number of blocks is (x / p × y / q), and each block image of the input frame Fa can be identified by block numbers 1 to (x / p × y / q). Yes.

  Specifically, the block matching circuit 34 compares the target block image of the current frame Fa with an arbitrary block image in the search area of the previous frame Fa, and detects the block position of the previous frame Fa that minimizes the mean square error. A vector connecting the target block of the current frame Fa and the detected block of the previous frame Fa is defined as a motion vector. The block matching circuit 34 supplies the block image of the current frame Fa to the spatial filter processing unit 41 of the pixel number conversion unit 22 and supplies the motion vector of the block image to the control unit 44.

  Next, in step S4, the control unit 44 of the pixel number conversion unit 22 moves the moving speed v (moving amount v in the X-axis direction) based on the motion vector supplied from the moving amount detecting unit 21 (block matching circuit 34). ) Then, the control unit 44 detects the thinning amount mx corresponding to the moving speed v (moving amount v) from the table representing the relationship between the moving speed v and the thinning amount m shown in FIG.

  In step S5, the control unit 44 detects the movement speed v (movement amount v) in the Y-axis direction based on the motion vector supplied from the movement amount detection unit 21 (block matching circuit 34). Then, the control unit 44 detects the thinning amount my corresponding to the moving speed v (moving amount v) from the table representing the relationship between the moving speed v in the Y-axis direction and the thinning amount m.

  Next, in step S <b> 6, the control unit 44 of the pixel number conversion unit 22 controls the spatial filter processing unit 41 to block the block image ((ip × jq) pixels of the input frame Fa supplied from the movement amount detection unit 21. ) Is converted into an image of (mxp × myq) pixels (see the left side of FIG. 20) (the block image is subjected to spatial filter processing).

  For example, when the thinning amount mx = 4, the control unit 44 controls the spatial filter processing unit 41 to perform spatial filter processing for obtaining an image of (4p × q) pixels. In addition, when the thinning amount mx = 3, the control unit 44 controls the spatial filter processing unit 41 to perform spatial filter processing for obtaining an image of (3p × q) pixels. Here, the pixel in the X-axis direction has been described as an example, but the same applies to the pixel in the Y-axis direction.

  Next, in step S7, the control unit 44 controls the spatial thinning processing unit 42 and is supplied from the spatial filter processing unit 41 according to the thinning amounts mx and my determined in steps S4 and S5 (mxp × myq). The pixel image is thinned out. Thus, the spatial thinning processing unit 42 uses the thinning amount mx in the X-axis direction and the thinning-out operation in the Y-axis direction with respect to the block image of (mxp × myq) pixels supplied from the spatial filter processing unit 41. A thinning process is performed with the amount my. As a result, a block image of (p × q) pixels is obtained (see the right side of FIG. 20).

  For example, when the thinning amount mx = 4, the thinning processing is performed on the (4p × q) pixels supplied by the spatial filter processing unit 41 with the thinning amount 4 in the X-axis direction, and (p × q) A block image of pixels is generated. When the thinning amount mx = 3, the thinning processing is performed on the (3p × q) pixel image supplied by the spatial filter processing unit 41 with the thinning amount 3 in the X-axis direction, and (p × q ) A block image of pixels is generated.

  When space thinning is performed, aliasing components are generated. However, since the thinning amounts mx and my are thinning amounts satisfying Expression (8), the aliasing components do not cancel each other.

  The spatial thinning processing unit 42 supplies a block image of (p × q) pixels obtained as a result of the thinning processing to the frame memory 43. The frame memory 43 stores the block image at a predetermined position and synthesizes one display frame Fb.

  In step S8, the block matching circuit 34 of the movement amount detector 21 increments the block number by one.

  Thereafter, the process returns to step S2, and the subsequent processing is performed.

  If it is determined in step S2 that motion vectors have been detected for all blocks, the process ends.

  As described above, the resolution of one input frame Fa is converted for each block by the processing in steps S3 to S7, so that one display frame Fb is generated and displayed at the display image frame rate. Are output to the image display unit 13. As a result, the observer can perceive an image having a resolution exceeding the resolution of the display subject image Wb, but the spatial resolution perceived by the observer due to the super-resolution effect is the block image ((p × q) pixels of the display frame Fb. ) Corresponds to the resolution of an image of (mxp × myq) pixels in which the X-axis direction is mx times and the Y-axis direction is my times.

  As described above, the motion is detected for each block of the image, and appropriate spatial filter processing and spatial thinning processing are performed in the spatial filter processing unit 41 and the spatial thinning processing unit 42, respectively. With respect to the block, the super-resolution effect is obtained, and with respect to the block having a small motion, the aliasing distortion is reduced, and as a whole, a high-quality image can be perceived.

  Furthermore, the super-resolution effect can be easily obtained without changing the structure of the display element as in the prior art.

  FIG. 24 shows a configuration example of an image processing apparatus 51 to which the present invention is applied. In this image processing apparatus, an image output unit 61 is provided instead of the image display unit 13 of the image display apparatus 1 of FIG. The other parts are the same as in the case of the image display device 1 of FIG.

  The image conversion unit 12 of the image processing device 51 converts the input frame Fa input from the image input unit 11 into a display frame Fb having a resolution that can be displayed on the display device 52, as in the example of FIG. . The image conversion unit 12 outputs the generated display frame Fb to the image output unit 61.

  The image output unit 61 converts the image signal of the display frame Fb supplied from the image conversion unit 12 into an image signal format that can be received by the display device 52, and outputs the image signal format to the display device 52. The display device 52 is basically configured in the same manner as the image display unit 13 of FIG. 18, and displays an image corresponding to the image signal supplied from the image processing device 51 (image output unit 61) on the display device.

  That is, the present invention can also be applied to an image processing apparatus that does not include a display device.

  The series of processes described above can be realized by hardware, but can also be realized by software. When a series of processing is realized by software, a program constituting the software is installed in a computer, and the management device 2 described above is functionally realized by executing the program by the computer.

  FIG. 25 is a block diagram illustrating a configuration of an embodiment of a computer 101 that functions as the image display device 1 and the image processing device 51 as described above. An input / output interface 116 is connected to a CPU (Central Processing Unit) 111 via a bus 115, and the CPU 111 receives commands from an input unit 118 such as a keyboard and a mouse via the input / output interface 116. When input, it is stored in a recording medium such as a ROM (Read Only Memory) 112, a hard disk 114, or a magnetic disk 131, an optical disk 132, a magneto-optical disk 133, or a semiconductor memory 134 attached to the drive 120, for example. The program is loaded into a RAM (Random Access Memory) 113 and executed. Thereby, the various processes described above (for example, the process shown by the flowchart of FIG. 23) are performed. Further, the CPU 111 outputs the processing result to a display unit 117 such as an LCD (Liquid Crystal Display) via the input / output interface 116 as necessary. The program is stored in advance in the hard disk 114 or the ROM 112 and provided to the user integrally with the computer 101 or as a package medium such as the magnetic disk 131, the optical disk 132, the magneto-optical disk 133, and the semiconductor memory 134. Or can be provided to the hard disk 114 via the communication unit 119 from a satellite, a network, or the like.

  In the present specification, the step of describing the program provided by the recording medium is not limited to the processing performed in chronological order according to the described order, but is not necessarily performed in chronological order. It also includes processes that are executed individually.

It is a figure explaining the rule of a block. It is a figure explaining the principle of resolution conversion processing. It is a figure explaining a sampling position. It is a figure explaining a return component. It is a figure explaining the change of the phase of a return component. It is another figure explaining the change of the phase of a return component. It is another figure explaining the change of the phase of a return component. It is another figure explaining the change of the phase of a return component. It is another figure explaining the change of the phase of a return component. It is another figure explaining the rotation interval of the phase of a return component. It is another figure explaining the rotation interval of the phase of a return component. It is another figure explaining a return component. It is another figure explaining the rotation interval of the phase of a return component. It is a figure explaining the phase of the return component for obtaining a super-resolution effect. It is a figure which shows the speed range which cannot acquire a super-resolution effect. It is another figure which shows the speed range which cannot acquire a super-resolution effect. It is another figure which shows the speed range which cannot acquire a super-resolution effect. It is a block diagram which shows the structural example of the image display apparatus to which this invention is applied. It is a block diagram which shows the structural example of the motion detection part and pixel number conversion part of FIG. It is a figure explaining the operation | movement of the pixel number conversion part of FIG. It is a figure which shows the relationship between the moving speed of an image in case a super-resolution effect can be acquired, and the thinning-out amount. It is another figure which shows the relationship between the moving speed of an image in case a super-resolution effect can be acquired, and the thinning-out amount. 20 is a flowchart for explaining the operation of the image processing unit in FIG. 19. It is a block diagram which shows the structural example of the image processing apparatus to which this invention is applied. And FIG. 16 is a block diagram illustrating a configuration example of a personal computer.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 1 Image display apparatus, 11 Image input part, 12 Image conversion part, 13 Image display part, 21 Movement amount detection part, 22 Pixel number conversion part, 31 Frame memory, 32 Delay circuit, 33 Frame memory, 34 Block matching circuit, 41 Spatial filter processing unit, 42 Spatial thinning processing unit, 43 frame memory, 44 control unit, 51 image processing device, 52 display device, 61 image output unit

Claims (16)

An image input step of inputting an image signal having a first resolution;
A resolution conversion step of converting the input image signal into an image signal having a second resolution lower than the first resolution;
An image display control step of displaying on the display means an image corresponding to the image signal whose resolution has been converted at a predetermined frame rate,
The resolution conversion step detects a movement amount of at least a part of the region of the image signal input in the image input step, and uses the predetermined amount of space with a spatial thinning amount determined based on the detected movement amount. An image display method characterized by performing a spatial thinning process that gives a visual effect such that display means for displaying at a rate has a resolution exceeding the second resolution. The horizontal thinning amount in the horizontal direction is determined based on the horizontal movement amount of the region, and the vertical spatial thinning amount is determined based on the vertical movement amount of the region. Item 2. The image display method according to Item 1. The image display method according to claim 1, wherein when the frame rate is high, an amount of space thinning in the space thinning process is increased. In the spatial thinning process for an arbitrary frame, the spatial thinning amount is determined so that all the aliasing components of the spatial thinned signal in a predetermined number of frames that are temporally adjacent to the frame temporally cancel each other. The image display method according to claim 1, wherein: The image display method according to claim 1, wherein the resolution conversion step performs the spatial thinning process on a signal obtained by subjecting the input image signal to a spatial filter process. The image display method according to claim 1, wherein the resolution conversion step divides the image signal input in the image input step into a plurality of regions, and detects the movement amount for each of the divided regions. An image input step of inputting an image signal having a first resolution;
A resolution conversion step of converting the input image signal into an image signal having a second resolution lower than the first resolution;
An image display control step of displaying the resolution-converted image signal on a display means at a predetermined frame rate,
The resolution conversion step divides the image signal input in the image input step into a plurality of regions, detects a movement amount for each region, and calculates a spatial thinning amount of each region based on the detected movement amount. Display means for setting and displaying each predetermined area at the predetermined frame rate based on the set spatial thinning amount after performing spatial filter processing on each area of the input image signal An image display method characterized by performing a spatial thinning process that gives a visual effect that has a resolution exceeding the second resolution. Image input means for inputting an image signal having a first resolution;
Resolution conversion means for converting the image signal input to the image input means into an image signal having a second resolution lower than the first resolution;
An image display control unit for displaying an image signal whose resolution is converted by the resolution conversion unit on a display unit at a predetermined frame rate,
The resolution conversion unit includes a movement amount detection unit that detects a movement amount of at least a partial region of the image signal input by the image input unit, and a spatial thinning amount determined based on the detected movement amount. An image display apparatus comprising: a resolution conversion processing unit that performs a spatial thinning process that provides a visual effect such that the display unit that displays at the predetermined frame rate has a resolution exceeding the second resolution. The horizontal spatial thinning amount is determined based on the horizontal movement amount of the region, and the vertical spatial thinning amount is determined based on the vertical movement amount of the region. The image display device according to claim 8. The image display apparatus according to claim 8, wherein when the frame rate is high, a spatial thinning amount in the spatial thinning process is increased. In the spatial thinning process for an arbitrary frame, the spatial thinning amount is determined so that all the aliasing components of the spatial thinned signal in a predetermined number of frames that are temporally adjacent to the frame temporally cancel each other. The image display device according to claim 8, wherein: The resolution conversion unit further includes a storage unit for storing a table of spatial thinning amounts corresponding to the movement amount, and the spatial thinning amount is determined by referring to the table. The image display device according to claim 8. The image display device according to claim 8, wherein the resolution conversion processing unit performs the spatial thinning process on a signal obtained by subjecting the input image signal to a spatial filter process. The image display apparatus according to claim 8, wherein the movement amount detection unit divides the image signal input by the image input unit into a plurality of regions, and detects a movement amount for each of the divided regions. Image input means for inputting an image signal having a first resolution;
Resolution conversion means for converting the image signal input to the image input means into an image signal having a second resolution lower than the first resolution;
An image display control unit for displaying an image signal whose resolution is converted by the resolution conversion unit on a display unit at a predetermined frame rate,
The resolution conversion means includes
A movement amount detector that divides the image signal input to the image input means into a plurality of regions and detects a movement amount for each region;
Based on the detected amount of movement, a spatial thinning amount of each region is set, and after applying spatial filtering processing to each region of the input image signal, based on the set spatial thinning amount And a resolution conversion processing unit that performs a spatial thinning process that provides a visual effect such that the display means for displaying the image at the predetermined frame rate for each region has a resolution exceeding the second resolution. Display device. On the computer,
An image input control step for controlling input of an image signal having a first resolution;
A resolution conversion control step for controlling conversion of the input image signal into an image signal having a second resolution lower than the first resolution;
An image display program for executing a process including an image display control step of displaying an image corresponding to the image signal having undergone resolution conversion on a display unit at a predetermined frame rate,
The resolution conversion control step detects a movement amount of at least a partial region of the image signal input in the processing of the image input control step, and is a spatial thinning amount determined based on the detected movement amount. A program for performing a spatial thinning process for providing a visual effect such that the display means for displaying at the predetermined frame rate has a resolution exceeding the second resolution.

Images