JP2000194845A - Image processor, image processing method and image processing system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processor, an image processing method and an image processing system capable of improving image quality in converting to a high resolution still image, based on plural low resolution images. SOLUTION: Among the plural frames of moving images stored in a storage part 101, the motion vector of an m-th frame and an (m+n)-th frame is extracted in a motion vector computation part 102 and the motion vector is divided by a frame number among the frames in a vector division part 103. Then, based on the divided vector, the pixels of an (m+a)-th frame (a<n) are respectively arranged in an arrangement part B105 and synthesized with the pixels of the m-th frame in a synthesis part 106.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

The present invention relates to an image processing apparatus and method, and an image processing system, and more particularly to an image processing apparatus and method for converting image information into high resolution, and an image processing system.

[0002]

2. Description of the Related Art Conventionally, various methods have been proposed as a method of converting the resolution of input low-resolution information into high-resolution information.

In the resolution conversion method conventionally proposed, high resolution is realized by interpolating pixels with respect to low-resolution information, and the type of target image (for example, each pixel is The conversion processing method is different depending on a multivalued image having gradation information, a binary image binarized by a pseudo halftone, a binary image binarized by a fixed threshold, a character image, and the like.

As a method of pixel interpolation in a conventional resolution conversion method, a nearest interpolation method in which the same pixel value closest to an interpolation point is arranged as shown in FIG. 11 or an interpolation method as shown in FIG. Four points surrounding the insertion point (the pixel values of the four points are A, B, C,
D), the interpolation point E is calculated by the following calculation.
Is commonly used to determine the pixel value of the pixel. E = (1-i) (1-j) A + i (1-j) B + j (1-i) C + ijD Equation 1 (However, if the distance between pixels is 1, interpolation points Let E be a distance from A to i in the horizontal direction and j in the vertical direction (i ≦
1, j ≦ 1)) Further, as represented by the sampling theorem, as a means for converting a sampled discrete signal into a continuous signal, the signal is passed through an ideal low-pass filter that can be expressed by a SINC function. Thus, a continuous signal can be reproduced. Further, since the calculation of the SINC function takes a long processing time, there is a method of approximating an interpolation function represented by the SINC function and calculating an interpolation value only by a simple product-sum operation.

[0005] For example, according to "Image Analysis Handbook: Mikio Takagi and Hirohisa Shimoda, University of Tokyo Press", an approximation of an interpolation function can be realized in the cubic convolution interpolation method. A method for calculating an interpolation value by the interpolation method will be described with reference to FIG. FIG.
In the pixel arrangement shown in (1), P is an interpolated point (interpolation point), and P11 to P44 indicate the pixel values of the surrounding 16 pixels.
Then, the interpolation points are interpolated using a tertiary convolution function represented by the following equation. In the following equation, x ^ y
Denote x to the y-th power. (However, [] indicates a Gaussian symbol and takes an integer part.) However, as a result of the resolution conversion by the above three kinds of interpolation methods, all of the results are blurring by interpolation and a block-like shape depending on the input low resolution. Jaggy etc. will occur,
High-resolution high-resolution information could not be created.

Accordingly, the present applicant has developed a method for generating resolution information from low resolution information without interpolating blur caused by interpolation processing and enabling resolution conversion without jaggies. JP-A-7-93531, JP-A-7-1
No. 07268 and JP-A-7-105359.

The basic idea in these proposals is as follows:
After removing the resolution-dependent component from the input source information,
This is a method in which the number of pixels is increased to an output resolution equivalent, information corresponding to the resolution (high resolution) to be converted in that state is estimated, and finally image information subjected to resolution conversion is created. For example, removing the dependency of the input resolution can be realized by smoothing with a low-pass filter (LPF), and increasing the number of pixels can be realized by linear interpolation. The high-resolution information is estimated by simply binarizing the information after interpolation and performing different processing for pixels classified as "1" and pixels classified as "0", and outputs the processed data. Calculate the pixel value.

Further, the applicant has proposed a method for producing a good edge while maintaining the continuity of pixel values in Japanese Patent Application Laid-Open No. 9-252400. According to the proposal, a pixel at m points (m ≧ 1) (where the pixel value at the observation point n among the m points is P (n)) is detected from the peripheral pixels of the low-resolution target pixel, and the target pixel is detected. An output value h (k) is calculated by the following equation based on the interpolation value C (k) at each interpolation point k obtained by interpolating a pixel into a plurality of pixels. h (k) = Σα (n) P (n) + βC (k) (The range of Σ is from n = 1 to m) Equation 5 where α (n) and β are arbitrary coefficients, and β ≠ 0

However, the resolution conversion by the above-described conventional interpolation method has the following disadvantages. That is, no matter how much high-resolution information is created, there is a limit to high image quality.

As is apparent from the sampling theorem, since information having an input resolution equal to or higher than the Nyquist limit does not exist in an input image, all information generation having a frequency higher than the Nyquist frequency is based on estimation. Therefore, the less complex C
It is easy to convert a flat artificial image such as a G image, an illustration image, or an animation image into a jaggy-less image, but it is difficult to increase the quality of a natural image by estimating information equal to or more than the Nyquist limit. That is, no matter what method is used, an image obtained by inputting low-resolution information and converting it to high-resolution clearly has lower image quality than an image originally input as high-resolution information.

[0010] On the other hand, with the recent spread of digital video cameras and the like, it is possible to easily input a captured moving image to a computer in a continuous frame unit. Therefore, it is possible to print out one frame of a moving image by a printer, but the input resolution of the imaging system tends to increase, but is still lower than the output resolution of the printer, which is increasing year by year. is the current situation.

Therefore, instead of creating one high-resolution still image from one low-resolution still image as described in the above-mentioned conventional example, a plurality of continuous low-resolution still images captured from a moving image are obtained. It is conceivable to create one high-resolution still image from.

As a technique for creating a wider-range panoramic image from a plurality of still images, a technique for synthesizing a background image in consideration of panning of a moving image: Yoshizawa, Hanamura, Tominaga, and Shingaku Spring Full Preliminary Collection 7-51 (1990) "and" Generation method of panoramic image by divisional imaging: Nakamura, Kaneko, Hayashi, and IEICE Spring Full Preliminary Proceedings 7-165 (1991) ".
However, these are proposals of a technique for creating a panoramic image in which the imaging range is enlarged more than one still image. The imaging range is the same, information of a plurality of still images is combined, and the resolution of the image is determined by interpolation. It does not improve.

As a technique for creating a high-resolution still image from a low-resolution moving image, there is a proposal in Japanese Patent Application Laid-Open No. 5-260264. In the proposal, comparing successive images,
Based on the difference, parameters of affine transformation and translation are detected, and these images are synthesized.
An example in which the synthesis method is used for interpolation is also mentioned.

However, this proposal has the following problems.

That is, in the method of using the above-mentioned synthesizing method for interpolation, by comparing continuous images enlarged by the above-described interpolation method shown in FIGS.
The above-described parameters are calculated, the interpolation position is determined, and the interpolation positions are synthesized. Since the interpolation calculation itself does not create new high-resolution information, it is difficult to accurately determine coordinates to be combined.

Here, "interpolating" means interpolating between pixels. However, in the interpolation by the above-described synthesis method, there is no information between pixels of the input resolution when comparing continuous images. In other words, considering the case where two types of images A and B are combined, the determination of where to interpolate the pixels of the image B between the pixels of the image A is merely a comparison between the enlarged images. Is difficult.

This is because the minimum unit of the vector amount of the motion vector is a pixel unit, and there is no resolution finer than the distance between pixels. That is, if the resolution of the vector does not have an accuracy equal to or less than the pixel-to-pixel distance, the effect of interpolating using a plurality of still images is diminished. There is almost no difference in image quality from the case of creating high resolution still images.

SUMMARY OF THE INVENTION The present invention has been made to solve the above problems, and it is possible to improve the image quality when converting a plurality of low-resolution images into one high-resolution still image. It is an object of the present invention to provide an image processing apparatus and method, and an image processing system.

The present invention also provides an image processing apparatus and method for converting a plurality of low-resolution images into a single high-resolution still image by performing interpolation with the resolution of a motion vector smaller than the pixel-to-pixel distance. , And an image processing system.

It is another object of the present invention to provide an image processing system capable of outputting a high-quality image between models having different resolutions at the time of input and output.

[0021]

In order to achieve the above object, an image processing apparatus according to the present invention has the following arrangement.

That is, input means for inputting a plurality of low-resolution images, detecting means for detecting a difference between the plurality of low-resolution images, and generation for generating a high-resolution image from the low-resolution images based on the differences. Means.

For example, the input unit inputs images of a plurality of frames in a moving image, the detection unit detects a motion vector between the plurality of frames, and the generation unit detects the motion vector of the plurality of frames based on the motion vector. A high-resolution still image is generated from the image.

For example, the input means inputs a plurality of continuous frames from a first frame to a second frame of a moving image, and the detecting means calculates a motion vector between the first frame and the second frame. Detecting, wherein the generation unit generates a still image by interpolating pixels of a frame between the first frame and the second frame into an image of the first frame based on the motion vector. And

Further, the generating means includes a dividing means for dividing the motion vector by the number of frames between the first frame and the second frame, and a first arranging means for arranging the pixels of the first frame. And the first frame and the second frame based on the motion vector divided by the dividing means.
Second arranging means for arranging the pixels of the frames between the frames, and synthesizing means for synthesizing the pixels arranged by the first and second arranging means to generate one still image. It is characterized by.

For example, if the dividing means divides the motion vector into n, the second arranging means arranges pixels of the a-th frame from the first frame at a / n positions of the motion vector. It is characterized by doing.

[0027]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An embodiment according to the present invention will be described below in detail with reference to the drawings.

It should be noted that the image processing apparatus according to the present invention is mainly provided with a printer or a printer connected directly or via a computer to an analog video camera or digital video camera for capturing moving images. It is efficient to provide it inside an image output device such as a video printer. In addition, an image processing device serving as an intermediate adapter when a video camera is connected to a printer, or application software in a host computer,
It can also be incorporated as printer driver software for outputting to a printer.

<First Embodiment> [Apparatus Configuration] FIG. 1 is a block diagram showing a functional configuration of a computer which is an image processing apparatus of the present embodiment.
The operation procedure will be described with reference to FIG. In the present embodiment, an example will be described in which an image captured by a digital video camera is transmitted to a computer and converted to a resolution equivalent to a printer by application software in the computer.

In the figure, reference numeral 100 denotes an input terminal for inputting a moving image picked up by a video camera into a computer.
A user reproduces a moving image captured by digital video from a recording medium and sends a command to capture an image in a desired scene. In synchronization with this fetch instruction, image information of a plurality of continuous frames immediately after the fetch instruction is stored in the storage unit 101 in the computer. Reference numeral 102 denotes a motion vector calculation unit, which is a unit that measures a movement amount of a partial movement as a vector based on a difference between two types of images. The details of the motion vector calculation unit 102 will be described later. 10
Reference numeral 3 denotes a vector division unit that divides the calculated vector into a plurality of vectors, the details of which will be described later.

Reference numeral 104 denotes an arrangement unit A, which controls the arrangement of pixels of a captured image. Reference numeral 105 denotes an arrangement unit B,
In accordance with the vector amount divided by the vector dividing unit 103, a pixel arrangement to be interpolated in the image in which the pixels are arranged by the arrangement unit A104 is controlled. Reference numeral 106 denotes a synthesizing unit, which synthesizes images in which pixels are arranged by the arranging units A104 and B105. Reference numeral 107 denotes an interpolation unit which, when the image synthesized by the synthesizing unit 106 does not yet have the interpolation point information filled up to the desired resolution, calculates the interpolation point information that is not embedded by interpolation. By generating enough pixels to achieve a desired resolution by the interpolation unit 107, a higher resolution in the present embodiment is achieved. Reference numeral 108 denotes an output terminal for transmitting image information having a high resolution to a printer or the like.

Reference numeral 110 denotes a CPU, which controls the operations of the above-described components according to a control program stored in the ROM 111. RAM 112 is CPU 110
Used as a work area.

[Motion Vector Calculation] Next, the processing in the motion vector calculation unit 102 will be described in detail.

Various methods have been proposed for calculating a motion vector. In the following, we introduce the Residual Sequential Test (SSDA method) in template matching described in "Image Analysis Handbook: Mikio Takagi and Hirohisa Shimoda Supervised by the University of Tokyo Press".

As shown in FIG. 2, a template or block of N1 × N1 pixels is set to a larger M1.
× M1 pixel search range in the input image (M1-N1 + 1) ^ 2
By moving the above, by obtaining the upper left position of the template image such that the residual represented by the following Expression 6 is minimized,
It is considered that the superposition (matching) has been achieved. R (a, b) = ΣΣ | I (a, b) (m1, n1) -T (m1, n1) |… Equation 6 (However, the left Σ is from m1 = 0 to N1-1, and the right Σ is n1 = 0 to N1-1
Where (a, b) indicates the upper left position of the template image in the input image, I (a, b) (m1, n1) is a partial image of the input image, and T (m1, n1) Is a template image.

At this time, if the superposition is displaced, the residual increases rapidly when sequentially adding each pixel. Therefore, if the residual exceeds a certain threshold value in the middle of the addition shown in equation (6). A method of judging that the superposition is insufficient and discontinuing the addition and proceeding to the next operation (a, b) is a residual sequential test method.

That is, here, when two types of images are determined to be continuous images on a moving image, the following method is used.
It is easy to quantify the geometric deviation between the two.

As a method of calculating a motion vector, a method with higher accuracy has been proposed, but in the present embodiment, the method based on the above-described template matching is used for ease of explanation.

The motion vector obtained by the above-described template matching is for detecting a superposition in which the residual is minimized, and the resolution of the vector is in units of pixels as described above. That is, it does not have a resolution finer than the distance between pixels. Although it is very effective to use template matching as motion compensation during encoding, finer resolution is required for application to interpolation, ie, an interpolation technique for filling information between pixels.

Therefore, in the present embodiment, two consecutive
Instead of calculating the motion vector of the kind image, the motion vector is calculated for two kinds of images that are temporally separated in sampling.

That is, as two types of images transmitted from the storage unit 101 to the motion vector calculation unit 102, one of
An image immediately after the user issues a capture command (referred to as an m-th frame), and the other image is referred to as an image ((m + n) -th frame after an n-th frame from the m-th frame, where n>
1). Here, the case where n = 3 will be described as an example. The motion vector calculation unit 102 detects what kind of vector movement has occurred between the image of the m-th frame and the image of the (m + 3) -th frame. Things. Naturally, the resolution of the motion vector calculated at this time is in pixel units.

[Vector division] Next, the vector division unit 10
3 will be described in detail.

The vector dividing unit 103 divides a vector amount moved between n frames into a vector amount per frame based on the motion vector amount calculated by the motion vector calculating unit 102 by the value of n. And a divider.

As described above, the feature of the present invention resides in that the resolution of a motion vector is set finer than the distance between pixels, and interpolation is performed. Therefore, in the present embodiment, it is assumed that the motion vector moves linearly in time.
If the motion vector moves linearly as described above, the motion vector is calculated with a time difference of a plurality of frames and converted for each frame, so that a vector finer than the pixel between consecutive frames can be calculated. It becomes possible.

Since the present embodiment is established based on the above assumptions, the following slight restrictions are provided. That is, when the motion vector amount between n frames is too large compared to the value of n, the accuracy of the vector amount per frame is reduced.

However, if the target image is not a fast-moving one such as a sports scene, but rather a relatively small one such as a commemorative photograph, a landscape, a plant, a still object, etc., an absolute number of blocks in the image are absolute. Small moving amount. Therefore, in the motion vector detection method based on template matching shown in FIG. 2, since the motion amount naturally differs in block units, it may be possible to adopt a configuration in which a block having a movement amount larger than a predetermined value is not divided.

[Outline of Pixel Arrangement] The arrangement unit A104 arranges pixels in the m-th frame, and the arrangement unit B105 arranges interpolation pixels for the image in the m-th frame in accordance with the vector division amount. That is, the frame input to the arrangement unit B105 is the (m + a) -th frame (a = 1 to (n−1)), and
1) The number of frames is sequentially input, and the pixel values of the interpolation pixels are arranged according to the division amount.

At this time, the division amount transmitted from the vector division unit 103 is 1 / n of the motion vector,
At 105, this is multiplied by a. That is, the motion vector × (a / n) is the arrangement position when the (m + a) th frame is input.

Here, assuming that n = 3, the arrangement portion B10
5 receives 1/3 as the motion vector division amount and obtains (m
In the (+1) th frame, 1/3 of the motion vector is arranged, and in the (m + 2) th frame, 2/3 of the motion vector is arranged.

FIG. 3 is a diagram showing an example of the movement of an actual image, in which a photographed object (or a camera) is gradually moving in an oblique direction. (A) to (d) of FIG. 3 represent images of the mth to (m + 3) th frames, respectively.

FIG. 4 is a diagram showing the state of the motion vector in the example shown in FIG. 4A is an image of the m-th frame, and FIG. 4B is an image of the (m + 3) -th frame. Here, it is assumed that the images shown in FIGS. 4A and 4B are used as the two types of images for calculating the motion vector. In FIG. 4, a block surrounding an object is a block for calculating a motion vector, and corresponds to the N1 × N1 block shown in FIG. Here, for ease of explanation, it is assumed that the motion vector is intentionally common to all blocks. The motion vectors obtained by (a) and (b) of FIG. 4 are shown in (c) of FIG.

FIG. 5 shows how the motion vector shown in FIG. 4C is divided. 5A shows the calculated motion vectors for three frames, FIG. 5B shows the vector obtained by reducing the vector of FIG. 5A to １, and FIG. 5 (a)
Is reduced to 2/3. That is, FIG.
Since the vector amount shown in (a) has moved between three frames, the vector amount is divided by the number of frames.

FIG. 6 shows a state in which blocks are moved according to the vector division amount calculated as shown in FIG. 6A shows an image that does not move at all, FIG. 6B shows an image obtained by moving the vector amount by /, FIG. 6C shows an image obtained by moving the vector amount by /, FIG. 6D is an image obtained by moving the vector amount by one. Here, FIG.
(A) and (d) correspond to (a) and (d) in FIG. 3, respectively. However, (b) and (c) of FIG.
(b) and (c) do not completely match. As described above, the present embodiment is based on the assumption that the motion vector is linear on the time axis (however, in a short time and when the moving distance is very small). The position of c)) is based on a guess.

[Details of Pixel Arrangement] Hereinafter, the pixel arrangement in the present embodiment will be described in more detail.

FIG. 7 is a diagram showing the details of the vector division, where the intersections of the vertical and horizontal lines indicate the positions of the respective pixels. The arrow between the circles in FIG.
It shows a motion vector calculated between images of (m + n) frames. Hereinafter, the description will be made on the assumption that n = 3.

The motion vector shown in FIG.
The pixel has moved up by two pixels. In other words, this indicates that the vector movement corresponding to the arrow has occurred in three frames. In FIG. 7B, an arrow connecting a circle and a triangle indicates that the arrow in FIG.
(A) shows the amount moved by / of the vector shown in FIG. That is, in the (m + 1) th frame, it is assumed that the pixel marked with a circle has moved to the mark marked with a triangle. Similarly, FIG.
In FIG. 7, the arrow connecting the mark and the mark indicates the amount moved by 2/3 of the vector shown in FIG. That is,
In the (m + 2) -th frame, it is assumed that the pixel marked by ○ has moved to the mark ×.

FIG. 8 is a diagram showing a state in which information for three frames of m, (m + 1) and (m + 2) frames is arranged and synthesized based on the amount of divided vectors shown in FIG. . That is, the information of the m-th frame (marked by ○) is arranged without the movement amount, the information of the (m + 1) th frame (marked by △) is arranged at a position moved by / of the vector, and
The information of the frame (marked by x) is arranged at a position shifted by 2/3 of the vector. By controlling the arrangement in this way, interpolation between pixels can be performed as shown in FIG.

In the example shown in FIG. 8, three times interpolation is realized in the vertical direction. However, the interpolation operation in the horizontal direction or in the vertical direction is executed by the interpolation unit 107. As the interpolation method at this time, the methods shown in FIGS.

In the interpolation method according to the present embodiment, pixel information is not always arranged at a desired interpolation position between pixels. In that case, by performing an interpolation operation,
The information of the desired interpolation position is calculated. However, when a plurality of still images are combined, the pixel referred to in the interpolation calculation is closer to the interpolation position than when a single still image is obtained. Therefore, a higher definition image can be created.

In the present embodiment, an example has been described in which the resolution is increased by interpolating pixels. However, it is needless to say that the present invention can be similarly applied to the case of realizing enlargement / reduction.

As described above, according to the present embodiment,
By extracting the motion vectors for a plurality of frames and dividing by the number of frames, the resolution of the motion vector amount of the continuous frames can be set finer than one pixel unit. Therefore, by synthesizing a plurality of frames with a resolution finer than one pixel, a high-resolution image can be generated in higher detail.

Further, as compared with the case where the motion vector of a continuous frame is calculated, the number of times of calculating the motion vector is reduced, so that high-speed processing can be performed.

Accordingly, for example, one piece of high-resolution still image information can be easily created from low-resolution still image information captured by a video camera and output to a printer or the like.
In an image processing system for performing communication between models having different resolutions at the time of input / output, it is possible to output a high-quality image.

<Second Embodiment> Hereinafter, a second embodiment according to the present invention will be described.
An embodiment will be described.

FIG. 9 is a block diagram showing a functional configuration of a computer which is an image processing apparatus according to the second embodiment. Components similar to those shown in FIG. 1 according to the first embodiment are denoted by the same reference numerals. The description is omitted.

The second embodiment is characterized in that, when one still image is created, two types of images for which a motion vector is to be calculated are sequentially switched and further combined.

In FIG. 9, a counter 900 counts the number of arrangements based on each divided vector in arrangement section B105.

Here, similarly to the above-described first embodiment,
Example in which the motion vectors of the m-th frame and the (m + n) -th frame are calculated, and (n-1) -th frames from the (m + 1) -th frame to the (m + n-1) -th frame are arranged based on the divided vectors. Will be described.

When the counter 900 counts the arrangement of (n-1) sheets, the output frame control unit 901
Notify. Upon receiving the notification, the output frame control unit 901 selects, from among the frames stored in the storage unit 902,
Furthermore, two frames for which the next motion vector is to be calculated are specified. The next two frames are based on the (m + n) th frame, which is one of the previous two frames, and then n
A motion vector with the (m + 2n) th frame that has progressed by the frame is calculated.

The arrangement section A104 and the arrangement section B105
In, the pixel arrangement based on the motion vector is performed and synthesized by the synthesizing unit 106.
In (6), (m + n-
1) A composite image based on the frames up to the frame is completed. Therefore, a pixel arrangement based on the (m + n) th frame to the (m + 2n-1) th frame is further combined with the combined image.

As described above, in the second embodiment, images that have progressed by n frames are sequentially compared to determine the amount of motion, and based on the divided vector obtained by dividing the amount of motion by n,
Pixels are sequentially arranged. That is, as shown in FIG.
Assuming that b is an integer, a motion vector is obtained by the (m + n × b) th frame and the (m + n × (b + 1)) th frame, and based on the divided vector, the (m + n × b + a) th frame is obtained by the (n−1) th frame. It will be arranged every time. Then, by repeating this process by incrementing b by one and up to a predetermined upper limit, a higher-quality high-resolution image can be obtained.

As described above, according to the second embodiment, in addition to the effects obtained by the above-described first embodiment, by setting the upper limit value of b to be large, it is possible to use one still image. Since the number of frames is increased, the image quality after interpolation is further improved.

In the second embodiment, the values of n and b are
For example, it is possible to provide an image processing apparatus capable of performing resolution conversion with the highest image quality by setting an optimal value obtained experimentally.

In the second embodiment, the counter 900 is used.
Has been described as counting the number of arrangements in the arrangement unit B105, but the numerical value to be counted is not limited to the number of arrangements, and may be anything as long as the start timing of the next motion vector calculation can be defined.

<Third Embodiment> Hereinafter, a third embodiment according to the present invention will be described.
An embodiment will be described.

FIG. 10 is a block diagram showing a functional configuration of a computer which is an image processing apparatus according to the third embodiment.
The same components as those in FIG. 1 shown in the above-described first embodiment are denoted by the same reference numerals, and description thereof is omitted.

The third embodiment is characterized in that two types of images for which a motion vector is to be calculated are switched according to the magnitude of the calculated motion vector amount. That is, the comparison target frame is fed back based on the vector amount calculated by the motion vector calculation unit 1001.

Here, similarly to the above-described first embodiment,
In the motion vector calculation unit 1001, the m-th frame
Consider a case where a motion vector is calculated between the (m + n) th frame. At this time, (m +
When it is determined that the movement amount is larger than the preset threshold value, that is, the calculated motion vector value is larger in the (n) th frame, the output frame control unit 1002 is notified. In the output frame control unit 1002,
(m + n) One frame backward from the (m + n) th frame (m + n
-1) The storage unit 1 uses the frame number as a vector operation target.
003. Then, the storage unit 1003 stores (m + n-
1) The frame is output and the motion vector calculation unit 1001
Then, the m-th frame and the (m + n-1) -th frame are compared.

That is, the frame to be compared with the m frame in the motion vector calculation unit 1001 depends on the situation.
The (m + n−c) th frame (c = 0 to n−2). The two frames are compared while increasing the value of c by one. Even when c = n−2, if the motion vector is still larger than the predetermined threshold, the synthesis is not performed. , And outputs the image of the m-th frame as it is.

In the arranging section B105, the (m + a) th frame (a = 1 to (nc-1)) is input for (nc-1) frames, and the vector is divided by (nc). Pixel values are arranged according to the division amount.

As described above, according to the third embodiment, in addition to the effects obtained by the above-described first embodiment, a motion vector is calculated only between frames having a more appropriate moving amount and an appropriate By performing the insertion process, the image quality after interpolation is further improved.

In each of the above-described embodiments, the processing of calculating the motion vector amount for a frame temporally advanced from the m-th frame has been described. Of course, the calculation is also possible.

The method of calculating the motion vector is as follows.
Although template matching has been described as an example, other methods may of course be used. For example, in addition to the case where the movement between frames is only a parallel movement, a method of calculating the movement of the rotation system as an affine transformation coefficient, dividing the coefficient, and arranging pixels may be considered.

In each of the above-described embodiments, an example has been described in which an image captured by a video camera is temporarily recorded on a recording medium such as a video tape, and the recording medium is reproduced to store a desired plurality of frames. However, the plurality of frames to be processed in the present invention are not limited to the reproduced image from such an intermediate medium, and a plurality of frames are directly stored from a captured image according to a user's specification to create a high-resolution still image. It may be.

[0085]

[Other Embodiments] Even if the present invention is applied to a system including a plurality of devices (for example, a host computer, an interface device, a reader, a printer, etc.), an apparatus (for example, a copying machine) Machine, facsimile machine, etc.).

Further, an object of the present invention is to supply a storage medium storing a program code of software for realizing the functions of the above-described embodiments to a system or an apparatus, and to provide a computer (or CPU) of the system or apparatus.
And MPU) read and execute the program code stored in the storage medium.

In this case, the program code itself read from the storage medium implements the functions of the above-described embodiment, and the storage medium storing the program code constitutes the present invention.

As a storage medium for supplying the program code, for example, a floppy disk, hard disk, optical disk, magneto-optical disk, CD-ROM, CD
-R, a magnetic tape, a nonvolatile memory card, a ROM, or the like can be used.

When the computer executes the readout program code, not only the functions of the above-described embodiment are realized, but also the OS (Operating System) running on the computer based on the instruction of the program code. ) May perform some or all of the actual processing, and the processing may realize the functions of the above-described embodiments.

Further, after the program code read from the storage medium is written into a memory provided in a function expansion board inserted into the computer or a function expansion unit connected to the computer, based on the instruction of the program code, It goes without saying that the CPU included in the function expansion board or the function expansion unit performs part or all of the actual processing, and the processing realizes the functions of the above-described embodiments. When the present invention is applied to the storage medium, the storage medium stores program codes corresponding to the flowcharts described above.

[0091]

As described above, according to the present invention, a high-quality high-resolution still image can be generated based on a plurality of low-resolution images.

Further, it is possible to perform interpolation in which the resolution of the motion vector is made finer than the distance between pixels.

Further, it is possible to output a high quality image between models having different resolutions at the time of input / output.

[0094]

[Brief description of the drawings]

FIG. 1 is a block diagram showing a functional configuration of an image processing apparatus according to an embodiment of the present invention;

FIG. 2 is a diagram illustrating a motion vector calculation;

FIG. 3 is a diagram showing an example of a continuous image.

FIG. 4 is a diagram for explaining a motion vector;

FIG. 5 is a diagram illustrating vector division.

FIG. 6 is a diagram illustrating an arrangement for each frame;

FIG. 7 is a diagram illustrating vector division in pixel units.

FIG. 8 is a diagram illustrating the arrangement and combination of pixels.

FIG. 9 is a block diagram illustrating a functional configuration of an image processing apparatus according to a second embodiment.

FIG. 10 is a block diagram illustrating a functional configuration of an image processing apparatus according to a third embodiment;

FIG. 11 is a view for explaining a conventional nearest neighbor interpolation method;

FIG. 12 is a diagram illustrating a conventional bilinear interpolation method;

FIG. 13 is a view for explaining a conventional cubic convolution interpolation method;
It is.

[Explanation of symbols]

 Reference Signs List 100 input terminal 101, 902, 1003 storage unit 102, 1001 motion vector calculation unit 103 vector division unit 104 placement unit 105 placement unit B 106 synthesis unit 107 interpolation unit 108 output terminal 900 counter 901, 1002 output frame control unit

Claims (21)

[Claims] An input unit that inputs a plurality of low-resolution images; a detection unit that detects a difference between the plurality of low-resolution images; and a generation unit that generates a high-resolution image from the low-resolution images based on the differences. And an image processing device. 2. The input unit inputs images of a plurality of frames in a moving image, the detecting unit detects a motion vector between the plurality of frames, and the generating unit detects the motion vector of the plurality of frames based on the motion vector. The image processing apparatus according to claim 1, wherein a high-resolution still image is generated from the image. 3. The input means inputs a plurality of continuous frames from a first frame to a second frame of a moving image, and the detecting means detects a motion vector between the first frame and the second frame. Detecting, wherein the generating unit generates a still image by interpolating pixels of a frame between the first frame and the second frame into an image of the first frame based on the motion vector. The image processing device according to claim 2. 4. The generating means further comprises: a dividing means for dividing the motion vector by the number of frames between the first frame and the second frame; and a first arrangement for arranging pixels of the first frame. Means, means for arranging pixels of a frame between the first frame and the second frame based on the motion vector divided by the means for dividing, and first and second arranging means 4. An image processing apparatus according to claim 3, further comprising: synthesizing means for synthesizing the pixels arranged according to (1) and generating one still image. 5. When the dividing unit divides the motion vector into n, the second arranging unit arranges pixels of an a-th frame from the first frame at a / n positions of the motion vector. 5. The method according to claim 4, wherein
The image processing apparatus according to any one of the preceding claims. 6. The image processing apparatus according to claim 4, wherein said generating means further comprises an interpolation means for further performing an interpolation operation on the image synthesized by said synthesis means to increase the resolution. 7. The method according to claim 7, wherein the generating unit further includes the first and second units.
4. The image processing apparatus according to claim 3, further comprising a frame set control unit that controls selection of a frame set including the frames. 8. The image processing apparatus according to claim 7, wherein the synthesizing unit synthesizes a still image synthesized for each frame group selected by the frame group selecting unit. 9. The image processing apparatus according to claim 8, wherein the frame group control means controls the first and second frame intervals to be the same in each of the frame groups. 10. The frame set control means controls the second frame of the currently selected frame set to be the first frame of the next selected frame set. Item 10. The image processing device according to Item 9. 11. The image processing apparatus according to claim 7, wherein said frame set control means selects said frame set based on a motion vector detected by said detection means. 12. The apparatus according to claim 11, wherein said frame set control means performs control so as to narrow the first and second frame intervals in said frame set when said motion vector is equal to or greater than a predetermined value. The image processing apparatus according to any one of the preceding claims. 13. The frame set control means controls the second frame in the frame set to be closer to the first frame when the motion vector is equal to or larger than a predetermined value. 13. The image processing device according to 12. 14. The first and second frames in the frame set.
14. The method according to claim 13, wherein when the frame interval becomes two frames, if the motion vector is equal to or more than a predetermined value, the generation unit outputs the image of the first frame as it is. Image processing device. 15. An inputting step of inputting a plurality of low-resolution images, a detecting step of detecting a difference between the plurality of low-resolution images, and a generating step of generating a high-resolution image from the low-resolution images based on the difference. And an image processing method. 16. An image processing system in which an imaging device that captures a moving image and an image processing device that processes the captured moving image are connected, wherein the image processing device is configured to capture an image captured by the imaging device. Input means for inputting a plurality of frames of a moving image; detecting means for detecting a motion vector between the plurality of frames; generating a still image having a higher resolution than the image from the plurality of frames based on the motion vector An image processing system comprising: a generation unit. 17. A recording medium on which a program code for image processing is recorded, wherein the program code detects a code between an input step of inputting a plurality of low-resolution images and a difference between the plurality of low-resolution images. A recording medium comprising: a code of a detection step of performing a detection; and a code of a generation step of generating a high-resolution image from the low-resolution image based on the difference. 18. An image processing apparatus for creating a single piece of high-resolution still image information based on a plurality of pieces of low-resolution image information, wherein said storage means stores image information for a plurality of continuous frames in a moving image. From the stored plural frames, the m-th frame and m + n
Calculating means for selecting a frame (m and n are natural numbers and n> 1) and calculating a motion vector between the frames; dividing means for dividing the calculated motion vector based on the value of n; A first arranging unit for arranging the pixels of the frame, and m according to the motion vector divided by the dividing unit.
A second arranging means for arranging the pixels of the + a-th frame (a is a natural of a <n); An image processing apparatus comprising: 19. An image processing apparatus for creating a single piece of high-resolution still image information based on a plurality of pieces of low-resolution image information, wherein the storage means stores image information for a plurality of continuous frames in a moving image. M + n × b-th frame (m and n are natural numbers, b is an integer and n> 1) among the stored plural frames, and m + n
× (b + 1) th calculating means for selecting a frame and calculating a motion vector between the frames; dividing means for dividing the calculated motion vector based on the value of n; And m according to the motion vector divided by the dividing means
+ N × b + a frame (a is a natural number of a <n), a second arranging unit, and a control unit that increases the value of b to control selection of two frames to be operated by the operation unit. An image processing apparatus comprising: a synthesizing unit that synthesizes pixels of each frame in which the pixels are arranged by the first and second arrangement units. 20. An image processing apparatus for creating a single piece of high-resolution still image information based on a plurality of pieces of low-resolution image information, wherein the storage means stores image information for a plurality of continuous frames in a moving image. From the stored plural frames, the m-th frame and m + n
-C-th frame (m and n are natural numbers, c is an integer, n
> 1, 0 ≦ c ≦ n−2) to calculate a motion vector between the frames, a dividing unit to divide the calculated motion vector based on the value of n, And m in accordance with the first arranging means for arranging m and the motion vector divided by the dividing means.
A second arranging means for arranging pixels of the + a-th frame (a is a natural of a <n); a synthesizing means for synthesizing pixels of each frame arranged by the first and second arranging means; Control means for controlling to increase c if the motion vector is equal to or greater than a predetermined value. 21. The apparatus according to claim 18, wherein said dividing means divides said motion vector by n, and said second arranging means arranges a pixel at a / n position of said motion vector. 20. The image processing device according to any one of 20.