JPH1198395A - Picture processor - Google Patents

Abstract

PROBLEM TO BE SOLVED: To realize video image with high resolution by a simplified inexpensive mechanism part in a picture processor for obtaining the high resolution of a moving image. SOLUTION: A storing part 110 stores plural timewisely different frame data D1 and D2, and a movement detecting part 121 detects a relative moving vector V between the frames with 1/2 (or 1/n) pixel precision based on the two frame data D1 and D2(or the partial data of a frame), and a high resolution picture interpolating part 122 shifts one frame data D1 (or the partial data of the frame) only by the moving vector V, and urites the frame data D1 with the other frame data D2. Also, it is desired that a camera fine adjusting mechanism part (eccentric motor) 239 fine adjusts a camera 220a or an image pickup element or a lens.

Description

DETAILED DESCRIPTION OF THE INVENTION

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image processing apparatus, and more particularly to an image processing apparatus for increasing the resolution of a moving image.

In recent years, digital images have been widely applied in the fields of communication, medical care, industry and the like, and accompanying this, moving images for transmitting more sophisticated information as image data have been developed. There is an increasing need for an image processing device capable of high resolution.

[0002]

2. Description of the Related Art FIG. 6 shows an example of a conventional image processing apparatus. In FIG. 1A, a zoom mechanism 240 as a high-resolution unit enlarges an input image itself, and the TV camera 200 that has input the image enlarges an image signal of a part of the enlarged image as an image processing device 100. Frame memory unit 11
0, and the frame memory unit 110 stores the video signal as frame data (hereinafter, also referred to as image data).

Further, the frame memory unit 110 stores the stored image data in a normal display unit 310 having a high resolution.
To display. That is, the partial video is displayed with a high resolution by being enlarged.

However, enlarging an image by the zoom mechanism 240 reduces reliability due to the presence of a movable part of the optical system, increases system cost,
In the case where an industrial television camera (ITV) is used for monitoring, for example, there is a drawback such that it becomes impossible to monitor an image other than a portion displayed by zooming.

FIGS. 2 (2) and 3 (3) show an image processing apparatus for increasing the resolution of captured image data in a data processing section. According to this apparatus, the entire image is increased in resolution. It can be displayed on a high-resolution image display device.

That is, in FIG. 1B, an image processing apparatus 100 connected to a TV camera 200 includes a frame memory 110, a bilinear interpolation unit 13 as a data processing unit.
0 and an enlarged frame memory 110c. The frame memory 110 stores the video signal input from the TV camera 200 as image data and sends the image data to the bilinear interpolation unit 130.

[0007] The bilinear interpolation section 130 interpolates the input image data by bi-linea-interporation, and sends the image data having a high resolution to the enlarged frame memory section 110c. The enlarged frame memory unit 110c stores the input high-resolution image data and sends the high-resolution image data to the high-resolution image display device 300. The display device 300 displays a pseudo-zoomed partial image or a high-resolution whole image.

FIG. 3C shows an image processing apparatus for increasing the resolution of an image proposed in Japanese Patent Application Laid-Open No. 2-276385. In this example, an image processing apparatus 100 is shown.
Is a TV camera 200 and two image sensors 220a,
It is connected between 220b and the high resolution image display device 300, and comprises two frame memories 110a and 110b and a pixel switching unit 140.

The image pickup device 220a and the image pickup device 220b
The output signal of the TV camera 200 is set so as to be input in such a manner that it is shifted by one pixel in one direction and one television scanning line in a direction orthogonal thereto. Therefore, the image sensor 2
The video signals sampled by 20a and 220b are shifted from each other by half a pixel.

The frame memories 110a and 110b receive video data from the image pickup devices 220a and 220b, respectively, and store them as image data separated by half a pixel from each other. The pixel switching unit 140 includes the frame memories 110a, 110
Single image data with a higher resolution is generated from the image data of 10b and sent to the high-resolution image display device 300 for display.

As described above, the resolution can be increased by increasing the number of imaging elements and frame memories.

[0012]

In such a conventional image processing apparatus, when the resolution is increased by bilinear interpolation as in the conventional example of FIG. The image will be blurred and the magnification will be 2
It is considered that about four times is a practical limit.

On the other hand, in the case of an apparatus using a plurality of image pickup elements in which blur is unlikely to occur (FIG. 3 (3)), it is necessary to increase the number of image pickup elements in order to increase the resolution and to use a special optical mechanism. However, there is a problem that the cost is high because of the necessity, and it is difficult to adopt it in an actual system.

Accordingly, the present invention provides an image processing apparatus comprising a storage unit for storing a video signal as frame data and a data processing unit for performing data processing on the frame data and outputting display data. Is realized using a simplified and inexpensive mechanism.

[0015]

[Means for Solving the Problems]

[1] In order to solve the above-described problem, in the image processing apparatus 100 according to the present invention illustrated in FIG. 1, the storage unit 110 stores the two frame data D1,
D2, the data processing unit 120 detects a relative motion vector V with half-pixel accuracy based on the frame data D1 and D2, and shifts one frame data by the motion vector V. And a high-resolution image interpolating unit 122 for merging with the other frame data.

That is, the TV camera 200a transmits the photographing target 400 and the background to the image processing apparatus 100 as video signals. In the image processing apparatus 100, the storage unit 11
0 indicates that the received video signal has a different time T-
The motion detection unit 121 of the data processing unit 120 detects the motion of the entire frame image from the frame data D1 and D2 as a motion vector V with half-pixel accuracy.

The high-resolution image interpolation unit 122 of the data processing unit 120 spatially and relatively shifts the frame data D2 read from the storage unit 110 by the motion vector V specified by the motion detection unit 121, and shifts the frame data D1 from the frame data D1. The image data D3 that has been merged and has a higher resolution is generated and output to the high-resolution image display device 300. Display device 300
Displays all the high-resolution images including the imaging target 400.

As a result, the image processing apparatus 100 can generate a high-resolution video without changing the resolution of the TV camera (image pickup device) to a high resolution. This is achieved by converting the resolution in the time direction of the TV camera image into the resolution in the space direction via a motion vector.

[2] In the present invention, the above-mentioned invention [1]
, The motion detection unit detects a partial motion vector corresponding to the partial data of the frame data, and the high-resolution image interpolation unit calculates the partial data of one frame data as the partial motion vector. It is possible to merge only by shifting.

That is, in FIG.
Samples a video object 400 (for example, a person) that moves slightly and a background image that is stationary and sends it to the storage unit 110 as a video signal, and the storage unit 110 converts the video signal into frames at times T−1 and T. It is stored as data D1 and D2. The motion detection unit 121 detects the “partial motion vector V” for the partial data D11 and D21 corresponding to the video object 400 with an accuracy of half a pixel.

The high-resolution image interpolating unit 122 includes a storage unit 1
For example, partial data D21 cut out from 10 as a person
Is spatially and relatively shifted by a motion vector V designated by the motion detection unit 121 to generate image data D3 merged with the frame data D1, and the high-resolution image display device 30
Output to 0. The display unit 300 displays all images in which only the imaging target 400 has been increased in resolution.

[3] As described above, in the image processing apparatus according to the present invention [1] or [2], in FIG. Uses that it exists. That is, "position shift" generated by a change in some physical quantity is used for an image taken every frame sampling. Therefore, when photographing a stationary photographing target 400 with a completely fixed camera 400, a motion vector with half-pixel accuracy cannot be generated, and the resolution cannot be improved.

Therefore, the image processing apparatus according to the present invention, according to the invention [1] or [2], generates a means for generating the two frame data by moving a camera for inputting a video signal (the camera shown in FIG. 1). A fine movement mechanism (eccentric motor) 230 can be provided.

[4] Further, in the image processing apparatus according to the present invention, in the above-mentioned invention [3], the generating means may be means for moving an image pickup device for inputting a video signal.

[5] In the image processing apparatus according to the present invention, in the above-mentioned invention [3], the generating means may be a means for moving a lens for capturing a video signal.

That is, as shown in FIG. 3, the camera image 410 is univocally determined by the optical positional relationship between the object 400, the optical lens 210, and the image pickup device 220 such as a charge coupled device (CCD). Is determined. Therefore, as a mechanism for generating a motion vector (position shift) between different frames in the time direction, movement of the imaging target 400, movement of the optical lens 210, movement of the image sensor 220, and movement of the optical axis can be considered.

As described above, in the present invention, (1) vibrating or moving the camera 200 itself, (2) vibrating or moving the image pickup device 220, and (3) the lens 21
It is possible to generate a motion vector by either vibrating or moving 0. It should be noted that the accuracy of the mechanism for oscillating or moving the camera, the image sensor, or the lens is not required, and only a force for generating a random motion vector in the x and y directions of about several pixels is required.

[6] In the image processing apparatus according to the present invention, in the above-mentioned invention [1], the storage unit stores three or more pieces of frame data different in time direction, and the motion detection unit A relative motion vector between reference frame data, which is one of three or more of the frame data, and another of the frame data is detected with an accuracy of n (n is a positive integer) one pixel. However, the high-resolution image interpolating unit may shift the other frame data by the corresponding motion vector and merge the other frame data with the reference frame data.

[7] Further, in the image processing apparatus according to the present invention, in the above-mentioned invention [6], the motion detection unit may determine whether the partial data of the reference frame data and the partial data of the other frame data are different. The partial motion vector of
It is also possible to detect each pixel with the precision of the pixel, and the high-resolution image interpolation unit shifts the partial data of the other frame data by the corresponding partial motion vector and merges the partial data with the reference frame data. .

That is, in the present inventions [6] and [7], the reference frame data is determined from a plurality of frame data different in the time direction, and the reference frame data (or partial data of the reference frame data) is determined. Is detected with a precision of 1 / n pixel between the frame data (or partial data of other frame data).

Then, a plurality of other frame data (or partial data of other frame data) are spatially moved by a corresponding motion vector (or a partial motion vector) and merged with the reference frame data, thereby achieving high resolution. Can be done.

[0032]

FIG. 4 shows an embodiment of the operation of a data processing section 120 (see FIGS. 1 and 2) in an image processing apparatus according to the present invention. In this embodiment, as a basic technique, a motion evaluation technique and a motion contour part extraction technique (cutting out a part of a person or the like) which are frequently used in recent image compression means are used. Used.

FIGS. 1A and 1B show T in FIG.
Image (real image) at time T-1 input to V camera 200
R and an image (real image) C at time T are shown. FIGS. 3 (3) and 4 (4) show images R and C sampled as pixels by an image sensor (not shown) of the camera 200, and the frame data D1 and the image frames at time T-1 and T, respectively. The sampling images R (x, y) and C (x, y) stored as D2 are shown.

In FIG. 4, each pixel is indicated by a portion surrounded by a thick line, and a grid interval including a thin line is assumed on the assumption that the image data from the camera is enlarged and displayed twice. Indicates the display resolution of the high-resolution image display device 300 (see FIG. 2).

FIG. 11 (1) shows one pixel designated by a thick line, and this pixel is composed of partial pixels P0 to P3 obtained by dividing into four lines by a thin line. The partial pixels P0 to P3 correspond to the resolution of the display unit 300 described above. The image element (not shown) of the camera has a partial pixel P
The video signal input to 0 (shaded portion) is sampled.

The sampling image R (x, x,
y) ((3) in the same figure) and sampling image C at time T
The procedure for obtaining a relative motion vector between (x, y) ((4) in FIG. 4) with half-pixel accuracy will be described below.

A motion evaluation value E (dx, dy) is obtained from the image C (x, y) at time T and the image R (x, y) at time T-1 based on the following equation (1). E (dx, dy) = ΣABS (Cx, y−R (x + dx), (y + dy)) Equation (1)

By searching for a combination of dx and dy that minimizes E (dx, dy), a motion vector V '(dx, dy) with one-pixel accuracy can be obtained. In this embodiment, the motion between the images R (x, y) and C (x, y) is x, y
Since the direction is one pixel or less, the motion vector V ′ (x, y) with one-pixel accuracy is (0, 0).

From the motion vector V 'with one-pixel accuracy and the sampled image R (x, y), the images R (x, y), C
A motion vector V with an optimum half-pixel accuracy between (x, y) is obtained below.

The sampled image R (x, y) and the sampled image R (x, y) are moved in the x and y directions by moving vectors (x = １ ／, y = 0), (x = 0, y = 1, respectively). /
2), sampling images R (x + 1, y), (x, y + 1), (x + y, y) shifted by (x = 1/2, y = 1/2)
1, Y + 1) and the image PB (x,
y), PC (x, y), PD (x, y) (FIG.
(33)) is generated based on the following equations (2) to (4).

PB (x, y) = (R (x, y) + R (x + 1, y)) / 2 Equation (2) PC (x, y) = (R (x, y) + R (X, y + 1) / 2 Expression (3) PD (x, y) = (R (x, y) + R (x + 1, y) + R (x, y + 1) + R (x + 1, y + 1)) / 4 Expression (4)

As in the case of the image R (x, y), the motion evaluation values EB (dx, dy), EC (dx, dy), and ED (dx, dy) are expressed by the following equations (5) to (7). Ask based on.

EB (dx, dy) = ΣABS (C (x, y) −PB (x, y)) Equation (5) EC (dx, dy) = ΣABS (C (x, y) −PC (x, y)) Equation (6) ED (dx, dy) = ΣABS (C (x, y) −PD (x, y)) Equation (7)

The optimal half-pixel unit motion vector dV '
(Dx ′, dy ′) is the obtained motion evaluation value EB (dx, dy),
It is determined by the following equation (8) based on the evaluation value that minimizes the value of EC (dx, dy) and ED (dx, dy). When EB (dx, dy) is minimum: dV '(dx', dy ') = (1/2, 0) When EC (dx, dy) is minimum: dV' (dx ', dy') = ( 0,1 / 2) When ED (dx, dy) is minimum: dV '(dx', dy ') = (1 / 2,1 / 2) Equation (8)

In FIG. 4, ED (dx, dy) is minimized, and the motion vector dV '= (1 / 2,1 /
2). From the above results, the image C (x,
y) and the motion vector V with half-pixel accuracy of the image R (x, y)
Can be obtained by the following equation (9). V = (dx + dx ′, dy + dy ′) Expression (9)

In this embodiment, since it is known that the motion vector V '(dx, dy) = (0, 0) with one-pixel accuracy, the motion vector V with half-pixel accuracy is (0+
(１ ／, 0 + ／) = (１ ／,）).

The image D (x, y) having a higher resolution is obtained by using the following equation (10a) and interpolation data in half-pixel units from the motion vector V, the image R (x, y), and C (x, y). Is generated in the next step (10b). D (x, y) = R (x, y) Expression (10a) D (x-dx-dx '), (y-dy-dy') = Cx, y Expression (10b) Where x and y are integers.

That is, the image C (x, y) at time T is moved by the direction vector opposite to the half-pixel precision motion vector V and merged with the image R (x, y) at time T-1. An image D (x, y) ((5) in the same figure) can be generated. Note that the image R (x, y) may be moved by the vector V and merged with the image C (x, y).

The above operation is performed by using the image R (x,
y) In addition to the image R 'at time T-2, T-3
(x, y), R "(x, y) (not shown) to detect motion vectors of other pixels P1 and P2 ((11) in FIG. 11) in half-pixel units, and It is also possible to generate an image D (x, y) at the time T whose resolution has been improved by combining them.

Further, the above procedure is performed for the image R (x, y),
Although C (x, y) has been described as an image of the entire frame,
The images R (x, y) and C (x, y) may be applied as a partial image of a person or the like extracted by extracting a motion outline part. In this case, the resolution of only the partial image is increased.

For example, when the camera is used as a TV conference or a surveillance camera, the TV camera itself is stationary, and the photographing target is a person or the like. When a motion vector exists, only the image of a person or the like to be noted is increased in resolution.

Instead of a half-pixel accuracy motion vector, a 1 / n-pixel accuracy motion vector can be calculated from a plurality of image data to improve the resolution n times.
Furthermore, by connecting an eccentric motor or the like to a TV camera, a lens, or an image sensor and applying a slight vibration, it is possible to ensure that a motion vector of the entire frame is provided between two different frame data in the time direction.

FIG. 5 shows a high resolution image display device 3 by the data processing section 120 at twice the resolution of the camera.
A display example in the case of displaying at 00 (see FIG. 2) is shown. FIGS. 1 (1) to 3 (3) show FIGS. 4 (1), 4 (3),
Same as (4). FIG. 5D shows an image R (x, y) at time T−1 sampled at the camera resolution.
B generated by simply enlarging half the pixels of
An example in which (x, y) is displayed on the display device 300 is shown. This is the same as the conventional display without increasing the resolution.

FIG. 5 (5) shows a sampled image C (x, y) at time T and a sampled image R (x, y) at time T-1.
y), an image D (x, y) obtained by interpolating the image is shown. In the conventional image B, four adjacent points have the same color information, and the boundary is jagged on the stairs. However, in the display image D (x, y), the boundary is a real image at time T-1. It can be understood that it is getting closer and smoother.

In the image processing apparatus of the present invention, the operation of detecting the motion vector of the entire frame and increasing the resolution and the operation of detecting the motion vector of a portion in the frame and increasing the resolution of only the portion are independent of each other. Have to work,
It is also possible to increase the resolution and enlarge the display of the image by using these together. For example, using a vibrating camera, a plurality of parts having different movements, such as a person or an object, and the background thereof can be independently increased in resolution.

[0056]

As described above, according to the image processing apparatus of the present invention, the storage unit stores a plurality of temporally different frame data, and the motion detection unit stores the two frame data (or the frame portion). Data)
A relative motion vector between frames is detected with (or 1 / n) pixel accuracy, and the high-resolution image interpolating unit shifts one frame data (or partial data of the frame) by the motion vector and shifts the other frame data (or partial data of the frame) by the other. Since the image data is merged with the frame data, it is possible to generate high-resolution image data at low cost without having to improve the resolution of the image sensor for the entire screen (or part).

Preferably, the vibration generating means (for example, an eccentric motor) is configured to finely move the camera, the image pickup device or the lens, so that high-resolution image data can be generated by a simplified and inexpensive mechanism. Becomes possible.

Further, in a surveillance application of a camera in which a wide-angle image is captured using a fish-eye lens and a part of the image is monitored, the magnification of the conventional image processing apparatus is increased from several times to ten and several times. Become. This causes blurring. The occurrence of this blur can be suppressed by reducing the enlargement magnification by improving the resolution of the image using the image processing apparatus of the present invention.

[0059]

[Brief description of the drawings]

FIG. 1 is a block diagram illustrating an operation principle (1) of an image processing apparatus according to the present invention.

FIG. 2 is a block diagram showing an operation principle (2) of the image processing apparatus according to the present invention.

FIG. 3 is a block diagram showing an operation principle (3) of the image processing apparatus according to the present invention.

FIG. 4 is a diagram illustrating an operation example of a data processing unit in the image processing apparatus according to the present invention.

FIG. 5 is a diagram illustrating a display example of a data processing unit according to the present invention.

FIG. 6 is a block diagram showing a conventional image processing apparatus.

[Explanation of symbols]

REFERENCE SIGNS LIST 100 Image processing device 110 Storage unit 110 a, 110 b Frame memory unit 110 c Enlarged frame memory unit 120 Data processing unit 121 Motion detection unit 122 High resolution image interpolation unit 123 Bilinear interpolation unit 124 Pixel switching unit 200 TV camera 210 Optical lens 220 Imaging Element (CCD) 230 Vibration generating means (camera fine movement mechanism, eccentric motor) 240 Zoom mechanism 300 High resolution image display device 400 Object to be photographed 410 Camera image In the drawings, the same reference numerals indicate the same or corresponding parts.

Claims (7)

[Claims] 1. An image processing apparatus comprising: a storage unit that stores a video signal as frame data; and a data processing unit that performs data processing on the frame data and outputs display data. The data processing unit detects a relative motion vector with a half-pixel accuracy based on the two frame data, and stores one frame data with only the motion vector. An image processing apparatus comprising: a high-resolution image interpolating unit that shifts and combines the other frame data. 2. The apparatus according to claim 1, wherein the motion detecting section detects a partial motion vector corresponding to the partial data of the frame data, and the high-resolution image interpolating section detects the partial motion vector of the one frame data. An image processing apparatus characterized in that data is shifted and merged by the partial motion vector. 3. The image processing apparatus according to claim 1, further comprising means for generating the two frame data by moving a camera for inputting the video signal. 4. An image processing apparatus according to claim 3, wherein said generating means is means for moving an image sensor for inputting said video signal. 5. An image processing apparatus according to claim 3, wherein said generating means is means for moving a lens which takes in said video signal. 6. The storage unit according to claim 1, wherein the storage unit stores three or more frame data different in a time direction, and the motion detection unit is one of the three or more frame data. A relative motion vector between the reference frame data and the other frame data is detected with an accuracy of one pixel of n (n is a positive integer), and the high-resolution image interpolating unit detects the other motion vector. An image processing apparatus, wherein frame data is shifted by the corresponding motion vector and merged with the reference frame data. 7. The motion detecting section according to claim 6, wherein the motion detecting section sets a partial motion vector between the partial data of the reference frame data and the partial data of the other frame data to n (n is a positive integer). The high-resolution image interpolating unit detects each pixel with an accuracy of 1 / pixel, and shifts the partial data of the other frame data by the corresponding partial motion vector to merge the partial data with the reference frame data. Characteristic image processing device.