JP3067833B2 - Image processing apparatus and method - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a television camera, a VT
The present invention relates to an image processing apparatus and method for outputting a moving image input from R or the like as a still image to a CRT or a printer.

[0002]

2. Description of the Related Art Conventionally, image signals applied to a printer or the like have been mainly input from a still image input device such as an image scanner.
A wide variety of video equipment such as video cameras and VTRs have been used. In addition, with the increase in definition of an image of a high-definition television or the like, there is a demand for printing a high-definition television image by a printer or a printing device.

[0003]

However, since the input image of the television camera is a moving image, while the output image of the printer is a still image, a device for converting the moving image into a still image is required. . At the time of moving image / still image conversion, there is a problem in that noise components are more visually conspicuous in a still image than in a moving image. That is, in a moving image, noise in one frame becomes difficult to see due to the integration effect of the visual system in the time axis direction, whereas in a still image from which one frame is extracted, there is no visual system integration effect, and image quality degradation due to noise is conspicuous. .

[0004] Noise removal methods include inter-frame smoothing and intra-frame smoothing, but they have advantages and disadvantages. That is, in the inter-frame smoothing, if there is a moving portion in the image, the image position between the frames shifts. Therefore, not only a portion from which noise is not removed remains but also an unclear portion such as blur or color blur occurs. In the intra-frame smoothing, high-frequency components are removed, so that an edge portion of an image becomes unclear. Various filtering methods for preserving edges and performing intra-frame smoothing have been proposed. However, since only information within the frame is used, noise removal is insufficient.

Further, the conventional method has a problem that the processing speed is slow.

Accordingly, an object of the present invention is to provide an image processing apparatus and method for solving such a problem.

[0007]

An image processing apparatus according to the present invention comprises: input means for inputting image data of a plurality of screens ;
Orthogonal transform means for orthogonally transforming image data of at least two screens of the plurality of screens; and motion detecting means for detecting motion of an image based on the image data of at least two screens orthogonally transformed by the orthogonal transform means. , The orthogonal
Image based on the image data orthogonally transformed by the transformation means.
Edge detecting means for detecting an edge of an image, the image data that has been orthogonally converted by the orthogonal transform unit, the motion
And a noise reduction unit for performing a noise reduction process according to a detection result of the edge detection unit .

[0008] The image processing method according to the present invention, an input process for inputting image data of a plurality of <br/> screens, the plurality of image
And orthogonal transform processing to orthogonally transform the image data of at least two screen surfaces, small orthogonally converted by the orthogonal transform
Image movement based on image data of at least two screens
Motion detection processing for detecting the
Detect image edges based on the converted image data
The motion detection processing and the edge detection are performed on the image data orthogonally transformed by the edge detection processing and the orthogonal transformation processing.
And having a noise reduction process applied noise reduction processing according to the result of the edge detection process.

[0009]

[0010]

Embodiments of the present invention will be described below with reference to the drawings.

FIG. 1 is a block diagram showing the configuration of an embodiment of the present invention and its peripheral devices. Reference numeral 10 denotes a moving image / still image conversion device according to an embodiment of the present invention, 12 denotes a moving image input device such as a television camera, a video camera, or a VTR, and 14 denotes a still image output device such as a color printer or a color monitor. It is.

In the moving image / still image converter 10 of this embodiment, 16 is an interface with the moving image input device 12, 18 is an interface with the still image output device 14, 20 is an image data bus, and 22 is a control signal bus. , 2
Reference numeral 4 denotes a control circuit for controlling the whole, and reference numeral 26 denotes an image memory capable of storing several frames of moving images in frame units.

Reference numeral 28 denotes a motion detection circuit for detecting the motion of an image between frames. Reference numeral 30 denotes an edge detection circuit for detecting an edge within a frame.
Are connected to the image data bus 20 and the control signal bus 22 via the orthogonal transform circuit 31. Reference numeral 32 denotes a noise removal circuit, which includes an intra-frame processing circuit 34 for performing noise reduction processing within a frame and an inter-frame processing circuit 36 for performing noise reduction processing between frames. The noise removing circuit 32 is connected to the image data bus 20 and the control signal bus 22 via the inverse orthogonal transform circuit 37.

Reference numeral 38 denotes a moving image / still image conversion device 1
0 (specifically, the control circuit 24) is an operation device including key switches for inputting various operation instructions.

In the above embodiment, the coefficient generating circuits 44, 58,
70 and 76 and the reference value generation circuit 52 are, specifically,
It comprises a RAM or a ROM that holds a predetermined coefficient or reference value in a look-up table format using i and j as addresses.

In this embodiment, the orthogonal transform circuit 31 performs a two-dimensional discrete cosine transform (hereinafter abbreviated as DCT) on the image data on the image data bus 20, and performs an inverse orthogonal transform circuit 3.
7 performs an inverse two-dimensional discrete cosine transform (hereinafter abbreviated as an inverse DCT) on the data from the noise removal circuit 32 and outputs the result to the image data bus 20. The unit of orthogonal transform is N
It is a square of pixels × N pixels. If N is too large, the amount of calculation becomes enormous, and if N is too small, the effect of DCT is difficult to obtain. Therefore, in this embodiment, N is set to about 4 to 8. In the following description, the image of the attention area in the real space is represented by Aij and Bij, and the image after the orthogonal transformation is represented by Xij and Yij. After the orthogonal transformation, the signal becomes a signal on the frequency axis.
In Yij, the frequency increases as i and j increase. Regarding the two-dimensional discrete cosine transform and its inverse transform itself,
Since it has nothing to do with the present invention, a detailed description is omitted.

FIG. 2 is a detailed circuit block diagram of a portion including the motion amount detecting circuit 28, the edge amount detecting circuit 30, and the noise removing circuit 32. Needless to say, image data that has been orthogonally transformed by the orthogonal transformation circuit 31 is input to the motion amount detection circuit 28 and the edge amount detection circuit 30.

In the motion amount detecting circuit 28, reference numeral 40 denotes a difference circuit for calculating a difference between frames in the frequency space (difference between Xij and Yij); 42, an absolute value circuit for obtaining the absolute value of the output of the differentiator 40; Is a coefficient generation circuit for generating a coefficient αij, 46 is a multiplication circuit for multiplying the output of the absolute value circuit 42 by the coefficient αij generated by the coefficient generation circuit 44, and 48 is an integration circuit for multiplying the output of the multiplication circuit 46 in the attention area. It is.
The output of the integrating circuit 48 is the detected motion amount MV.

In the edge amount detection circuit 30, X is 50
An absolute value circuit for taking the absolute value of ij, 52 is a reference value generating circuit for generating a reference value Eij necessary for edge amount detection, and 54 is a reference value Eij generated by the reference value generating circuit 52 from the output of the absolute value circuit 50. Difference circuit for subtraction, 56 is difference circuit 5
A judgment circuit for judging the sign (positive / negative) of the output of 4; 58, a coefficient generation circuit for generating a coefficient βij;
A multiplier circuit for multiplying the output of the multiplier 6 by the coefficient βij; 62, an integrating circuit for integrating the output of the multiplier circuit 60 within the region of interest;
Is a group forming circuit for performing grouping according to the judgment result of the judgment circuit 56. The output of the integrating circuit 62 is the detected edge amount EV.

In the inter-frame processing circuit 36, reference numeral 66 denotes an Xij buffer, and 68 denotes an output Xij−
Yij buffer, 70 is output M of motion amount detection circuit 28
A coefficient generation circuit for generating a coefficient Tij according to V; a multiplication circuit 72 for multiplying the coefficient Tij by the output Xij-Yij of the buffer 68; a difference circuit 74 for subtracting the output of the multiplication circuit 72 from the output Xij of the buffer 66 is there.

In the in-frame processing circuit 34, a coefficient 76 generates a coefficient Kij from the edge amount EV output from the integration circuit 62 of the edge amount detection circuit 30 and the group flag Gij output from the group formation circuit 64. A generation circuit 78 is an output Sij of the inter-frame processing circuit 36.
Is a multiplication circuit for multiplying the coefficient Kij by the output of the buffer 78. The output of the multiplication circuit 80 is a signal Zij obtained by removing noise from the image data after the orthogonal transformation.

The operation of this embodiment will be described. When a user inputs a moving image input request using the operation device 38, the control circuit 24 activates the interface 16 and the image memory 26 via the control signal bus 22 to give an access right to the image data bus 20. The interface 16 fetches a moving image signal from the moving image input device 12 in synchronization with a synchronizing signal such as a frame synchronizing signal, and outputs the moving image signal onto the image data bus 20. Of course, the moving image input device 1
If the output signal of the second is an analog signal, the interface 16 converts the signal into a digital signal and outputs the digital signal on the image data bus 20.

The image memory 26 stores the image data bus 20
The above image data is sequentially captured, and successive images are stored for several frames (at least two frames) in frame units. When a predetermined number of frames of image data are stored in the image memory 26, the control circuit 24 makes the interface 16 and the image memory 26 non-active via the control signal bus 22. Thus, the input of the moving image is completed.

Next, when the user inputs a request for conversion into a still image from the operation device 38, the moving image / still image conversion device 10
Forms a still image from which noise has been removed from a plurality of frame images stored in the image memory 26, and outputs the still image to the still image output device 14. That is, the control circuit 24 activates the image memory 26, the orthogonal transformation circuit 31, the motion amount detection circuit 28, the edge amount detection circuit 30, the noise removal circuit 32, and the inverse orthogonal transformation circuit 37 via the control signal bus 22, and The access right to the data bus 20 is given. The orthogonal transformation circuit 31 reads the first frame image Aij and the second frame image Bij stored in the image memory 26 in predetermined small area units, orthogonally converts them into Xij and Yij, and stores them in the internal memory. . The motion amount detection circuit 28
Although the details will be described later, the difference Xij-Yij of the image in the frequency space and the motion amount MV are calculated and output from Xij and Yij. Although the details will be described later, the edge amount detection circuit 30
From j, an edge amount EV and a feature component Gij are calculated and output. The inter-frame processing circuit 36 generates a signal Sij obtained by smoothing Xij and Yij between frames according to the motion amount MV.
The intra-frame processing circuit 34 outputs the inter-frame smoothed signal Sij according to the edge amount EV and the characteristic component Gij.
Are subjected to intra-frame processing (eg, intra-frame smoothing processing and edge enhancement processing).

The inverse orthogonal transform circuit 37 performs an inverse orthogonal transform on the output Zij of the intra-frame smoothing circuit 34 and writes the result into the output frame storage area of the image memory 26 via the image data bus 20. By performing the same processing for all the image areas of the frame for each small area of interest, the noise-reduced frame image is stored in the output frame storage area, and the noise reduction processing is completed.

When the noise reduction processing is completed, the control circuit 2
4 is a control signal bus 22 through which a motion amount detection circuit 28;
The edge amount detection circuit 30, the orthogonal transformation circuit 31, the noise removal circuit 32, and the inverse orthogonal transformation circuit 37 are deactivated, the interface 18 is activated, and the output frame stored in the output frame storage area of the image memory 26 is read. The data is sequentially read out to the image data bus 20. The interface 18 outputs the image data on the image data bus 20 to the still image output device 14 in synchronization with the still image output device 14. Thus, the still image output device 14 forms and outputs a visible still image. When the signal input format of the still image output device 14 is an analog format, the interface 18 outputs an analog image signal by an internal D / A converter.

Next, specific operations of the motion amount detecting circuit 28, the edge amount detecting circuit 30, and the noise removing circuit 32 will be described.

The motion amount MV and the edge amount EV are calculated according to the following equations. MV = Σαij × ABS (Xij−Yij) (1) EV = Σβij × REC (ABS (Xij) −Eij)) (2) where Σ is i = 0 to N−1 and j = 0 to N− ABS () represents an absolute value, and REC () is a rectifying function of REC (x) = 0 (x ≦ 0) REC (x) = x (x> 0). Αij and βij are weighting factors, and Eij is a positive value, which is a reference value (a kind of threshold) of an edge component.

The operation of the motion amount detection circuit 28 will be described. The motion amount detection circuit 28 outputs the image Xij, Y of the attention area of the first and second frames in the frequency space from the orthogonal transformation circuit 31.
ij are sequentially read. The difference circuit 40 calculates the difference Xij of each element.
−Yij, and the absolute value circuit 42 outputs the difference Xij−Y
ij is output, and the multiplication circuit 46 outputs the absolute value
A coefficient generation circuit 4 is applied to the output ABS (Xij-Yij)
4 is multiplied by a coefficient αij. The coefficient αij is D
It is used to match the scale of the CT element or to increase the evaluation of the element corresponding to the frequency component to be emphasized.
The coefficient αij is determined in advance for i and j. The integrating circuit 48 outputs the output of the multiplying circuit 46 from i = 0 to N−1, j =
Integration is performed for 0 to N-1. As a result, the motion amount MV shown in Expression (1) is obtained. The obtained motion amount MV is applied to the coefficient generation circuit 70 of the inter-frame processing circuit 36,
The coefficient generation circuit 70 temporarily stores the motion amount MV.

The edge amount detection circuit 30 is a motion amount detection circuit 2
8 and operates as follows. That is, the absolute value circuit 50 outputs the absolute value ABS (Xij) of Xij. The coefficient generation circuit 52 generates a coefficient Eij,
Is a coefficient E from the output ABS (Xij) of the absolute value circuit 50.
ij is subtracted. The judgment circuit 56 outputs the output A of the difference circuit 54.
It is determined whether BS (Xij) -Eij is positive or negative. If positive, ABS (Xij) -Eij is output to the multiplying circuit 60 and the flag Gij of the group forming circuit 64 is set (to 1), and negative or If it is zero, it outputs 0 to the multiplication circuit 60 and resets the flag Gij of the group formation circuit 64 (sets it to 0). According to the flag Gij, elements are grouped into elements that contribute to the edge (elements with Gij = 1) and elements that do not contribute (elements with Gij = 0).

The multiplication circuit 60 outputs the output RE of the judgment circuit 56.
C (ABS (Xij) -Eij) is multiplied by the coefficient βij from the coefficient generation circuit 58, and the multiplication circuit 62
The outputs of 0 are integrated for i = 0 to N−1 and j = 0 to N−1. The output of the integrating circuit 62 becomes the edge amount EV for the region of interest. The coefficient generating circuit 76 of the intra-frame processing circuit 34 temporarily stores the edge amount EV from the integrating circuit 62 and the flag Gij from the group forming circuit 64.
Note that the reference value Eij and the coefficient βij for detecting the edge amount are:
It is determined in advance by experiments or the like.

When the motion amount MV, the edge amount EV, and the flag Gij are determined as described above, the noise removing circuit 32 performs a noise removing process. That is, the coefficient generation circuit 7
0 outputs a coefficient Tij according to the motion amount MV, and the multiplication circuit 72 multiplies the signal Xij-Yij from the buffer 68 by the coefficient Tij. The subtraction circuit 74 outputs the output Tij × (Xij) of the multiplication circuit 72 from Xij from the buffer 66.
-Yij) is subtracted. The subtraction circuit 74 uses Xij-Ti
j × (Xij−Yij) is output. The output Sij of the subtraction circuit 74 is applied to the buffer 78 of the intra-frame processing circuit 34 and is temporarily stored. Output Sij of subtraction circuit 74
Corresponds to image data that has been subjected to inter-frame processing in the frequency space.

In the intra-frame processing circuit 34, the coefficient generation circuit 76 generates a coefficient Kij according to the edge amount EV and the group flag Gij, and the multiplication circuit 80 generates the coefficient Ki.
j is multiplied by Sij from buffer 78. That is, the output Zij of the multiplication circuit 80 is Kij × Sij. Z
ij corresponds to image data obtained by performing intra-frame processing on image data that has been subjected to inter-frame processing in the frequency space.

FIG. 3 shows an example of a characteristic curve of a coefficient Tij indicating the degree of inter-frame processing in the frequency space. As the amount of motion MV increases, the coefficient Tij decreases, and the degree of inter-frame processing in the frequency space also decreases. Specifically, when the motion amount MV is between 0 and the predetermined value MV1, the coefficient Tij = 1/2, and therefore Sij is (Xij + Y
ij) / 2, which corresponds to inter-frame averaging in the frequency space. When the motion amount MV is larger than the predetermined value MV2, the coefficient Tij = 0, and Sij = Xij
This is equivalent to no inter-frame processing. When the motion amount MV is between MV1 and MV2, the coefficient T
ij takes a continuous value between 1/2 and 0, and Sij continuously changes in the range of (Xij + Yij) / 2 to Xij. This means that the degree of inter-frame processing changes continuously. Note that the coefficient Tij and the thresholds MV1, M
V2 can be set for each element of the DCT, but FIG.
An example for one element is illustrated. By performing such inter-frame processing in the DCT space (frequency space), frequency components having low correlation between the first frame and the second frame are suppressed, whereby a noise suppression effect is obtained in the inverse DCT real space.

FIG. 4 shows an example of a characteristic curve of a coefficient Kij indicating the degree of intra-frame processing in the frequency space. In this example, the edge amount EV is divided into five regions. The coefficient Kij and its threshold values EV1 to EV4 can be set for each element of the DCT. Here, an example for one element is shown.

If the edge amount EV is less than EV1, the coefficient Ki
j = 0 (however, K ₀₀ = 1), and the output Zij = 0 (where Z ₀₀ = S ₀₀ ) of the intra-frame processing circuit 34 is obtained. This means that frequency components other than the DC component (Z ₀₀ ) have been removed, and the element Ci obtained by inverse DCT of Zij
j is block-smoothed in the region of interest.

When the edge amount EV is equal to or larger than EV1 and equal to or smaller than EV2, the coefficient Kij continuously changes from 0 to 1. However, K ₀₀ = 1. The degree of suppression of frequency components other than the DC component (Z ₀₀ ) changes continuously according to the edge amount EV. The smaller the EV, the greater the degree of suppression. As a result, the element Cij obtained by performing the inverse DCT on Zij has been subjected to block smoothing according to the edge amount EV in the region of interest, and the smaller the EV, the greater the degree of block smoothing.

When the edge amount EV is equal to or larger than EV2 and equal to or smaller than EV3, the coefficient Kij is constant at 1 and Zij = Sij. This means that no intra-frame processing is performed.

When the edge amount EV is equal to or larger than EV3 and equal to or smaller than EV4, the coefficient Kij continuously changes from 1 to Kh (> 1) for the elements of the group G1 (Gij = 1) contributing to the edge. The non-contributing group G2 (Gij =
The coefficient Kij continuously changes from 1 to 0 for the element of (0). However, K ₀₀ = 1. As a result, the degree of increase of the frequency component that contributes to the edge and the degree of suppression of the frequency component that does not contribute to the edge continuously change according to the edge amount EV. The degree increases. As a result, Zij is converted to the inverse DC
The element Cij obtained by T is the edge amount E within the region of interest.
The degree of edge enhancement changes continuously according to V, and the larger the EV, the stronger the edge enhancement.

When the edge amount EV is equal to or larger than EV4, the coefficient Kij = Kh is set for the elements of the group G1 (Gij = 1) contributing to the edge, and the group G2 not contributing to the edge is set.
The coefficient Kij = 0 for the element of (Gij = 0).
However, K ₀₀ = 1. Thereby, the DC component is preserved, but the frequency component that contributes to the edge increases, and the frequency component that does not contribute to the edge is removed. As a result, Zi
An element Cij obtained by performing inverse DCT on j is edge-emphasized in the attention area.

It goes without saying that the function of each circuit block may be realized by individual hardware or by software processing by a dedicated or general-purpose digital processing unit. Processes at appropriate locations may be simultaneously executed by pipeline processing. In this embodiment, the number of images to be processed is two, but of course, noise reduction processing may be performed sequentially using images of three or more frames. A field unit may be used for interlaced scanning moving images.

Also, in the above embodiment, the orthogonal transform is 2
Although the dimensional discrete cosine transform is used, the present invention is not limited to this, and another orthogonal transform such as a two-dimensional Hadamard transform may be used. In the above embodiment, the N × N
Is used as a unit of image processing after orthogonal transformation, and the square block of is also used as a unit of image processing after orthogonal transformation. The processing may be performed in units of small blocks, and the units of orthogonal transform and / or the units of image processing may overlap each other for the entire frame.

The amount of motion MV and the amount of edge EV may be determined according to other equations, instead of the equations defined in equations (1) and (2). For example, the motion amount may be calculated in a frequency space obtained by orthogonally transforming a frame difference image in a real space. The edge amount may be obtained by imagining a specific edge and paying attention to a specific DCT element (frequency component) corresponding thereto.

The inter-frame processing and the intra-frame processing in the noise elimination method are not limited to the configuration of the above embodiment. For example, weighted average coefficients in inter-frame processing,
And the proportionality factor of the linear transformation of the intra-frame processing may be a function of the size of the DCT element. Further, a variable filter according to the amount of motion or the amount of edge may be used. In other words,
The parameters controlled by the edge amount and the motion amount are not limited to the weighted average coefficient, but may be other plural parameters, or may be parameters in which the edge amount and the motion amount are complexly related. The processing order in which the intra-frame processing is performed after the inter-frame processing is not limited to this, and may be reversed or partially related. The size of the attention area, which is a processing unit, is not limited to N pixels × N pixels. That is, the size and shape of the attention area may be set according to the inter-frame processing and the intra-frame processing to be used. For example, a configuration in which the processing unit itself is controlled by the amount of motion and the amount of edge may be employed.

It goes without saying that the function of one or a plurality of circuit blocks described in the present embodiment may be realized by software processing.

In this embodiment, since the DCT transform is used as the orthogonal transform, the processing can be speeded up.

[0047]

As can be easily understood from the above description, according to the present invention, the motion and the edge of an image are detected based on the image data subjected to orthogonal transformation, and the orthogonal transformation is performed according to the detection result. since subjected to noise reduction processing on the image data, it is possible to image quality deterioration obtain a clear image inconspicuous. In addition, image motion and edge detection processing
Image data used for noise reduction and image data used for noise reduction processing
Share the orthogonally transformed image data
Reduces the number of orthogonal transformation processing units and the number of orthogonal transformation processes.
Processing speed, circuit scale down
Small and cost reduction can be achieved.

[Brief description of the drawings]

FIG. 1 is a configuration block diagram of an embodiment of the present invention.

FIG. 2 is a configuration block diagram of a main part of FIG. 1;

FIG. 3 is a characteristic diagram of a coefficient Tij of inter-frame processing.

FIG. 4 is a characteristic diagram of coefficients Kij for intra-frame processing.

[Description of Signs] 10: Moving image / still image conversion device 12: Moving image input device
14: Still image output device 16, 18: Interface 20: Image data bus 22: Control signal bus 24: Control circuit 26: Image memory 28: Motion amount detection circuit 30: Edge amount detection circuit 31: Orthogonal transformation circuit 32: Noise Removal circuit 34:
In-frame processing circuit 36: Inter-frame processing circuit 3
7: Inverse orthogonal transformation circuit 38: Operation device 40: Subtraction circuit 42: Absolute value circuit 44: Coefficient generation circuit 46: Multiplication circuit 48: Multiplication circuit 50: Absolute value circuit 52: Reference value generation circuit 54: Subtraction circuit 56: Judgment Circuit 58: Coefficient generation circuit 60: Multiplication circuit 62:
Integration circuit 64: Group formation circuit 66, 68: Buffer 70: Coefficient generation circuit 72: Multiplication circuit 74: Subtraction circuit 76: Coefficient generation circuit 78: Buffer 80:
Multiplication circuit

──────────────────────────────────────────────────続 き Continuation of the front page (56) References JP-A-64-73972 (JP, A) JP-A-4-132460 (JP, A) JP-A-2-29177 (JP, A) JP-A-3-3 67376 (JP, A) JP-A-62-104283 (JP, A) JP-A-1-143358 (JP, A) JP-A-61-158274 (JP, A) (58) Fields investigated (Int. H04N ^7/ 24-7/68 H04N 1/41-1/419 H04N 5/76-5/956 H04N 5/14-5/217 G06F 15/62

Claims (2)

(57) [Claims] An image memory capable of storing a predetermined number of images, an orthogonal transformation unit for orthogonally transforming the images stored in the image memory, and a motion amount and a motion amount based on image information orthogonally transformed by the orthogonal transformation unit. And / or an image processing apparatus comprising: a detection unit that detects an edge amount; and a noise reduction unit that performs a noise reduction process on an image orthogonally transformed by the orthogonal transformation unit in accordance with a detection result of the detection unit. . 2. The method according to claim 1, wherein the given image data is subjected to an orthogonal transform, a motion in the target image is detected from the converted time-series data of a plurality of screens, and noise is reduced in accordance with the motion. Image processing method.