JP2000311241A - Image processor - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processor which can decrease the quantity of data to be transferred via a bus and also can perform plural image processing operations including the scaling processing in real time and via a pipeline without increasing the memory capacity. SOLUTION: A memory 6 consisting of (4×5) pieces of independent memory elements is prepared at the preceding stage of a scaling processing part 8 to perform the 16-point cubic interpolation processing. The memory 6 is used with switching carried out in every row between (4×4) pieces of memory elements which read out the input data for the interpolation processing and (4×1) pieces of memory elements which write the output data given from the preceding processing part. Then (4×4) pieces of data which are stored in a read memory element of the memory 6 and necessary for the interpolation processing are selectively read out to carry out the interpolation processing. At the same time, the output data given from the preceding processing part are written in the remaining write memory elements to carry out the pipeline processing.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image processing apparatus used for an electronic image pickup apparatus such as a digital camera, and more particularly to an image processing apparatus including a scaling processing capable of realizing pipeline processing in real time with a small memory capacity. Related to the device.

[0002]

2. Description of the Related Art Generally, as an image processing procedure including an enlargement / reduction process in an electronic imaging device such as a digital camera using a solid-state imaging device such as a CCD, first, as shown in FIG. The captured image signal is subjected to a pre-process
It is stored once in 103. Next, image data is read from the frame memory 103, and image processing 104 including enlargement / reduction processing is sequentially performed.
To record the data on a recording medium such as the memory card 106.

In order to realize such an image signal processing procedure, an image processing apparatus as shown in FIG. 14, for example, is conventionally used. That is, the conventional image processing apparatus includes a bus 201, a CPU 202, a pre-processing circuit 203, and a plurality of image processing circuits 204-1 to 204-2 including a scaling unit.
204-n, a JPEG processing unit 205 and a frame memory 206 are connected to each other. And CPU20
Under the control of 2, the imaging signal from the CCD imaging device is processed by the pre-processing circuit 203 and then temporarily recorded in the frame memory 206 via the bus 201. Next, the image data is read from the frame memory 206, input to the image processing circuit 204-1 via the bus 201 for processing, and rewritten to the frame memory 206 again via the bus 201. Similarly, image processing circuits 204-2 to 20-20 including a frame memory 206 and an enlargement / reduction processing unit are connected via a bus 201.
4-n sequentially exchanges data, and finally J
The PEG processing unit 205 performs JPEG processing and records the processed data in the frame memory 206 or a memory card.

[0004] Next, the specific contents of the conventional enlargement / reduction processing will be described. FIG. 15 is a diagram showing a mode in which enlargement / reduction processing is performed by Cubic interpolation processing.
The ic interpolation process uses the data of 16 pixels indicated by AP to interpolate, for example, one point indicated by Z in the center. In order to interpolate one point of the image, the data of the surrounding 16 pixels is required. Is needed. In FIG. 15, xa, xb, x
c, xd, ya, yb, yc, yd are interpolation coefficients;
Assuming that the data of the 16 pixels of P are represented by A to P, the interpolation data at the Z position is represented by the following equation (1). Z = xa · ya · A + xb · ya · B + xc · ya · C + xd · ya · D + xa · yb · E + xb · yb · F + xc · yb · G + xd · yb · H + xa · yc · I + xb · yc · J + xc · yc · K + xd · yc · L + xa · yd · M + xb · yd · N + xc · yd · O + xd · yd · P (1)

[0005] Therefore, in order to perform such a Cubic process, it is necessary to be able to perform random access when fetching 16 pixel data from the memory. Conventionally, an example of a format in which pixel data is stored in a memory before performing such a scaling process is shown in FIG.
Shown in The memory shown in FIG. 16 has 16 pixels required for interpolation, and therefore, 16 memory elements are prepared, and is configured by 16 independent memory elements having a depth in the address direction. . Then, the pixel data is stored in the memory having such a configuration. As a method of storing the pixel data, a storage method as shown in FIG. 17 can be considered. FIG. 17 is a diagram showing the upper left portion of the image in detail. Pixel data of a block indicated by pixels A1 to P1 is stored at address 1 of the memory shown in FIG. Next, the pixel data of the block indicated by the pixels A2 to P2 is written into the address 2 with the same mapping. Hereinafter, the same operation is repeated to write the pixel data of the entire image area into the memory.

By storing the pixel data in the memory in this way, the data required for interpolation of a certain point in any area of the image must be stored in any one of the address portions of the memory elements corresponding to AP. Just take it out. For example, in FIG. 17, A1 to P1, A2 to P
2, A (m + 1) to P (m + 1), A (m + 2) to P
When interpolation processing is performed using data of four pixels from four blocks of (m + 2), K1, L1,
The data of O1 and P1 are obtained from address 1, I2, J2, M
2, N2 data is obtained from address 2 by C (m + 1), D
The data of (m + 1), G (m + 1), and H (m + 1) are A (m + 2), B (m + 2),
The data of E (m + 2) and F (m + 2) is the address (m +
Although the address is different, like taking out from 2),
By extracting data one by one from the 16 memory elements, data necessary for interpolation can be obtained.

[0007] By the way, when the interpolation processing is performed in such a manner, the pipeline processing is attempted.
As shown in FIG. 18, output data from the preceding image processing circuit is switched between the memories a and b using two memories a and b (double buffers) and written, and the data is written to one of the memories. Read the data written in the other memory b when
It must be input to the scaling processor.
However, in the enlargement / reduction processing, in the interpolation processing such as the boundary portion of each block, a state occurs in which some data necessary for the interpolation processing must be used again for the interpolation processing of the next area. That is, for example, among the data written to and read from the memory b, the data required for the next processing indicated by hatching (the glue allowance) cannot be written to the memory a again from the previous processing unit. An operation such as transfer to the memory a via the buffer is required. Such an operation is extremely complicated,
In reality, it is difficult to perform pipeline processing involving such an operation. Therefore, conventionally, such pipeline processing is not performed, and output data of the previous stage is temporarily stored in a frame memory, and data is transferred from the frame memory each time. A method of reading and performing enlargement / reduction processing is used.

[0008]

In the conventional image processing apparatus, image processing including scaling processing by a plurality of image processing circuits as described above is executed by reading or writing data from a frame memory via a bus. It is supposed to be. Therefore, when performing real-time processing, there is a problem that the amount of data transferred through the bus is extremely large, and power consumption increases with processing time.

SUMMARY OF THE INVENTION The present invention has been made in order to solve the above-mentioned problems in the conventional image processing apparatus, and has been made in consideration of a plurality of image processing methods including an enlargement / reduction process without reducing a data transfer amount of a bus and increasing a memory capacity. It is an object of the present invention to provide an image processing apparatus capable of performing pipeline processing in real time.

[0010]

SUMMARY OF THE INVENTION In order to solve the above-mentioned problems, the present invention provides a method for spatially converting image data output from a solid-state imaging device and recorded in a frame memory by a plurality of image processing units including a scaling unit. Access to the image processing apparatus that performs the basic image processing and outputs the image at the preceding stage of the enlargement / reduction processing unit, (the number of rows of the interpolation size) × (the number of columns obtained by adding at least 1 to the interpolation size) A memory unit composed of various memory elements, the memory is configured to be switchable in units of columns between a memory unit for writing output data from a preceding image processing unit and a memory unit for reading input data to a scaling unit, The data of the (memory number of rows of the interpolation size) × (the number of columns of the interpolation size) memory elements necessary for the interpolation stored in the reading memory section are selectively read out and subjected to the interpolation processing, and at the same time, the remaining In which the output data from the preceding image processing unit is written to the write memory unit and the pipeline processing is executed.

With this configuration, it is possible to realize an image processing apparatus capable of performing image processing including enlargement / reduction processing capable of performing pipeline processing in real time using a small-capacity memory.

[0012]

Next, an embodiment will be described. First, a schematic embodiment according to the present invention will be described with reference to FIG. This embodiment shows a case where two image processing units including a scaling unit are used as image processing units. In the image processing apparatus according to the present embodiment, a CPU 2 that controls each unit connected to the bus 1
, The signal obtained by processing the image signal from the CCD image sensor by the pre-processing circuit 3 is temporarily stored in the frame memory 4 via the bus 1, and the image data read from the frame memory 4 is then stored via the bus 1. The operation up to the input to the image processing circuit 5 is the same as that of the conventional image processing apparatus. In the image processing apparatus according to the present invention, the image processing circuit 5 as the first stage image processing circuit is connected in series to the enlargement / reduction processing unit 8 as the next stage image processing unit via the memory 6 and the selector 7. First, image processing is performed by the pipeline method, and the JPEG processing unit 9 performs JPEG processing.
After the processing, the image data is recorded on the frame memory 4 or the memory card via the bus 1.

As described above, the amount of data transferred via the bus 1 is determined by the transfer from the frame memory 4 to the first stage image processing circuit 5 and the JPEG processing unit 9 from the frame memory 4.
Alternatively, only the transfer to the memory card is performed, so that the data transfer amount can be considerably reduced as compared with the conventional example in which data is exchanged between the frame memory and the enlargement / reduction processing unit 8.

Next, the configuration of the enlargement / reduction processing unit and the memories and selectors arranged at the preceding stage will be described in detail. First, as shown in FIG.
The image data is stored in the memory 6 configured using 20 (ROW × COL) independent memory elements A to T as described below. That is, the enlargement / reduction processing unit 8 performs the Cubic interpolation processing, and since the processing requires 16 × 4 pixel data, the pixel data of the blocks A1 to P1 of the image shown in FIG.
At the address 1 of each of the memory elements A to P,
The pixel data of the blocks 2 to P2 are stored in the addresses 2 of the memory elements AP. Rows of images in the same way (ROW)
The pixel data of the block in the direction is sequentially stored in each address of the memory elements A to P. Then, as shown in FIG. 4, the image data at the address 1 of the memory elements A to P constituting the memory 6 is read out via the selector 7, and the enlargement / reduction processing unit 8 performs the Cubic interpolation processing. The selector 7 is composed of 16 selectors for selecting and outputting one of the 20 inputs. During the reading of the image data from the memory elements A to P, the addresses 1, 2,... Of the remaining memory elements Q, R, S, T of the 20 independent memory elements A to T.
., The corresponding image data Q1 to T1, Q2 to T2,... Are written and stored as shown in FIG.

When the interpolation processing in the first column direction is completed, 16 pixel data written in the independent memory elements E to T are sequentially read out, and the Cubic interpolation processing in the second column direction is performed. At this time, the pixel data A1 to D1, A2 to D2 stored in each address of the memory elements A to D are stored.
Are unnecessary, and as shown in FIG. 5B, the addresses of the pixel data Q1 to T1, Q2 to T2,. Pixel data A1 to D1, A2 to D2,... Adjacent in the row (COL) direction
.. Are newly written and stored during the reading of the pixel data for the Cubic interpolation processing in the second column direction.
Similarly, as shown in FIG. 5C, 16 pieces of pixel data are read out, and one column of pixel data is sequentially written newly during the Cubic interpolation processing. The Cubic interpolation processing can be executed in a pipeline manner over the entire image area.

FIG. 6 shows an access mode in the memory 6 composed of 16 independent memory elements A to T during the above-mentioned Cubic interpolation processing. Also, pixel data read out from the memory 6 for the Cubic interpolation process via the selector 7 and each pixel data when the Cubic interpolation process is performed by the enlargement / reduction processing unit using the 16 pixel data read out. In FIG. 7A, the switching mode of the interpolation coefficient by which
(B) shows. FIG. 7A shows memory elements A to D.
7 is in a write state, and FIG. 7B shows a case in which the memory elements E to H are in a write state.

Next, the execution / stop control of the processing of the image processing unit including the enlargement / reduction processing unit will be described. In image processing, enlargement / reduction processing is a special processing, and the number of input data and the number of output data change. That is, in the case of enlargement processing, pixel data is densely captured, so that a plurality of pieces of data processed from 16 pieces of pixel data may be output, whereas in the case of reduction processing, 16 pieces of data may be output. In some cases, no data processed from the pixel data is output. Therefore, data is discontinuously output in the scaling processing as described above, so that data cannot be continuously transmitted and received between the preceding image processing unit and the scaling processing unit. Similarly, the process cannot be continuously executed in synchronization with the execution of the process of the filter processing unit.

In this embodiment, the memory 6 provided between the preceding image processing unit and the enlargement / reduction processing unit stores 1 data in addition to the 16 data required for the Cubic interpolation processing by the enlargement / reduction processing unit. Since it has write memory elements for columns,
During the execution of the processing steps of the block of pixel data required for the ubic interpolation processing and the writing of data from the pre-processing unit, the pre-processing unit and the enlargement / reduction processing unit determine whether the next processing block can be executed. Judge and execute /
Stop control can be performed.

Next, based on the timing charts of FIGS. 8A and 8B, control of execution / stop of the preceding image processing unit and the enlargement / reduction processing unit will be described. (A) of FIG.
Is a timing chart showing execution / stop control at the time of enlargement processing. At the time of enlargement processing, pixels are taken densely compared to the original pixels, so that the Cubic interpolation processing is performed a plurality of times using the same 16 pixel data. Will do. Then, since new data cannot be received during the processing period, when performing the Cubic interpolation processing with the same data a plurality of times, it is necessary to stop the execution of the processing of the preceding image processing unit. Therefore, when it is necessary to stop the processing of the preceding image processing unit, the execution / stop determination unit (provided in the enlargement / reduction processing unit) outputs a processing execution stop determination signal, and outputs the processing execution stop signal via the CPU. The execution of the processing of the image processing unit is temporarily stopped.

FIG. 8B is a timing chart showing the execution / stop control at the time of the reduction processing. At the time of the reduction processing, pixels are coarsely taken as compared with the original pixels. Data may not be used for processing. In that case, since the Cubic interpolation calculation processing is completed early, it is necessary to suspend the processing of the scaling processing section until the next processing data output from the preceding image processing section is written to the memory. . Therefore, when such a state is determined in advance by the execution stop determination unit, the execution of its own process by the enlargement / reduction processing unit is temporarily stopped.

In the above-described embodiment, the enlargement / reduction processing section uses the 16 × 4 pixel data as a Cubic.
An example in which interpolation processing is performed has been described.
As shown in (4), it is also possible to perform a four-point interpolation process using data of four 2 × 2 pixels A to D. In this case, the interpolation data of the Z position is obtained by converting the pixel data to A
If represented by .about.D, it is represented as the following equation (2). Z = xa · ya · A + xb · ya · B + xa · yb · C + xb · yb · D (2)

The memory used in this case is shown in FIG.
As shown in (A) of FIG. 2, the storage of pixel data in this memory is made up of four 2 × 2 pixels in this four-point interpolation process. Since the data is required, the pixel data of the blocks A1 to D1 of the image shown in FIG. 10B is stored in the address 1 of each of the memory elements A to D of the memory, and then the block pixel data of A2 to D2 is stored in the memory element. The pixel data of the blocks in the column direction of the image are sequentially stored at the addresses of the memory elements A to D in the same manner. Then, when the data of the pixels A1 to D1 stored in the four memory elements A to D of this memory are read out for the four-point interpolation process, the pixels C1 and D1 in the image shown in FIG.
The data of one row of pixels E1, F1, E2, F2,... Adjacent to 1, C2, D2,. It is written.

Next, four-point interpolation processing is performed using the data C1, D1, E1, and F1 stored in the memory elements C to F. During the Cubic interpolation processing, the pixels E1 and F of the image are processed.
Pixels A1, B1, A2, B adjacent to COL direction 1
The data of 2,... Is written and stored in the memory elements A and B. FIG. 11 shows an access mode to such a memory, and FIG. 12 shows a schematic block configuration of the memory 16, the selector 17, and the enlargement / reduction processing unit 18 in the case of 2 × 2 four-point interpolation processing.

In the above embodiment, the memory arranged before the enlargement / reduction processing unit is constituted by (interpolated size rows) × (interpolated size + 1 columns) independent memory elements. However, it is also possible to use a memory composed of independent memory elements having the number of columns of (interpolation size + 2 or more), and it is possible to perform a read / write operation in the same manner.

[0025]

As described above with reference to the embodiments, according to the present invention, it is possible to reduce a data transfer amount of a bus and to perform a plurality of image processes including a scaling process without increasing a memory capacity. An image processing device capable of performing pipeline processing in real time can be realized.

[Brief description of the drawings]

FIG. 1 is a block diagram showing a schematic configuration of an embodiment of an image processing apparatus according to the present invention.

FIG. 2 is a diagram showing a configuration of a memory arranged at a stage preceding a scaling processing unit in the embodiment shown in FIG. 1;

FIG. 3 is a diagram showing a pixel data array of an image stored in a memory shown in FIG. 2;

FIG. 4 is a schematic block diagram showing a configuration of a memory, a selector, and a scaling unit in the embodiment shown in FIG. 1;

FIG. 5 is a diagram illustrating addressing of pixel data stored in a memory illustrated in FIG. 2;

FIG. 6 is a timing chart showing an access mode of the memory shown in FIG. 2;

FIG. 7 is a diagram showing a mode of switching between memory storage data and an interpolation coefficient by a selector.

FIG. 8 is a timing chart showing an execution / stop control mode of a preceding-stage image processing unit and an enlargement / reduction processing unit during enlargement processing and reduction processing.

FIG. 9 is a diagram illustrating a 4-point Cubic interpolation processing mode.

FIG. 10 is a diagram illustrating a configuration of a memory used when performing 4-point Cubic interpolation processing, and a pixel data array of an image stored in the memory.

11 is a timing chart showing an access mode of the memory shown in FIG.

FIG. 12 illustrates a memory, a selector, and a memory that perform 4-point Cubic interpolation processing.
FIG. 3 is a schematic block diagram showing the configuration of an enlargement / reduction processing unit.

FIG. 13 is an explanatory diagram showing a conventional image processing procedure including a scaling process.

FIG. 14 is a schematic block diagram illustrating an image processing apparatus including a conventional enlargement / reduction processing unit.

FIG. 15 is a diagram illustrating a mode of enlargement / reduction processing by 16-point Cubic interpolation processing.

16 is a diagram illustrating a configuration of a memory arranged in a stage preceding the enlargement / reduction processing unit illustrated in FIG. 14;

17 is a diagram illustrating a method of storing image data in the memory illustrated in FIG. 16;

18 is a diagram illustrating an aspect in a case where pipeline processing is performed using the memory illustrated in FIG. 16;

[Explanation of symbols]

 Reference Signs List 1 bus 2 CPU 3 pre-processing circuit 4 frame memory 5 image processing circuit 6, 16 memory 7, 17 selector 8, 18 enlargement / reduction processing unit 9 JPEG processing unit

 ────────────────────────────────────────────────── ─── Continued on the front page F term (reference) 5B057 BA02 CA16 CB16 CD06 CH05 CH11 CH18 5C023 AA02 AA31 BA02 CA02 DA04 DA08 5C076 AA21 AA22 BA03 BA04 BA05 BA06 BB04

Claims (4)

[Claims] An image processing apparatus that performs spatial image processing on image data output from a solid-state imaging device and recorded in a frame memory by a plurality of image processing units including a scaling unit, and outputs the processed image data. A memory consisting of memory elements that can be accessed simultaneously (the number of rows of the interpolation size) × (the number of columns obtained by adding at least 1 to the interpolation size) is provided in the preceding stage of the enlargement / reduction processing unit. A memory unit for writing output data from the unit and a memory unit for reading input data to the scaling unit can be switched on a column-by-column basis. Number) × (the number of columns of the interpolation size) The data of the memory elements are selectively read out and subjected to interpolation processing, and at the same time, the output data from the preceding image processing section are written to the remaining writing memory sections. The image processing apparatus characterized by being configured to execute the pipeline processing write. 2. The image processing apparatus according to claim 1, wherein the enlargement / reduction processing unit is configured to perform one-point interpolation processing from an image area having m rows and n columns (where m and n are positive integers). In the first column to the m-th row in the first column of the first to m-th memory elements of the memory,
The data of the m + 1 to 2m rows of the first column are written into the second addresses of the first to m-th memory elements, respectively. A predetermined number is sequentially written to the m-th memory element while incrementing the address, and the data in the second column is written to the (m + 1) -th to (2m) -th memory elements in the same procedure. Data of m × (n
+1) After writing data to the memory elements, the memory control means for writing data from the (n + 2) th column to the first to mth memory elements in the same procedure again is provided. 2. An image processing apparatus according to claim 1, wherein all the data necessary for the interpolation can be read simultaneously even when the interpolation is performed. 3. During interpolation processing for a fixed number of data in the column direction and during writing of a fixed number of output data of the preceding image processing unit,
It is determined whether or not it is possible to write the next column of data output from the preceding image processing unit to the memory and to execute the processing in the scaling processing unit. 3. The image processing apparatus according to claim 2, further comprising means for controlling execution / stop. 4. During interpolation processing for a fixed number of data in the column direction and during writing of a fixed number of output data of the preceding image processing unit,
It is determined whether writing of the next column of data output from the preceding image processing unit to the memory and execution of the processing by the scaling processing unit are possible, and execution / stop of the processing of the scaling processing unit is controlled. 3. An image processing apparatus according to claim 2, further comprising: means.