JP4464325B2 - Image processing device - Google Patents

Description

  The present invention relates to an image processing apparatus that performs image processing on pixel data of a color image and stores it in a memory.

  In recent years, SDRAMs (synchronous DRAMs) that can read and write data faster than DRAMs have been developed and used for color image processing (see Patent Document 1). That is, for example, pixel data of a color image obtained from a digital camera is temporarily stored in the SDRAM, read out from the SDRAM, and subjected to image processing.

In order to execute image processing at high speed, a configuration is proposed in which a plurality of arithmetic units are provided in the image processing circuit, and each arithmetic unit simultaneously performs image processing on a plurality of partial images obtained by dividing one image. Has been.
JP 2002-157163 A

  In order to further reduce the size of the digital camera, it is preferable that each electronic component including the SDRAM is as small as possible. Conventionally, however, an SDRAM cannot be reduced in size because it requires an area for storing both pixel data and image processing data.

  An object of the present invention is to provide an image processing apparatus capable of reducing the capacity of a memory such as an SDRAM.

  The image processing apparatus according to the present invention divides one image by a horizontal line, and pixel data of a first partial image located above and pixel data of a second partial image located below the first partial image Is an image processing apparatus that simultaneously performs image processing by separate arithmetic means, and includes a first memory area for storing pixel data constituting each line of the first partial image, and a second partial image Provided to store pixel data constituting each line, and has an upper memory area for storing pixel data of the uppermost line and a lower memory area for storing pixel data of lines other than the uppermost line A second memory area and a temporary area, the first and second memory areas are formed in a common memory, and the pixel data of the lowermost line of the first partial image is stored in the first memory area; The address to be stored is adjacent to the address in which the pixel data of the uppermost line of the second partial image is stored in the second memory area, and the first and second partial images are respectively connected from the uppermost line to the lowermost line. The first image processing data obtained by performing the image processing sequentially toward the pixel image and processing the pixel data of the first partial image is sequentially stored for each line in the first memory area. Partial image processing data obtained by image processing relating to pixel data is temporarily stored in the temporary area, and second image processing data obtained by image processing relating to pixel data of lines other than the uppermost part is stored in the lower memory area for each line. The partial image processing data stored sequentially in the temporary area are stored in the upper memory after the storing operation in the lower memory area is completed. It is characterized by being stored in the region.

  The temporary area is formed in a memory, for example. The temporary area may be a part of an area provided for storing compressed image data obtained by image compression of the image processing data corresponding to the first and second partial images.

  The memory is, for example, an SDRAM.

  Preferably, the number of the uppermost line is plural.

  Image processing includes, for example, edge enhancement.

  Preferably, the image processing data is stored at the same address as the corresponding pixel data in the first and second memory areas.

  The image processing method according to the present invention divides one image by a horizontal line, and pixel data of a first partial image located above and pixel data of a second partial image located below the first partial image Are processed simultaneously by different computing means, and a first memory area for storing pixel data constituting each line of the first partial image and a pixel constituting each line of the second partial image An image processing method in which a second memory area for storing data is formed in a common memory, wherein the pixel data of the uppermost line of the second partial image is transferred to an upper memory area of the second memory area And the pixel data of the lines other than the uppermost part of the second partial image are stored in the lower memory area of the second memory area, and the pixels of the lowermost line of the first partial image are stored in the first memory area. Data stored The address is adjacent to the address where the pixel data of the uppermost line of the second partial image is stored in the second memory area, and the first and second partial images are moved from the uppermost line to the lowermost line, respectively. The first image processing data obtained by sequentially performing image processing toward the first partial image and processing the pixel data of the first partial image is sequentially stored for each line in the first memory area, and the pixel of the uppermost line Partial image processing data obtained by image processing relating to data is temporarily stored in the temporary area, and second image processing data obtained by image processing relating to pixel data of lines other than the uppermost part is stored in the lower memory area for each line. The partial image processing data stored sequentially and stored in the temporary area is stored in the upper memory area after the storing operation to the lower memory area is completed. It is characterized in that.

  According to the present invention, since the image processing data is overwritten on the area where the pixel data is stored in the memory, the memory capacity for storing the pixel data and the image processing data can be reduced.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.
FIG. 1 is a block diagram illustrating a schematic configuration of a digital camera including an image processing apparatus according to an embodiment of the present invention.

  Red (R), green (G), and blue (B) color filters arranged in accordance with the Bayer arrangement are provided on the light receiving surface of the image sensor (CCD) 10. A photographing lens 11 and a shutter 12 are provided in front of the light receiving surface, and these are driven by a drive circuit 13. That is, the focusing operation of the photographic lens 11 and the opening / closing operation of the shutter 12 are controlled by the drive circuit 13. When the shutter 12 is opened, the light beam that has passed through the photographic lens 11 is incident on the image sensor 10. 11, an image signal that is an analog electrical signal corresponding to the two-dimensional image obtained by 11 is generated.

  In the analog signal processing circuit 14 having a correlated double sampling circuit (CDS) and an AD converter (ADC), the image signal is removed of noise components and converted into digital pixel data (raw data). Pixel data is temporarily stored in an image processing memory (SDRAM) 16 connected to a digital signal processing circuit (DSP) 15, read out from the memory 16, and subjected to various image processing described later in the digital signal processing circuit 15. The processed image processing data is stored in the memory 16 as compressed JPEG data.

  JPEG data is read from the memory 16 and recorded in the memory card 17. The image processing data is read from the memory 16 and displayed as a still color image on a liquid crystal display (LCD) 18. Further, the image signal read from the image sensor 10 is directly input to the liquid crystal display device 18 via the analog signal processing circuit 14 and the digital signal processing circuit 15, and the monitor image is displayed as a moving image on the liquid crystal display device 18. .

  The digital signal processing circuit 15 is provided with two arithmetic units (cores) 21 and 22 having the same function. These calculation units 21 and 22 execute calculation processing such as white balance adjustment on the pixel data.

  The digital signal processing circuit 15 is connected to an operation switch 19 for performing various operations such as a recording operation of image processing data to the memory card 17.

  FIG. 2 is a diagram showing a procedure of image processing executed in the calculation units 21 and 22. In the digital signal processing circuit 15, pixel data (raw data) is first subjected to white balance adjustment in step S1. Next, in step S2, pixel interpolation is executed. That is, as will be described later, R and B pixel data are obtained by interpolation for pixels in which R and B pixel data are missing. Further, G pixel data is obtained by interpolation for the R pixel and the B pixel. Thereby, R, G, and B pixel data are obtained for all the pixels.

  In step S3, a color correction matrix is applied to the pixel data obtained as a result of the processing in step S2, and errors caused by the characteristics of the color filter are removed. In step S4, gamma correction is performed on the pixel data subjected to the color correction matrix. In step S5, RGB / YUV conversion is performed, and R, G, and B pixel data are converted into luminance data (Y), color difference data, that is, U data (= B−Y), and V data (= R) by a known method. -Y). That is, luminance data, U data, and V data are generated for one pixel.

  In step S6, edge enhancement is performed on the luminance data. In step S7, image compression is performed on the luminance data, U data, and V data according to the JPEG algorithm to obtain JPEG data.

With reference to FIG. 3, the pixel interpolation performed in step S2 will be described. In FIG. 3, the letters R, G, and B indicate red, green, and blue colors, respectively. The subscripts attached to the characters R, G, and B are the coordinates of the pixels of the two-dimensional image formed by the pixel data (raw data) input to the digital signal processing circuit 15. The origin of the coordinates is (0, 0), and the G pixel (G ₀₀ ) is located at the origin. The horizontal positive direction is right and the vertical positive direction is downward. In the pixel data of the two-dimensional image, G pixels and R pixels are alternately arranged in the odd-numbered columns from the top, and B pixels and G pixels are alternately arranged in the even-numbered columns. For example, in the third column from the top, they are arranged in the order of G ₂₀ , R ₂₁ , G ₂₂ , R ₂₃ , G ₂₄ .

Interpolation of R and B pixel data is performed for pixel G using pixel data of pixels adjacent vertically or horizontally. For example, the R pixel data of the G pixel (G ₁₁ ) can be obtained by taking the arithmetic average of the pixel data of the R pixels positioned above and below it. That is, R ₁₁ = (R ₀₁ + R ₂₁ ) / 2
It is. The B pixel data of the G pixel (G ₁₁ ) is obtained by taking the arithmetic mean of the pixel data of the B pixels located on the left and right. That is, B ₁₁ = (B ₁₀ + B ₁₂ ) / 2
It is.

For the B pixel and the R pixel, an arithmetic average of pixel data of four pixels adjacent in the oblique direction is taken. For example, the R pixel data of the B pixel (B ₁₂ ) is R ₁₂ = (R ₀₁ + R ₀₃ + R ₂₁ + R ₂₃ ) / 4.
It is. The pixel data of B of the R pixel (R ₂₁ ) is B ₂₁ = (B ₁₀ + B ₁₂ + B ₃₀ + B ₃₂ ) / 4.
It is.

On the other hand, interpolation of G pixel data is performed using pixel data of G pixels located above and below and to the left and right. For example, the G pixel data of the B pixel (B ₁₂ ) can be obtained by taking the arithmetic average of the pixel data of the G pixels located above and below and on the left and right. That is, G ₁₂ = (G ₀₂ + G ₁₁ + G ₂₂ + G ₁₃ ) / 4
It is.

Edge enhancement will be described with reference to FIGS. Edge enhancement is performed on the luminance data as described above. FIG. 4 shows the luminance data Y _{11 to} Y ₃₃ in the 3 × 3 pixels surrounded by the frame F in FIG. FIG. 5 shows an operator (filter matrix) for performing edge enhancement. As can be understood from FIG. 5, for example, in order to perform edge enhancement on the pixel of the luminance data Y ₂₂ , the following calculation may be performed.
2 × Y ₂₂ − (1/8) × (Y ₁₁ + Y ₁₂ + Y ₁₃ + Y ₂₁ + Y ₂₃ + Y ₃₁ + Y ₃₂ + Y ₃₃ )

Edge enhancement, to run with respect to all the pixels of the 3 × 3 in the frame F in FIG. 3 for example, only 16 pixels, such as G _00, R _01, G ₀₂ surrounding the outside of the frame F is insufficient . For example, in order to perform edge enhancement on a G pixel (G ₁₁ ), G pixel data is required for all eight pixels located around the G pixel. However, in the example of FIG. 3, since pixel data of G cannot be obtained for R ₀₁ and B ₁₀ , luminance data cannot be obtained, and therefore edge enhancement cannot be performed. That is, in order to perform edge enhancement, a pixel that surrounds the pixel twice is necessary. In the case of an image composed only of the pixels shown in FIG. G ₂₂₎ only.

  In this way, in order to perform edge enhancement on a two-dimensional image, in addition to the pixels of the two-dimensional image, pixel data of two pixels adjacent to each other on the upper and lower sides and the left and right sides of the two-dimensional image, that is, “margin (ring Pixel) ”is further needed. Therefore, for example, in the case of a two-dimensional image in which the number of pixels in the horizontal direction is H and the number of pixels in the vertical direction is V, data of (H + 4) pixels × (V + 4) pixels is required. In other words, image processing data of a two-dimensional image can be obtained by also using pixel data forming a double frame surrounding the outside of the two-dimensional image.

  In the present embodiment, as described above, the digital signal processing device 15 is provided with the two arithmetic units 21 and 22 having the same function. That is, one image is divided into two partial images, and pixel data of each partial image is read from the image processing memory 16. These pixel data are simultaneously subjected to image processing by the arithmetic units 21 and 22 and stored in the image processing memory 16. Next, the storage state of the pixel data and the image processing data in the memory 16 will be described in detail with reference to FIGS.

  It is assumed that the number of pixels in one image is 2048 pixels in the horizontal direction and 1536 pixels in the vertical direction. As shown in FIG. 6, one image includes a first partial image located on the upper side and a lower side of the first partial image by a horizontal line H passing between the 768th line and the 769th line from the top. Is divided into a second partial image located at. Image processing requires four more pixels in the horizontal and vertical directions, respectively. Accordingly, the memory 16 stores pixel data of 2052 × 1540 pixels.

  The pixel data (2048 pixels × 768 pixels) (reference A1) constituting each line of the first partial image includes the “margin” having a width of two pixels surrounding the left side, the upper side, and the right side of the first partial image. 1 memory area A2. That is, the first memory area A2 is an area surrounded by a broken line located above the horizontal line H in FIG. Similarly, pixel data (2048 pixels × 768 pixels) (reference numeral B1) constituting each line of the second partial image is also stored in the second memory area B2 together with the “margin” of the width of two pixels surrounding the outer periphery. The stored memory area B2 is an area surrounded by a two-dot chain line located below the horizontal line H.

  The first memory area A2 and the second memory area B2 are formed in the memory 16, that is, a common memory. That is, the address at which the pixel data of the lowermost line of the first partial image is stored in the first memory area A2 is the pixel data of the uppermost line of the second partial image stored in the second memory area B2. Adjacent to the address.

  The second memory area B2 is used to store the pixel data of the uppermost two lines of the second partial image and the pixel data of 768 lines other than the uppermost memory area B21. And a lower memory area B22. In the image processing, the pixel data in the upper memory area B21 and the pixel data in the lower memory area B22 are processed separately.

  Each of the first and second partial images is sequentially subjected to image processing from the uppermost line toward the lowermost line. In the first process, in order to perform image processing on the uppermost line of the first partial image, pixel data of five lines from the top are read from the first memory area A2. Here, the arithmetic unit 21 is provided with a buffer memory (not shown), and the read pixel data of the five lines are temporarily stored in the buffer memory. Then, image processing up to edge enhancement is performed using the stored pixel data of the five lines, and the third line from the top of the first memory area A2 (that is, the top of the first partial image). Line) address. That is, the address where the image processing data is stored is the same as the address where the pixel data is stored.

  Simultaneously with the image processing of the uppermost line of the first partial image, the image processing of the uppermost line of the second partial image is also executed. That is, pixel data of two lines from the bottom of the first partial image is read from the first memory area A2, and pixel data of three lines from the top of the second partial image is the second Read from the memory area B2. Here, the arithmetic unit 22 is also provided with a buffer memory (not shown), and the read pixel data of the five lines are temporarily stored in the buffer memory. Then, image processing up to edge enhancement is performed using the pixel data of these five lines, but the image processing data is stored at the address of the first line in the temporary area C1 (FIG. 7). The temporary area C1 is formed in the memory 16, but is an area different from the first and second memory areas A2 and B2.

  In the next processing, in order to perform image processing on the second line from the top of the first partial image, the pixel data read and stored from the first line from the top of the first memory area A2 is stored in the buffer memory. And the pixel data of the sixth line from the top of the first memory area A2 are read from the first memory area A2 and stored in the buffer memory. Then, the pixel data of the five lines stored in the buffer memory are read, and image processing is executed. The image processing data is stored at the address of the second line from the top of the first partial image in the first memory area A2. At the same time, image processing of the second line from the top of the second partial image is also executed. That is, the pixel data of the second line from the bottom of the first partial image stored in the buffer memory of the arithmetic unit 22 is erased from the buffer memory, and the fourth one from the top of the second partial image. The pixel data of this line is read from the second memory area B2 and stored in the buffer memory. Then, the pixel data of five lines (pixel data of the bottom line of the first partial image and pixel data of four lines from the top of the second partial image) stored in the buffer memory are used. The image processing is executed, and the image processing data is stored at the address of the second line in the temporary area C1.

  Further, in the next processing, in order to perform image processing on the third line from the top of the first partial image, the pixel data read and stored from the second line from the top of the first memory area A2 is buffered. While erasing from the memory, the pixel data of one line of the seventh line from the top is read from the first memory area A2 and stored in the buffer memory. Then, image processing is executed using the pixel data of five lines from the top to the seventh stored in the buffer memory. The image processing data is stored at the address of the third line from the top of the first partial image in the first memory area A2. At the same time, image processing of the third line from the top of the second partial image is also executed. That is, the pixel data of the lowermost line of the first partial image stored in the buffer memory of the arithmetic unit 22 is erased from the buffer memory, and the fifth one from the top of the second partial image is deleted. The pixel data of the line is read from the second memory area B2 and stored in the buffer memory. Then, image processing is executed using five lines from the top of the second partial image stored in the buffer memory. The image processing data is stored at the third address in the second memory area B2. Therefore, pixel data is still stored at the first and second addresses in the second memory area B2.

  In the subsequent processing, image processing data is obtained for the fourth line from the top of the first partial image, and stored in the address of the fourth line from the top of the first partial image in the first memory area A2. . At the same time, the image processing data of the fourth line from the top of the second partial image is obtained and stored at the fourth address in the second memory area B2.

  As described above, regarding the pixel data of the first partial image, the first image processing data obtained by the image processing is sequentially stored for each line in the first memory area A2. On the other hand, regarding the pixel data of the second partial image, the partial image processing data obtained by the image processing relating to the pixel data of the uppermost line is stored in the temporary area C1, and the image processing relating to the pixel data of the lines other than the uppermost line. The second image processing data obtained by the above is sequentially stored for each line in the lower memory area B22.

  That is, as shown in FIG. 8, the memory 16 stores the image processing data of the first partial image P1 (2048 pixels × 768 pixels) and the lower image P21 excluding the uppermost two lines of the second partial image. Are stored. The partial image processing data of the upper two-line upper image P22 of the second partial image is stored in the temporary area C1.

  After the storage operation in the first memory area A2 and the lower memory area B22 is completed, the partial image processing data stored in the temporary area C1 is stored in the upper memory area B21. Accordingly, in the memory 16, the image processing data is stored at the same address as the corresponding pixel data (2048 pixels × 1536 pixels) in the first and second memory areas A2 and B2.

  Therefore, when the image processing data stored in the first and second memory areas A2 and B2 are read out in the order of addresses and reproduced by the liquid crystal display device 18, as shown in FIG. An image formed by connecting two partial images P2 (consisting of an upper image P22 and a lower image P21) is displayed.

  FIG. 10 shows the configuration of the image processing memory 16. The image processing memory 16 has first and second memory areas A2 and B2 and a compressed image recording area C2. The first and second memory areas A2 and B2 are areas for storing pixel data (raw data) or image processing data Y data, U data, and V data. The compressed image recording area C2 is provided for storing compressed image data (JPEG data) obtained by image compressing image processing data corresponding to the first and second partial images. The temporary area C1 is a part of the compressed image recording area C2.

  As described above, in the present embodiment, the image processing data obtained by performing the image processing on the pixel data of the uppermost two lines of the second partial image is temporarily stored in the temporary area C1, and is stored in the first memory. After the storage operation to the area A2 and the lower memory area B22 is completed, the image processing data of the temporary area C1 is stored in the upper memory area B21. Therefore, since image processing is performed on all pixel data of one image, a line that is not subjected to image processing does not occur in the central portion of the reproduced image.

  In addition, the present embodiment is configured such that image processing is simultaneously performed in the first and second calculation units 21 and 22 on two partial images obtained by dividing one image into two parts. An area for temporarily storing the uppermost image processing data of the partial image is provided. Therefore, it is not necessary to store both the image processing data and the pixel data in the image processing memory 16, and the capacity of the image processing memory 16 can be minimized, and the image processing apparatus can be downsized. It becomes possible.

  In the above embodiment, the raw data is obtained by adding pixel data of two pixels on the top and bottom and the left and right of the two-dimensional image. However, depending on the content of the image processing, three pixels or four pixels are used. What added pixel data may be adopted. In this case, the capacity of the temporary area C1 is set to a size of 3 lines or 4 lines according to the width of “margin”.

  The number of image divisions is not limited to 2, and the present invention can be applied to, for example, four divisions.

1 is a block diagram illustrating a schematic configuration of a digital camera including an image processing apparatus according to an embodiment of the present invention. It is a figure which shows the procedure of the image process performed in a digital signal processing circuit. It is a figure for demonstrating the pixel interpolation performed in step S2 of FIG. It is a figure which shows the pixel data of G of the pixel in the frame of FIG. It is a figure which shows the operator for performing edge emphasis. It is a figure which shows the storage state of the pixel data in the image processing memory. It is a figure which shows a temporary area | region. It is a figure which shows the state which stored the image processing data of the 1st partial image, and the image processing data of the lower part image except 2 lines of the uppermost part among 2nd partial images in the image processing memory. It is a figure which shows the state which reproduced | regenerated image processing data with the liquid crystal display device. It is a figure which shows the structure of the memory for image processing.

Explanation of symbols

15 Digital signal processing circuit 16 Image processing memory

Claims (8)

One image is divided by a horizontal line, and pixel data of the first partial image located on the upper side and pixel data of the second partial image located on the lower side of the first partial image are separated by separate calculation means. An image processing apparatus that simultaneously performs image processing,
A first memory area for storing pixel data constituting each line of the first partial image;
Provided to store pixel data constituting each line of the second partial image, and stores an upper memory area for storing pixel data of the uppermost line and pixel data of lines other than the uppermost line. A second memory area having a lower memory area for
With a temporary area,
The first and second memory regions are formed in a common memory;
The address at which the pixel data of the lowermost line of the first partial image is stored in the first memory area stores the pixel data of the uppermost line of the second partial image in the second memory area. Adjacent to the address,
Each of the first and second partial images is sequentially processed from the uppermost line toward the lowermost line,
The first image processing data obtained by image processing the pixel data of the first partial image is sequentially stored for each line in the first memory area,
Partial image processing data obtained by image processing relating to pixel data of the uppermost line is temporarily stored in the temporary area,
Second image processing data obtained by image processing relating to pixel data of lines other than the top is sequentially stored for each line in the lower memory area,
The partial image processing data stored in the temporary area is stored in the upper memory area after the storage operation in the lower memory area is completed.   The image processing apparatus according to claim 1, wherein the temporary area is formed in the memory.   The temporary area is a part of an area provided for storing compressed image data obtained by image compression of the image processing data corresponding to the first and second partial images. The image processing apparatus according to claim 1.   The image processing apparatus according to claim 1, wherein the memory is an SDRAM.   The image processing apparatus according to claim 1, wherein the number of the uppermost line is plural.   The image processing apparatus according to claim 1, wherein the image processing includes edge enhancement.   The image processing apparatus according to claim 1, wherein the image processing data is stored at the same address as the corresponding pixel data in the first and second memory areas. One image is divided by a horizontal line, and pixel data of the first partial image located on the upper side and pixel data of the second partial image located on the lower side of the first partial image are separated by separate calculation means. A first memory area for storing image data constituting each line of the first partial image at the same time and a pixel data constituting each line of the second partial image An image processing method in which the second memory area is formed in a common memory,
The pixel data of the uppermost line of the second partial image is stored in the upper memory area of the second memory area, and the pixel data of the lines other than the uppermost part of the second partial image is stored in the second area. Stored in the lower memory area of
The address at which the pixel data of the lowermost line of the first partial image is stored in the first memory area stores the pixel data of the uppermost line of the second partial image in the second memory area. Adjacent to the address,
Each of the first and second partial images is sequentially processed from the uppermost line to the lowermost line,
First image processing data obtained by image processing the pixel data of the first partial image is sequentially stored for each line in the first memory area;
Partial image processing data obtained by image processing relating to pixel data of the uppermost line is temporarily stored in a temporary area,
Second image processing data obtained by image processing relating to pixel data of lines other than the top is sequentially stored for each line in the lower memory area,
The image processing method, wherein the partial image processing data stored in the temporary area is stored in the upper memory area after the storage operation in the lower memory area is completed.

Images