JP5231330B2 - Image processing device - Google Patents

Description

  The present invention relates to an image processing apparatus capable of directly connecting a plurality of image processes.

  In recent years, with the improvement in the number of pixels and the continuous shooting performance of an imaging apparatus typified by a digital camera or the like, an image processing apparatus capable of processing image data with a huge number of pixels in a short time is required. Yes.

  With regard to such high-speed image data processing technology, for example, Patent Document 1 proposes a technology for converting a plurality of image processes into pipeline processing. In Patent Document 1, a plurality of image processing units that perform a plurality of spatial filter processing such as low-pass filter processing and an image compression processing unit are directly connected. In Patent Document 1, image data is input from the last stage of the image processing unit to the image compression processing unit for each compression unit (generally called MCU (Minimum Coded Unit)) necessary for the compression processing in the image compression processing unit. As can be done, image data is read from the frame memory in units of blocks having the number of pixels equal to the number of pixels in the column direction in the column direction and the number of pixels for one line in the row direction.

  Here, when the spatial filter processing is performed on the image data, the size of the image data after processing is generally smaller than the size of the image data before processing. This is because image processing such as spatial filter processing cannot process the peripheral portion of the image data input to the image processing unit. In the case of a configuration in which an image processing unit and an image compression processing unit are directly connected as in Patent Document 1, the number of pixels in the column direction of image data input to the image compression processing unit from the last stage of the image processing unit is a compression unit. Must be equal to the number of pixels in the column direction (or an integer multiple thereof). For this reason, in patent document 1, the peripheral part of each block is processed redundantly. However, when the peripheral portion of each block is processed in an overlapping manner, a reduction in image processing efficiency is inevitable.

  As a proposal for eliminating the duplication processing of image data in the spatial filter processing, for example, the technique of Patent Document 2 is proposed. In this patent document 2, among the intermediate data being processed by the filter processing means, the intermediate data of the overlapping portion used also for the processing of the next divided image is stored in the frame memory (image memory). . And in this patent document 2, at the time of the process of the next division image, it processes by using the next division image and the intermediate data stored in the frame memory (image memory). Eliminates the need for multiple processing.

JP 2000-31327 A JP 2002-304624 A

  Here, when the method of Patent Document 2 is applied to an image processing apparatus in which a plurality of image processing units such as Patent Document 1 are directly connected and connected, intermediate data sequentially generated by processing in each image processing unit is converted into an image. It is necessary to store sequentially in the memory. For this reason, the bandwidth of the bus connected to the image memory is squeezed and the processing speed is likely to decrease. Also, the capacity of the image memory is easy to press.

  The present invention has been made in view of the above circumstances, and an object thereof is to provide an image processing apparatus capable of improving the image processing efficiency when a plurality of image processing units are directly connected and connected.

In order to achieve the above object, an image processing apparatus according to a first aspect of the present invention is an image processing apparatus that performs a plurality of image processings on image data that has been captured and stored in a frame memory. The image data of the divided block lines obtained by dividing the image data of the plurality of block lines having the first number of pixels in the column direction and the second number of pixels in the row direction into a plurality of regions in the row direction is in column order. A plurality of stages of image processing units that sequentially perform image processing on the image data of the divided block lines that are input and input in the column order, and are provided corresponding to each of the plurality of stages of image processing units. Among the image data of the current divided block line input to each of the image processing units in the stage, the column is also used in image processing for the image data of the divided block line next to the current divided block line ; And a margin buffer for storing image data of the margin portion of the direction, each of the image processing unit of the plurality of stages, the image processing on the image data of the next divided block line, the next divided block Image processing is performed on image data obtained by combining line image data and image data of the margin portion.

  According to the present invention, it is possible to provide an image processing apparatus capable of improving the image processing efficiency when a plurality of image processing units are directly connected and connected.

It is a figure which shows the structure of the imaging device provided with the image processing apparatus which concerns on one Embodiment of this invention. It is a figure for demonstrating a division | segmentation block line. It is a figure for demonstrating MCU. It is a figure for demonstrating a compression unit. 3 is a diagram illustrating a detailed configuration of one image processing unit and a margin storage buffer in the image processing unit 106. FIG. 3 is a diagram illustrating a concept of operation of an image processing unit 106. FIG. 6 is a timing chart showing the operation of the image processing unit 106. It is a figure for demonstrating the process order of the compressed image data at the time of image processing for every divided block line. It is a figure which shows schematic structure of a JPEG rearrangement part. FIG. 10 is a diagram showing a data format of compressed image data written to the frame memory 105 by the rearrangement unit 1091. It is a figure which shows the relationship between a row direction compression unit number, a column direction compression unit number, and a compression unit line. It is a figure which shows the detailed structure of the rearrangement part 1091. FIG. It is a figure which shows the detailed structure of the rearrangement part 1092. FIG. 10 is a flowchart illustrating the operation of the rearrangement unit 1092. It is a figure which shows the state of the compression image data on the frame memory 105 until the completion of rearrangement of the division | segmentation block line 2-1. It is a figure which shows the modification in the case of dividing | segmenting and processing not only in a row direction but in a column direction. It is a figure which shows the structure of the modification in case only one paste margin preservation | save buffer is provided. It is a figure for demonstrating the glue margin in the case of a distortion correction process.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.
FIG. 1 is a diagram illustrating a configuration of an imaging apparatus including an image processing apparatus according to an embodiment of the present invention. 1 includes a lens 101, an image sensor 102, a preprocessing unit 103, a bus 104, a frame memory 105, an image processing unit 106, a JPEG interface (I / F) 107, and JPEG processing. Unit 108, JPEG rearrangement unit 109, media interface (I / F) 110, recording medium 111, and CPU 112.

  The lens 101 focuses the optical image of the subject on the image sensor 102. The image sensor 102 is configured by attaching, for example, a Bayer array color filter to a light receiving surface formed by two-dimensionally arranging photoelectric conversion elements such as photodiodes. The image sensor 102 converts the light collected by the lens 101 into an electrical signal (image signal) and outputs it to the preprocessing unit 103. The image sensor 102 may be a CMOS image sensor or a CCD image sensor.

  The preprocessing unit 103 performs analog processing such as correlated double sampling (CDS) processing and gain control (AGC) processing on the image signal from the image sensor 102, and further performs A / D conversion on the analog processed image signal. To generate digital signal image data.

  The bus 104 is a transfer path for transferring various data generated inside the imaging apparatus to each block in the imaging apparatus. The bus 104 is connected to the preprocessing unit 103, the frame memory 105, the image processing unit 106, the JPEG processing unit 108, the JPEG rearrangement unit 109, the media I / F 110, and the CPU 112.

  The frame memory 105 stores various data such as image data obtained by the preprocessing unit 103 and image data processed by the JPEG processing unit 108. The frame memory 105 has a capacity that can store image data for at least one frame.

  The image processing unit 106 is configured by directly connecting a plurality of image processing units 1061a, 1062a,..., 106na, and converts Bayer array image data into YCbCr data for the image data read from the frame memory 105. A plurality of image processing such as processing and low-pass filter processing are sequentially performed. Note that the image processing unit 106 in the present embodiment reads the image data stored in the frame memory 105 in units of divided block lines. FIG. 2A shows the concept of divided block lines. One divided block line in the present embodiment is configured as image data obtained by dividing one block line into three in the row direction. One block line has the number of pixels equal to an integral multiple of the compression unit in the column direction (first pixel number), and the number of pixels for one row (second pixel number) in the row direction. Configured. Here, the compression unit means unit image data to be compressed by the JPEG processing unit 108. In the present embodiment, one compression unit is configured by arranging one or more MCUs in the row direction. It is assumed that

  In the JPEG standard, discrete cosine transform (DCT) is performed using 8 × 8 pixels as one unit. A collection of a plurality of 8 × 8 pixel image data is generally called an MCU. For example, in the case of image data in the YC422 format (image data in which the data number ratio between luminance and color difference obtained after image processing is Y: Cb: Cr = 4: 2: 2), Y shown in FIG. The total of × 8 pixels, Cb is 8 × 8 pixels, and Cr is 8 × 8 pixels is 1 MCU. In addition, in the case of image data in YC420 format (image data in which the data number ratio between luminance and color difference obtained after image processing is Y: Cb: Cr = 4: 1: 1 and Cb = Cr) is shown in FIG. , Y is 16 × 16 pixels, Cb is 8 × 8 pixels, and Cr is 8 × 8 pixels.

  In the JPEG standard, compression processing is performed in units of MCUs, and a marker called a restart marker is inserted at the end of the compression processing. At the time of decompressing the image data, it is possible to correctly decompress the compressed image data by detecting the restart marker. In the present embodiment, one compression unit is determined by appropriately setting the interval at which the restart marker is inserted.

  4A to 4C show examples of compression units. FIG. 4A shows an example in which one compression unit is constituted by one MCU. In this case, a restart marker is inserted when the compression of the image data for one MCU shown in FIG. FIG. 4B shows an example in which one compression unit is constituted by 2 MCUs. In this case, a restart marker is inserted when the compression of image data for 2 MCUs is completed. FIG. 4C shows an example in which one compression unit is constituted by 4 MCUs. In this case, a restart marker is inserted when compression of image data for 4 MCUs is completed. Not limited to those shown in FIGS. 4A to 4C, one compression unit may be an integer multiple of the MCU.

  The image processing unit 106 according to the present embodiment reads image data for each divided block line by reading the image data so that the divided block lines are read in the column order as indicated by arrows in FIG. I do. By performing such reading, the image processing unit 106 can output image data corresponding to a compression unit.

  In the present embodiment, paste margin storage buffers 1061b, 1062b,..., 106nb are provided corresponding to the plurality of image processing units 1061a, 1062a,. Details of the operation of the image processing unit 106 including the operations of the margin storage buffers 1061b, 1062b,..., 106nb will be described later.

  The JPEG I / F 107 is connected to the final stage image processing unit constituting the image processing unit 106, and is configured to have the same number of buffer memories as the number of compression units in the column direction included in each divided block line. Yes. Each buffer memory has at least a capacity sufficient to store one piece of image data for a compression unit. By using the buffer memory having such a configuration, the image data of the divided block lines processed by the image processing unit 106 is divided for each compression unit.

  The JPEG processing unit 108 compresses the image data processed by the image processing unit 106 and read from the JPEG I / F 107 for each compression unit at the time of recording the image data, and the compressed image data obtained by the compression processing is converted into a frame. Write to the memory 105. The JPEG processing unit 108 also reads the compressed image data recorded on the recording medium 111 and decompresses the read compressed image data and then writes it to the frame memory 105 when reproducing the image data.

  The JPEG rearrangement unit 109 performs rearrangement so that the compressed image data processed by the JPEG processing unit 108 is correctly reproduced during reproduction. Details of the rearrangement will be described later.

  The media I / F 110 controls writing and reading of data on the recording medium 111. The recording medium 111 is, for example, a recording medium composed of a memory card that can be attached to and detached from the imaging apparatus.

  The CPU 112 comprehensively controls various sequences of the imaging apparatus such as operation control of the image sensor 102, the preprocessing unit 103, and the image processing unit 106.

  Next, the image processing unit 106 will be further described. FIG. 5 is a diagram illustrating a detailed configuration of one image processing unit and a margin storage buffer in the image processing unit 106. 5 illustrates the configuration of an image processing unit that performs 3 × 3 filter processing as image processing.

  As shown in FIG. 5, the image processing unit includes an input buffer 201, an image data serial / parallel conversion unit 202, a margin data serial / parallel conversion unit 203, and a product-sum operation unit 204. The margin storage buffer has a margin storage memory 205.

  The input buffer 201 holds image data for divided block lines input as serial data according to the order shown in FIG. 2B from the frame memory 105 or the image processing unit in the previous stage.

  The image data serial conversion unit 202 converts the serial data image data output from the input buffer 201 into parallel data, thereby generating 3 × 3 pixel image data necessary for the filter processing. The image data-serial conversion unit 202 includes two line buffers 301a and 301b, nine flip-flops (FF) 302a to 302i, six selectors 303a to 303f, and a data switching control unit 304. Yes. Here, of the nine FFs, the FF 302a is connected to the input buffer 201, and FFs 302d and 302g are connected in series via the selectors 303a and 303d, respectively, after the FF 302a. A line buffer 301 a is connected to the input buffer 201. The FF 302b is connected to the line buffer 301a, and FFs 302e and 302h are connected in series via selectors 303b and 303e, respectively, after the FF 302b. The line buffer 301b is connected to the line buffer 301a. The FF 302c is connected to the line buffer 301b, and FFs 302f and 302i are connected in series via selectors 303c and 303f, respectively, after the FF 302c. Further, the FFs 302 a to 302 i are also connected to the product-sum operation unit 204.

  Each of the line buffers 301a and 301b has a capacity capable of holding image data corresponding to the number of pixels in the column direction (one line) of one divided block line. The line buffer 301a holds the image data from the input buffer 301 for one line, and then delays the input image data by sequentially outputting the held image data to the line buffer 301b and the FF 302b. In addition, the line buffer 301b holds the image data from the line buffer 301a for one line, and then sequentially outputs the held image data to the FF 302c, thereby further delaying the input image data. Each time the image data is input, the FFs 302a to 302i sequentially output the image data for one pixel that has been held so far. Such line buffers 301a and 301b and FFs 302a to 302i generate 3 × 3 pixel image data.

  Here, if the size of the filter is not 3 × 3, it is necessary to change the number of line buffers and FFs accordingly. In general, if the filter size is m × n, the image data serial-parallel conversion unit 202 needs (n−1) line buffers and (m × n) FFs.

  The selectors 303a to 303f respectively connect the image data output from the FFs 302a to 302f in the image data serial-parallel converter 202 and the image data of the margin part output from the FFs 304a to 304f in the margin data serial-para converter 203. Either one is selected according to the data switching control signal. The data switching control unit 304 performs switching control of the data switching control signal. Here, details of the data switching control signal switching control by the data switching control unit 304 will be described later.

  The margin data serializer conversion unit 203 converts the image data of the margin portion output as serial data from the margin storage memory 205 of the margin storage buffer into parallel data. This margin data serial-serial conversion unit 203 is composed of six FFs 304a to 304f. Each time image data is input, the FFs 304a to 304f sequentially shift the image data for one pixel that has been held so far. These FFs 304a to 304f generate image data of 2 × 3 pixels from the image data of the margin portion input from the margin storage memory 205.

  Here, the number of FFs of the margin data serial conversion unit 203 is changed depending on the number of pixels of the image data of the margin portion described later.

  The product-sum operation unit 204 executes product-sum operation for filter operation on the 3 × 3 pixel image data output from the FFs 302a to FF302i. The coefficient multiplied to the image data output from each of the FF 302a to FF 302i is changed according to the type of filter applied to the image processing.

  The margin storage memory 205 of the margin storage buffer is composed of, for example, an SRAM (Static RAM), and stores a part of the image data output from the input buffer 201 as image data of the margin portion.

  Here, the concept of glue margin will be described. In general, when performing image processing including spatial filter processing such as low-pass filter processing, it is known that the data size of image data after image processing is smaller than the size of image data input to the image processing unit. . This is because, in the spatial filter processing, a portion where the filter processing cannot be performed occurs in the peripheral portions (upper end portion, lower end portion, right end portion, left end portion) of the image data input to the image processing unit. Therefore, in order to perform image processing on a certain area of image data obtained via the image sensor 102, an image signal of an area wider than the predetermined area is acquired in the image sensor 102, and from this predetermined area, In addition, it is necessary to perform image processing on image data for a wide area. In the present embodiment, the image data corresponding to the extra area is defined as a margin.

  Further, when the image processing unit 106 and the JPEG processing unit 108 are directly connected as in this embodiment, image data is read from the frame memory 105 in units of divided block lines as shown in FIG. ing. In this case, since an area that cannot be processed every time image processing is performed on the image data of the divided block lines, an image of the margin portion (hatched portion in FIG. 2A) is generated every time image processing is performed on the image data of the divided block lines. Data also needs to be processed. When such processing is performed, the glue margin overlaps between the lower end of a certain divided block line 1-n and the upper end of the divided block line 1- (n + 1) next to the divided block line 1-n.

  Here, the number of pixels in the column direction of the image data of the margin portion is determined by the number of taps of the filter used for the spatial filter processing and the number of times of the spatial filter processing. In general, the number of pixels in the column direction of the margin required for image processing using an m × m central interpolation filter (number of taps m) is (m−1) pixels. For example, in the case of the 3 × 3 filter described above, the number of pixels in the column direction of the image data in the margin portion is 2. In addition, the number of pixels in the column direction of the image data of the margin portion necessary for the image processing using the 5 × 5 filter is 4. Further, the number of pixels in the column direction of the image data of the margin portion when the plurality of spatial filter processes are directly connected is the sum of the margins of the respective spatial filters. For example, the number of pixels in the column direction of the image data in the margin portion when the 5 × 5 filter and the 3 × 3 filter are directly connected is 6.

  In the present embodiment, when image processing is performed on image data of a certain divided block line 1-n, a margin portion used also in image processing in the divided block line 1- (n + 1) next to the divided block line 1-n. Are stored in the margin storage memory 205, and in the image processing of the divided block line 1-n, the image processing is performed on the divided block line 1- (n-1) before the divided block line 1-n. The image processing is performed including the image data of the margin portion stored in the margin storage memory 205.

  When performing processing in units of divided block lines, in addition to the pasting margin in the column direction as described above, for example, the right end of the divided block line 1-n and the left end of the adjacent divided block line 2-n Adhesive margin is also required in the row direction. However, if this margin in the row direction is taken into account, the processing becomes complicated. Therefore, in this embodiment, the margin in the row direction is ignored.

  Next, details of the operation of the image processing unit 106 in the present embodiment will be described. FIG. 6 is a diagram illustrating the concept of the operation of the image processing unit 106. In the example of FIG. 6, the image processing using the 3 × 3 filter (the number of pixels in the column direction of the margin portion is 2 as described above) is performed on the image data of the divided block line 1- (n + 1). The case where it performs is illustrated.

  When image processing is performed on the image data of the divided block line 1-n, the image data of the divided block line 1-n stored in the input buffer 201 is read as serial data according to the order shown in FIG. Then, image processing is performed. At this time, when image data corresponding to the lower two lines of the divided block line 1-n (image data indicated by the hatched portion in FIG. 6) is read from the input buffer 201, an image corresponding to the lower two lines. The data is stored in the margin storage memory 205 as image data of the margin portion.

  Here, in the image processing of the image data of the divided block line 1-n, the image including the image data stored in the margin storage memory 205 in the image processing of the divided block line 1- (n-1) is also included. Process. The description related to the image processing for the divided block line 1-n is the same as the description related to the image processing for the next divided block line 1- (n + 1). Therefore, the description is omitted here.

  Next, image processing for the image data of the divided block line 1- (n + 1) will be described. As described above, image data is input from the input buffer 201 line by line in the order shown in FIG. The glue margin corresponding to the same line as the line read from the input buffer 201 before the image data corresponding to the first pixel (upper end) in each line of the divided block line 1- (n + 1) is read from the input buffer 201. Image data (for two pixels in the column direction) is read out as serial data from the margin storage memory 205 and sequentially input to the FF 304a. Also, the margin image data held in the FFs 304a to 304f is sequentially shifted to the subsequent stage. As a result, the margin image data input from the margin storage memory 205 is converted into parallel data.

  Thereafter, before the image data corresponding to the first pixel (upper end) in each line of the divided block lines 1- (n + 1) is read from the input buffer 201, the data switching control unit 304 has been input from the FFs 304a to 304f. The data switching control signal is switched so as to select the paste margin image data. When the image data corresponding to the second pixel in each line of the divided block line 1- (n + 1) is read from the input buffer 201, the data switching control unit 304 selects the image data input from the FFs 302a to 302f. The data switching control signal is switched as described above.

  Each time image data is input to the FFs 302a to 302i, the image data held in the FFs 302a to 302i until then is shifted to the subsequent stage. When image data for 3 × 3 pixels is input to the product-sum operation unit 204, product-sum operation (filter processing) is performed on the input image data for 3 × 3 pixels.

  Thereafter, the operations of reading the margin image data from the margin storage memory 205, selecting the margin image data by the data switching control unit 304, and selecting the image data read from the input buffer 201 are performed on the divided block lines. It repeats for every line which comprises. In this way, as shown in FIG. 6, the filtering process for every 3 × 3 pixels in the divided block line 1- (n + 1) is performed.

  Further, when image data corresponding to the two lower rows of the divided block line 1- (n + 1) (image data indicated by the hatched portion in FIG. 6) is read from the input buffer 201, it corresponds to the two lower rows. The image data is stored in the margin storage memory 205 as image data of the margin portion. The image data of the margin portion stored in the margin storage memory 205 is used when processing the next 1- (n + 2) divided block line image data.

  By performing image processing by the image processing unit 106 having the above-described configuration, it is not necessary to read the image data of the margin portion when reading the image data of each divided block line, and the image data of the margin portion is processed. Duplicate processing is not necessary.

  Furthermore, by storing the margin data in the margin storage buffers 1061b to 106nb, even if the number of directly connected image processing units increases, the bandwidth of the bus 104 is not compressed.

  Also, by storing the margin portion image data for each divided block line, the size of the margin storage buffer can be reduced as compared with the case where the margin portion image data is stored for each block line. Is possible.

  FIG. 7 is a timing chart showing the operation of the image processing unit 106 in the present embodiment. 7 shows an example in which 3 × 3 filter processing is performed in the image processing unit 1061a and 5 × 5 filter processing is performed in the image processing unit 1062a.

  As shown in FIG. 7, in the present embodiment, almost all parts of the image data input to the first stage image processing unit 1061a and processed can be handled as valid data. For this reason, it is not necessary to wait until a valid portion of image data is input in the subsequent image processing unit 1062a, and as a result, the efficiency of image processing can be improved.

Next, the JPEG rearrangement unit 109 will be further described.
For example, consider a case where image processing is performed for each divided block line obtained by dividing image data for one frame as shown in FIG. Here, in the example shown in FIG. 8A, image data for one frame is configured by arranging image data of H compression units in the row direction and V compression units in the column direction. In addition, image data of M compression units in the row direction and 4 compression units in the column direction are arranged to form one divided block line. In FIG. 8A, the image data of each compression unit is numbered 1-1 to HV from the upper left corner for later explanation. Although not shown in FIG. 8A, the divided block lines are also numbered 1-1 to HV from the upper left corner.

  As described above, in this embodiment, image data is input for each divided block line. As a result, the image data shown in FIG. 8A is subjected to various processes in the image processing unit 106, and then, as shown in FIG. 8B, the compression units 1-1, 2-1,. 1, 1-2, 2-2, ..., M-2, 1-3, 2-3, ..., M-3, 1-4, 2-4, ..., M-4, 1-5, 2- 5, ..., M-5, 1-6, ..., M-V, (M + 1) -1, (M + 2) -1, ..., N-1, (M + 1) -2, ..., NV, (N + 1) ),..., HV in this order and input to the JPEG processing unit 108 via the JPEG I / F 107. Then, the compression processing is sequentially performed as shown in FIG. Each time the compression processing of the image data of the compression unit is completed, a restart marker RSTm that is identification information indicating a division of the compression unit is inserted. The restart marker RSTm is numbered according to the insertion order.

  Here, since the reproduction of the image data is normally performed in the line order, if the compressed image data is recorded on the recording medium 111 in the order shown in FIG. 8C, it is appropriate when reproducing the image data. The image is not played back.

  Therefore, in the present embodiment, the JPEG rearrangement unit so that the compressed image data obtained by the compression processing in the order shown in FIG. 8C is recorded on the recording medium 111 in the row order corresponding to the reproduction. In 109, the compressed image data is rearranged.

  FIG. 9 is a diagram illustrating a schematic configuration of the JPEG rearrangement unit 109. As illustrated in FIG. 9, the JPEG rearrangement unit 109 includes two rearrangement units 1091 and 1092.

  The rearrangement unit 1091 generates rearrangement header data necessary for rearrangement of the compressed image data, and writes the generated rearrangement header data in the frame memory 105 together with the compressed image data input from the JPEG processing unit 108. Process.

  FIG. 10 shows the data format of the compressed image data written to the frame memory 105 by the rearrangement unit 1091. As shown in FIG. 10, the compressed image data written in the frame memory 105 has a rearrangement header data portion and a compressed image data portion. The rearrangement header data part has rearrangement header data corresponding to each divided block line.

  The rearrangement header data includes the number of compression units in the row direction, the number of compression units in the column direction, and the amount of compression unit line data. The row direction compression unit number indicates the number of compression units arranged in the row direction of the corresponding divided block line. The number of compression units in the column direction indicates the number of compression units arranged in the column direction in the divided block line. Further, the compression unit line data amount is a data amount of compressed image data (hereinafter referred to as a compression unit line) having one division block line in the row direction and one compression unit in the column direction. Only the number of compression unit lines in the middle (four in the example of FIG. 8) is recorded. FIG. 11 shows the relationship between the number of row-direction compression units, the number of column-direction compression units, and the compression unit line.

  Here, the number of compression units in the row direction and the number of compression units in the column direction are determined depending on how the divided block lines are divided. Further, the compression unit line data amount changes for each compression unit line. Therefore, the compression unit line data amount is determined by counting the data amount every time image data of each compression unit line is input.

  The compressed image data portion is a portion where the compressed image data processed by the JPEG processing unit 108 is written.

  Further, the rearrangement unit 1092 reads the rearrangement header data and the compressed image data for each compression unit written in the frame memory 105 as shown in FIG. 10 by the rearrangement unit 1091, and the compressed image for each compression unit. The data is rearranged in the normal order and written back to the frame memory 105.

Hereinafter, the details of each of the rearrangement units 1091 and 1092 will be described.
First, details of the rearrangement unit 1091 will be described. FIG. 12 is a diagram illustrating a detailed configuration of the rearrangement unit 1091. As shown in FIG. 12, the rearrangement unit 1091 includes a selector 401, data amount counters 402a to 402d, a header generation unit 403, a restart marker detection unit 404, a restart marker counter 405, a selector 406, DMA 407.

  The selector 401 selects one of the data amount counters 402a to 402d every time the restart marker detecting unit 404 detects the restart marker RSTm, and the compressed data processed by the JPEG processing unit 108 is selected by the selected data amount counter. Input image data.

  The data amount counters 402a to 402d are provided in the same number as the number of compression unit lines in the divided block lines (four in the example of FIG. 8), and count the data amount of the compressed image data input from the selector 401. Thus, the compression unit line data amount is counted.

  The header generation unit 403 generates rearrangement header data. Here, as described above, the number of compression units in the row direction and the number of compression units in the column direction of the header data for rearrangement are determined by the way of dividing the divided block lines. Therefore, registers for setting the row direction compression unit number and the column direction compression unit number are respectively provided in the header generation unit 403, and the CPU 112 previously stores the row direction compression unit number and the column direction compression unit in these registers. Set the number. Thereby, at the time of generating the rearrangement header data, it is possible to obtain the row direction compression unit number and the column direction compression unit number by referring to the register setting. The compression unit line data amount can be acquired as a count result in the data amount counters 402a to 402d.

  The restart marker detection unit 404 detects a restart marker RSTm inserted at the end of the compressed image data for each compression unit input from the JPEG processing unit 108. The restart marker counter 405 is a counter that is incremented every time the restart marker detection unit 404 detects the restart marker RSTm, and counts the number of restart markers inserted into the compressed image data.

  The selector 406 selectively outputs rearrangement header data generated by the header generation unit 403 and compressed image data input from the JPEG processing unit 108 to the DMA 407 according to the count of the restart marker counter 405. A DMA (Direct Memory Access) 407 writes the rearrangement header data generated by the header generation unit 403 and the compressed image data input from the selector 406 to the frame memory 105.

  The operation of rearrangement section 1091 in FIG. 12 will be described. The compressed image data for each compression unit processed by the JPEG processing unit 108 is input to the selectors 401 and 406 and the restart marker detection unit 404, respectively. The selector 406 first selects the compressed image data processed by the JPEG processing unit 108. Therefore, the compressed image data processed by the JPEG processing unit 108 is output to the DMA 407 via the selector 406. Thereby, the compressed image data is sequentially written into the frame memory 105 by the DMA 407.

  The selector 401 first selects the data amount counter 402a. Therefore, the compressed image data processed by the JPEG processing unit 108 is input to the data amount counter 402a via the selector 401, and the data amount is counted by the data amount counter 402a.

  When the restart marker detection unit 404 detects a restart marker inserted into the compressed image data, the selector 401 selects the data amount counter 402b. Therefore, the compressed image data processed by the JPEG processing unit 108 is input to the data amount counter 402b via the selector 401, and the data amount is counted by the data amount counter 402a.

  Thereafter, every time a restart marker is detected, the selector 401 sequentially switches the data amount counter. That is, every time a restart marker is detected, the data amount is counted by another data amount counter. For example, taking the divided block line 1-1 as an example, the compressed image data of the compression unit is sent from the JPEG processing unit 108 to the compression units 1-1, 1-2, 1-3, 1-4, 2-1, 2-2,..., M-1, M-2, M-3, and M-4 are input in this order. Therefore, the data amount (compression unit line data amount) counted by the data amount counter 402a is compression unit 1-1 + compression unit 2-1 +,... + Compression unit M-1. Further, the data amount counted by the data amount counter 402b is a compression unit 1-2 + compression unit 2-2 +,... + Compression unit M-2. The data amount counted by the data amount counter 402c is compression unit 1-3 + compression unit 2-3 +,... + Compression unit M-3. The data amount counted by the data amount counter 402d is compression unit 1-4 + compression unit 2-4 +,... + Compression unit M-4.

  The restart marker counter 405 counts the number of restart markers, and when the count number matches the number of compression units constituting the divided block line, the reordering header generated by the header generation unit 403. The selector 406 is switched so as to select data. As a result, the rearrangement header data generated by the header generation unit 403 is output to the DMA 407 via the selector 406. As a result, the rearrangement header data is sequentially written into the frame memory 105 by the DMA 407.

  The process related to one divided block line is completed by the series of operations described above. The series of operations described above are performed for the subsequent divided block lines. In this way, the compressed image data is written into the frame memory 105 as shown in FIG.

  Next, details of rearrangement section 1092 will be described. FIG. 13 is a diagram illustrating a detailed configuration of the rearrangement unit 1092. As illustrated in FIG. 13, the rearrangement unit 1092 includes a DMA 501, a selector 502, write address generation units 503 a to 503 d, a restart marker detection unit 504, a restart marker change unit 505, and a compression unit line head address. A generation unit 506, a selector 507, a selector 508, and a DMA 509 are included.

  The DMA 501 reads rearrangement header data and compressed image data for each compression unit from the frame memory 105. The selector 502 outputs the reordering header data read out via the DMA 501 and the compressed image data for each compression unit, the write address generation units 503a to 503d, the restart marker detection unit 504, and the restart marker change unit 505. And selectively output to the compression unit line head address generation unit 506.

  The write address generation units 503a to 503d generate a write destination address when the compressed image data for each compression unit is written back to the frame memory 105, and output this write destination address together with the corresponding compressed image data to the selector 507.

  The restart marker detection unit 504 detects a restart marker inserted at the end of the compressed image data in the compression unit. The restart marker changing unit 505 changes the number of the restart marker according to the generation result of the write destination address of the write address generating units 503a to 503d.

  The compression unit line head address generation unit 506 generates a head address after rearrangement of the compression unit lines from the data amount of the compression unit lines recorded in the rearrangement header data.

  The selector 507 outputs the write destination address generated by any one of the write address generation units 503a to 503d to the DMA 509 each time the restart marker detection unit 504 detects a restart marker, and also writes the write address generation unit 503a. Data corresponding to the address generated in ˜503d is output to the DMA 509.

  The selector 508 generates the compressed image unit line head address input from the selector 507, the compressed image data whose restart marker number has been changed by the restart marker changing unit 505, and the compression unit line head address generating unit 506. The rearranged head address is selectively output to the DMA 509.

  The DMA 509 corresponds to the DMA memory for writing the rearranged head address generated by the compression unit line head address generator 506 to the frame memory 105, and the address of the frame memory generated by the write address generators 503a to 503d. DMA transfer for writing compressed image data for each compression unit (compressed image data output from the restart marker changing unit 505 and rewritten with the restart marker number) to the frame memory 105, and generated by write address generating units 503a to 503d The DMA transfer is performed to overwrite the frame memory 105 with the assigned address as the new rearranged start address.

  The operation of rearrangement section 1092 in FIG. 13 will be described with reference to the flowchart in FIG. First, the rear address of each compressed unit line after sorting is generated from the data amount of the compressed unit line recorded in the sorting header data (step S101).

  In step S101, first, the selectors 502 and 508 are set so as to select the compression unit line head address generation unit 506. Thereafter, the rearrangement header data written in the frame memory 105 is read out via the DMA 501. As a result, the reordering header data read via the DMA 501 is output to the compression unit line head address generation unit 506. The compression unit line head address generation unit 506 sequentially calculates the head address of the compression unit line using the rearrangement header data. After calculating the start address of the compression unit line, the calculated start address of the compression unit line is sequentially written into the frame memory 105 via the DMA 509.

Here, an example of a method for calculating the compression unit head address will be described. For the sake of explanation, the compression unit line start addresses 1, 2,..., V are named from the compression unit line start addresses corresponding to the compression unit lines at the upper end of FIG. First, the compression unit line head address 1 corresponding to the upper end of FIG. 8 is the head address of the compressed image data after rearrangement, and is set by the CPU 112 as a register. Further, the compression unit line head address V (V> 1) of the Vth row in FIG. 8 is calculated according to the following equation.
(Compression unit line head address V-1) + (total amount of compression unit line data in the Vth row)
For example, the compression unit line head address 2 is (compression unit line head address 1) + (compression unit line data amount 1 in the divided block line 1-1) + (compression unit line data amount 1 in the divided block line 2-1). + (Compression unit line data amount 1 in the divided block line 3-1). The compression unit line head address 3 is (compression unit line head address 2) + (compression unit line data amount 2 in the divided block line 1-1) + (compression unit line data amount 2 in the divided block line 2-1). + (Compression unit line data amount 2 in the divided block line 3-1).

  Next, the compression unit line head address stored in the frame memory 105 is read to initialize the write address generators 503a to 503d (step S102).

  In step S102, first, the selector 502 is set so as to sequentially select the write address generation units 503a to 503d. Thereafter, the compression unit head address corresponding to one block line (four lines in the example of FIG. 8) is read out via the DMA 501. Accordingly, the write address generation units 503a to 503d are initialized by overwriting the held values of the write address generation units 503a to 503d with the compression unit line head address read from the frame memory 105.

  Next, the compressed image data is read from the frame memory 105, and the read compressed image data is sorted according to the compression unit line head address held in the write address generation units 503a to 503d and written back to the frame memory 105 (step S103).

  In step S103, first, the selector 507 is set to select the write address generating unit 503a, and the selector 502 is set to select the restart marker detecting unit 504 and the restart marker changing unit 505. Thereafter, the compressed image data of the compression unit is sequentially read out via the DMA 501. The reading order of the compressed image data at this time is the storing order (compression unit 1-1, 1-2,...) In the frame memory 105. Thereby, the compressed image data of the compression unit 1-1 read out through the DMA 501 is sequentially written from the compression unit line head address 1 of the frame memory 105. When the writing of the compressed image data of the compression unit 1-1 is completed, the restart marker detecting unit 504 detects the restart marker. At this point, the selector 507 is set to select the write address generation unit 503b. Thereby, the compressed image data of the compression unit 1-2 read out via the DMA 501 is sequentially written from the compression unit line head address 2 of the frame memory 105. Thereafter, each time a restart marker is detected by the restart marker detection unit 504, the selector 507 is set so that the write address generation units 503c and 503d are sequentially selected. Thus, the compressed image data of the compression unit 1-3 is written from the compression unit line head address 3 of the frame memory 105, and the compressed image data of the compression unit 1-4 is written from the compression unit line head address 4 of the frame memory 105.

  Further, when writing the compressed image data to the frame memory 105, the restart marker changing unit 505 changes the number of the restart marker. This is for the restart markers to be consecutive numbers after the compressed image data is rearranged. Note that the numbering of restart markers is first performed in the compression units (compression units 1-1, 2-1, 3-1,..., H-1) corresponding to the first line in the rearranged compressed image data. The number is changed so that the Lister marker is a serial number, and the compression unit corresponding to the second line (compression unit 1-2, 2-2, 3-) is continuous with the last compression unit of the first line. Numbers are changed so that the restart markers of 2, ..., H-2) are serial numbers. The same applies to the third and subsequent lines.

  After the compressed image data of one divided block line is written back to the frame memory 105, the current values of the write address generators 503a to 503d (these values are used when the next divided block line is written back). Is output from the DMA 509, and the compression unit line start address already stored in the frame memory 105 is overwritten (step S104).

  Next, it is determined whether or not writing back of the compressed image data of all the divided block lines is completed (step S105). If it is determined in step S105 that writing back of the compressed image data of all the divided block lines has not been completed, the process returns to step S102, and the process for the next divided block line is performed. For example, after the process of the divided block line 1-1, the process for the next divided block line 1-2 is performed. At this time, the write address generation units 503a to 503d are initialized by the compression unit line head addresses 5 to 8, and the compressed image data is written back. In addition, after the process of the divided block line 1-2, the process for the next divided block line 1-3 is performed. At this time, the write address generation units 503a to 503d are initialized by the compression unit line head addresses 9 to 12, and the compressed image data is written back. After the process up to the lowermost divided block line is completed, the process is performed on the upper divided block line 2-1 in the right adjacent column. The compression unit line head addresses 1 to 4 read at this time are updated at the time of processing in the divided block line 1-1. For this reason, the divided block line 2-1 is neatly arranged right next to the divided block line 1-1. FIG. 15 shows the state of the compressed image data on the frame memory 105 until the rearrangement of the divided block lines 2-1 is completed. As shown in FIG. 15, the compressed image data of the divided block line 1-2 adjacent to the divided block line 1-1 in the column direction is compressed unit line head addresses 5 to 8 different from the divided block line 1-1. Are written sequentially. On the other hand, the compressed image data of the divided block line 2-1 adjacent to the divided block line 1-1 in the row direction is sequentially written so as to follow the divided block line 1-1.

  If it is determined in step S105 that the compressed image data of all the divided block lines has been written back, the processing in FIG. 14 ends.

  As described above, according to the present embodiment, it is not necessary to read out the image data of the margin portion when reading out the image data of each block line, and it is not necessary to overlap the image data of the margin portion. Further, by dividing one block line into a plurality of divided block lines and performing processing, it is possible to reduce the amount of image data of the margin portion necessary for image processing. As a result, it is possible to reduce the size of the margin storage buffer compared to performing processing in units of block lines.

  Further, by performing processing in units of divided block lines, the arrangement order of compressed image data obtained after compression becomes an order that is not suitable for reproduction. In this embodiment, the compressed image data can be recorded on the recording medium 111 in a state where appropriate reproduction can be performed by rearranging the compressed image data by the JPEG rearrangement unit 109.

  Here, in the example of FIG. 8 described above, processing is performed for each divided block line obtained by dividing one block line into three equal parts in the row direction. The number of divisions is not limited to three, and may be two or four or more.

  Further, as shown in FIG. 16, the image data for one frame may be further divided and processed in the column direction. In the above-described embodiment, rearrangement is performed after image processing and compression for one frame are completed. On the other hand, by performing the processing as shown in FIG. 16, it is possible to rearrange each column direction division region. Thereby, for example, the rearrangement of the column direction division region 1 is performed when the compression processing of the column direction division region 1 is completed, and the rearrangement of the column direction division region 2 is performed when the compression processing of the column direction division region 2 is completed. , And the like are possible. In this case, the capacity necessary for the frame memory 105 to store the compressed image data before rearrangement is large enough to store the larger compressed image data in the column direction divided area 1 or the column direction divided area 2. I just need it. For this reason, it is possible to reduce the capacity of the frame memory 105 rather than rearranging after processing for one frame is completed. It is also possible to reduce the time delay from the start of the compression process until the rearrangement is completed.

  In the example of FIG. 1, a margin storage buffer is provided for each of a plurality of image processing units in the image processing unit 106. On the other hand, as shown in FIG. 17, only one adhesive margin storage buffer 106b having a total capacity of at least the margin storage buffers 1061b, 1062b,..., 106nb is provided. Data transfer with the processing unit may be performed via the bus 1063. In general, it is known that it is more area efficient to arrange one large-capacity memory than a plurality of small-capacity memories. Therefore, the area of the margin storage buffer can be reduced by using the modification.

  Here, the memory that can be used as the margin storage buffer 106b is not particularly limited as long as it is a highly integrated memory. Examples of the memory that can be used for the margin storage buffer 106b include eDRAM (Embedded DRAM), MRAM (Magnetoresistive RAM), FeRAM (Ferroelectric RAM), ReRAM (Resistance RAM), PRAM (Phase change RAM), and 1T-SRAM. (One Transistor SRAM), Z-RAM (Zero Capacitor RAM), TTRAM (Twin Transistor RAM), etc. can be considered.

  In the above-described embodiment, the spatial filter process is cited as an example of the image processing performed by the image processing unit 106. However, the technique of the above-described embodiment is also applied to the case where the distortion correction process is performed. Is possible. As shown in FIG. 18A, the distortion correction process is a process of correcting image data with distortion such as a barrel shape into image data without distortion. Even when such distortion correction processing is performed, image data of the margin portion as described above is required.

  Hereinafter, input image data necessary for performing distortion correction processing on the block lines 1 to 3 shown in FIG.

  When the distortion correction processing is performed on the block line 1, at least the block line 1 in a distorted state may be read and processed. However, the hardware configuration can be simplified if the processing is performed on rectangular data. For this reason, normally, rectangular data including a distorted block line 1 as indicated by reference numeral 601 in FIG. 18B is read and processed. Similarly, when processing the block line 2, rectangular data including the distorted block line 2 as indicated by reference numeral 602 in FIG. 10C is read and processed. Further, when processing the block line 3, rectangular data including the distorted block line 3 as indicated by reference numeral 603 in FIG. 18D is read and processed.

  When the image data is read out for each rectangular block line as described above, the block lines 1, 2, and 3 perform the overlapping process as shown in FIGS. 18C and 18D on the input image data. Necessary parts will be generated. In the same manner as described in the above-described embodiments, the image data corresponding to the overlapping portion is stored in the margin storage buffer as the margin portion image data, thereby eliminating the need for the overlapping process. Furthermore, it is possible to reduce the capacity of the margin storage buffer by performing processing for each divided block line obtained by dividing one block line in the row direction.

  Note that the distortion correction processing shown in FIG. 18 shows distortion correction processing when image data is distorted in a barrel shape. However, the present embodiment can be applied even when the image data is distorted in a pincushion shape, and can also be applied to chromatic aberration correction (distortion status differs for each color).

  Although the present invention has been described above based on the embodiments, the present invention is not limited to the above-described embodiments, and various modifications and applications are naturally possible within the scope of the gist of the present invention.

  Further, the above-described embodiments include various stages of the invention, and various inventions can be extracted by appropriately combining a plurality of disclosed constituent elements. For example, even if some configuration requirements are deleted from all the configuration requirements shown in the embodiment, the above-described problem can be solved, and this configuration requirement is deleted when the above-described effects can be obtained. The configuration can also be extracted as an invention.

  DESCRIPTION OF SYMBOLS 101 ... Lens, 102 ... Image sensor, 103 ... Pre-processing part, 104,1063 ... Bus, 105 ... Frame memory, 106 ... Image processing part, 107 ... JPEG interface (I / F), 108 ... JPEG processing part, 109 ... JPEG rearrangement unit, 110 ... media interface (I / F), 111 ... recording medium, 112 ... CPU, 201 ... input buffer, 202 ... image data serial / parallel conversion unit, 203 ... glue margin data / serial conversion unit, 203 ... glue margin Saved data serial conversion unit, 204 ... product-sum operation unit, 205 ... paste margin storage memory, 206 ... image data switching unit, 1061a, 1062a, 106na ... image processing unit, 1061b, 1062b, 106nb ... paste margin storage buffer

Claims (11)

An image processing apparatus that performs a plurality of image processing on image data captured and stored in a frame memory,
Image data of divided block lines obtained by dividing image data of a plurality of block lines each having a first number of pixels in the column direction and a second number of pixels in the row direction into a plurality of regions in the row direction A plurality of stages of image processing units that sequentially perform image processing on the image data of the divided block lines input in the column order;
Among the image data of the current divided block line provided corresponding to each of the plurality of stages of image processing units and input to each of the plurality of stages of image processing units, the next of the current divided block line A margin buffer for storing image data of a margin portion in the column direction which is also used in image processing for image data of divided block lines;
Comprising
In the image processing for the next divided block line image data, each of the plurality of stages of image processing units converts the image data of the next divided block line and the image data of the margin portion into combined image data. An image processing apparatus that performs image processing on the image processing apparatus.   An image compression processing unit that is connected to the final image processing unit in the multiple image processing unit and that compresses the image data for each divided block line input from the final image processing unit for each compression unit; The image processing apparatus according to claim 1, further comprising:   The reordering unit according to claim 2, further comprising a reordering unit that reorders the compressed image data for each compression unit obtained by processing by the image compression processing unit so as to be written in the frame memory in line order. Image processing apparatus. The image compression processing unit adds identification data to the compressed image data obtained by the compression process every time the image data of the compression unit is compressed,
The rearrangement unit generates the write address so that the compressed image data for each compression unit is written in the frame memory in the row order by counting the number of identification data, and performs the rearrangement. The image processing apparatus according to claim 3. The image data of each block line is further divided into a plurality of regions in the column direction,
The image processing apparatus according to claim 3, wherein the rearrangement unit performs the rearrangement for each region divided in the column direction.   The image processing apparatus according to claim 1, wherein one adhesive margin storage buffer is provided corresponding to each of the plurality of stages of image processing units.   The image processing apparatus according to claim 1, wherein only one common margin storage buffer is provided corresponding to each of the plurality of stages of image processing units. .   8. The image processing apparatus according to claim 7, wherein the margin storage buffer has at least a total capacity of image data of margin portions used in each of the plurality of stages of image processing units.   The image processing apparatus according to claim 2, wherein the first number of pixels is equal to the number of pixels in the column direction of the compression unit. The first pixel number is equal to an integer multiple of the number of pixels in the column direction of the compression unit,
3. The first stage image processing unit in the plurality of stages of image processing units reads image data of the divided block lines from the frame memory so that the compression units are input in a column order. The image processing apparatus described.   The image processing apparatus according to claim 2, wherein the image compression processing unit performs JPEG compression processing.

Images