JP6440465B2 - Image processing apparatus, image processing method, and program - Google Patents

Description

  The present invention particularly relates to an image processing apparatus, an image processing method, and a program suitable for use in correcting distortion caused by optical factors by signal processing.

  In recent years, in an imaging apparatus such as a digital camera or a video camera, distortion due to an optical factor occurs in a captured image due to the influence of distortion characteristics of an imaging lens. In an expensive high-performance lens, since the influence of distortion characteristics is small, such distortion is not conspicuous. However, when a low-cost lens or an optical zoom lens is used, distortion becomes conspicuous and image quality deteriorates. Thus, in recent years, for example, Patent Documents 1 and 2 have proposed image pickup apparatuses that correct distortion due to optical factors by signal processing.

  Patent Document 1 discloses a technique including a distortion correction range calculation unit that calculates a pre-distortion (input) image range necessary for the distortion correction processing unit to perform distortion correction processing. With this configuration, a corrected image (output image) obtained by distortion correction can be output without excess or deficiency with respect to the image output range. Then, it has been proposed to perform distortion correction processing that effectively uses the input image data that is the original data, and to perform effective distortion correction processing for pincushion distortion, barrel distortion, Jinkasa distortion, and the like. .

  Further, in the method described in Patent Document 2, the coordinates before correction corresponding to the corrected image coordinate system are stored, the coordinates before correction corresponding to each pixel of the corrected image are read from the storage unit, and the coordinates are output. Then, the pre-correction image data corresponding to the output pre-correction coordinates is output as pixel data of the post-correction image. Here, when the coordinates before correction output from the coordinate output unit include a numerical value after the decimal point, it corresponds to the coordinates before correction by calculating a weighted average based on data of a plurality of pixels in the vicinity of the coordinates before correction. The pixel data of the corrected image to be output is output.

JP 2005-44098 A International Publication No. 2008/139579

  However, in the technique disclosed in Patent Document 1, since the calculation of which position of the pre-correction image pixel is taken based on the coordinates of the post-correction image, the input unit and the distortion correction calculation unit Both require coordinates before distortion correction. For this reason, two coordinate arithmetic circuits are required, and the circuit scale and power consumption increase.

  Further, since the distortion rate varies depending on the optical zoom magnification, the technique disclosed in Patent Document 2 stores the pre-correction coordinates corresponding to the post-correction image coordinate system, and thus the correction according to the zoom magnification. It is necessary to memorize the previous coordinates. As a result, the storage capacity may increase. Therefore, Patent Document 2 discloses a technique for holding the coordinates of only the vertices divided into rectangular blocks. In this case, although it can deal with pincushion distortion and barrel distortion, It can be difficult.

  Furthermore, the pre-correction image is stored in the block line buffer, but the block line buffer requires the vertical line for the maximum coordinate movement between the pre-correction image and the post-correction image. Is concerned. For example, when the block line buffer is an SRAM, the number of horizontal pixels has increased with the recent increase in the number of pixels, and there is a problem that the storage capacity for dealing with an enormous amount of data is insufficient. If the block line buffer is a DRAM, the storage capacity is sufficient, but if the coordinates before correction are below the decimal point, the data of a plurality of neighboring pixels is read out from the DRAM for interpolation. There is a problem that access occurs and performance decreases.

  The present invention has been made in view of the above-described problems, and it is an object of the present invention to suppress the storage capacity, the circuit scale, and the power consumption and perform the distortion correction with high performance.

An image processing apparatus according to the present invention is an image processing apparatus that corrects distortion aberration of the optical system in image data obtained by imaging an object using an optical system, and stores the image data as a pre-correction image. Storage means; coordinate generation means for generating coordinates of the corrected image in which the distortion is corrected; and coordinates of the uncorrected image corresponding to the coordinates of the corrected image generated by the coordinate generation means. Based on a part of the coordinates of the rectangular area in the pre-correction image calculated by the calculating means, a reading control means for reading out the pre-correction image for each rectangular area from the first storage means, A second storage means for temporarily storing the pre-correction image read by the read control means for each rectangular area; and the pre-correction image calculated by the calculation means. Image processing means for reading out the pre-correction image from the second storage means and generating the post-correction image based on the coordinates in the rectangular area, wherein the coordinate generation means is the first storage means. Generating coordinates for the calculation means to calculate the coordinates for reading the pre-correction image from, and generating coordinates for the calculation means to calculate the coordinates for reading the pre-correction image from the second storage means. The calculation means calculates coordinates for reading the uncorrected image from the first storage means or coordinates for reading the uncorrected image from the second storage means , and the calculation means further comprises: Determining means for determining whether or not the coordinates of the corrected image generated by the coordinate generating means are coordinates of a part of the rectangular area in the corrected image; and determining one of the rectangular areas by the determining means. And holding means for temporarily holding the coordinates of a part of the rectangular area in the pre-correction image corresponding to the rectangular area in the post-correction image. Is configured to read the pre-correction image from the second storage unit based on the coordinates held in the holding unit and generate the post-correction image .

  According to the present invention, distortion correction can be performed with high performance while suppressing storage capacity, circuit scale, and power consumption.

It is a block diagram which shows the detailed structural example of the aberration correction part which concerns on embodiment. It is a block diagram which shows the structural example of the digital camera which concerns on embodiment. It is a block diagram which shows the detailed structural example of the coordinate calculation part before correction | amendment in 1st Embodiment. It is the figure which showed the center of the image, the pixel position of the image before correction | amendment, and the pixel position of the image after correction | amendment. It is the figure which showed typically the input / output relationship in a rectangular image holding part. In a 1st embodiment, it is a figure for explaining the processing situation of a coordinate operation part, and the timing which writes a rectangular picture into a rectangular picture holding part. It is a figure which shows the image before correction | amendment and the image after correction | amendment which the barrel distortion produced by the aberration. It is a flowchart which shows an example of the calculation processing procedure of the coordinate before correction | amendment of 1 rectangular image by an aberration correction part. It is the figure which showed the rectangular division | segmentation of the image before correction | amendment which the pincushion produced by the aberration, and the image after correction | amendment. It is a block diagram which shows the detailed structural example of the coordinate calculation part before correction | amendment in 2nd Embodiment. In 2nd Embodiment, it is a figure for demonstrating the processing condition of a coordinate calculating part, and the timing which writes a rectangular image in a rectangular image holding | maintenance part.

(First embodiment)
Hereinafter, a first embodiment of the present invention will be described with reference to the drawings. In this embodiment, a digital camera will be described as an example of an image processing device. However, an imaging device such as a digital video camera, a mobile phone with a camera, an in-vehicle camera, or a device that requires aberration correction, such as a projector, may be employed. It is.

FIG. 2 is a block diagram illustrating a configuration example of the digital camera 10 according to the present embodiment.
In FIG. 2, an image sensor 200 is a light receiving element such as a CCD or CMOS sensor that converts a received subject image into an electrical signal to create image data. The A / D converter 201 converts the analog output signal of the image sensor 200 into a digital signal.

  The image processing unit 202 includes an imaging / development processing unit 203, an aberration correction unit 100, and a DMAC 110. Furthermore, an image buffer memory, a compression / decompression unit that performs compression processing and decompression processing of image data, a display unit that displays on a monitor, a recording processing unit that records image data on a recording medium (all not shown), and the like are provided. Yes.

  The imaging / development processing unit 203 appropriately performs black level correction, shading correction, and the like on the image data converted by the A / D converter 201. Further, the imaging / development processing unit 203 performs the following processing on the corrected image data and the image data stored in the image buffer memory or the memory 107. Specifically, resize processing such as pixel interpolation, filter processing, and reduction, and color conversion processing, for example, development processing such as processing for conversion to a YCbCr format that is an optimal format for saving in compressed image data are appropriately performed. .

  The aberration correction unit 100 corrects distortion aberration for the image data. Details of this processing will be described later. The DMAC 110 is a Direct Memory Access controller that performs data transfer. The image data is output to the bus 105 by the DMAC 110 and temporarily stored in the memory 107 via the memory control unit 106. The image data temporarily stored in the memory 107 is output from the memory 107 to the bus 105 by the DMAC 110, and the image data is output to the image processing unit 202 via the DMAC 110. The DMAC 110 has a plurality of DMA channels in order to perform data transfer for a plurality of image processes.

  The bus 105 is a bus that serves both as a system bus and an image data bus. However, the system bus and the image data bus may be configured as different buses. The memory control unit 106 writes data into the memory 107 or reads data from the memory 107 in accordance with an instruction from the system control unit 204 or the DMAC 110. Note that output data from the A / D converter 201 may be directly written into the memory 107. The memory 107 is a storage device having a storage capacity sufficient to store a predetermined number of still images, data for a predetermined time, data such as sound, constants for operating the system control unit 204, programs, and the like. For example, a DRAM or the like is used.

  The nonvolatile memory control unit 206 writes data into the nonvolatile memory 207 or reads data from the nonvolatile memory 207 in accordance with an instruction from the system control unit 204. The nonvolatile memory 207 is an electrically erasable / recordable memory, and for example, an EEPROM or the like is used. In addition, the nonvolatile memory 207 stores constants, programs, and the like for operating the system control unit 204.

  The system control unit 204 includes a microcomputer that controls the operation of the digital camera 10, and gives various instructions to each functional block constituting the digital camera 10 to execute various control processes. Further, the system control unit 204 controls the image processing unit 202, the memory control unit 106, the nonvolatile memory control unit 206, the operation unit 205, and the image sensor 200 connected via the bus 105. The microcomputer realizes each process of the present embodiment by executing the program recorded in the nonvolatile memory 207 described above.

  The operation unit 205 includes switches and buttons operated by the user, and is used for operations such as power ON / OFF and shutter ON / OFF.

FIG. 1 is a block diagram illustrating a detailed configuration example of the aberration correction unit 100 according to the present embodiment. In addition, about the structure same as FIG. 2, the same code | symbol is attached | subjected and detailed description is abbreviate | omitted.
In FIG. 1, a post-correction coordinate generation unit 111 includes a first post-correction coordinate generation unit 101 and a second post-correction coordinate generation unit 102, and generates the coordinates of the post-correction image.

  The first corrected coordinate generation unit 101 generates the coordinates of the corrected image. The coordinates of the post-correction image are used to cause the pre-correction coordinate calculation unit 103 to calculate coordinates for reading out the pre-correction image from the memory 107 and writing it to the rectangular image holding unit 108. Specifically, the corrected image is divided into rectangles, and coordinates for each peripheral pixel in the rectangle are generated as some coordinates in the rectangular area. Furthermore, the first post-correction coordinate generation unit 101 can adjust the rectangular size according to the aberration correction rate under the control of the system control unit 204. The rectangular size may be the same size or different sizes for one image. The first post-correction coordinate generation unit 101 includes a buffer that holds coordinate information for each peripheral pixel in the rectangle.

  The second corrected coordinate generation unit 102 generates the coordinates of the corrected image. The coordinates of the post-correction image are used to cause the pre-correction coordinate calculation unit 103 to calculate the coordinates read by the pixel interpolation unit 109 in units of pixels when reading the pre-correction image held in the rectangular image holding unit 108. . Specifically, as in the first corrected coordinate generation unit 101, the corrected image is divided into rectangles, and the coordinates of all the pixels in the rectangle are generated. Note that the coordinates and rectangular division of the pre-correction image and the post-correction image will be described later with reference to FIGS.

  The pre-correction coordinate calculation unit 103 calculates the pre-correction coordinates. Details of the processing will be described later with reference to FIG. The rectangular read control unit 104 receives the calculation result of the pre-correction coordinate calculation unit 103 and generates a read address of the memory 107. Although the details will be described later, the rectangular image holding unit 108 is configured by, for example, an SRAM or the like because writing and reading are frequently performed, and an image before correction is written for each rectangle and temporarily held. In this embodiment, a 2-bank configuration is used, but the number of banks may be other than 2.

  Based on the coordinates calculated by the pre-correction coordinate calculation unit 103, the pixel interpolation unit 109 corrects a pixel to be corrected in a rectangular pre-correction image (rectangular image) held in the rectangular image holding unit 108, and a plurality of correction target pixels. Read out the surrounding pixels in the rectangle. Then, using the coordinate value of the pre-correction coordinate calculation unit 103, the pixel is corrected by bilinear interpolation or bicubic interpolation to obtain a corrected image.

  Here, with reference to FIG. 7 and FIG. 9, the coordinates and rectangular division of the pre-correction image and the post-correction image will be described. FIG. 7 is a diagram illustrating an uncorrected image 700 and an after-correction image 701 in which barrel distortion has occurred due to aberration. Here, the coordinates of the corrected image 701 are an example of the coordinates of the corrected image generated by the first corrected coordinate generation unit 101. The coordinates before correction corresponding to the coordinates (X, Y) of the corrected image 701 are the coordinates (x, y) of the image 700 before correction. The rectangular images 703, 707, and 709 indicate one rectangle when the corrected image is divided into rectangles. The rectangular images of the images before correction corresponding to the rectangular images 703, 707, and 709 are rectangular images 702, 706, and 708, respectively. is there.

  The enlarged images 704 and 705 are enlarged images of the corrected rectangular image 703. In the enlarged image 704, the coordinates generated by the first corrected coordinate generation unit 101 are indicated by diagonal lines, and the first corrected coordinate generation unit 101 generates only the coordinates of the peripheral pixels in the rectangular image 703. The enlarged image 705 indicates that the coordinates generated by the second corrected coordinate generation unit 102 are hatched, and indicates that the second corrected coordinate generation unit 102 generates all the coordinates in the rectangular image 703. Yes.

  FIG. 9 is a diagram illustrating rectangular division of the pre-correction image 903 and the post-correction image 900 in which pincushion is generated due to aberration. In the example shown in FIG. 9A, the corrected rectangular image 901 is a first rectangular image of the corrected image 900, and is divided into rectangles. The corrected rectangular image 902 is a second rectangular image of the corrected image 900, and similarly indicates that the rectangle is divided into rectangles.

  A pre-correction rectangular image 904 indicates a rectangular size held in the rectangular image holding unit 108. Among these, the rectangular image 906 is the first rectangular image read from the memory 107, and the shaded portion is an effective pixel used for correction, and a blank indicates an invalid pixel that is not used. Further, the rectangular image 907 is not held in the rectangular image holding unit 108 but becomes an unused area.

  A pre-correction rectangular image 905 indicates a rectangular size held in the rectangular image holding unit 108. The pre-correction rectangular image 905 is a second rectangular image corresponding to the post-correction rectangular image 902 that is further read from the memory 107, and the hatched portion is an effective pixel used for correction, and a blank indicates an invalid pixel that is not used.

  On the other hand, in the example shown in FIG. 9B, the corrected rectangular image 908 is a first rectangular image of the corrected image 900, and is divided into squares. A corrected rectangular image 909 is a second rectangular image of the corrected image 900, which is divided into squares.

  The pre-correction rectangular image 910 indicates a rectangular size held in the rectangular image holding unit 108. The pre-correction rectangular image 910 is a first rectangular image corresponding to the post-correction rectangular image 908 that is read from the memory 107, and the hatched portion is an effective pixel used for correction, and a blank indicates an invalid pixel that is not used.

  The pre-correction rectangular image 911 indicates the rectangular size held in the rectangular image holding unit 108. The pre-correction rectangular image 911 is a second rectangular image corresponding to the post-correction rectangular image 909, which is further read from the memory 107, and the hatched portion is an effective pixel used for correction, and a blank indicates an invalid pixel that is not used.

  As shown in FIGS. 9A and 9B, when reading from the memory 107 in a shape close to a square, the rectangular image holding unit 108 can be used effectively because the rectangle includes many effective pixels. Further, if the number of coordinates for holding the peripheral coordinates in the rectangle of the peripheral coordinate holding unit 304 to be described later is the same as the area (storage area), the square is smaller than the rectangle. Therefore, in the present embodiment, the first corrected coordinate generation unit 101 reads a rectangular image from the memory 107 in a shape close to a square.

FIG. 3 is a block diagram illustrating a detailed configuration example of the pre-correction coordinate calculation unit 103 in the present embodiment. In addition, about the structure same as FIG.1 and FIG.2, the same code | symbol is attached | subjected and detailed description is abbreviate | omitted.
In FIG. 3, the rectangular periphery determination unit 301 determines whether the given coordinates are peripheral coordinates in the rectangle based on the coordinate information input from the second corrected coordinate generation unit 102. Here, the condition for determining the peripheral coordinates in the rectangle is a case where no coordinates are given from the second corrected coordinate generation unit 102 and there is no pre-correction image (rectangular image) in both 2 Banks of the rectangular image holding unit 108. . The determination result is used as a selection condition for the first selector 302 and the second selector 300. Further, along with the determination, the coordinate information input from the second corrected coordinate generation unit 102 is output to the first selector 302.

  The first selector 302 switches according to the determination result of whether or not the rectangle periphery determination unit 301 determines the peripheral coordinates in the rectangle. Specifically, in the case of peripheral coordinates, the coordinate information input from the first corrected coordinate generation unit 101 is output to the coordinate calculation unit 303, and in the case of not peripheral coordinates, the coordinates input from the rectangular periphery determination unit 301 are output. Information is output to the coordinate calculation unit 303.

  The coordinate calculation unit 303 calculates the pixel position of the uncorrected image with respect to the input coordinates by a general aberration correction method. Details of the calculation method of the coordinate calculation unit 303 will be described later. The peripheral coordinate holding unit 304 is configured by, for example, an SRAM or the like, and holds the output result of the coordinate calculation unit 303 as peripheral coordinates when the rectangular peripheral determination unit 301 determines that the peripheral coordinates are peripheral coordinates. Then, the peripheral coordinates held by the peripheral coordinate holding unit 304 are output to the second selector 300 as coordinate information.

  The second selector 300 outputs the coordinate information output from the peripheral coordinate storage unit 304 to the rectangular image storage unit 108 when the rectangular peripheral determination unit 301 determines that the pixel is a peripheral pixel. When the rectangular periphery determination unit 301 determines that the coordinates are not peripheral coordinates, the output result of the coordinate calculation unit 303 is output to the rectangular image holding unit 108.

  Next, an example of processing of the coordinate calculation unit 303 will be described with reference to FIG. Other methods may be adopted as long as aberration correction is possible. In general, when attention is paid to the pixel of the pre-correction image, the image to be originally formed on the target pixel is moved and formed on a line connecting the target pixel and the center of the image by each aberration.

  FIG. 4 is a diagram showing the center 400 of the image, the pixel position 402 of the image before correction, and the pixel position 401 of the image after correction. In FIG. 4, the output pixels of the image sensor 200 are represented by a two-dimensional orthogonal coordinate system image that defines a horizontal axis along the scanning direction and defines a vertical axis perpendicular to the scanning direction.

  It is generally known that the correction amount C for correcting the pixel position 402 of the uncorrected image is obtained from the image height R representing the distance from the center of the image and the characteristics of the optical system. It is possible to calculate the pixel position 402 of the uncorrected image by performing coordinate conversion from the angle θ between the horizontal direction of the center 400 of the image and the direction of the pixel position 401 of the corrected image from the center 400 of the image and the correction amount C. it can. In addition, the pixel position 402 of the image before correction is not necessarily a pixel unit of the image sensor, and may be moved between pixels of the image sensor, and the decimal point is calculated.

  Next, the control of the present embodiment will be described with reference to FIGS. FIG. 5 is a diagram schematically showing an input / output relationship in the rectangular image holding unit 108. In FIG. 5, Banks 500 and 501 are buffers formed of RAM or the like.

  First, as shown in FIG. 5A, the first rectangular image is written from the memory 107 to the Bank 500 and held. Subsequently, as shown in FIG. 5B, when the second rectangular image is written from the memory 107 to the Bank 501, the second rectangular image is retained, and the first rectangular image is read from the Bank 500 by the pixel interpolation unit 109 in units of pixels. It is. Further, as shown in FIG. 5C, when the third rectangular image is written from the memory 107 to the Bank 500, the third rectangular image is retained, and the second rectangular image is read from the Bank 501 by the pixel interpolation unit 109 in units of pixels. . Thereafter, the state shown in FIGS. 5B and 5C is repeated until one image is processed.

FIG. 6 is a diagram for explaining the processing status of the coordinate calculation unit 303 and the timing of writing a rectangular image to the rectangular image holding unit 108 in the present embodiment.
In FIG. 6, a timing chart 610 indicates coordinate calculation for reading an uncorrected image based on the coordinate information input from the first corrected coordinate generation unit 101 by the coordinate calculation unit 303 as High or Low. Indicates the calculation period.

  The timing chart 611 indicates coordinate calculation for reading an uncorrected image by High or Low based on the coordinate information input from the rectangular periphery determination unit 301 by the coordinate calculation unit 303, and High indicates a calculation period. . Further, the timing chart 612 shows the timing of writing the pre-correction image in the rectangular image holding unit 108, and High shows the writing period.

  In periods 600 and 601, the coordinate calculation unit 303 applies the coordinates generated by the first corrected coordinate generation unit 101 in order to read the first rectangular image and the second rectangular image of the pre-correction image from the memory 107. This is the period during which the calculation is performed. That is, the period 600 indicates the first rectangular coordinate calculation period, and the period 601 indicates the second rectangular coordinate calculation period.

  A period 602 indicates a period in which the first rectangular image of the pre-correction image is written in the Bank 500 of the rectangular image holding unit 108, and represents the state of FIG. A period 603 indicates a period during which the second rectangular image of the pre-correction image is written in the bank 501 of the rectangular image holding unit 108, and represents the state of FIG.

  In addition, the period 603 is also a period during which the coordinate calculation unit 303 performs calculation so that the pixel interpolation unit 109 reads the first rectangular image in units of pixels as illustrated in the timing chart 611. Further, as shown in the timing chart 610, in the period 603, the coordinate calculation unit 303 performs calculation in order to write the third rectangular image from the memory 107 to the Bank 500 of the rectangular image holding unit 108. Note that the coordinate calculation unit 303 performs either one of the processes, so that the timing chart 610 and the timing chart 611 do not become High at the same time. Hereinafter, the process switching will be described.

  A period 604 is a period during which the coordinate calculation unit 303 calculates to write the third rectangular image from the memory 107 to the Bank 500 of the rectangular image holding unit 108. Further, a period 604 indicates a period in which the peripheral coordinates of the first rectangular image are output from the peripheral coordinate holding unit 304 and the peripheral coordinates of the third rectangular image are held when the peripheral coordinates are output.

  A period 605 is a period during which the coordinate calculation unit 303 performs calculation so that the pixel interpolation unit 109 reads the first rectangular image from the rectangular image holding unit 108 in units of pixels. That is, after the period 603, during the High period of the timing chart 610, the coordinate calculation unit 303 performs calculations, the peripheral coordinates are held in the peripheral coordinate holding unit 304, and the peripheral coordinates of the previous rectangular image are output. Is shown. Note that a high period of the timing chart 610 in the periods 600 and 601 is a period in which the calculation is performed by the coordinate calculation unit 303 and the peripheral coordinates are held in the peripheral coordinate holding unit 304.

  A period 606 indicates a period in which the third rectangular image of the pre-correction image is written in the Bank 500 of the rectangular image holding unit 108, and shows the state of FIG. In the example illustrated in FIG. 6, the writing timing of the rectangular image holding unit 108 to the Bank 500 and the timing at which the pixel interpolation unit 109 reads the rectangular image in units of pixels are the same. However, there may be a case where writing to the Bank 500 of the rectangular image holding unit 108 ends early, or a timing when the pixel interpolation unit 109 ends reading out the rectangular image in units of pixels. In this case, control is performed so that writing and reading do not occur in the same Bank.

FIG. 8 is a flowchart illustrating an example of a calculation process procedure of the coordinates before correction of one rectangular image by the aberration correction unit 100.
First, in step S800, the first post-correction coordinate generation unit 101 generates and holds coordinate information for each peripheral pixel in the rectangle of the post-correction image. For example, in the case of the third rectangular image, the period held in the first corrected coordinate generation unit 101 is held in the period 602 in FIG.

  Subsequently, in step S <b> 801, the pre-correction coordinate calculation unit 103 determines whether there is data in at least one Bank of the rectangular image holding unit 108. As a result of the determination, if there is no data in at least one Bank, the process proceeds to Step S804, and if any Bank has data, the process proceeds to Step S802.

  In step S <b> 802, the pre-correction coordinate calculation unit 103 waits until coordinate information is input from the second post-correction coordinate generation unit 102. When coordinate information is input from the second corrected coordinate generation unit 102, the process proceeds to the next step S803.

  Next, in step S803, the pre-correction coordinate calculation unit 103 determines whether or not the input coordinates are peripheral coordinates in a rectangle. As a result of the determination, if the coordinates are peripheral coordinates, the process proceeds to step S804. If the coordinates are not peripheral coordinates, the process proceeds to step S806. Note that the processes in steps S801 to S803 are performed by the rectangular periphery determination unit 301 of the pre-correction coordinate calculation unit 103.

  In step S804, the coordinate calculation unit 303 inputs coordinate information from the first corrected coordinate generation unit 101 and performs coordinate calculation. In step S805, the coordinate calculation unit 303 outputs the calculation result to the rectangular readout control unit 104, and outputs and holds the calculation result in the peripheral coordinate holding unit 304. At this time, the pre-correction coordinate calculation unit 103 outputs the peripheral coordinates held in the peripheral coordinate holding unit 304 to the rectangular image holding unit 108.

  On the other hand, in step S806, the coordinate calculation unit 303 inputs coordinate information from the rectangular periphery determination unit 301, and the coordinate calculation unit 303 performs coordinate calculation. In step S <b> 807, the coordinate calculation unit 303 outputs the calculation result to the rectangular image holding unit 108.

  In the next step S808, it is determined whether or not all the calculations for the coordinates in the rectangle have been completed. As a result of the determination, if all the calculations are not completed, the process returns to step S801. On the other hand, if all the calculations are completed as a result of the determination in step S808, the calculation of the pre-correction coordinates of one rectangular image is ended.

  As described above, according to the present embodiment, the calculation of the coordinates when reading the pre-correction image from the memory 107 and the calculation of the coordinates when reading the pre-correction image from the rectangular image holding unit 108 are performed as one pre-correction. It can be executed by a coordinate calculation unit (circuit). Thereby, a circuit scale and power consumption can be reduced.

  In addition, the first post-correction coordinate generation unit 101, the second post-correction coordinate generation unit 102, the rectangular image holding unit 108, the rectangular periphery determination unit 301, the first selector 302, the peripheral coordinate storage unit 304, and the second selector 300 Thus, random access to the memory 107 is reduced. This makes it possible to maintain the same performance as when two pre-correction coordinate calculation units are mounted.

(Second Embodiment)
Next, a second embodiment of the present invention will be described. In the present embodiment, only the parts different from the first embodiment described above will be described, and the same parts will be denoted by the same reference numerals and detailed description thereof will be omitted.

FIG. 10 is a block diagram illustrating a detailed configuration example of the pre-correction coordinate calculation unit 103 in the present embodiment.
As shown in FIG. 10, the pre-correction coordinate calculation unit 103 according to the present embodiment includes an input determination unit 1000 and a coordinate calculation unit 303.

  The input determination unit 1000 determines whether the coordinate information is output from either the first post-correction coordinate generation unit 101 or the second post-correction coordinate generation unit 102, and uses the previously input coordinate information as the coordinate calculation unit 303. Output to. When coordinates are input from the first corrected coordinate generation unit 101 and the second corrected coordinate generation unit 102 substantially simultaneously, the coordinate information output from the first corrected coordinate generation unit 101 is given priority. The result is output to the calculation unit 303.

  Further, the second post-correction coordinate generation unit 102 outputs the same coordinates until the pre-correction coordinates corresponding to the coordinates input to the pre-correction coordinate calculation unit 103 are output. By outputting the same coordinates until calculated by the pre-correction coordinate calculation unit 103 in this way, it is possible to calculate all the coordinates of the first post-correction coordinate generation unit 101 and the second post-correction coordinate generation unit 102. It becomes.

FIG. 11 is a diagram for explaining the processing status of the coordinate calculation unit 303 and the timing of writing a rectangular image to the rectangular image holding unit 108 in the present embodiment.
In FIG. 11, a timing chart 1110 indicates coordinate calculation for reading an uncorrected image based on the coordinate information input from the first corrected coordinate generation unit 101 by the coordinate calculation unit 303 as High or Low. Indicates the calculation period.

  The timing chart 1111 indicates coordinate calculation for reading an uncorrected image by High or Low based on the coordinate information input from the second corrected coordinate generation unit 102 by the coordinate calculation unit 303, and High indicates the calculation period. Show. Further, a timing chart 1112 shows a timing chart for writing the pre-correction image in the rectangular image holding unit 108, and High shows a writing period.

  In periods 1100 and 1101, the coordinate calculation unit 303 calculates the coordinates of the first corrected coordinate generation unit 101 in order to read the first rectangular image and the second rectangular image of the image before correction from the memory 107, respectively. It is a period. That is, the period 1100 indicates the first rectangular coordinate calculation period, and the period 1101 indicates the second rectangular coordinate calculation period.

  A period 1102 indicates a period in which the first rectangular image of the pre-correction image is written in the Bank 500 of the rectangular image holding unit 108, and represents the state of FIG. A period 1103 indicates a period in which the second rectangular image of the pre-correction image is written in the bank 501 of the rectangular image holding unit 108, and represents the state of FIG.

  Here, a period 1105 in the period 1103 indicates a period during which the coordinate calculation unit 303 calculates the pixel interpolation unit 109 to read out the first rectangular image in units of pixels as illustrated in the timing chart 1111. Yes. Further, a period 1104 in the period 1103 is a period calculated by the coordinate calculation unit 303 in order to write the third rectangular image from the memory 107 to the Bank 500 of the rectangular image holding unit 108 as shown in the timing chart 1110. Show. Note that the coordinate calculation unit 303 performs either one of the processes, so the timing chart 1110 and the timing chart 1111 do not become High at the same time.

  A period 1106 is a period during which the third rectangular image of the pre-correction image is written in the Bank 500 of the rectangular image holding unit 108, and represents the state of FIG. Here, a period 1108 in the period 1106 indicates a period during which the coordinate calculation unit 303 calculates the pixel interpolation unit 109 so as to read out the second rectangular image in units of pixels, as shown in the timing chart 1111. Yes. Further, a period 1107 in the period 1106 is a period calculated by the coordinate calculation unit 303 in order to write the fourth rectangular image from the memory 107 to the Bank 500 of the rectangular image holding unit 108 as shown in the timing chart 1110. Show.

  In the example illustrated in FIG. 11, the timing at which the rectangular image holding unit 108 writes to the Bank 500 is the same as the timing at which the pixel interpolation unit 109 reads the rectangular image in units of pixels. However, there may be a case where writing to the Bank 500 of the rectangular image holding unit 108 ends early, or a timing when the pixel interpolation unit 109 ends reading out the rectangular image in units of pixels. In this case, control is performed so that writing and reading do not occur in the same Bank.

  As described above, according to the present embodiment, the calculation of the coordinates when reading the pre-correction image from the memory 107 and the calculation of the coordinates when reading the pre-correction image from the rectangular image holding unit 108 are performed as one pre-correction. It can be executed by the coordinate calculation unit. Thereby, a circuit scale and power consumption can be reduced.

(Other embodiments)
Although the first and second embodiments have been specifically described above, the present invention can be applied to other distortion corrections such as rolling distortion, and is not limited to the embodiments. Needless to say, various modifications can be made without departing from the scope of the invention.

  The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

103 Pre-correction coordinate calculation unit 104 Rectangle readout control unit 107 Memory 108 Rectangular image holding unit 109 Pixel interpolation unit 111 Corrected coordinate generation unit

Claims (10)

An image processing apparatus for correcting distortion aberration of the optical system in image data obtained by imaging an object using an optical system,
First storage means for storing the image data as an uncorrected image;
Coordinate generating means for generating coordinates of the corrected image in which the distortion is corrected;
Computing means for computing coordinates of the pre-correction image corresponding to coordinates of the post-correction image generated by the coordinate generation means;
Read control means for reading out the pre-correction image for each rectangular area from the first storage means based on some coordinates of the rectangular area in the pre-correction image calculated by the calculation means;
Second storage means for temporarily storing the pre-correction image read by the read control means for each rectangular area;
Image processing means for reading out the pre-correction image from the second storage means and generating the post-correction image based on the coordinates in the rectangular area in the pre-correction image calculated by the calculation means;
The coordinate generation means generates coordinates for the calculation means to calculate coordinates for reading the pre-correction image from the first storage means, and coordinates for reading the pre-correction image from the second storage means. Generating coordinates for the calculation means to calculate,
The calculating means calculates coordinates for reading the uncorrected image from the first storage means, or coordinates for reading the uncorrected image from the second storage means ,
Further, the calculation means includes:
Determining means for determining whether or not the coordinates of the corrected image generated by the coordinate generating means are coordinates of a part of a rectangular area in the corrected image;
When the determination unit determines that the coordinates are part of the rectangular area, the storage temporarily stores some coordinates of the rectangular area in the pre-correction image corresponding to the rectangular area in the corrected image. Means,
The image processing device, wherein the image processing unit reads the pre-correction image from the second storage unit based on the coordinates held in the holding unit, and generates the post-correction image . In the case where the first rectangular area in the pre-correction image is stored in the second storage means, the calculation means has the coordinates of the post-correction image different from the first rectangular area by the determination means. When it is determined that the coordinates of a part of the rectangular area corresponding to the rectangular area are, the calculation means includes one of the second rectangular areas in the uncorrected image corresponding to the rectangular area in the corrected image. Calculate the coordinates of the part,
The image processing unit reads the pre-correction image from the second storage unit based on the coordinates of the first rectangular area in the pre-correction image held in the holding unit, and generates the post-correction image. The image processing apparatus according to claim 1 . The calculation means prioritizes the calculation of coordinates for reading the uncorrected image from the first storage means over the calculation of coordinates for reading the uncorrected image from the second storage means. The image processing apparatus according to claim 1 or 2 . An image processing apparatus for correcting distortion aberration of the optical system in image data obtained by imaging an object using an optical system,
First storage means for storing the image data as an uncorrected image;
Coordinate generating means for generating coordinates of the corrected image in which the distortion is corrected;
Computing means for computing coordinates of the pre-correction image corresponding to coordinates of the post-correction image generated by the coordinate generation means;
Read control means for reading out the pre-correction image for each rectangular area from the first storage means based on some coordinates of the rectangular area in the pre-correction image calculated by the calculation means;
Second storage means for temporarily storing the pre-correction image read by the read control means for each rectangular area;
Image processing means for reading out the pre-correction image from the second storage means and generating the post-correction image based on the coordinates in the rectangular area in the pre-correction image calculated by the calculation means;
The coordinate generation means generates coordinates for the calculation means to calculate coordinates for reading the pre-correction image from the first storage means, and coordinates for reading the pre-correction image from the second storage means. Generating coordinates for the calculation means to calculate,
The calculating means calculates coordinates for reading the uncorrected image from the first storage means, or coordinates for reading the uncorrected image from the second storage means,
Images processor characterized in that the rectangular region is a square region. It said calculation means, an image processing apparatus according to any one of claims 1-4, characterized in that it is constituted by a single circuit. The first storage means is a DRAM, an image processing apparatus according to any one of claim 1 to 5, wherein said second storage means, which is a SRAM. An image processing method of an image processing apparatus for storing image data obtained by imaging an object using an optical system as a pre-correction image in a first storage unit and correcting distortion aberration of the optical system in the image data,
A coordinate generation step of generating coordinates of the corrected image in which the distortion is corrected;
A calculation step of calculating coordinates of the pre-correction image corresponding to the coordinates of the corrected image generated in the coordinate generation step;
Based on the coordinates of a part of the rectangular area in the pre-correction image calculated in the calculation step, the pre-correction image is read from the first storage unit for each rectangular area, and temporarily stored in the second storage unit. A control process for storing;
An image processing step of reading out the pre-correction image from the second storage means and generating the post-correction image based on the coordinates in the rectangular area in the pre-correction image calculated in the calculation step;
In the coordinate generation step, coordinates for calculating the coordinates for reading the pre-correction image from the first storage means in the calculation step and coordinates for reading the pre-correction image from the second storage means To generate coordinates for calculating in the calculation step,
In the calculation step, the coordinates for reading the pre-correction image from the first storage means or the coordinates for reading the pre-correction image from the second storage means are calculated ,
Furthermore, in the calculation step,
A determination step of determining whether or not the coordinates of the corrected image generated in the coordinate generation step are coordinates of a part of a rectangular region in the corrected image;
When it is determined in the determination step that the coordinates are a part of the rectangular area, the coordinates of the rectangular area in the pre-correction image corresponding to the rectangular area in the corrected image are temporarily stored in the holding unit. Holding process to hold,
In the image processing step, the post-correction image is generated by reading the pre-correction image from the second storage unit based on the coordinates held in the holding unit . An image processing method of an image processing apparatus for storing image data obtained by imaging an object using an optical system as a pre-correction image in a first storage unit and correcting distortion aberration of the optical system in the image data,
A coordinate generation step of generating coordinates of the corrected image in which the distortion is corrected;
A calculation step of calculating coordinates of the pre-correction image corresponding to the coordinates of the corrected image generated in the coordinate generation step;
Based on the coordinates of a part of the rectangular area in the pre-correction image calculated in the calculation step, the pre-correction image is read from the first storage unit for each rectangular area, and temporarily stored in the second storage unit. A control process for storing;
An image processing step of reading out the pre-correction image from the second storage means and generating the post-correction image based on the coordinates in the rectangular area in the pre-correction image calculated in the calculation step;
In the coordinate generation step, coordinates for calculating the coordinates for reading the pre-correction image from the first storage means in the calculation step and coordinates for reading the pre-correction image from the second storage means To generate coordinates for calculating in the calculation step,
In the calculation step, the coordinates for reading the pre-correction image from the first storage means or the coordinates for reading the pre-correction image from the second storage means are calculated,
An image processing method, wherein the rectangular area is a square area. A program for controlling an image processing apparatus that stores image data obtained by capturing an object using an optical system as a pre-correction image in a first storage unit and corrects distortion aberration of the optical system in the image data. And
A coordinate generation step of generating coordinates of the corrected image in which the distortion is corrected;
A calculation step of calculating coordinates of the pre-correction image corresponding to the coordinates of the corrected image generated in the coordinate generation step;
Based on the coordinates of a part of the rectangular area in the pre-correction image calculated in the calculation step, the pre-correction image is read from the first storage unit for each rectangular area, and temporarily stored in the second storage unit. A control process for storing;
Based on the coordinates in the rectangular area of the pre-correction image calculated in the calculation step, the computer executes an image processing step of reading the pre-correction image from the second storage means and generating the post-correction image Let
In the coordinate generation step, coordinates for calculating the coordinates for reading the pre-correction image from the first storage means in the calculation step and coordinates for reading the pre-correction image from the second storage means To generate coordinates for calculating in the calculation step,
In the calculation step, the coordinates for reading the pre-correction image from the first storage means or the coordinates for reading the pre-correction image from the second storage means are calculated,
Furthermore, in the calculation step,
A determination step of determining whether or not the coordinates of the corrected image generated in the coordinate generation step are coordinates of a part of a rectangular region in the corrected image;
When it is determined in the determination step that the coordinates are a part of the rectangular area, the coordinates of the rectangular area in the pre-correction image corresponding to the rectangular area in the corrected image are temporarily stored in the holding unit. Holding process to hold,
In the image processing step, the pre-correction image is read out from the second storage unit based on the coordinates held in the holding unit and the post-correction image is generated . A program for controlling an image processing apparatus that stores image data obtained by capturing an object using an optical system as a pre-correction image in a first storage unit and corrects distortion aberration of the optical system in the image data. And
A coordinate generation step of generating coordinates of the corrected image in which the distortion is corrected;
A calculation step of calculating coordinates of the pre-correction image corresponding to the coordinates of the corrected image generated in the coordinate generation step;
Based on the coordinates of a part of the rectangular area in the pre-correction image calculated in the calculation step, the pre-correction image is read from the first storage unit for each rectangular area, and temporarily stored in the second storage unit. A control process for storing;
Based on the coordinates in the rectangular area of the pre-correction image calculated in the calculation step, the computer executes an image processing step of reading the pre-correction image from the second storage means and generating the post-correction image Let
In the coordinate generation step, coordinates for calculating the coordinates for reading the pre-correction image from the first storage means in the calculation step and coordinates for reading the pre-correction image from the second storage means To generate coordinates for calculating in the calculation step,
In the calculation step, the coordinates for reading the pre-correction image from the first storage means or the coordinates for reading the pre-correction image from the second storage means are calculated,
The rectangular area is a square area.

Images