JP2008092602A - Image processor, camera system, and image processing method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image processor, a camera system, and an image processing method for correcting distortion of the image at low cost, and for producing a high-quality image in real time. <P>SOLUTION: The image processor corrects a distorted original image. The processor is provided with a horizontal correction means for correcting the distortion of the original image in the horizontal direction by performing an interpolating calculation to the original image, by means of a horizontal correction parameter indicating a horizontal correction amount in a pixel point constituting the original image; and a vertical correction means for correcting the distortion in the vertical direction by performing the interpolating calculation to the image corrected through the horizontal correction means, by means of a vertical correction parameter indicating a vertical correction amount in the pixel point constituting the original image. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to an image processing apparatus, a camera system, and an image processing method used for a video camera, a digital still camera, a silver salt camera, and the like.

  Conventionally, in an image captured by a video camera, a digital still camera, a silver salt camera, or the like, distortion has occurred due to the distortion aberration characteristic of the imaging lens. Here, the distortion is inconspicuous in a high-precision and high-performance lens, but it is difficult to completely avoid the influence of image distortion when using a low-cost lens or using an optical zoom lens.

  Therefore, in recent years, an image processing apparatus that corrects the distortion by signal processing has been proposed. FIG. 33 shows a configuration of a conventional image processing apparatus 100. As shown in FIG. 33, the conventional image processing apparatus 100 includes a lens 200, an image sensor 300, a data conversion unit 400, a signal processing unit 500, an image memory 600, a control microcomputer 700, a synchronization signal generation unit 800, and a correction data table 1010. A recording unit 1100, a reproduction unit 1200, and a display system processing unit 1300.

  Here, the outline of the operation of the image processing apparatus 100 will be described with reference to the flowchart of FIG. First, in step S <b> 1, an analog image signal for the subject 101 is input via the lens 200 and the image sensor 300. In step S2, the data conversion unit 400 converts the analog image signal into a digital image signal to generate an image 102.

  Next, in step S <b> 3, the signal processing unit 500 performs a correction operation on the distorted image 102 using a distortion correction vector (hereinafter also simply referred to as “correction vector”) stored in the correction data table 1010. In step S4, the control microcomputer 700 determines whether or not to end the input of the image. If it is determined not to end the process, the process returns to step S1.

  The above is the outline of the operation of the conventional image processing apparatus 100 shown in FIG. 33. The operation will be described in detail below.

  The lens 200 collects the reflected light from the subject 101 and projects it on the image sensor 300. The image sensor 300 is composed of a CCD, a CMOS sensor, or the like, and captures the projected video to generate an analog image signal. The data conversion unit 400 converts the analog signal supplied from the image sensor 300 into a digital image signal, and generates an image 102. On the other hand, the control microcomputer 700 issues a command for instructing a predetermined operation in response to an input to an external user interface.

  Further, the signal processing unit 500 stores the digital image signal generated by the data conversion unit 400 in the image memory 600 in accordance with the command supplied from the control microcomputer 700. Then, the signal processing unit 500 reads out correction vectors corresponding to all pixels recorded in advance in the correction data table 1010 from the table, obtains a necessary image signal from the image memory 600 according to the correction information, and then reads the image The distortion of the image 102 output from the data conversion unit 400 is corrected by executing geometric correction on the signal by a two-dimensional interpolation method.

  Here, the image signal generated in the signal processing unit 500 is supplied to the display system processing unit 1300 so that the image is displayed on the monitor, or is supplied to the recording unit 1100 so that an external tape, disk, or the like is displayed. It is recorded on a medium 1400 such as a memory. The image signal recorded on the medium 1400 is reproduced by the reproduction unit 1200, and the reproduction signal is supplied to the display system processing unit 1300, whereby the reproduced image is displayed on the monitor.

  The synchronization signal generation unit 800 generates an internal synchronization signal in accordance with a clock signal CLK supplied from the outside, and supplies the internal synchronization signal to the image sensor 300, the data conversion unit 400, and the signal processing unit 10.

  FIG. 35 is a block diagram showing a configuration of the signal processing unit 500 shown in FIG. As shown in FIG. 35, the signal processing unit 500 includes a timing control unit 510, an interpolation phase / input data coordinate calculation unit 520, a data acquisition unit 530, an interpolation coefficient generation unit 540, a data interpolation calculation unit 550, an output data buffer 560, And a data writing unit 570.

  Here, the data writing unit 570 supplies the digital image signal supplied from the data conversion unit 400 to the image memory 600 together with the write control signal Sw, and causes the image memory 600 to store the digital image signal.

  The timing control unit 510 generates a control timing signal St according to the internal synchronization signal supplied from the synchronization signal generation unit 800, and the interpolation phase / input data coordinate calculation unit 520 corresponds to the supplied control timing signal St. The coordinates of the output image are calculated, and a correction vector request signal Sa for requesting a correction vector for the obtained coordinates is supplied to the correction data table 1010.

  The correction data table 1010 obtains a correction vector corresponding to the correction vector request signal Sa from the built-in table and supplies the correction vector to the data acquisition unit 530 and the interpolation coefficient generation unit 540. The data acquisition unit 530 acquires the interpolation data corresponding to the integer component of the correction vector output from the correction data table 1010 from the image memory 600 by supplying the read control signal Sr to the image memory 600. The data acquisition unit 530 supplies the acquired interpolation data to the data interpolation calculation unit 550.

  On the other hand, the interpolation coefficient generation unit 540 generates an interpolation coefficient according to the decimal component of the correction vector supplied from the correction data table 1010 and supplies the generated interpolation coefficient to the data interpolation calculation unit 550. Then, the data interpolation calculation unit 550 performs an interpolation operation according to the interpolation data supplied from the data acquisition unit 530 and the interpolation coefficient supplied from the interpolation coefficient generation unit 540. Note that a two-dimensional interpolation calculation is executed as the interpolation calculation.

  Hereinafter, image conversion by two-dimensional interpolation will be described with reference to FIG. FIG. 36A shows images before and after the two-dimensional interpolation, and FIG. 36B shows an enlarged view of a part of FIG.

  Here, for example, when an arrow connecting the points a1 to a4 shown in FIG. 36A is an output image, the points on the image 102 corresponding to the points a1 to a4 constituting the output image are Let it be point A1 to point A4. Therefore, FIG. 36 (a) shows a case where the original image composed of arrows connecting the points A1 to A4 is converted into an output image connecting the points a1 to a4 by two-dimensional interpolation.

  At this time, when the image of each point of the output image is determined using two (2 × 2) image data in the x and y directions, for example, four grid points K00, K01, K10, The image data at the point a1 is determined using the image data at K11. Note that the same calculation is performed for the points A2 to A4, thereby determining the image data of the points a2 to a4. Here, the four grid points K00, K01, K10, and K11 are determined according to the correction coordinates output from the correction data table 1010.

  As shown in FIG. 36 (b), when the distances between the lattice point K00 and the lattice point K10 and between the lattice point K10 and the lattice point K11 are 1, both in the x direction and the y direction. The position of the point A1 is specified by decimal parameters Px and Py, respectively. At this time, the weighting (interpolation coefficient) Cn (n = 1 to 4) of each image data of the grid points K00, K01, K10, and K11 used when calculating the image data of the point a1 is supplied from the correction data table 1010. Determined by the decimal component of the correction vector, that is, the decimal parameters Px and Py.

  Further, data obtained as a result of the interpolation calculation by the data interpolation calculation unit 550 is held in the output data buffer 560 and is output to the display system processing unit 1300 or the recording unit 1100 at a predetermined timing.

  Here, the conventional data interpolation calculation unit 550 is configured as shown in FIG. FIG. 37 shows a configuration in which an image of each point of the output image is determined using image data consisting of a total of 16 pieces (4 × 4) arranged in the x and y directions.

  As shown in FIG. 37, the conventional data interpolation calculation unit 550 includes four line memories 900, a total of 16 registers 901 connected in series to the output nodes of each line memory 900, and each register 901. 16 multiplication circuits 902 for multiplying the image data outputted from the corresponding interpolation coefficient CHn (n = 00 to 33), respectively, and an addition circuit 904 for adding the data obtained by the 16 multiplication circuits 902 And a dividing circuit 905 that divides the data obtained by the adding circuit 904.

  However, according to the conventional image processing apparatus as described above, it is possible to correct the distortion of the image in real time. However, since it is necessary to have a correction vector corresponding to all pixels, the circuit scale is increased and the cost is increased. There is a problem that it takes.

  Further, when the position of the lens 200 is changed or when the lens is exchanged, it is necessary to update the correction vector according to the change in the distortion aberration characteristic of the lens. Is required.

  Further, the update of the correction data table 1010 is executed by the control microcomputer 700 according to an instruction from the user interface. However, since a large communication capacity is required between the control microcomputer 700 and the correction data table 1010, control is performed. There is also a problem that real-time processing by the microcomputer 700 becomes difficult.

  In addition, there is a method of sequentially calculating the correction vector instead of providing the correction data table 1010. However, with this method, real-time processing without so-called frame delay is difficult, and a large amount of hardware is required to realize real-time processing. There is a problem that it becomes costly.

  Also, as described above, in two-dimensional interpolation, image data at a plurality of points on a two-dimensional plane on which an image is formed is used to correct one point of image data, but in order to obtain a high-quality image. Since image data at a number of points is necessary, there is a problem that the access frequency to the image memory 600 is high and the operation speed cannot be increased.

  In addition, when performing two-dimensional interpolation, it is necessary that the port width of the image memory 600 has a bandwidth that is several times the output rate. That is, for example, when one pixel image data is generated from four pixel image data in two-dimensional interpolation, the port width needs to be four times the bandwidth of one pixel.

  Thus, when two-dimensional interpolation is performed, a certain condition is required for the port width. Therefore, a high-order tap (“tap” means the number of data in one direction to be subjected to image processing. This means that it is very difficult to use a high-performance filter.

  The present invention has been made to solve the above-described problems. An image processing apparatus, a camera system, and an image processing method for correcting image distortion at low cost and generating a high-quality image in real time are provided. The purpose is to provide.

  The present invention is an image processing apparatus that corrects a distorted original image, and performs an interpolation operation on the original image using a horizontal correction parameter that indicates a correction amount in a horizontal direction at a pixel point constituting the original image. By applying the correction, a horizontal correction unit that corrects distortion in the horizontal direction of the original image, and an amount of correction in the vertical direction at the pixel points constituting the original image with respect to the image obtained by the correction by the horizontal correction unit are shown. And a vertical correction unit that corrects distortion in the vertical direction of the original image by performing an interpolation calculation using a vertical correction parameter.

  Preferably, the horizontal correction unit expands or contracts the original image in the horizontal direction by adjusting a horizontal interval between pixel points for which image data is obtained by the interpolation calculation, and the vertical correction unit performs the interpolation. The original image is expanded or contracted in the vertical direction by adjusting the interval in the vertical direction of pixel points for which image data is obtained by calculation.

  Preferably, the horizontal correction means includes first data acquisition means for selectively acquiring image data at the pixel point according to an integer component of the horizontal correction parameter, and an interpolation coefficient according to a decimal component of the horizontal correction parameter. The interpolation calculation is performed using the first interpolation coefficient generation means for generating the image data, the image data acquired by the first data acquisition means, and the interpolation coefficient generated by the first interpolation coefficient generation means. First interpolation calculation means, wherein the vertical correction means selectively acquires image data at the pixel point according to an integer component of the vertical correction parameter, and a decimal number of the vertical correction parameter. Second interpolation coefficient generation means for generating an interpolation coefficient according to the component; the image data acquired by the second data acquisition means; and the second interpolation. And a second interpolation operation means for performing the interpolation calculation using said interpolation coefficients generated by the number generator.

  Preferably, the apparatus further comprises storage means for storing a horizontal correction image obtained by correction by the horizontal correction means, and the vertical correction means obtains the horizontal correction image according to the vertical correction parameter from the storage means. Acquisition means, and interpolation calculation means for performing interpolation calculation using the vertical correction parameter for the horizontal correction image acquired by the data acquisition means.

  Preferably, the horizontal correction unit corrects distortion in the horizontal direction of the original image by performing an interpolation calculation, and the vertical correction unit corrects distortion in the vertical direction of the original image by performing an interpolation calculation. to correct.

  The present invention is also an image processing apparatus for correcting an original image having distortion, and performing an interpolation operation using a vertical correction parameter indicating a correction amount in a vertical direction at a pixel point constituting the original image on the original image. And a vertical correction unit for correcting distortion in the vertical direction of the original image, and a horizontal correction amount at pixel points constituting the original image with respect to the image obtained by the correction by the vertical correction unit. And a horizontal correction means for correcting distortion in the horizontal direction of the original image by performing an interpolation calculation using a horizontal correction parameter indicating.

  Preferably, the vertical correction unit expands or contracts the original image in the vertical direction by adjusting a vertical interval between pixel points for which image data is obtained by the interpolation calculation, and the horizontal correction unit includes the interpolation The original image is expanded and contracted in the horizontal direction by adjusting the horizontal interval between pixel points for which image data is obtained by calculation.

  Preferably, the vertical correction means includes first data acquisition means for selectively acquiring image data at the pixel point according to an integer component of the vertical correction parameter, and an interpolation coefficient according to a decimal component of the vertical correction parameter. The interpolation calculation is performed using the first interpolation coefficient generation means for generating the image data, the image data acquired by the first data acquisition means, and the interpolation coefficient generated by the first interpolation coefficient generation means. First correction calculation means, wherein the horizontal correction means selectively acquires image data at the pixel point according to an integer component of the horizontal correction parameter, and a decimal number of the horizontal correction parameter. Second interpolation coefficient generation means for generating an interpolation coefficient according to the component; the image data acquired by the second data acquisition means; and the second interpolation. And a second interpolation operation means for performing the interpolation calculation using said interpolation coefficients generated by the number generator.

  Preferably, the apparatus further comprises storage means for storing a vertical correction image obtained by correction by the vertical correction means, and the horizontal correction means obtains the vertical correction image according to the horizontal correction parameter from the storage means. Acquisition means, and interpolation calculation means for performing interpolation calculation using the horizontal correction parameter for the vertical correction image acquired by the data acquisition means.

  Preferably, the vertical correction unit corrects distortion in the vertical direction of the original image by performing an interpolation calculation, and the horizontal correction unit corrects distortion in the horizontal direction of the original image by performing an interpolation calculation. to correct.

  The present invention also provides a lens that collects light from a subject, an image sensor that generates a captured image signal from projection light from the lens, and data conversion that converts the captured image signal from the image sensor into a digital image signal. A camera system comprising: a circuit; a signal processing circuit that generates a recording image data by processing a digital image signal from the data conversion circuit; and a recording unit that records the recording image data from the signal processing circuit. The signal processing circuit includes image correction means for correcting an original image having lens distortion, and the image correction means has a horizontal correction parameter indicating a horizontal correction amount at a pixel point constituting the original image. A horizontal correction means for correcting distortion in the horizontal direction of the original image by performing an interpolation operation on the original image, and a correction by the horizontal correction means. A vertical correction for correcting distortion in the vertical direction of the original image by performing an interpolation operation using a vertical correction parameter indicating a correction amount in the vertical direction at the pixel points constituting the original image on the obtained image Means.

  The present invention also provides a lens that collects light from a subject, an image sensor that generates a captured image signal from projection light from the lens, and data conversion that converts the captured image signal from the image sensor into a digital image signal. A camera system comprising: a circuit; a signal processing circuit that generates a recording image data by processing a digital image signal from the data conversion circuit; and a recording unit that records the recording image data from the signal processing circuit. The signal processing circuit includes image correction means for correcting an original image having lens distortion, and the image correction means has a vertical correction parameter indicating a correction amount in a vertical direction at a pixel point constituting the original image. A vertical correction unit that corrects distortion in the vertical direction of the original image by performing an interpolation operation on the original image, and a correction by the vertical correction unit. A horizontal correction for correcting distortion in the horizontal direction of the original image by performing an interpolation operation using a horizontal correction parameter indicating a horizontal correction amount at pixel points constituting the original image on the obtained image Means.

  The present invention is also an image processing method for correcting a distorted original image, wherein an interpolation operation is performed on the original image using a horizontal correction parameter indicating a horizontal correction amount at a pixel point constituting the original image. The first step of correcting distortion in the horizontal direction of the original image, and the vertical correction amount at the pixel points constituting the original image with respect to the image obtained in the first step. And a second step of correcting distortion in the vertical direction of the original image by performing an interpolation calculation using the vertical correction parameter shown.

  Preferably, at least in the first step, the original image is expanded or contracted in the horizontal direction by adjusting an interval in a horizontal direction between pixel points for which image data is obtained by the interpolation operation, or in the second step, The original image is expanded and contracted in the vertical direction by adjusting the interval in the vertical direction of pixel points for which image data is obtained by the interpolation calculation.

  Preferably, the method further includes a step of storing a horizontal correction image obtained by the correction in the first step in a storage unit, and the second step includes the step of storing the horizontal correction image according to the vertical correction parameter from the storage unit. And a data acquisition step for performing an interpolation operation using the vertical correction parameter on the horizontal correction image acquired in the data acquisition step.

  Preferably, the first step corrects a distortion in the horizontal direction of the original image by performing an interpolation operation, and the second step performs an interpolation operation in the vertical direction of the original image. Correct distortion.

  The present invention is also an image processing method for correcting a distorted original image, wherein an interpolation operation is performed on the original image using a vertical correction parameter indicating a correction amount in a vertical direction at a pixel point constituting the original image. By performing the first step of correcting distortion in the vertical direction of the original image, and with respect to the image obtained in the first step, the horizontal correction amount at the pixel points constituting the original image is set. And a second step of correcting distortion in the horizontal direction of the original image by performing an interpolation operation using the horizontal correction parameter shown.

  Preferably, at least in the first step, the original image is expanded or contracted in the vertical direction by adjusting an interval in a vertical direction of pixel points for which image data is obtained by the interpolation operation, or in the second step, The original image is expanded or contracted in the horizontal direction by adjusting the horizontal interval between pixel points for which image data is obtained by the interpolation calculation.

  Preferably, the method further comprises a step of storing a vertical correction image obtained by the correction in the first step in a storage unit, wherein the second step includes the vertical correction image corresponding to the horizontal correction parameter from the storage unit. And a data acquisition step for performing an interpolation operation using the horizontal correction parameter for the vertical correction image acquired in the data acquisition step.

  Preferably, the first step corrects a distortion in the vertical direction of the original image by performing an interpolation operation, and the second step performs an interpolation operation in the horizontal direction of the original image. Correct distortion.

  According to the image processing apparatus, the camera system, and the image processing method according to the present invention, the distortion of the original image can be corrected in real time at a low cost, so that a high-quality image can be easily obtained.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In the drawings, the same reference numerals indicate the same or corresponding parts.

  FIG. 1 is a block diagram showing a configuration of an image processing system according to an embodiment of the present invention. As shown in FIG. 1, an image processing system according to an embodiment of the present invention includes an image processing device 2, a preprocessing device 3, and a medium 1400. The image processing device 2 includes a lens 200, an image sensor 300, and a data conversion unit. 400, a signal processing unit 10, an image memory 7, a control microcomputer 8, a correction parameter decoder 9, a synchronization signal generation unit 800, a recording unit 1100, a reproduction unit 1200, and a display system processing unit 1300. A parameter encoder 5 and a correction parameter derivation unit 6 are included.

  Here, the lens 200 condenses the reflected light from the subject 101 and projects it on the image sensor 300, and may have a zoom function as well as a single focus lens. In addition, the image pickup device 300 is configured by a CCD, a CMOS sensor, or the like, and captures a projected video and generates an analog image signal in accordance with an internal synchronization signal supplied from the synchronization signal generation unit 800.

  The data conversion unit 400 is connected to the image sensor 300 and converts the analog image signal generated by the image sensor 300 into a digital image signal in accordance with the internal synchronization signal supplied from the synchronization signal generation unit 800 to generate the image 102. .

  The signal processing unit 10 is connected to the control microcomputer 8, the data conversion unit 400, the image memory 7, the correction parameter decoder 9, and the synchronization signal generation unit 800. The signal processing unit 10 stores the digital image signal supplied from the data conversion unit 400 in the image memory 7 in accordance with the command supplied from the control microcomputer 8 and the correction amount supplied from the correction parameter decoder 9. Correction processing for the image signal stored by the parameter is executed. Then, the signal processing unit 10 supplies the image signal obtained by the correction to the display system processing unit 1300 and the recording unit 1100. The signal processing unit 10 will be described in detail later.

  On the other hand, the correction parameter deriving unit 6 calculates in advance a correction amount vector corresponding to each position of all pixels from data relating to the distortion aberration of the lens 200. The correction parameter encoder 5 is connected to the correction parameter deriving unit 6 and the user interface, and compresses (encodes) the correction amount vector supplied from the correction parameter deriving unit 6 in accordance with the control signals Ln and Lw supplied from the user interface. The compressed data Pc is supplied to the correction parameter decoder 9.

  Note that the calculation in the correction parameter deriving unit 6 and the above encoding are both operations with a very large load, but may be calculated using a separate personal computer or the like, and do not affect the real-time processing by the image processing apparatus 2. .

  Further, in the image processing system according to the embodiment of the present invention, the preprocessing device 3 is not an essential component, and the compressed data Pc is supplied to the correction parameter decoder 9 from outside the image processing device 2. Embodiments are conceivable as well.

  The control microcomputer 8 outputs a command or the like for instructing a predetermined operation to the signal processing unit 10 in accordance with a control signal from the user interface, and supplies the position information of the lens 200 to the correction parameter decoder 9.

  The correction parameter decoder 9 is connected to the correction parameter encoder 5, the control microcomputer 8, and the signal processing unit 10. Then, the correction parameter decoder 9 expands (decodes) the encoded compressed data Pc supplied from the correction parameter encoder 5 into a correction amount parameter corresponding to each pixel in accordance with the information supplied from the control microcomputer 8. The correction amount parameter is supplied to the signal processing unit 10.

  Here, the correction parameter decoder 9 supplies the correction amount parameter to the signal processing unit 10 regardless of the interpolation method executed in the signal processing unit 10. The correction parameter encoder 5 and the correction parameter decoder 9 will be described in detail later.

  The recording unit 1100 is connected to the signal processing unit 10, and the image signal generated by the signal processing unit 10 is transferred to a medium (recording medium) 1400 such as a tape, a flexible disk, a DVD (digital versatile disk), a hard disk, or a memory. Record. The image signal generated by the signal processing unit 10 can be recorded on the medium 1400 using the Internet, wireless communication, or the like.

  Further, the playback unit 1200 is connected to the medium 1400, plays back an image signal stored in the medium 1400, and supplies the image signal to the display system processing unit 1300. The display system processing unit 1300 is connected to the signal processing unit 10 and the reproduction unit 1200, and displays the image signal supplied from the signal processing unit 10 or the reproduction unit 1200 on a monitor.

  The synchronization signal generation unit 800 generates an internal synchronization signal according to the clock signal CLK supplied from the outside, and supplies the internal synchronization signal to the image sensor 300, the data conversion unit 400, and the signal processing unit 10.

  FIG. 2 is a block diagram showing a configuration of the signal processing unit 10 shown in FIG. As shown in FIG. 2, the signal processing unit 10 includes a horizontal one-dimensional interpolation unit 501 and a vertical one-dimensional interpolation unit 502. The image memory 7 includes an image memory 601 composed of a horizontal processing FIFO memory and an image memory 602 composed of a vertical processing line buffer. The correction parameter decoder 9 includes an image distortion correction parameter decoder 33 for x direction and an image memory 602 for y direction. Image distortion correction parameter decoder 34.

  Note that the image memory 602 has a capacity sufficient to store data for a minimum number of lines necessary to realize vertical distortion correction, which will be described later. The capacity is generally determined according to the frequency of the output synchronization signal supplied to the output data buffer 32.

  The horizontal one-dimensional interpolation unit 501 includes a data writing unit 21, an operation control unit 22, an interpolation phase / input data coordinate calculation unit 23, a data acquisition unit 24, an interpolation coefficient generation unit 25, and a data interpolation calculation unit 26. The one-dimensional interpolation unit 502 includes an arithmetic control unit 27, an interpolation phase / input data coordinate calculation unit 28, a data acquisition unit 29, an interpolation coefficient generation unit 30, a data interpolation calculation unit 31, and an output data buffer 32.

  Here, the data writing unit 21 is connected to the data conversion unit 400, and the calculation control unit 22 is connected to the synchronization signal generation unit 800. The interpolation phase / input data coordinate calculation unit 23 is connected to the calculation control unit 22 and the control microcomputer 8, and the data acquisition unit 24 is connected to the interpolation phase / input data coordinate calculation unit 23, the image memory 601, and the image distortion correction parameter decoder 33. Connected. The interpolation coefficient generation unit 25 is connected to the image distortion correction parameter decoder 33, and the data interpolation calculation unit 26 is connected to the data acquisition unit 24 and the interpolation coefficient generation unit 25.

  The image memory 601 is connected to the data writing unit 21 and the data acquisition unit 24, and the image memory 602 is connected to the data interpolation calculation unit 26 and the data acquisition unit 29. The image distortion correction parameter decoder 33 is connected to the interpolation phase / input data coordinate calculation unit 23 and the data acquisition unit 24.

  On the other hand, the calculation control unit 27 is connected to the synchronization signal generation unit 800, and the interpolation phase / input data coordinate calculation unit 28 is connected to the calculation control unit 27 and the control microcomputer 8. The data acquisition unit 29 is connected to the interpolation phase / input data coordinate calculation unit 28, the image memory 602, and the image distortion correction parameter decoder 34, and the interpolation coefficient generation unit 30 is connected to the image distortion correction parameter decoder 34. The data interpolation calculation unit 31 is connected to the data acquisition unit 29 and the interpolation coefficient generation unit 30, and the output data buffer 32 is connected to the data interpolation calculation unit 31 and the synchronization signal generation unit 800.

  The output node of the output data buffer 32 is connected to the display system processing unit 1300 and the recording unit 1100. The image distortion correction parameter decoder 34 is connected to the interpolation phase / input data coordinate calculation unit 28.

  In the signal processing unit 10 having the above-described configuration, the horizontal one-dimensional interpolation unit 501 first performs a one-dimensional interpolation operation in the horizontal direction (x direction), and then the vertical one-dimensional interpolation unit 502 performs the vertical direction (y direction). ) One-dimensional interpolation operation is executed. Here, an outline of the calculation by the signal processing unit 10 will be described with reference to FIG. FIG. 3 illustrates a case where the image data of each point of the output image is determined using image data consisting of a total of 16 pieces (4 × 4) arranged in the x and y directions.

  FIG. 3A shows that the image data of points B10 to B40 corresponding to the points B1 to B4 constituting the original image with distortion is calculated by the correction in the x direction, and FIG. ) Indicates that the image data of the points b1 to b4 are calculated corresponding to the points B10 to B40 by further correcting in the y direction.

  More specifically, for example, the image data of the point B10 is calculated by performing a predetermined interpolation operation on the image data of four grid points straddling the point B1 that is continuous in the horizontal direction, and similarly, the image data of the point B2 to B4 corresponds to the points B2 to B4. Thus, the image data of the points B20 to B40 are calculated.

  Next, as shown in FIG. 3B, for example, with respect to the point B30, the image data at four lattice points (points K20 to K23) in the broken line straddling the point B30 continuous in the vertical direction is set to a predetermined value. The image data of the point b3 is calculated by performing the interpolation calculation. Similarly, image data of points b1, b2, and b4 are calculated corresponding to points B10, B20, and B40, respectively.

  Here, the one-dimensional interpolation calculation in the horizontal direction as described above is realized by the horizontal processing circuit 40 shown in FIG. 4 included in the data interpolation calculation unit 26. As shown in FIG. 4, the horizontal processing circuit 40 includes a line memory 900, four registers 901 connected in series to an output node of the line memory 900, and an interpolation coefficient CHk ( 4 multiplication circuits 902 for multiplying k = 0 to 3), and an addition circuit 903 for adding the data obtained by the four multiplication circuits 902.

  The one-dimensional interpolation calculation in the vertical direction as described above is realized by a circuit shown in FIG. 25 described later, which will be described in detail later.

  Next, an outline of the operation of the signal processing unit 10 shown in FIG. 2 will be described. First, the image data input from the data conversion unit 400 to the horizontal one-dimensional interpolation unit 501 is supplied to the image memory 601 together with the write control signal by the data writing unit 21, and is written to the image memory 601 in accordance with the write control signal.

  At this time, the data acquisition unit 24 supplies the read control signal to the image memory 601 for horizontal processing, so that the image memory 601 corresponds to the x-direction correction amount parameter Xm supplied from the image distortion correction parameter decoder 33. To obtain image data arranged in the horizontal direction as interpolation data.

  The data interpolation calculation unit 26 executes horizontal one-dimensional interpolation calculation using the interpolation coefficient supplied from the interpolation coefficient generation unit 25, and the image memory 602 for vertical processing stores the calculation result.

  Next, in the vertical one-dimensional interpolation unit 502, the data acquisition unit 29 is arranged in the vertical direction from the image memory 602 for vertical processing according to the correction amount parameter Ym for y direction supplied from the image distortion correction parameter decoder 34. Image data is acquired as interpolation data. Then, the data interpolation calculation unit 31 performs vertical one-dimensional interpolation calculation using the interpolation coefficient supplied from the interpolation coefficient generation unit 30, and the output data buffer 32 outputs the calculation result according to the output synchronization signal. .

  As described above, since the interpolation calculation executed by the horizontal one-dimensional interpolation unit 501 and the vertical one-dimensional interpolation unit 502 is a one-dimensional interpolation calculation, a 4-tap filter such as cubic interpolation or a higher order filter is used. A tap count filter can be used.

  That is, since the one-dimensional interpolation calculation is realized by a simple circuit as described above, the calculation using a high-order tap filter, which is difficult with the two-dimensional interpolation calculation, can be easily realized, so that a higher quality image can be obtained. . Note that since a general pixel number conversion circuit or the like is a circuit that performs one-dimensional interpolation, the existing circuit may be shared for the above-described calculation.

  In the above description, the embodiment in which the one-dimensional interpolation calculation in the vertical direction is executed after the one-dimensional interpolation calculation in the horizontal direction has been described. Conversely, the one-dimensional interpolation calculation in the vertical direction is executed first, Thereafter, a one-dimensional interpolation calculation in the horizontal direction may be executed. In this case, the image data output from the data conversion unit 400 is input to the vertical one-dimensional interpolation unit 502, and after being subjected to the vertical one-dimensional interpolation calculation, is temporarily stored in the image memory 601 for horizontal processing. The image data stored in the image memory 601 is further subjected to horizontal one-dimensional interpolation calculation by the horizontal one-dimensional interpolation unit 501, completely corrected for distortion, and output to the outside of the signal processing unit 10. .

  In addition, the arithmetic processing described above may be applied not only to one system of data but also to each system for color signals (RGB, YUV). Further, when the interpolation calculation is performed on the moving image, the calculation may be executed in synchronization with the vertical synchronization signal.

  By the way, an imaging apparatus such as a video camera or a digital still camera is often equipped with a so-called optical zoom function and a camera shake correction function. Here, when the optical zoom is performed by the above function, the distortion characteristics of the lens vary depending on whether it is tele (zoom up) or wide (zoom down). That is, generally, when the lens 200 moves in the wide direction, barrel distortion occurs in the image, and when the lens 200 moves in the tele direction, pincushion distortion occurs in the image.

  At this time, since the image quality deteriorates when the image is not corrected by an appropriate correction vector corresponding to the optical zoom, the correction parameter decoder 9 selects an optimal correction amount parameter corresponding to the lens position. .

  Specifically, the correction parameter decoder 9 receives information indicating the position of the lens 200 from the control microcomputer 8 and selectively decodes the compressed data Pc supplied from the correction parameter encoder 5 according to the position information.

  As described above, according to the image processing apparatus 2 shown in FIG. 1, even when the characteristic of the lens 200 varies, only the correction amount parameter decoded according to the characteristic is used for the interpolation calculation. Therefore, the amount of data used for the calculation can be minimized, and as a result, the manufacturing cost can be reduced.

  Next, the camera shake correction function will be described. In general, as a method of correcting image distortion due to camera shake, a method of optically correcting the image by controlling the position of a lens or the like like an active prism method or an active lens method, and an active image area method are used. Thus, there is a method of electrically correcting the obtained image signal by performing predetermined processing.

  Here, the optical correction method is difficult to be realized by the image processing apparatus 2 according to the embodiment of the present invention because the lens characteristics vary depending on the position of the lens 200.

  On the other hand, the above-described electrical correction method is realized by signal processing for cutting out a part (effective area) of an image from the entire image based on information on a camera shake position detected by an angular velocity sensor or the like. At this time, since the target of image processing changes according to the position of the effective area, etc., it is necessary to change the correction vector used when performing the interpolation calculation according to the target.

  Accordingly, the correction parameter decoder 9 further receives camera shake position information from the control microcomputer 8, and realizes camera shake correction by selectively decoding the compressed data Pc supplied from the correction parameter encoder 5 in accordance with the position information. To do.

  In the image processing apparatus 2 according to the present embodiment, even when the lens 200 is replaced, the correction parameter decoder 9 selectively decodes the compressed data Pc according to the new lens 200 and the like. If so, a high-quality image can be easily obtained even after the parts are replaced.

  Next, the operation of the horizontal one-dimensional interpolation unit 501 shown in FIG. 2 will be described in detail with reference to the flowcharts of FIGS. 5 and 6. First, the arithmetic control unit 22 generates a control timing signal in accordance with the internal synchronization signal supplied from the synchronization signal generation unit 800. The interpolation phase / input data coordinate calculation unit 23 operates according to the control timing signal supplied from the calculation control unit 22 and performs interpolation in the coordinate system when the image input to the signal processing unit 10 is not distorted. Calculate point coordinates with a decimal point.

  Specifically, in step S1, the interpolation phase / input data coordinate calculation unit 23 cuts out the image as coordinates (x, y) on the image subjected to distortion correction and equal magnification conversion as shown in FIG. The upper left coordinates (Sx, Sy) of the image CI are initialized, and a correction parameter request signal Rx is supplied to the image distortion correction parameter decoder 33. On the other hand, in step S2, the image distortion correction parameter decoder 33 obtains the supplied correction parameter request signal Rx and the correction amount parameter Xm corresponding to the coordinates (Sx, Sy), and sends them to the data acquisition unit 24 and the interpolation coefficient generation unit 25. Supply.

  Here, the image distortion correction parameter decoder 33 includes, for example, a ROM (Read Only Memory), and a comparison table between the x coordinate and the correction amount parameter Xm may be stored in advance in the ROM. The correction amount parameter Xm may be approximated as a function having an x coordinate, and the correction amount parameter Xm may be obtained using the function, which will be described in detail later.

  Next, in step S3, the data acquisition unit 24 uses the coordinates (X, Y) supplied from the interpolation phase / input data coordinate calculation unit 23 in accordance with the correction amount parameter Xm supplied from the image distortion correction parameter decoder 33. A correction amount vector (Xm, 0) is added. Accordingly, as shown in FIG. 7B, the coordinates (X + Xm, Y) of the point corresponding to the coordinates (X, Y) in the original image OI before correction, that is, the correction vector is obtained. .

  Instead of the data acquisition unit 24, the image distortion correction parameter decoder 33 obtains the correction vector according to the x coordinate supplied from the interpolation phase / input data coordinate calculation unit 23, and sends the correction vector to the data acquisition unit 24. You may make it supply.

  At this time, the data acquisition unit 24 determines whether or not the integer value of the x coordinate has been changed by adding Xm. If it is determined that the value has changed, the process proceeds to step S5. Advances to step S6.

  In step S5, the data acquisition unit 24 further determines whether or not the integer value has changed by 2 or more. When it is determined that the integer value has changed by 2 or more, the process proceeds to step S8. Advances to step S7. On the other hand, in step S <b> 6, the image memory 601 supplies the same interpolation data output in the previous cycle to the data acquisition unit 24 again in response to the hold signal Sh supplied from the data acquisition unit 24.

  In the above, the data acquisition unit 24 generates an address of data to be read from the image memory 601 according to the integer value of the x component (X + Xm) of the generated correction vector, and supplies a read control signal to the image memory 601. Interpolation data corresponding to the address is acquired.

  Here, the image memory 601 sequentially outputs the interpolation data corresponding to the address while incrementing the address one by one from the head address, and the hold signal Sh is supplied from the data acquisition unit 24, so that the increment is temporarily performed. Stop.

  The image memory 601 may receive a read start address from the data acquisition unit 24 and output a predetermined number of continuous data with the read start address as the head address.

  Here, the hold signal Sh and the read start address are obtained from the integer component of the correction amount parameter Xm output from the image distortion correction parameter decoder 33.

  On the other hand, the interpolation coefficient generation unit 25 treats the decimal component of the correction amount parameter Xm supplied from the image distortion correction parameter decoder 33 as the phase of the horizontal interpolation filter, and generates an interpolation coefficient according to the decimal component. Such an operation is applied when the image 102 input to the signal processing unit 10 is in the RGB format. On the other hand, in the case of the YUV format, the filter phase of the luminance signal Y can be handled in the same way as the filter phase of the RGB format. For the color difference signal Cb / Cr, not only the decimal component but also the integer component of the correction amount parameter Xm. The phase can be calculated in combination.

  In step S7, the data interpolation calculation unit 26 performs a one-dimensional interpolation operation according to the interpolation data supplied from the data acquisition unit 24 and the interpolation coefficient, and the process proceeds to step S9.

  Here, in the above one-dimensional interpolation calculation, for example, in the YUV format, the luminance data Dt of 8 pixels in the horizontal direction from the vicinity of the correction vector (X + Xm, Y) as shown in FIG. And an 8-tap interpolation operation with the decimal component as a phase is executed. Note that the result obtained by the interpolation calculation is used as luminance data of the output image, and the horizontal distortion is corrected thereby.

  On the other hand, in step S 8, the data acquisition unit 24 supplies the skip signal sk to the interpolation phase / input data coordinate calculation unit 23, the image distortion correction parameter decoder 33 and the data interpolation calculation unit 26, and these interpolation phase / input data coordinate calculation unit 23. And the operations of the image distortion correction parameter decoder 33 and the data interpolation calculation unit 26 are stopped.

  Here, if it is determined in step S5 that the x coordinate has changed by 2 or more, it means that the center coordinate actually interpolated is moved by 2 pixels or more, and therefore the data interpolation calculation unit 26 applies the data to the image memory 602. Data output is interrupted. In addition, when the center coordinates actually interpolated are moved by two or more pixels, the decimal component (interpolation phase) of the correction amount parameter Xm output from the image distortion correction parameter decoder 33 is held until the next cycle. The operation of the distortion correction parameter decoder 33 is stopped.

  In step S13, the interpolation phase / input data coordinate calculation unit 23 adds the horizontal enlargement / reduction parameter Ha to the x coordinate, and the process proceeds to step S2. The enlargement / reduction parameter Ha is determined by a ratio of lengths in the horizontal direction of the original image having distortion with respect to the image after correction, and is set to a value smaller than 1 when the image is enlarged in the horizontal direction after correction. On the contrary, when the image is reduced, the value is larger than 1, and when the image is scaled, the value is 1.

  In step S9, the data interpolation calculation unit 26 stores the obtained image data in the image memory 602 including a vertical processing line buffer. In step S10, the interpolation phase / input data coordinate calculation unit 23 determines whether image data for one line, that is, the number of output horizontal pixels HS has been output to the image memory 602 based on the current x coordinate, If it is determined that one line of data has been output, the process proceeds to step S11. If it is determined that one line of data has not been output, the process proceeds to step S13.

  In step S11, the interpolation phase / input data coordinate calculation unit 23 sets the x coordinate to Sx and adds 1 to the y coordinate. In step S12, the interpolation phase / input data coordinate calculation unit 23 further determines whether one frame of image data, that is, the number of output vertical lines, has been output to the image memory 602 based on the y coordinate. The operation is terminated when it is determined that the amount of data has been output, and the processing proceeds to step S13 when it is determined that the data for one frame has not been output.

  As described above, the horizontal one-dimensional interpolation unit 501 performs horizontal image distortion correction processing and horizontal enlargement / reduction processing at the same time by performing horizontal one-dimensional interpolation operation on the original image having distortion, The obtained image is stored in the image memory 602 for vertical processing.

  A specific example of the equal magnification conversion by the horizontal one-dimensional interpolation is shown in FIG. 8 shows conversion relating to the luminance signal, FIG. 8A shows the interpolation data D0 to D9 input to the signal processing unit 10, and FIGS. 8B and 8F show the correction amount parameter Xm. FIG. 8C and FIG. 8D show sampling positions and numbers of data constituting the corrected image, respectively.

  8E shows the x coordinate (xt) supplied from the interpolation phase / input data coordinate calculation unit 23 to the image distortion correction parameter decoder 33. FIG. 8G is generated by the data acquisition unit 24. The x coordinate (correction parameter) of the correction vector, FIG. 8H shows the address of the interpolation data in the image before correction, and FIG. 8I shows the interpolation phase.

  For example, as shown in FIG. 8, the correction amount parameter Xm of the data located at the point where the x coordinate is 2.0 in the corrected image is 1.25. As a result, the x coordinate of the corresponding point of the point in the image before correction is obtained as 3.25 by adding the correction amount parameter Xm to 2.0. At this time, the integer component (3) of the x coordinate (3.25) indicates the address of the data in the image before correction, and 0.25 indicates the interpolation phase. Therefore, the luminance signal at the point where the x coordinate is 2.0 in the image after correction is a one-dimensional interpolation in which the phase of the horizontal interpolation filter is set to 0.25 for a plurality of continuous data having three adjacent x addresses in the image before correction. It will be obtained by calculation.

  FIG. 9 is a timing chart showing the operation timing of the equal magnification conversion shown in FIG. Here, FIG. 9A shows an internal synchronization signal supplied to the arithmetic control unit 22, FIG. 9B shows a control timing signal generated by the arithmetic control unit 22, and FIG. 9C shows a data acquisition unit. FIG. 9D shows data for interpolation input from the image memory 601 to the data acquisition unit 24, and FIG. 9E shows the interpolation phase / input data coordinate calculation unit 23. The x-coordinates (xt) supplied to the image distortion correction parameter decoder 33 are respectively shown.

  FIG. 9F shows the correction amount parameter Xm output from the image distortion correction parameter decoder 33, FIG. 9G shows the correction parameter generated by the data acquisition unit 24, and FIG. 9 (i) is the interpolation phase, FIGS. 9 (j) and 9 (k) are the skip signal sk and hold signal Sh generated by the data acquisition unit 24, respectively. l) is 2-tap data read from the image memory 601, FIG. 9 (m) is data output from the data interpolation calculation unit 26 to the image memory 602, and FIG. 9 (n) is internally generated by the data interpolation calculation unit 26. Each output enable signal is shown. In order to simplify the description, it is assumed that 2-tap data shown in FIG. 9L is used in the interpolation calculation for obtaining one data.

  As shown in FIG. 9B, when the control timing signal is activated to a high level according to the internal synchronization signal at time T1, interpolation phase / input data coordinate calculation is performed as shown in FIG. 9E. The unit 23 sequentially supplies the x coordinate (xt) incremented by 0.0 to 1.0 to the image distortion correction parameter decoder 33.

  Accordingly, as shown in FIG. 9F, the image distortion correction parameter decoder 33 obtains the corresponding correction amount parameter Xm, and thereafter, the data acquisition unit 24 calculates the correction parameter shown in FIG. 9G. Here, as shown in FIG. 9H, the data acquisition unit 24 specifies 0 as the leading address of the interpolation data in the image before correction from the integer component of the correction parameter. Then, as shown in FIGS. 9C and 9D, the data acquisition unit 24 supplies the image memory 601 with the address 0 specified as described above, together with the activated read control signal.

  As a result, as shown in FIG. 9D, the image memory 601 sequentially outputs the interpolation data sequentially from the data D0 corresponding to the head address 0 to the data acquisition unit 24.

  9 (g) and 9 (j), when the data acquisition unit 24 determines that the integer component of the correction parameter has increased by 2 or more at time T2, the high-level (H) skip signal sk Is supplied to the interpolation phase / input data coordinate calculation unit 23, the data interpolation calculation unit 26, and the image distortion correction parameter decoder 33. As a result, as shown in FIGS. 9 (e) to 9 (g), the correction parameter generation operation is stopped for one cycle from time T3, and also shown in FIGS. 9 (m) and 9 (n). As described above, when the output enable signal is inactivated to a low level, data output from the data interpolation calculation unit 26 to the image memory 602 is stopped.

  Further, as shown in FIGS. 9 (g) and 9 (k), the data acquisition unit 24 uses the integer component of the correction parameter (8.75) generated at time T4 as the integer of the correction parameter (8.25) one cycle before. It is determined that the component is the same as the component, and the hold signal Sh is activated to a high level at time T4. Thus, as shown in FIG. 9L, at time T5, the data acquisition unit 24 acquires the same 2-tap interpolation data D8 and D9 from the image memory 601 as in the previous cycle.

  10 shows a specific example of enlargement conversion by horizontal one-dimensional interpolation as in FIG. 8, and FIG. 11 shows the operation timing of the enlargement conversion as in FIG. In this example of enlargement conversion, as shown in FIG. 10E, data with data numbers 2 to 6 is enlarged in the horizontal direction with the horizontal enlargement / reduction parameter Ha being 0.5. Here, FIG. 10B shows the correction amount parameter Xm for 10 data with data numbers 0 to 9, and FIG. 10F shows the interpolation point by the enlargement, that is, the x coordinate is 2.0 to 6.5. The correction amount parameter Xm at 10 points at intervals of 0.5 is shown.

  In such enlargement conversion, since the integer component of the correction parameter does not change at times T2, T3, T4, T5, and T6 as shown in FIG. 11 (g), the hold signal for one cycle at each time. Sh is activated to a high level.

  Next, the operation of the vertical one-dimensional interpolation unit 502 shown in FIG. 2 will be described in detail with reference to the flowcharts of FIGS. First, the arithmetic control unit 27 generates a control timing signal in accordance with the internal synchronization signal supplied from the synchronization signal generation unit 800. The interpolation phase / input data coordinate calculation unit 28 operates in accordance with the control timing signal supplied from the calculation control unit 27 and performs interpolation in the coordinate system when the image input to the signal processing unit 10 is not distorted. Calculate point coordinates with a decimal point.

  Specifically, in step S1, the interpolation phase / input data coordinate calculation unit 28 cuts out the image as coordinates (x, y) on the image subjected to distortion correction and equal magnification conversion as shown in FIG. The upper left coordinates (Sx, Sy) of the image CI are initialized, and a correction parameter request signal Ry is supplied to the image distortion correction parameter decoder 34. On the other hand, in step S 2, the image distortion correction parameter decoder 34 obtains a correction amount parameter Ym corresponding to the y coordinate in accordance with the supplied correction parameter request signal Ry, and supplies it to the data acquisition unit 29 and the interpolation coefficient generation unit 30. .

  Here, the image distortion correction parameter decoder 34 may include, for example, a ROM (Read Only Memory), and a comparison table between the y coordinate and the correction amount parameter Ym may be stored in advance in the ROM. The correction amount parameter Ym may be approximated as a function having a y coordinate, and the correction amount parameter Ym may be obtained using the function, which will be described in detail later.

  Next, in step S3, the data acquisition unit 29 corresponds to the correction amount parameter Ym supplied from the image distortion correction parameter decoder 34 to the coordinates (X, Y) supplied from the interpolation phase / input data coordinate calculation unit 28. A correction amount vector (0, Ym) is added. As a result, as shown in FIG. 14B, the coordinates (X, Y + Ym) of the point corresponding to the coordinates (X, Y) in the original image OI before correction, that is, the correction vector is obtained. At this time, the data acquisition unit 29 generates an address of data to be read from the image memory 602 according to the integer value of the y component (Y + Ym) of the generated correction vector, and supplies it to the image memory 602 together with a memory control signal.

  Instead of the data acquisition unit 29, the image distortion correction parameter decoder 34 obtains the correction vector in accordance with the y coordinate supplied from the interpolation phase / input data coordinate calculation unit 28, and the correction vector is obtained as the data acquisition unit 29 or the like. You may make it supply to.

  In step S 4, a plurality of interpolation data arranged in a plurality of lines in the vertical direction at the coordinate X are simultaneously output to the data acquisition unit 29 in accordance with the address supplied to the image memory 602 for vertical processing. .

  Here, the image memory 602 receives the head address to start reading from the data acquisition unit 29, and sequentially outputs the interpolation data corresponding to the address by incrementing the address by 1, or increments the address. Without this, a predetermined number of continuous data are output from the received head address. Here, the head address is obtained from the integer component of the correction amount parameter Ym output from the image distortion correction parameter decoder 34.

  On the other hand, the interpolation coefficient generation unit 30 treats the decimal component of the correction amount parameter Ym supplied from the image distortion correction parameter decoder 34 as the phase of the vertical interpolation filter, and generates an interpolation coefficient according to the decimal component.

  In step S5, the data interpolation calculation unit 31 executes a one-dimensional interpolation calculation according to the interpolation data supplied from the data acquisition unit 29 and the interpolation coefficient. The interpolation calculation is not applied only when the image 102 input to the signal processing unit 10 is in the RGB format. That is, in the case of the YUV format, when the data density in the vertical direction of the luminance signal and the color difference signal is the same, the filter phase of the luminance signal can be used as the filter phase of the color difference signal, and when the data density is different, the correction amount parameter The filter phase of the color difference signal is calculated by using not only the decimal component of Ym but also the integer component.

  In the YUV format, as shown in FIG. 14C, luminance data Dt of 8 pixels, for example, in the vertical direction from the vicinity of the correction vector (X, Y + Ym) is used as interpolation data, and the decimal component is used as the interpolation data. An 8-tap interpolation operation with the phase is executed. Note that the result obtained by the interpolation calculation is used as luminance data and color difference data of the output image, thereby correcting vertical distortion.

  Next, in step S6, the output data buffer 32 outputs the image data obtained by the interpolation calculation. In step S7, the interpolation phase / input data coordinate calculation unit 23 determines whether image data for one line, that is, the output horizontal pixel number HS has been output based on the current x coordinate, and data for one line. If it is determined that the data is output, the process proceeds to step S8. If it is determined that the data for one line is not output, the process proceeds to step S10.

  In step S8, the interpolation phase / input data coordinate calculation unit 28 sets the x coordinate to Sx, and adds the enlargement / reduction parameter Va in the vertical direction to the y coordinate. On the other hand, in step S10, the horizontal enlargement / reduction parameter Ha is added to the x coordinate, and the process returns to step S2. The enlargement / reduction parameter Va is determined by a ratio of lengths of the original image having distortion to the corrected image in the vertical direction, and is set to a value smaller than 1 when the image is enlarged in the vertical direction after correction. On the contrary, when the image is reduced, the value is larger than 1, and when the image is scaled, the value is 1.

  In step S9, the interpolation phase / input data coordinate calculation unit 28 further determines whether image data for one frame, ie, the number of vertical lines (number of vertical pixels) has been output from the output data buffer 32 based on the y coordinate. If it is determined that one frame of data has been output, the operation is terminated. If it is determined that one frame of data has not been output, the process proceeds to step S10.

  Note that the one-dimensional interpolation in the vertical direction as described above does not involve the interpolation of the data in the horizontal direction and the enlargement / reduction of the image. Therefore, in the horizontal scanning shown in FIG. Is repeated. However, when the correction amount parameter Ym is large, it may take a long time to read appropriate interpolation data depending on the storage location in the image memory 602. In such a case, the data acquisition unit 29 supplies the standby signal WT that has been activated to the interpolation phase / input data coordinate calculation unit 28 and the image distortion correction parameter decoder 34, and the interpolation phase / The operations of the input data coordinate calculation unit 28 and the image distortion correction parameter decoder 34 are interrupted.

  As described above, the vertical one-dimensional interpolation unit 502 simultaneously performs vertical image distortion correction processing and vertical enlargement / reduction processing by performing vertical one-dimensional interpolation operation on the original image having distortion, Generate and output a fully distorted image.

  A specific example of the equal magnification conversion by the vertical one-dimensional interpolation is shown in FIG. Here, FIG. 15 is a graph showing conversion relating to the luminance signal, where the horizontal axis represents the x coordinate and the vertical axis represents the corrected y coordinate (Y + Ym).

  In FIG. 15, 10 points with a y coordinate of 0 and an x coordinate of 0.0 to 10.0 indicate points on the corrected image, and the arrows indicate the correction amount up to the point on the original image corresponding to each point. Parameter Ym is shown. That is, for example, the point at the coordinate (1.0, 0) in the image after correction corresponds to the point at the coordinate (1.0, 7.1) in the original image before correction, the correction amount parameter is 7.1, and the interpolation phase is the fractional component 0.1. Is done.

  Next, the preprocessing device 3 and the correction parameter decoder 9 shown in FIG. 1 will be described in detail. First, the outline of the operations of the preprocessing device 3 and the correction parameter decoder 9 will be described with reference to the flowchart shown in FIG.

  As shown in FIG. 16, in step S <b> 1, the correction parameter encoder 5 reads correction amount vectors for all pixel points from the correction parameter deriving unit 6. Next, as shown in step S2, the correction parameter encoder 5 determines a grid line for dividing the correction amount vector of all the pixel points for each section. The determination of the lattice lines will be described in detail later.

  In step S3, the correction parameter encoder 5 compresses the correction amount vector of each section divided by the grid lines and supplies it as compressed data Pc to the correction parameter decoder 9. In step S4, the image sensor 300 captures an image. . The compression of the correction amount vector will be described in detail later.

  In step S5, the data converter 400 converts the analog image signal generated by the imaging into a digital image signal. In step S6, the correction parameter decoder 9 determines a grid necessary for reading the correction amount parameter to the signal processing unit 10, and normalizes the coordinates supplied from the signal processing unit 10 in step S7 according to the grid.

  In step S8, the correction parameter decoder 9 decodes the compressed data Pc supplied from the correction parameter encoder 5 using the lattice, and supplies the obtained correction amount parameter to the signal processing unit 10. In step S9, the signal processing unit 10 performs an interpolation operation on the original image using the correction amount parameter. Here, in step S10, the control microcomputer 8 determines whether or not the input of the original image to the signal processing unit 10 is to be ended. If it is determined that the input is to be ended, the operation of the image processing apparatus 2 is ended and the input is performed. If it is determined not to end the process, the process returns to step S4.

  FIG. 17 is a block diagram showing a configuration of the correction parameter encoder 5 shown in FIG. As shown in FIG. 17, the correction parameter encoder 5 includes a lattice division unit 11 and a parameter compression unit 12. Here, the lattice division unit 11 is connected to the user interface, and the parameter compression unit 12 is connected to the lattice division unit 11 and the correction parameter derivation unit 6. Hereinafter, the operation of the correction parameter encoder 5 will be described in detail with reference to FIGS.

  First, the lattice division unit 11 determines lattice lines for dividing the image 102 obtained by the data conversion unit 400 into a plurality of regions. Then, the parameter compression unit 12 compresses the correction amount vector of the image using the grid points for each area divided by such grid lines, and supplies the obtained compressed data Pc to the correction parameter decoder 9.

  According to such a method, the number of correction amount vectors to be held by the correction parameter decoder 9 can be reduced, and correction vectors in the x and y directions can be obtained as in the case of holding correction point vectors for all points. The calculation can be performed separately, and a high-speed interpolation calculation can be realized.

  Hereinafter, the lattice division operation by the lattice division unit 11 shown in FIG. 17 will be described. Actually, the distortion of the image 102 generated by the data conversion unit 400 occurs point-symmetrically with respect to the center (origin), and therefore, ¼ of the image 102 as shown in FIG. Only a region, for example, the first quadrant Q1 is sufficient as a target region for lattice division.

  That is, since the distortion is determined by the distance from the center, the image processing in the first quadrant Q1 can be applied as it is to the image processing in the other quadrants by reversing the signs of the x coordinate and / or the y coordinate. .

  The grid division determination method includes a method of dividing a predetermined region equally in the x and y directions (equal division), a method of dividing each grid so that the width of each grid is a power of 2 (power division), and an optimum There is a method of dividing at the dividing position (optimum division).

  Here, the grid division unit 11 receives a signal Lw for designating a grid division method and a signal Ln for designating the number of grid divisions from the user interface, and as shown in FIG. 50 is used to divide the image 102 into a designated number of divisions.

  At this time, only the correction amount vector at the lattice point obtained by the lattice division as described above, that is, the correction amount vector for each direction (1 / grid width) is used for the interpolation calculation. In the power division, since the grid width is set to a power of 2, calculation of the correction amount vector at each grid point is facilitated, so that the circuit scale can be reduced.

  Hereinafter, the optimum dividing method executed by the lattice dividing unit 11 will be described with reference to the flowcharts of FIGS. 19 and 20.

  In step S1, the scanning direction in image processing is first determined as the x direction. Next, in step S2, a correction amount parameter for one line L1 at the upper end of the screen shown in FIG. 18A is acquired, and the x dependency of the correction amount parameter is examined. For example, FIG. 21A shows an example of x dependency of the correction amount parameter Xm (x) when the reference point is x = 0.

  In step S3, a target point is set two pixels to the right of the reference point (origin), and all points between the reference point and the target point (one section) are quadratic polynomials (hereinafter referred to as “section quadratic polynomials”). ”).

  At this time, if the condition that the difference (also referred to as cost) between the value of the correction amount parameter Xm (x) and the correction amount obtained by the quadratic polynomial is smaller than a predetermined value in the section, the target point is determined. Further, the cost calculation is repeated by shifting one pixel to the right. In this way, the maximum point satisfying the above condition is searched (rightward search).

  In step S4, the reference point is shifted to the target point, and a right direction search in the next section is executed. By such a method, for example, the points X1, X2, and X3 shown in FIG. 21B are sequentially determined, and the correction amount parameter Xm (x) as a function of x is approximated by a quadratic polynomial for each section. Is done.

  In step S5, it is determined whether or not the target point is the right end. If it is determined that the target point is the right end, the process proceeds to step S6. If it is determined that the target point is not the right end, the process returns to step S3.

  In step S6, the right end data is used as a reference point, a target point is set to the left of two pixels of the reference point, and a left direction search is executed in the same manner as the right direction search. Then, after a certain segment is determined by the cost calculation, the reference point is shifted to the target point in step S7, and a leftward search in the next segment is executed. By such a method, for example, the points X5 and X4 shown in FIG. 21C are sequentially determined, and the correction amount parameter Xm (x) as a function of x is approximated by a quadratic polynomial for each section. .

  In step S8, it is determined whether or not the target point is the left end. If it is determined that the target point is the left end, the process proceeds to step S9. If it is determined that the target point is not the left end, the process returns to step S6.

  Next, in step S9, as shown in FIG. 21 (d), the points obtained by the above-mentioned right direction search are compared with the points obtained by the left direction search, so that the total cost is minimized. Find the position (optimum point). Here, for example, as shown in FIG. 21D, the point X6 is determined by comparing the point X4 and the point X1, and the point X7 is determined by comparing the point X5 and the point X2.

  In step S10, it is determined whether or not the search direction of the division position is the x direction. If it is determined that it is the x direction, the process proceeds to step S11, and if it is determined that it is the y direction instead of the x direction. Ends the operation.

  In step S11, a correction amount parameter for one line at the right end of the division target region is acquired, the y dependency of the correction amount parameter is examined, and the process returns to step S3. A function in which the reference point is y = 0, the horizontal axis is the y coordinate, and the vertical axis is the correction amount parameter Xm (y) is shown in the same manner as in FIG. The search operation is executed at the same time. In this way, the lattice division unit 11 determines the division positions in the x direction and the y direction to determine the lattice 50. The determined lattice position is supplied to the parameter compression unit 12 as lattice information Li.

  The parameter compressing unit 12 illustrated in FIG. 17 holds only the correction amount vector at each lattice point according to the lattice information Li supplied from the lattice dividing unit 11. And the parameter compression part 12 determines the line segment L2 which comprises the grating | lattice 50 as a process target, as FIG. 22 (a) shows. Here, for example, when the x-coordinates at both ends of the line segment L2 are X0 and X2, and the correction amount parameters at these ends are Xm0 and Xm2, respectively, the relationship between the x-coordinate and the correction amount parameter at each point on the line segment L2. For example, as shown in FIG. At this time, coefficients Ca, Cb, and Cc satisfying the following expression (1) are calculated by setting the x coordinate of the pixel on the line segment L2 to X1 and the correction amount parameter to Xm1.

なお、図２２（ｂ）及び図２２（ｃ）に示されるように、Ｘ１はＸ０からＸ２まで順次１だけインクリメントされ、線分Ｌ２上の各点において補正量パラメータの大きさと該区分２次多項式による近似値との差が順次比較される。そして、該差が最小となる点のｘ座標及び補正量パラメータが、それぞれＸ１及びＸｍ１として上記式（１）に代入される。

As shown in FIGS. 22B and 22C, X1 is sequentially incremented by 1 from X0 to X2, and the magnitude of the correction amount parameter and the piecewise quadratic polynomial at each point on the line segment L2. The difference from the approximate value is sequentially compared. Then, the x coordinate and the correction amount parameter of the point where the difference becomes the minimum are substituted into the above equation (1) as X1 and Xm1, respectively.

  Here, the parameter compression unit 12 calculates and holds the coefficients Ca, Cb, Cc for all the line segments forming the lattice 50, and supplies these coefficients Ca, Cb, Cc to the correction parameter decoder 9 as compressed data Pc. .

  FIG. 23 is a block diagram showing a configuration of the image distortion correction parameter decoder 33 for the x direction shown in FIG. As shown in FIG. 23, the image distortion correction parameter decoder 33 includes a distortion parameter buffer 61, a lattice determination unit 62, a normalization unit 63, a function conversion unit 64, and a plane interpolation unit 65.

  Here, the distortion parameter buffer 61 is connected to the control microcomputer 8 and the correction parameter encoder 5, and the lattice determination unit 62, the normalization unit 63, and the function conversion unit 64 are all connected to the distortion parameter buffer 61. Further, the lattice determination unit 62 is connected to the signal processing unit 10, and the normalization unit 63 is connected to the lattice determination unit 62. The function conversion unit 64 is connected to the normalization unit 63, and the plane interpolation unit 65 is connected to the function conversion unit 64. The signal processing unit 10 is connected to the plane interpolation unit 65.

  The image distortion correction parameter decoder 33 having the above configuration decodes the compressed data Pc supplied from the correction parameter encoder 5 to restore the correction amount parameter in the x direction at each point on the screen. This will be described in detail below.

  The image distortion correction parameter decoder 34 for y direction shown in FIG. 2 has the same configuration as the image distortion correction parameter decoder 33 for x direction, and operates in the same manner as the image distortion correction parameter decoder 33. .

  First, the distortion parameter buffer 61 receives the compressed data Pc, the lattice position information Lp indicating the position of the lattice corresponding to the compressed data Pc, and lattice constant information Lc consisting of the reciprocal of the width of the lattice from the correction parameter encoder 5. And a command signal Cd is input from the control microcomputer 8.

  The lattice determination unit 62 receives the x-coordinate (xt) and y-coordinate (yt) of the point for which a corrected image is to be obtained from the signal processing unit 10 together with the correction parameter request signal Rx, and determines a lattice frame including the point. . Here, the lattice determination unit 62 determines the lattice frame by comparing the supplied coordinates (xt, yt) with the lattice information LI supplied from the distortion parameter buffer 61.

  Next, the normalizing unit 63 normalizes the coordinates (xt, yt) by the following equation (2) in order to execute a predetermined interpolation calculation within the range of the lattice frame determined by the lattice determining unit 62. However, here, as shown in FIG. 24A, the coordinates of the four corners of the lattice frame including the coordinates (xt, yt) are (X0, Y0), (X0, Y2), (X2, Y0) and (X2). , Y2).

なお、上記式（２）における１／（Ｘ２−Ｘ０）及び１／（Ｙ２−Ｙ０）の値は補正パラメータエンコーダ５に含まれた格子分割部１１において算出され、正規化部６３は歪みパラメータバッファ６１から該値を格子定数情報Ｌｃとして受領する。これより、正規化部６３において上記値を用いた乗算を実行することによって、座標（ｐｘ，ｐｙ）が算出される。

Note that the values of 1 / (X2−X0) and 1 / (Y2−Y0) in the above equation (2) are calculated by the lattice division unit 11 included in the correction parameter encoder 5, and the normalization unit 63 stores the distortion parameter buffer. The value is received from 61 as lattice constant information Lc. Thus, the coordinates (px, py) are calculated by executing multiplication using the above values in the normalizing unit 63.

  As shown in FIG. 24B, the function conversion unit 64 uses the correction amount parameters f (x) and g (x), m as functions of x or y in the lattice frame including the coordinates (xt, yt). (Y) and n (y) are obtained. The function converter 64 receives the coefficients Ca, Cb, and Cc in each of the four functions from the distortion parameter buffer 61 as coefficient information CL.

  The function conversion unit 64 obtains the correction amount parameter for the coordinates (xt, yt) using the four functions. In order to ensure the continuity of the function in the x direction and the y direction, the four functions f , G, m, n are converted into approximate functions F, G, M, N in consideration of weighting as shown in the following equation (3), for example. In Equation (3), fa, fb, and fc indicate coefficients corresponding to the coefficients Ca, Cb, and Cc in the function f. Similarly, ga, gb, and gc are functions of the function g, and ma, mb, and mc are functions. m, na, nb, and nc represent the coefficients of the function n, respectively.

なお、関数変換部６４は正規化部６３より供給された座標（ｐｘ，ｐｙ）を、そのまま平面補間部６５へ供給する。

The function conversion unit 64 supplies the coordinates (px, py) supplied from the normalization unit 63 to the plane interpolation unit 65 as they are.

  Then, the plane interpolation unit 65 uses the functions F, G, M, and N obtained by the function conversion unit 64 and the information indicating the coordinates (px, py) to express the coordinates ( A correction amount parameter Xm at xt, yt) is calculated.

平面補間部６５は、このような方法により算出された補正量パラメータＸｍを、該パラメータの算出動作が終了したことを示すイネーブル信号ＥＮと共に信号処理部１０へ供給する。なお、ｙ方向用の画歪補正パラメータデコーダ３４は、上記と同様な方法により補正量パラメータＹｍを算出し、イネーブル信号ＥＮと共に信号処理部１０へ供給する。

The plane interpolation unit 65 supplies the correction amount parameter Xm calculated by such a method to the signal processing unit 10 together with the enable signal EN indicating that the parameter calculation operation has been completed. The image distortion correction parameter decoder 34 for y direction calculates the correction amount parameter Ym by the same method as described above, and supplies it to the signal processing unit 10 together with the enable signal EN.

  The functions f, g, m, and n forming the lattice frame may be approximated by an nth order polynomial (n is a natural number) in addition to being approximated by a piecewise second order polynomial as described above.

  FIG. 25 is a diagram illustrating a configuration of the image memory 602, the data acquisition unit 29, and the data interpolation calculation unit 31 illustrated in FIG. FIG. 25 shows a configuration in the case where the image processing apparatus 2 generates image data of each pixel by interpolation calculation using 16 pixel image data of (4 × 4) taps.

  As shown in FIG. 25, the image memory 602 includes a selector 67 and five memories that are one larger than the number of vertical taps, that is, an A memory 71 and a B memory 72, a C memory 73, a D memory 74, and an E memory 75. The data acquisition unit 29 includes a control unit 80, an A buffer 81, a B buffer 82, a C buffer 83, a D buffer 84, an E buffer 85, a cycle division unit 560, and selectors 96 to 99. Note that the cycle division unit 560 includes selectors 91 to 95.

  Here, the data acquisition unit 29 includes five buffers (A buffer 81 to E buffer 85) that are one more than the number of vertical taps as described above and the corresponding five selectors 91 to 95, and the number of vertical taps 4 Two selectors 96 to 99 are included.

  The data interpolation calculation unit 31 includes four registers 901, a multiplication circuit 902, and an addition circuit 43.

  In the above, the selector 67 is connected to the data interpolation calculation unit 26 and the control unit 80, and the A memory 71 and the B memory 72, the C memory 73, the D memory 74, and the E memory 75 are connected to the selector 67.

  The control unit 80 is connected to the image distortion correction parameter decoder 34, the A buffer 81 is connected to the A memory 71, and the B buffer 82 is connected to the B memory 72. Similarly, the C buffer 83 is connected to the C memory 73, the D buffer 84 is connected to the D memory 74, and the E buffer 85 is connected to the E memory 75.

  The selector 91 is connected to the A buffer 81, the selector 92 is connected to the B buffer 82, and the selector 93 is connected to the C buffer 83. Similarly, the selector 94 is connected to the D buffer 84 and the selector 95 is connected to the E buffer 85. The selectors 96 to 99 are connected to the five selectors 91 to 95, respectively. The selectors 91 to 99 are controlled by the control unit 80, respectively.

  A register 901 is connected to each of the selectors 96 to 99, and a multiplication circuit 902 is connected to each register 901. The four multiplier circuits 902 are connected to one adder circuit 43.

  Here, as described above, the data subjected to the horizontal interpolation processing by the data interpolation calculation unit 26 is written into the image memory 602, and at the same time, the data obtained from the image memory 602 by the data acquisition unit 29 is vertically interpolated. Since the processing is performed, the image distortion correction is executed without causing a frame delay as a processing waiting time.

  Hereinafter, operations of the image memory 602, the data acquisition unit 29, and the data interpolation calculation unit 31 illustrated in FIG. 25 will be described in detail. First, the data subjected to the horizontal interpolation processing is sequentially supplied from the data interpolation calculation unit 26 to the selector 67, and the data is transferred from the A memory 71 to the E memory 75 by the selector 67 controlled by the control unit 80. It is sorted and stored in one memory.

  The data stored in the A memory 71 is supplied to the selector 91 via the A buffer 81, and the data stored in the B memory 72 is supplied to the selector 92 via the B buffer 82. Similarly, data stored in the C memory 73 is supplied to the selector 93 via the C buffer 83, and data stored in the D memory 74 is supplied to the selector 94 via the D buffer 84 and stored in the E memory 75. The processed data is supplied to the selector 95 via the E buffer 85.

  Here, each of the selectors 91 to 95 included in the cycle dividing unit 560 divides data read from the A buffer 81 to the E buffer 85 in units of, for example, two pixels in accordance with control by the control unit 80. Data for one pixel is supplied to the selectors 96 to 99 for each cycle.

  The selectors 96 to 99 selectively output the data supplied from the selectors 91 to 95 to the register 901 under the control of the control unit 80. As a result, four data, which is the number of taps necessary for the vertical interpolation process, are selectively supplied to the data interpolation calculation unit 31.

  Further, each data stored in the register 901 is multiplied with the interpolation coefficients C0 to C3 in each multiplication circuit 902, and the four products are added by the addition circuit 43, whereby an interpolation operation in the vertical direction is performed. Is supplied to the output data buffer 32.

  Here, the operation of the image processing apparatus 2 according to the embodiment of the present invention will be described with reference to FIG. In FIGS. 26A to 26D, image data for one frame is shown.

  First, as shown in FIG. 26A, when image data is input to the signal processing unit 10 from time T1, the horizontal one-dimensional interpolation unit 501 performs horizontal interpolation processing from time T2. Then, as shown in FIG. 26C, the image subjected to the horizontal interpolation process is sequentially written from the A memory 71 included in the image memory 602 to the E memory 75 after time T2. Here, for example, data for vertical processing is read from the image memory 602 to the data acquisition unit 29 in an odd cycle, and data subjected to horizontal processing from the data interpolation calculation unit 26 is written to the image memory 602 in an even cycle. A distortion correction process with a two-cycle period is executed.

  At this time, as shown in FIG. 27, from time T3 when the data Dmx corresponding to the number of lines corresponding to the maximum amount of distortion in the vertical direction in the maximum distortion curve 104 of the horizontal line in the image 102 is stored in the image memory 602, each time Interpolation operations in the vertical direction of the line are sequentially executed. Accordingly, the delay time in the interpolation calculation is from time T1 to time T3, and it is not necessary to use the time (frame delay) in which the horizontal interpolation processing is performed on the data for one frame as a waiting time. Image distortion correction can be executed.

  Also, as a whole, the image memory 602 is the number obtained by adding the number of lines corresponding to the maximum distortion amount and the number of vertical processing taps (for example, 4 taps) in the vertical direction, and is input to the signal processing unit 10 in the horizontal direction. It has a memory capacity for storing data for the number of pixels in the horizontal direction of the image. Note that the five memories from A memory 71 to E memory 75 shown in FIG. 25 have the same capacity, for example, and the port width of each memory is, for example, 32 bits.

  Here, FIG. 28 shows a method for storing the data in the area 102P of the image 102 in the image memory 602. In FIG. 28, “A” to “E” mean “A memory” 71 to “E memory” 75 shown in FIG. Further, when the port width of each memory is 32 bits and the data for one pixel consists of 16 bits including the Y signal (luminance information) and the C signal (color difference information) as described above, the selector 67 has two pixels. The data is sequentially stored from the A memory 71 to the E memory 75 in units of minutes.

  That is, as shown in FIG. 28, the selector 67 first stores the data from the 0th to 23rd pixels of the 0th line in the A memory 71, and then stores the data from the 0th to 23rd pixels of the 1st line to the Bth line. Store in the memory 72. Similarly, the selector 67 stores data from the 0th pixel to the 23rd pixel on the second line in the C memory 73, stores data from the 0th pixel to the 23rd pixel on the third line in the D memory 74, and stores the data on the fourth line. Are stored in the E memory 75. The selector 67 stores the data of each line sequentially from the A memory 71 to the E memory 75 for each line in the same manner.

  Hereinafter, the reason why the number of buffers one more than the number of vertical taps is required for the data acquisition unit 29 will be described. When the image 102 having distortion is viewed locally, as shown in patterns 1 to 3 in FIGS. 29A to 29C, the image data is in the vertical direction between two pixels adjacent in the horizontal direction. It has never moved more than two pixels.

  That is, as shown in the pattern 1 of FIG. 29A, the image data does not move in the vertical direction between pixels adjacent in the horizontal direction, or the pattern 2 of FIG. 29B and FIG. Although it moves by one pixel in the vertical direction as shown in pattern 3, as shown in FIGS. 30 (a) and 30 (b), the image data is 2 in the vertical direction between adjacent pixels in the horizontal direction. It does not move more than the amount of pixels.

  Here, in the vertical 4-tap processing, as shown in FIG. 31, filtering processing is executed using data of four pixels including three peripheral pixels Ip adjacent in the vertical direction to the central pixel Ic. .

  At this time, as shown in FIG. 32, the five memories A memory 71 to E memory 75 included in the image memory 602 each have, for example, a 32-bit port. By accessing, 16-bit image data for two pixels is output through each port.

  That is, as shown in FIG. 32, image data Ia0 and Ia1 each consisting of 16 bits are read from the A memory 71 in units of two pixels by one access, and an image consisting of 16 bits each from the B memory 72. Data Ib0 and Ib1 are read in units of two pixels, and image data Ic0 and Ic1 each having 16 bits are read from the C memory 73 in units of two pixels. Similarly, image data Id0 and Id1 each having 16 bits are read from the D memory 74 in units of two pixels, and image data Ie0 and Ie1 each having 16 bits are read from the E memory 75 in units of two pixels. .

  In this way, by reading out the image data for two adjacent pixels by one more than the number of taps in the vertical direction, the change between the adjacent pixels changes from pattern 1 shown in FIGS. 29 (a) to 29 (c). In any case up to 3, the same processing is executed in each column. That is, for example, as shown in the shaded area in FIG. 32, filtering processing is performed on image data in four pixels arranged in the vertical direction from the pixel one pixel above the center pixel Ic to the pixel two pixels below in each column. As a result, image data of two pixels adjacent in the horizontal direction is generated.

  Note that the control unit 80 identifies in advance which of the patterns 1 to 3 shown in FIGS. 29 (a) to 29 (c) corresponds to the change between adjacent pixels.

  More specifically, the control unit 80 receives the y coordinates of two central pixels Ic in two columns adjacent in the horizontal direction from the image distortion correction parameter decoder 34, and controls the selectors 96 to 99 according to the difference between the y coordinates. As a result, the image data indicated by the hatched portion in FIG. 32 is selectively supplied to the data interpolation calculation unit 31 as a filtering processing target.

  In the above description, the 4-tap filtering process has been described as an example. However, in the image processing method according to the embodiment of the present invention, the image memory 602, the data acquisition unit 29, and the data interpolation calculation unit 31 correspond to the number of taps. It goes without saying that the present invention can be applied to filtering processing other than four taps by changing the data input / output cycle to the image memory 602.

  As described above, according to the image processing system according to the embodiment of the present invention, the one-dimensional interpolation operation is performed in the horizontal and vertical directions on the captured image with the optical distortion, and the correction vector is efficiently used. Therefore, distortion correction for not only a still image but also a moving image that requires real-time processing is realized with a simple configuration, and a high-quality image without distortion can be easily obtained.

  In addition, according to the image processing system according to the embodiment of the present invention, image distortion can be corrected in real time by signal processing, so that the degree of freedom in lens design can be increased, and the lens can be downsized and the lens can be reduced. The cost can be easily reduced.

1 is a block diagram illustrating a configuration of an image processing system according to an embodiment of the present invention. It is a block diagram which shows the structure of the signal processing part shown by FIG. It is a figure explaining the outline | summary of the one-dimensional interpolation calculation performed by the signal processing part shown by FIG. It is a figure which shows the structure of the horizontal processing circuit contained in the data interpolation calculation part shown by FIG. FIG. 3 is a first flowchart showing an operation of the horizontal one-dimensional interpolation unit shown in FIG. 2. FIG. FIG. 3 is a second flowchart showing the operation of the horizontal one-dimensional interpolation unit shown in FIG. 2. It is a figure explaining the operation | movement shown by FIG.5 and FIG.6. It is a figure which shows an example of the equal magnification conversion in horizontal one-dimensional interpolation. FIG. 9 is a timing chart showing the operation timing of equal magnification conversion shown in FIG. 8. It is a figure which shows an example of the horizontal expansion conversion in horizontal one-dimensional interpolation. FIG. 11 is a timing chart showing the operation timing of horizontal enlargement conversion shown in FIG. 10. 3 is a first flowchart showing an operation of the vertical one-dimensional interpolation unit shown in FIG. 2. FIG. 4 is a second flowchart showing an operation of the vertical one-dimensional interpolation unit shown in FIG. 2. It is a figure explaining the operation | movement shown by FIG.12 and FIG.13. It is a figure which shows an example of the vertical expansion conversion in vertical one-dimensional interpolation. 2 is a flowchart showing an outline of operations of a preprocessing device and a correction parameter decoder shown in FIG. 1. It is a block diagram which shows the structure of the correction parameter encoder shown by FIG. It is a figure explaining the outline | summary of operation | movement of the lattice division part shown by FIG. It is a 1st flowchart which shows the method of optimal division | segmentation. It is a 2nd flowchart which shows the method of optimal division | segmentation. FIG. 21 is a first diagram illustrating the operation shown in FIGS. 19 and 20. FIG. 21 is a second diagram for explaining the operation shown in FIGS. 19 and 20. FIG. 3 is a block diagram showing a configuration of an image distortion correction parameter decoder for x direction shown in FIG. 2. FIG. 24 is a diagram for explaining the operation of the correction parameter decoder shown in FIG. 23. It is a figure which shows the structure of the image memory shown by FIG. 2, a data acquisition part, and a data interpolation calculation part. It is a timing diagram which shows the timing of a horizontal one-dimensional interpolation process and a vertical one-dimensional interpolation process. It is a figure explaining memory capacity required in order to perform horizontal one-dimensional interpolation and vertical one-dimensional interpolation. It is a figure explaining the data storage method to the image memory shown by FIG. It is a figure which shows the change pattern which the vertical direction can take in an adjacent pixel. It is a figure which shows the change pattern which cannot be the vertical direction in an adjacent pixel. It is a figure explaining a vertical 4 tap process. FIG. 26 is a diagram illustrating a method of reading data from the image memory illustrated in FIG. 25. It is a block diagram which shows the structure of the conventional image processing apparatus. It is a flowchart which shows the outline | summary of operation | movement of the image processing apparatus shown by FIG. It is a block diagram which shows the structure of the signal processing part shown by FIG. It is a figure which shows the principle of the image conversion by two-dimensional interpolation. FIG. 36 is a block diagram showing a configuration of a data interpolation calculation unit shown in FIG. 35.

Explanation of symbols

2,100 image processing device, 3 preprocessing device, 5 correction parameter encoder, 6 correction parameter derivation unit, 7,600, 601,602 image memory, 8,700 control microcomputer, 9 correction parameter decoder, 10,500 signal processing unit , 11 Grid division unit, 12 parameter compression unit, 21,570 data writing unit, 22, 27 calculation control unit, 23, 28, 520 interpolation phase / input data coordinate calculation unit, 24, 29, 530 data acquisition unit, 25, 30, 540 interpolation coefficient generation unit, 26, 31, 550 data interpolation calculation unit, 32, 560 output data buffer, 33, 34 image distortion correction parameter decoder, 40 horizontal processing circuit, 42, 43, 903, 904 addition circuit, 61 Distortion parameter buffer, 62 lattice determination unit, 63 normalization unit, 64 function conversion unit, 65 plane Interpolation unit, 67, 91-98 selector, 71 A memory, 72 B memory, 73 C memory, 74 D memory, 75 E memory, 80 control unit, 81 A buffer, 82 B buffer, 83 C buffer, 84 D buffer, 85 E buffer, 200 lens, 300 image sensor, 400 data conversion unit, 501 horizontal one-dimensional interpolation unit, 502
Vertical one-dimensional interpolation unit, 510 timing control unit, 800 synchronization signal generation unit, 900 line memory, 901 register, 902 multiplication circuit, 905 division circuit, 1010 correction data table, 1100 recording unit, 1200 playback unit, 1300 display system processing unit 1400 media.

Claims (20)

An image processing apparatus for correcting an original image having distortion,
Horizontal correction means for correcting distortion in the horizontal direction of the original image by performing an interpolation operation on the original image using a horizontal correction parameter indicating a horizontal correction amount at pixel points constituting the original image;
The image obtained by the correction by the horizontal correction means is subjected to an interpolation operation using a vertical correction parameter indicating a vertical correction amount at a pixel point constituting the original image, so that the original image in the vertical direction is corrected. An image processing apparatus comprising: vertical correction means for correcting distortion. The horizontal correcting means expands and contracts the original image in the horizontal direction by adjusting a horizontal interval between pixel points for which image data is obtained by the interpolation calculation,
The image processing apparatus according to claim 1, wherein the vertical correction unit expands or contracts the original image in a vertical direction by adjusting a vertical interval between pixel points for which image data is obtained by the interpolation calculation. The horizontal correction means includes
First data acquisition means for selectively acquiring image data at the pixel points according to an integer component of the horizontal correction parameter;
First interpolation coefficient generation means for generating an interpolation coefficient in accordance with the decimal component of the horizontal correction parameter;
First interpolation calculation means for executing the interpolation calculation using the image data acquired by the first data acquisition means and the interpolation coefficient generated by the first interpolation coefficient generation means,
The vertical correction means includes
Second data acquisition means for selectively acquiring image data at the pixel points according to an integer component of the vertical correction parameter;
Second interpolation coefficient generating means for generating an interpolation coefficient in accordance with the decimal component of the vertical correction parameter;
The second interpolation calculation means for executing the interpolation calculation using the image data acquired by the second data acquisition means and the interpolation coefficient generated by the second interpolation coefficient generation means. An image processing apparatus according to 1. Storage means for storing a horizontal correction image obtained by the correction by the horizontal correction means,
The vertical correction means includes
Data acquisition means for acquiring the horizontal correction image according to the vertical correction parameter from the storage means;
The image processing apparatus according to claim 1, further comprising: an interpolation calculation unit that performs an interpolation calculation using the vertical correction parameter for the horizontal correction image acquired by the data acquisition unit. The horizontal correction means corrects distortion in the horizontal direction of the original image by performing an interpolation operation,
The image processing apparatus according to claim 1, wherein the vertical correction unit corrects distortion in the vertical direction of the original image by performing an interpolation operation. An image processing apparatus for correcting an original image having distortion,
Vertical correction means for correcting distortion in the vertical direction of the original image by performing an interpolation operation on the original image using a vertical correction parameter indicating a vertical correction amount at a pixel point constituting the original image; ,
The image obtained by the correction by the vertical correction unit is subjected to an interpolation operation using a horizontal correction parameter indicating a horizontal correction amount at a pixel point constituting the original image, thereby causing the original image in the horizontal direction to be corrected. An image processing apparatus comprising: a horizontal correction unit that corrects distortion. The vertical correction means expands and contracts the original image in the vertical direction by adjusting a vertical interval between pixel points for which image data is obtained by the interpolation calculation, and
The image processing apparatus according to claim 6, wherein the horizontal correction unit expands and contracts the original image in a horizontal direction by adjusting a horizontal interval between pixel points for which image data is obtained by the interpolation calculation. The vertical correction means includes
First data acquisition means for selectively acquiring image data at the pixel points according to an integer component of the vertical correction parameter;
First interpolation coefficient generation means for generating an interpolation coefficient in accordance with the decimal component of the vertical correction parameter;
First interpolation calculation means for executing the interpolation calculation using the image data acquired by the first data acquisition means and the interpolation coefficient generated by the first interpolation coefficient generation means,
The horizontal correction means includes
Second data acquisition means for selectively acquiring image data at the pixel points according to an integer component of the horizontal correction parameter;
Second interpolation coefficient generating means for generating an interpolation coefficient according to the decimal component of the horizontal correction parameter;
The second interpolation calculation means for executing the interpolation calculation using the image data acquired by the second data acquisition means and the interpolation coefficient generated by the second interpolation coefficient generation means. An image processing apparatus according to 1. Storage means for storing a vertical correction image obtained by correction by the vertical correction means;
The horizontal correction means includes
Data acquisition means for acquiring the vertical correction image according to the horizontal correction parameter from the storage means;
The image processing apparatus according to claim 6, further comprising: an interpolation calculation unit that performs an interpolation calculation using the horizontal correction parameter on the vertical correction image acquired by the data acquisition unit. The vertical correction means corrects distortion in the vertical direction of the original image by performing an interpolation operation,
The image processing apparatus according to claim 6, wherein the horizontal correction unit corrects distortion in the horizontal direction of the original image by performing an interpolation operation. A lens that collects light from the subject,
An image sensor that generates a captured image signal from projection light from the lens;
A data conversion circuit that converts a captured image signal from the image sensor into a digital image signal;
A signal processing circuit for generating recorded image data by performing signal processing on the digital image signal from the data conversion circuit;
A recording unit that records recording image data from the signal processing circuit,
The signal processing circuit comprises image correction means for correcting an original image having lens distortion,
The image correcting means includes
Horizontal correction means for correcting distortion in the horizontal direction of the original image by performing an interpolation operation on the original image using a horizontal correction parameter indicating a horizontal correction amount at pixel points constituting the original image;
The image obtained by the correction by the horizontal correction means is subjected to an interpolation operation using a vertical correction parameter indicating a vertical correction amount at a pixel point constituting the original image, so that the original image in the vertical direction is corrected. A camera system comprising: vertical correction means for correcting distortion. A lens that collects light from the subject,
An image sensor that generates a captured image signal from projection light from the lens;
A data conversion circuit that converts a captured image signal from the image sensor into a digital image signal;
A signal processing circuit for generating recorded image data by performing signal processing on the digital image signal from the data conversion circuit;
A recording unit that records recording image data from the signal processing circuit,
The signal processing circuit comprises image correction means for correcting an original image having lens distortion,
The image correcting means includes
Vertical correction means for correcting distortion in the vertical direction of the original image by performing an interpolation operation on the original image using a vertical correction parameter indicating a correction amount in the vertical direction at the pixel points constituting the original image;
The image obtained by the correction by the vertical correction unit is subjected to an interpolation operation using a horizontal correction parameter indicating a horizontal correction amount at a pixel point constituting the original image, thereby causing the original image in the horizontal direction to be corrected. A camera system comprising: horizontal correction means for correcting distortion. An image processing method for correcting an original image having distortion,
A first step of correcting a distortion in the horizontal direction of the original image by performing an interpolation operation on the original image using a horizontal correction parameter indicating a horizontal correction amount at a pixel point constituting the original image; ,
The image obtained in the first step is subjected to an interpolation operation using a vertical correction parameter indicating a vertical correction amount at a pixel point constituting the original image, so that the original image is distorted in the vertical direction. A second step of correcting the image processing method. At least in the first step, the original image is expanded or contracted in the horizontal direction by adjusting a horizontal interval between pixel points for which image data is obtained by the interpolation calculation, or in the second step by the interpolation calculation. The image processing method according to claim 13, wherein the original image is expanded and contracted in the vertical direction by adjusting a vertical interval between pixel points for which image data is obtained. A step of storing a horizontal correction image obtained by the correction in the first step in a storage unit;
The second step includes
A data acquisition step of acquiring the horizontal correction image according to the vertical correction parameter from the storage means;
The image processing method according to claim 13, further comprising: an interpolation calculation step of performing an interpolation calculation using the vertical correction parameter for the horizontal correction image acquired in the data acquisition step. The first step corrects distortion in the horizontal direction of the original image by performing an interpolation operation,
The image processing method according to claim 13, wherein the second step corrects a distortion in the vertical direction of the original image by performing an interpolation operation. An image processing method for correcting an original image having distortion,
A first step of correcting distortion in the vertical direction of the original image by performing an interpolation operation on the original image using a vertical correction parameter indicating a vertical correction amount at a pixel point constituting the original image; ,
The image obtained in the first step is subjected to an interpolation operation using a horizontal correction parameter indicating a horizontal correction amount at a pixel point constituting the original image, whereby the distortion of the original image in the horizontal direction is performed. A second step of correcting the image processing method. At least in the first step, the original image is expanded and contracted in the vertical direction by adjusting a vertical interval between pixel points for which image data is obtained by the interpolation calculation,
The image processing method according to claim 17, wherein in the second step, the original image is expanded or contracted in the horizontal direction by adjusting a horizontal interval between pixel points for which image data is obtained by the interpolation calculation. A step of storing a vertical correction image obtained by the correction in the first step in a storage unit;
The second step includes
A data acquisition step of acquiring the vertical correction image according to the horizontal correction parameter from the storage means;
The image processing method according to claim 17, further comprising: an interpolation calculation step of performing an interpolation calculation using the horizontal correction parameter for the vertical correction image acquired in the data acquisition step. The first step corrects a distortion in the vertical direction of the original image by performing an interpolation operation;
The image processing method according to claim 17, wherein the second step corrects distortion in the horizontal direction of the original image by performing an interpolation operation.

Images