JP6524644B2 - Image processing apparatus and electronic device - Google Patents

Description

  The present invention relates to an image processing apparatus and an electronic device.

  As electronic devices, for example, digital cameras, miniaturization is rapidly advancing, and miniaturization is also required for the built-in imaging lens. In order to miniaturize the photographing lens, in place of using a combination of a large number of conventional lenses, it is often replaced with a small number of one or a small number of lenses.

  In addition, the diameter of the lens may be replaced with a smaller one for downsizing, or the material of the lens may be replaced with an inexpensive one for cost reduction. However, it is difficult to sufficiently suppress image quality deterioration caused by so-called chromatic aberration, distortion aberration, and the like with such a downsized and low cost photographing lens. Miniaturization and widening of the imaging apparatus are factors that increase distortion at the periphery of a captured image.

  Then, the imaging device of patent document 1 is proposed. The imaging device described in Patent Document 1 is an image according to each color of R (red), G (green), and B (blue) from an imaging element such as a charge coupled device (CCD) sensor or a complementary metal oxide semiconductor (CMOS) sensor. Take out the signal. Next, the digital data is temporarily converted into digital data and temporarily stored in individual field memories. Then, based on the drive state of the photographing lens, the entire field memory is individually vector-moved to enlarge or reduce each image. After that, it is described that color deviation and distortion generated in the photographing lens are corrected by combining R, G, and B again.

  Also, in an imaging apparatus such as a digital camera, digital zoom technology is already known which enlarges the magnification of an image by performing image processing after distortion correction on an image obtained from an imaging element.

  However, in the imaging device described in Patent Document 1, the data read from the imaging element is temporarily stored in a memory such as a synchronous dynamic random access memory (SDRAM) and then read out for distortion correction. After the distortion correction is performed, an operation of saving in the memory again is performed. Similarly, when digital zooming is performed, the image data stored in the memory is read out, and after an enlarged image is generated by a process such as interpolation, the image is stored again in the memory.

  For this reason, reading of data from the memory and writing of data to the memory are frequently performed, which causes a problem that a bandwidth for accessing the memory is insufficient. In particular, since distortion correction and digital zoom processing are performed on image data before compression, the access bandwidth to the memory is largely insufficient. For this reason, it is necessary to take measures such as speeding up of the system clock, and power consumption increases, and there is a problem that the system configuration becomes complicated.

  In addition, by increasing the number of pixels of the image sensor such as a CCD sensor or CMOS sensor, and widening the angle of the imaging lens, the range of enlargement magnification and reduction magnification when enlarging or reducing pixels according to the location of the image at distortion correction becomes large. . In this case, reduction processing of 1⁄2 or less may be performed depending on the pixel position. Therefore, if digital zooming is performed on the image after distortion correction by image processing, a position at which enlargement processing is performed appears after reduction processing, resulting in a problem that the image quality is degraded.

  The present invention aims to solve such problems. That is, an object of the present invention is, for example, to provide an image processing apparatus capable of suppressing an increase in access bandwidth to a memory and reducing deterioration in image quality.

In order to solve the problems described above, the present invention is an image processing apparatus including a deformation unit that performs predetermined deformation processing on an input image and outputs an output image, and the deformation unit is configured to When the output image is divided into a plurality of areas, and the number of pixels of the reference area to be referred to in the input image for each of the divided areas is smaller than the number of pixels of the divided area, subjected to contraction small processing deformation processing co, when the number of pixels of the reference region is large, and characterized by applying expansion increasing processing to the transformation processing co respect to the divided regions.

  According to the present invention, the expansion process or the reduction process is performed together with the deformation process for each divided area, so there is no need to access the memory for the deformation process and the expansion process or the reduction process, and the increase of the memory access bandwidth can be suppressed. it can. In addition, since it is determined whether enlargement processing or reduction processing is to be performed for each divided area, deterioration in image quality can be reduced.

FIG. 1 is a block diagram of an image processing apparatus according to a first embodiment of the present invention. It is explanatory drawing of the coordinate data table shown by FIG. It is explanatory drawing of the example of an input image. It is an explanatory view showing four points of four corners of 1 area. It is an example of the correspondence of the coordinate data of the area shown by FIG. 4, and an input image. It is explanatory drawing of the relationship between the pixel count of the reference area (reference area) in the case of only a deformation | transformation process, and the pixel count of an output area. FIG. 14 is an explanatory view of an enlargement operation in the free deformation unit 12; It is explanatory drawing of the relationship between the pixel count of the reference area in the case of the deformation process in the case of the expansion process or a reduction process, and the pixel count of an output area. It is the flowchart which showed the operation | movement of the free deformation part shown by FIG. It is explanatory drawing of the relationship between the pixel count of a reference area, and the pixel count of an output area in the case of the deformation process in the expansion process or the reduction process of the image processing apparatus concerning the 2nd Embodiment of this invention. It is the flowchart which showed the operation | movement of the free deformation part concerning the 2nd Embodiment of this invention. It is a block block diagram of the image processing apparatus concerning the 3rd Embodiment of this invention. It is an explanatory view showing an example of a blending picture. It is explanatory drawing which decomposed | disassembled the image shown in FIG. 13 into two images. It is explanatory drawing of the example of two images after processing is performed by the free deformation part shown in FIG. It is the timing chart which showed the operation of the image processing device concerning a 4th embodiment of the present invention. It is a block block diagram of the electronic device concerning one Embodiment of this invention.

First Embodiment
An image processing apparatus according to a first embodiment of the present invention will be described with reference to FIGS. 1 to 9. The image processing apparatus 1 according to the present embodiment has a coordinate data table 11, a free deformation unit 12, a memory controller 13, a memory 14, and an enlargement / reduction unit 15, as shown in FIG. ing.

  The coordinate data table 11 as coordinate storage means is a table in which an area which the free deformation unit 12 reads out to deform an input image is set. The coordinate data table 11 is stored, for example, in a non-volatile memory such as a ROM (Read Only Memory). Alternatively, it may be configured of volatile memory such as RAM and loaded from the outside when the power is turned on. The coordinate data table 11 is provided with a plurality of tables for each deformation process such as distortion correction and rotation.

  The free deformation unit 12 as a deformation unit causes the memory controller 13 to read the input image stored in the memory 14 based on the coordinate data set in the coordinate data table 11. The free deformation unit 12 internally has an internal memory such as a RAM, and the read input image is stored in the internal memory. Then, the free deformation unit 12 performs predetermined deformation processing accompanying distortion processing, rotation processing and the like, and enlargement processing or reduction processing to be described later, based on the input image stored in the internal memory. The image data after the above-described processing is stored in the memory 14 via the memory controller 13.

  The memory controller 13 causes the memory 14 to store, for example, an input image acquired by an external imaging unit or the like and input to the image processing apparatus 1. Further, the memory controller 13 reads an input image from the memory 14 or image data after processing by the free deformation unit 12 in accordance with instructions from the free deformation unit 12 and the enlargement / reduction unit 15.

  The memory 14 is, for example, an SDRAM or the like, and stores an input image and image data and the like after processing by the free deformation unit 12 under control of the memory controller 13.

  The enlargement / reduction unit 15 as the enlargement / reduction means performs enlargement processing or reduction processing on the input image stored in the memory 14 or the image data subjected to the processing by the free deformation unit 12 and externally outputs the image as an output image. Output.

  In the image processing apparatus 1 having the above-described configuration, the memory 14 may be provided outside. The image processing apparatus 1 may be configured as a semiconductor integrated circuit, or may be configured as a printed circuit board, a module, or the like.

  Alternatively, it can be configured as a computer program (image processing program) that causes the configuration of FIG. 1 to function on a CPU (Central Processing Unit) or memory of a computer. For example, the CPU can function as the free deformation unit 12, the memory controller 13, and the enlargement / reduction unit 15, and the memory can function as the coordinate data table 11 and the memory 14.

  Next, the coordinate data table 11 will be described with reference to FIGS. 2 and 3. FIG. 2 is an example of the coordinate data table 11. The coordinate data table 11 is composed of X coordinate data and Y coordinate data. Here, the X coordinate indicates the main scanning direction (for example, the horizontal direction of the image) of the image, and the Y coordinate indicates the sub scanning direction of the image (for example, the vertical direction of the image).

  The coordinate data table 11 indicates the pixel position of the area to be referred to in the input image when processing by the free deformation unit 12 by a pair of X coordinate data and Y coordinate data. A specific example will be described with reference to FIG. FIG. 3A shows an input image. FIG. 3 (b) is an output image corresponding to the hatched portion in FIG. 3 (a).

  In the case of FIG. 3, the X coordinate data and the Y coordinate data are set to 4 pixels × 4 pixels as one area (area). For example, a pixel to be referred to corresponding to the pixel a in FIG. 3B is set to X coordinate data · Y coordinate data −0 in FIG. Similarly, a pixel to be referenced corresponding to the pixel b is set to X coordinate data · Y coordinate data−1, a pixel to be referenced corresponding to the pixel c is set to X coordinate data • Y coordinate data −2, The pixel to be referred to corresponding to d is set to X coordinate data · Y coordinate data -3 and the pixel d is set to X coordinate data · Y coordinate data -15.

  The coordinate data of the pixel corresponding to the area on the right of the area described in FIG. 3B is set as the X-coordinate data, Y-coordinate data-16 to X-coordinate data, Y-coordinate data-31 of FIG. .

  The memory controller 13 calculates an address of the memory 14 based on the X coordinate data and the Y coordinate data described above, and reads an input image (pixel) of the coordinates indicated by the X coordinate data and the Y coordinate data.

The address calculation of the memory 14 by the memory controller 13 is, for example, the following equation (1) when the data format of the input image is YUV422.
Memory address = memory offset address + X coordinate data × 2 + (Y coordinate data × (number of X pixels of input image × 2)) (1)

  Here, the memory offset address is a predetermined offset value of the address, and the number of X pixels is the number of pixels in the X direction (main scanning direction).

  Further, in the present embodiment, both X coordinate data and Y coordinate data can be set to five decimal places. Therefore, when designated with decimal point accuracy (sub-pixel accuracy), the free deformation unit 12 performs interpolation calculation by the bicubic method well-known as a pixel interpolation method. The bicubic method requires adjacent peripheral pixels, and the present embodiment requires 9 pixels. Of course, other numbers of pixels such as 16 pixels may be used. That is, when the X coordinate data and the Y coordinate data are specified with decimal point precision, it is necessary to read data of 9 pixels per one coordinate set in the X coordinate data and the Y coordinate data. Note that if there are no digits after the decimal point in X coordinate data and Y coordinate data, interpolation by bicubic method is not performed, and reference (readout) of pixels required only for interpolation by bicubic method is performed. Absent.

  As described above, since nine pixels stored in the memory 14 are referred to one pixel of output pixel data by interpolation calculation by the bicubic method, for example, X coordinate data, Y coordinate data-0 and X coordinate data, There is a high possibility that the data to be referred to overlap with Y coordinate data -1. Therefore, in the present embodiment, the free deformation unit 12 divides the output image into area units of 4 × 4 pixels, and handles image data in area units. The area of the 4 × 4 pixel output image is called an output area or an output area image. Then, 16 reference pixel data (X coordinate data and Y coordinate data 0 to 15) are retrieved from the memory 14 from the coordinate data table 11 and stored in the internal memory of the free deformation unit 12 to access the memory 14 I am improving the efficiency.

  This output area (output area image) is preset, but is not limited to 4 × 4 pixels. Further, this output area (output area image) is a reference in the enlargement process or the reduction process in the free deformation unit 12. Although the size of the output area is equal to the size of the output result of the free deformation unit 12 in the case of equal magnification processing in the free deformation unit 12, the size of the output area and the enlargement process or the reduction process are performed. The size of the output result of the free deformation unit 12 may be different. Details will be described later.

  In the example shown in FIG. 2, 16 coordinate data per area are set in the coordinate data table 11. However, as shown in FIG. 11 may be set, and the other 12 points may be calculated. FIG. 4 is a view showing one of the output area images.

  In FIG. 4, the pixel f refers to the input image at the coordinates indicated by X coordinate data and Y coordinate data −0 in FIG. 2. Similarly, the pixel g refers to the input image at coordinates indicated by X coordinate data / Y coordinate data-1, and the pixel h refers to the input image at coordinates indicated by X coordinate data / Y coordinate data-2. The pixel i is referred to an input image at coordinates indicated by X coordinate data and Y coordinate data -3.

  The remaining 12 pixels calculate coordinate data by, for example, linear interpolation. An interpolation method other than linear interpolation may be used. In addition, parameters may be provided to change the interpolation method for each area.

  FIG. 5 shows a specific example of linear interpolation. FIG. 5A shows specific coordinate data in FIG. 4 (the upper part is X coordinate data, and the lower part is Y coordinate data). The coordinate data of the position corresponding to the pixels f, g, h, i in FIG. 4 in FIG. 5A is a value set in advance in the coordinate data table 11. Coordinate data of other positions are values calculated by linear interpolation. In the case of FIG. 5A, an area having a shape as shown in FIG. 5B is referred to in the input image (note that FIG. 5B includes the interpolation by the bicubic method). Absent).

  FIG. 6 shows the relationship between the number of pixels in the reference area (reference area) and the number of pixels in the output area in the case of only the deformation processing. Here, the output area refers to one divided area when the output image is divided into predetermined predetermined sizes as described above, and in the example of FIG. 6, the number of pixels is 4 × 4 pixels (rectangular). Let it be a pixel. The upper part of FIG. 6 shows the reference area, and the lower part shows the result of processing by the free deformation section 12 with reference to the reference area.

  FIG. 6A shows the case where the number of pixels in the output area <the number of pixels in the reference area. 6B shows the number of pixels in the output area> the number of pixels in the reference area, FIG. 6C shows the number of pixels in the output area = the number of pixels in the reference area, and FIG. This is the case where the number <the number of pixels in the reference area.

  The reference area may not necessarily be rectangular as shown in FIG. 6 because interpolation by the bicubic method is necessary when the reference area is specified with the above-described sub-pixel accuracy. Also, depending on the contents of the transformation process, an area other than 4 × 4 pixels may be referred to. Therefore, the number of pixels in the reference area may be calculated based on the coordinate data table 11, for example, using a well-known coordinate method.

  In FIG. 6, since the enlargement process and the reduction process are not performed, all the processing results in the lower stage are fixed at 4 × 4 pixels. That is, only the deformation (for example, FIG. 5) based on the reference area corresponding to the coordinates shown in the coordinate data table 11 is performed.

  Next, the enlargement operation in the free deformation unit 12 will be described with reference to FIG. For example, in the case of enlarging from 4 × 4 pixels to 5 × 5 pixels, the enlargement processing can be performed by using the coordinate data of the four corners set in the coordinate data table 11 as it is.

  As shown in FIG. 5, when only four points which are four corners of one area are set in the coordinate data table 11 (when the coordinate position of the input image corresponding to the four vertices of the rectangle is stored), other than four corners The enlargement process is performed by increasing the number of pixels. For example, as shown in FIG. 7, in the case of 4 × 4 pixels, the coordinate data of 2 pixels between each vertex was calculated by linear interpolation, but when expanding to 5 × 5 pixels, the distance between each vertex is calculated. The coordinate data may be calculated with three pixels. By doing this, it is possible to carry out an enlargement process of 1.25 times.

  In the case of FIG. 7, the same data can be used for the coordinate data concerning the pixels f, g, h and i in the case of 4 × 4 pixels and in the case of 5 × 5 pixels. Similarly, if it is treated as 6 × 6 pixels, it can be enlarged to 1.75 times if it is treated as 1.5 times and as 7 × 7 pixels. That is, since coordinate data is calculated by linear interpolation from four points at four corners, the enlargement operation can be performed by changing the size of the output image in units of area output from the free deformation unit 12. Instead of linear interpolation, another interpolation method may be used.

  Although the above is an example of the enlargement process, the reduction process is also possible in the same way. That is, the interpolation processing may be performed to reduce the number of pixels between the four points at the four corners. That is, the deformation means performs enlargement processing or reduction processing on the divided area based on the coordinate positions of the four vertexes of the rectangle stored in the coordinate storage means.

  FIG. 8 shows the relationship between the number of pixels in the reference area and the number of pixels in the output area in the case of deformation processing involving enlargement processing or reduction processing in the free deformation unit 12. FIG. 8 is an example of the case where the image processing apparatus 1 as a whole performs an enlargement process of 1.25 times by, for example, digital zoom. The upper part of FIG. 8 shows the reference area, and the lower part shows the result of processing by the free deformation section 12 with reference to the reference area.

  FIG. 8A shows the case where the number of pixels in the output area <the number of pixels in the reference area. 8 (b) shows the number of pixels in the output area> the number of pixels in the reference area, FIG. 8 (c) shows the number of pixels in the output area = the number of pixels in the reference area, FIG. 8 (d) shows the pixels in the output area This is the case where the number <the number of pixels in the reference area. Here, the number of pixels in the output area is 4 × 4 pixels = 16 pixels as in FIG.

The judgment criteria of the enlargement process or the reduction process of the free deformation unit 12 in the present embodiment are performed by the following three.
(1) When the condition of the number of pixels in the output area <the number of pixels in the reference area is satisfied, the free deformation unit 12 performs enlargement processing. That is, when the number of pixels in the reference area is larger than the number of pixels in the divided area, the enlargement process is performed on the divided area.
(2) When the condition of the number of pixels in the output area> the number of pixels in the reference area is satisfied, the free deformation unit 12 performs a reduction process. That is, when the number of pixels in the reference area is smaller than the number of pixels in the divided area, reduction processing is performed on the divided area.
(3) When the condition of the number of pixels in the output area = the number of pixels in the reference area is satisfied, the free deformation unit 12 neither performs the enlargement process nor the reduction process (equal magnification).

  If the above condition is applied to FIG. 8, in the case of FIG. 8A, the enlargement process of 1.25 times is performed because the number of pixels in the output area (16 pixels) <the number of pixels in the reference area (27 pixels). It will be. In the case of FIG. 8B, since the number of pixels in the output area (16 pixels)> the number of pixels in the reference area (9 pixels), the reduction process is performed. In the case of reduction processing, the size of the reference area is reduced. In the case of FIG. 8C, neither the enlargement process nor the reduction process is performed because the number of pixels in the output area (16 pixels) = the number of pixels in the reference area (16 pixels). In the case of FIG. 8D, since the number of pixels in the output area (16 pixels) <the number of pixels in the reference area (21 pixels), the enlargement processing of 1.25 times is performed.

  If the number of pixels in the output area in FIG. 8A or 8D <the number of pixels in the reference area, the image data output from the free deformation unit 12 is subjected to reduction processing on the data in the reference area. It can be regarded as equivalent to For example, it is assumed that the reference area image is 25 × 25 pixels and the output area image is 4 × 4 pixels, and enlargement processing is performed twice while performing predetermined deformation such as distortion correction.

  In this case, if only the deformation processing is performed in the free deformation unit 12 and the magnification / reduction unit 15 in the subsequent stage doubles, the (4/25) × (4/25) pixel is performed in the deformation processing in the free deformation unit 12 After the reduction process, the enlargement process is performed twice. On the other hand, when the free deformation unit 12 performs the deformation process and the enlargement process, the free deformation unit 12 performs a process of (8/25) × (8/25) pixels. Then, since the enlargement / reduction unit 15 does not perform any processing at equal magnification, it can be determined that the image quality is improved compared to enlargement by the enlargement / reduction unit 15 when enlargement is performed by the free deformation unit 12.

  If the number of pixels in the output area in FIG. 8B> the number of pixels in the reference area, the reference area image is 3 × 3 pixels, so the free deformation unit 12 may be 3 × 3 pixels the same as the size of the reference area image. In this case, the processing is substantially equal. Therefore, it can be determined that the image quality does not deteriorate even if it is reduced from the output area image size. Then, the enlargement / reduction unit 15 separately performs enlargement processing on the area of the output image output from the free deformation unit 12.

  In the case where the number of pixels in the output area in FIG. 8C = the number of pixels in the reference area, the free deformation unit 12 performs equal magnification processing, so it can be determined that the image quality does not improve even if enlarged by the free deformation unit 12 . Therefore, the enlargement / reduction unit 15 separately performs enlargement processing on the area of the output image output by the free deformation unit 12.

  The operation (image processing method) of the above-described free deformation unit 12 is summarized in the flowchart of FIG. In the flowchart of FIG. 9, first, the free deformation unit 12 reads the reference area of the area to be processed from the memory 14 via the memory controller 13 in step S101 based on the coordinate data table 11. Next, the free deformation unit 12 determines whether or not enlargement processing such as digital zoom is performed in step S102, and if there is (in the case of YES), the number of pixels in the output area and the number of pixels in the reference area in step S103. Compare On the other hand, when there is no enlargement process (in the case of NO), only the deformation process is performed in step S106, and the process proceeds to step S107.

  In step S103, the free deformation unit 12 performs enlargement processing and deformation processing in step S104 if the number of pixels in the output area <the number of pixels in the reference area. On the other hand, if the number of pixels in the output area> the number of pixels in the reference area, reduction processing and deformation processing are performed in step S105. Here, enlargement or reduction refers to processing in comparison with the number of pixels in the output area described above.

  If the number of pixels in the output area = the number of pixels in the reference area, the free deformation unit 12 performs only the deformation processing in step S106. Then, in step S107, the free deformation unit 12 determines whether there is a next area, and if there is, the process returns to step S101 to repeat the above-described process for the next area, and ends the process if there is no such area.

  As apparent from the above description, steps S101 to S107 correspond to the deformation step. In addition, the free deformation unit 12 divides the output image into a plurality of output areas, and performs the enlargement process or the reduction process on the output area based on the reference area referred to in the input image for each of the output areas. Processing is given.

  According to the present embodiment, in the image processing apparatus 1, the free deformation unit 12 divides the output image into a plurality of parts, and divides the divided areas based on the reference areas to be referred to in the input image for each of the divided areas. Zoom in or out. In this way, the deformation process and the enlargement process or the reduction process are performed for each divided area, and therefore there is no need to access the memory 14 for the deformation process and the enlargement process or the reduction process, thereby suppressing an increase in memory access bandwidth. be able to. In addition, since it is determined whether to enlarge or reduce for each divided area, it is possible to reduce the deterioration of the image quality.

  The free deformation unit 12 performs reduction processing and deformation processing if the number of pixels in the output area> the number of pixels in the reference area, and if the number of pixels in the output area <the number of pixels in the reference area, enlargement processing and deformation processing are performed. Do. By doing this, in the case where the number of pixels in the output area> the number of pixels in the reference area, the free deformation unit 12 makes the size the same as the size of the reference area, so the image quality is reduced even from the output area image size. It does not deteriorate. Further, in the case where the number of pixels in the output area <the number of pixels in the reference area, the enlargement processing is performed only by the free deformation unit 12, so the image quality is improved as compared with enlargement separately by the enlargement / reduction unit 15.

  In addition, the free deformation unit 12 performs only the deformation process when the number of pixels in the output area> the number of pixels in the reference area. By doing this, since the free deformation unit 12 performs equal magnification processing, the image quality is not improved even if the free deformation unit 12 enlarges the image. Therefore, the enlargement / reduction unit 15 may separately perform processing.

Second Embodiment
Next, an image processing apparatus according to a second embodiment of the present invention will be described with reference to FIGS. 10 and 11. FIG. The same parts as those of the first embodiment described above are denoted by the same reference numerals and the description thereof will be omitted.

  The image processing apparatus 1 according to the present embodiment has the same block configuration as that of FIG. 1, but the processing in the free deformation unit 12 is different.

  FIG. 10 shows the relationship between the number of pixels in the reference area and the number of pixels in the output area in the case of deformation processing involving enlargement processing or reduction processing according to the present embodiment. FIG. 10 shows an example of the case where the image processing apparatus 1 as a whole is subjected to 1.25 × enlargement processing by digital zoom or the like as in FIG. The upper part of FIG. 10 shows the reference area, and the lower part shows the result of processing by the free deformation unit 12 with reference to the reference area. Further, the number of pixels in the output area is 4 × 4 pixels = 16 pixels as in the first embodiment.

The criteria for the enlargement process or the reduction process of the free deformation unit 12 in the present embodiment are performed by the following six. In the present embodiment, the main scanning direction (X coordinate) will be described as one direction, and the sub scanning direction (Y coordinate) as another direction orthogonal to the one direction.
(1) When the condition of X size of the image of the output area (number of pixels in the main scanning direction) <X size of the image of the reference area is satisfied, the free deformation unit 12 enlarges the X size (enlargement processing in X direction). That is, when the number of pixels in one direction of the reference area is larger than the number of pixels in one direction of the divided area, the enlargement process is performed in the one direction of the divided area.
(2) When the condition of X size of the image of the output area> X size of the image of the reference area is satisfied, the free deformation unit 12 reduces the X size (reduction process in the X direction). That is, when the number of pixels in one direction of the reference area is smaller than the number of pixels in one direction of the divided area, the reduction process is performed in one direction of the divided area.
(3) When the condition of X size of image of output area = X size of image of reference area is satisfied, the free deformation unit 12 does not change the X size.

(4) If the Y size of the image in the output area (number of pixels in the sub scanning direction) <the Y size of the image in the reference area is satisfied, the free deformation unit 12 enlarges the Y size (enlargement processing in the Y direction). That is, when the number of pixels in the other direction of the reference area is larger than the number of pixels in the other direction of the divided area, the enlargement processing is performed in the other direction of the divided area.
(5) If the condition of Y size of the image of the output area> Y size of the image of the reference area is satisfied, the free deformation unit 12 reduces the Y size (reduction processing in the Y direction). That is, when the number of pixels in the other direction of the reference area is smaller than the number of pixels in the other direction orthogonal to one direction of the divided area, the reduction process is performed in the other direction of the divided area.
(6) When the condition of Y size of image of output area = Y size of image of reference area is satisfied, the free deformation unit 12 does not change the Y size.

  If the above conditions are applied to FIG. 10, in the case of FIG. 10A, since the X size of the output area (4 pixels) <the X size of the reference area (6 pixels), the X size is changed from 4 pixels to 5 pixels. Increase (increase by 1.25 times). In addition, since Y size of output area (4 pixels)> Y size of reference area (3 pixels), Y size is reduced from 4 pixels to 3 pixels (reduction to fit the size of reference area).

  In the case of FIG. 10B, the X size is reduced from 4 pixels to 3 pixels because the X size (4 pixels) of the output area> the X size (3 pixels) of the reference area to shrink). In addition, since Y size of output area (4 pixels)> Y size of reference area (3 pixels), Y size is reduced from 4 pixels to 3 pixels (reduction to fit the size of reference area).

  In the case of FIG. 10C, since the X size of the output area (4 pixels) = the X size of the reference area (4 pixels), the X size is not changed. In addition, since the Y size of the output area (4 pixels) = the Y size of the reference area (4 pixels), the Y size is not changed.

  In the case of FIG. 10B, the X size is reduced from 4 pixels to 3 pixels because the X size (4 pixels) of the output area> the X size (3 pixels) of the reference area to shrink). In addition, Y size is increased from 4 pixels to 5 pixels (expanded by 1.25 times) because Y size of output area (4 pixels) <Y size of reference area (7 pixels).

  In the case of this embodiment, since the first embodiment is judged separately for X size and Y size, each judgment of enlargement, reduction and equal magnification can be considered in the same manner as the first embodiment. . In other words, if X size of output area <X size of reference area and Y size of output area <Y size of reference area, enlargement / reduction unit 15 does nothing with X size or Y size as equal Processing is not performed. Therefore, it can be determined that the image quality is improved compared to enlargement by the enlargement / reduction unit 15 when enlarged by the free deformation unit 12.

  In the case of X size of output area> X size of reference area and Y size of output area> Y size of reference area, in free deformation unit 12, X size or Y size is substantially equal magnification processing. It can be determined that the image quality does not deteriorate even if it is reduced from the output area image size. Then, the enlargement / reduction unit 15 separately performs enlargement processing on the output image output by the free deformation unit 12.

  In the case of X size of output area = X size of reference area and Y size of output area = Y size of reference area, the free deformation unit 12 performs equal magnification processing, so the free deformation unit 12 enlarges Also, it can be determined that the image quality does not improve. Therefore, the enlargement / reduction unit 15 separately performs enlargement processing on the output image output by the free deformation unit 12.

  For example, in the case of FIG. 10A, the enlargement / reduction unit 15 performs enlargement processing with the same size as the X size and 1.66 times the Y size. In the case of FIG. 10B, the enlargement process is performed 1.66 times for both the X size and the Y size. In the case of FIG. 10C, the enlargement process is performed at 1.25 times for both the X size and the Y size. In the case of FIG. 10D, the X size is subjected to 1.66 times enlargement processing, and the Y size is subjected to equal magnification processing. In this manner, the image processing apparatus 1A as a whole is subjected to the 1.25 × enlargement processing.

  The operation (image processing method) of the free deformation unit 12 according to the present embodiment is summarized in the flowchart of FIG. In the flowchart of FIG. 11, first, the free deformation unit 12 reads the reference area of the area to be processed based on the coordinate data table 11 in step S201. Next, the free deformation unit 12 determines whether or not enlargement processing such as digital zoom is present in step S202, and if there is (in the case of YES), the X size of the output area and the X size of the reference area in step S203. Compare On the other hand, if the enlargement process is not performed (in the case of NO), only the deformation process is performed in step S212, and the process proceeds to step S213.

  In step S203, the free deformation unit 12 increases the X size in step S204 if the X size of the output area <the X size of the reference area. If X size of output area> X size of reference area, the X size is reduced in step S205. If the X size of the output area = the X size of the reference area, the X size is not changed in step S206.

  The free deformation unit 12 compares the Y size of the output area with the Y size of the reference area in step S207, which has been advanced from steps S204 to S206. Then, in step S207, if Y size of output area <Y size of reference area, the Y size is increased in step S208. If Y size of output area> Y size of reference area, the Y size is reduced in step S209. When Y size of output area = Y size of reference area, the Y size is not changed in step S210.

  In step S211 that follows from step 208 to step S210, the free deformation unit 12 performs enlargement processing or reduction processing based on the X size and Y size determined in steps S204 to S206 and steps S208 to S210 and deformation processing based on the coordinate data table 11. I do.

  The free deformation unit 12 determines whether or not there is a next area in step S213. If there is, the process returns to step S201 to repeat the above-described process for the next area.

  As apparent from the above description, steps S201 to S213 correspond to the deformation step. In the flowchart shown in FIG. 11, the X size is determined first, but the Y size may be determined first. Further, although the X size and the Y size are determined independently in the flowchart shown in FIG. 11, they may be determined simultaneously.

  According to this embodiment, when comparing the output area and the reference area, not the number of pixels but X size and Y size are individually compared, and enlargement processing or reduction processing is performed based on the comparison results. Is going. By doing this, even when the reference area is not rectangular, the deterioration of the image quality can be reduced. Therefore, the deterioration of the image quality can be reduced in more various deformation processes.

Third Embodiment
Next, an image processing apparatus according to a third embodiment of the present invention will be described with reference to FIG. 12 to FIG. The same parts as those in the first and second embodiments described above are designated by the same reference numerals and description thereof is omitted.

  A block diagram of the image processing apparatus 1A according to the present embodiment is shown in FIG. In the image processing apparatus 1A shown in FIG. 12, a blending unit 16 is added to the image processing apparatus 1A shown in FIG. 1, and the memory controller 13 is changed to a memory controller 13a. The memory controller 13a is configured to be able to input a plurality of input images (input images A and B). A plurality of input images may be sequentially input by one single input like the memory controller 13 shown in FIG.

The blending unit 16 as blending means performs blending processing of a plurality of input images. The blending process is a process of superimposing a plurality of input images into one output image in which the plurality of input images are mixed. For example, when two images are blended, a blending coefficient is set, the following equation (2) is calculated, and pixel data (image data) of the calculation result is output to the memory 14 through the memory controller 13.
(Pixel data of one image x blending coefficient) + (pixel data of the other image x (1-blending coefficient)) (2)

  FIG. 13 is an example of a blending image. FIG. 13 blends the edges of two images, an A image and a B image. That is, the right end of the A image and the left end of the B image are superimposed. This superimposed region is hereinafter referred to as a blending image. In the present embodiment, the image shown in FIG. 13 or the like is not an image before the process of the free deformation unit 12 but an image after processing (after deformation).

  That is, as shown in FIG. 14, the A image is composed of 15 areas of areas 1 to 5, areas 11 to 15, and areas 21 to 25. Also, the B image is composed of 15 areas of areas 6 to 10, areas 16 to 20, and areas 26 to 30. In this case, areas 5 and 6, areas 15 and 16, and areas 25 and 26 overlap with each other.

  When the blending process is performed in the blending unit 16, if the enlargement process or the reduction process in the free deformation unit 12 described in the first and second embodiments is performed, for example, the area between two images is enlarged etc. in the area included in the blending image. The magnification of may be different. If magnifications such as enlargement differ, either one needs to be subjected to enlargement processing or reduction processing at the time of blending processing, and processing efficiency as the image processing apparatus 1A decreases.

  Therefore, in the present embodiment, in the free deformation unit 12, the area included in the blending image is adjusted to one of the magnifications. Here, the magnification is represented as a value larger than 1 in the enlargement processing, a value less than 1 in the reduction processing, and 1 as an equal magnification.

  As a specific example, in the case of FIG. 13, when the magnification of the area of the A image> the magnification of the area of the B image, the enlargement processing or the reduction processing of the area is performed at the magnification of the area of the A image. When the magnification of the area of the A image <the magnification of the area of the B image, the enlargement process or the reduction process of the area is performed at the magnification of the area of the B image. That is, in the divided area mixed by the mixing means, the deformation means performs the enlargement process or the reduction process according to the largest magnification among the magnifications of the enlargement or the reduction calculated for each divided area of each output image. .

  As processing of the free deformation unit 12 in FIG. 14, first, processing of areas 1 to 4 of the A image is performed. Next, the processing of area 5 is performed later, and then the processing of area 6 of B image is performed, and the magnifications of area 5 and area 6 are combined and two areas are output to the blending unit 16. Then, processing of areas 7 to 10 of the B image is performed. That is, the processing of the area 5 of the A image and the area 6 of the B image included in the blending image is performed by the blending unit 16. The processes of the areas 1 to 4 of the A image and the processes of the areas 7 to 10 of the B image are the same as those in the first or second embodiment, and the process by the blending unit 16 is not performed. As described above, in the present embodiment, the areas superimposed on each other in the blending image are continuously processed.

  FIG. 15 shows an example in which the process as described above is performed. Areas 5 and 6 included in the blending image, areas 15 and 16 and areas 25 and 26 have the same magnification.

  According to the present embodiment, when the free deformation unit 12 performs blending processing, in the area included in the blending image, the free deformation unit 12 performs enlargement or reduction in accordance with the higher magnification among the calculated magnifications. By doing this, there is no need to perform enlargement processing or reduction processing in the blending unit 16, and the processing efficiency of the image processing apparatus 1A can be improved without complicating the processing of the blending unit 16.

  In the example described above, the blending process of two images has been described, but even in the case of three or more blending processes, the processing efficiency can be improved by similarly adjusting to a high magnification.

Fourth Embodiment
Next, an image processing apparatus according to a fourth embodiment of the present invention will be described with reference to FIG. The same parts as those in the first to third embodiments described above are denoted by the same reference numerals and description thereof is omitted.

  In the present embodiment, the block configuration may be either FIG. 1 or FIG. 12, and the operation of the free deformation portion 12 is different.

  When the operation of the free deformation unit 12 described in the first to third embodiments is performed, the access bandwidth to the memory 14 may be insufficient. For example, in the case of a digital camera, when the input image input from the imaging unit is a moving image, thirty or more images are input in one second. Therefore, the image processing apparatus 1 (1A) needs to complete the processing within 33 milliseconds at the latest. However, depending on the processing content of the free deformation unit 12, the access amount to the memory 14 may increase and the processing may not be completed within 33 milliseconds.

  In the present embodiment, if memory access is insufficient with the immediately preceding input image, the free deformation unit 12 does not process the next input image, and prohibits enlargement processing from the next input image and reduces it. Allow only processing or equal-magnification processing.

  A specific example is shown in FIG. FIG. 16 is a timing chart showing the processing state of the frame start signal and the free deformation unit 12. The frame start signal is a signal indicating the start of a frame, and is input from, for example, an imaging unit or the like in the front stage of the image processing apparatus 1 (1A).

  In FIG. 16, the processing of the free deformation unit 12 is not completed within the maximum processing time (one frame period) for the first input image. Therefore, since the second input image has passed one frame period or more, the process is not performed in the free deformation unit 12. Then, although the free deformation unit 12 resumes the process from the third input image, the enlargement process is not performed. If enlargement processing is required, the enlargement / reduction unit 15 separately performs it. That is, when the processing of one input image has passed a predetermined time or more, the deformation means limits the enlargement processing to the subsequent input images.

  Note that the predetermined period for determining that the enlargement processing is not performed is not limited to one frame period, and the overhead of the processing of the read / write of the memory 14 and the processing of the enlargement / reduction unit 15 etc., and the image processing device 1 (1A) It may be suitably changed according to the performance required of.

  According to the present embodiment, the free deformation unit 12 restricts the enlargement processing to be prohibited for the subsequent input images when the processing of one input image has elapsed for one frame period or more. By doing this, the amount of data written to the memory 14 by the free deformation unit 12 and the amount of data read by the enlargement / reduction unit 15 are reduced, and it is possible to eliminate the bandwidth shortage of memory access.

  In addition, since the free deformation unit 12 does not process the input image immediately after the processing of the input image for which one frame period or more has elapsed, the image is degraded when the input image is a moving image or the like input in real time. Can be minimized.

  Lastly, an embodiment of an electronic device having any one of the image processing apparatuses 1 (1A) according to the above-described four embodiments will be described with reference to FIG. Although FIG. 17 shows an example of the image processing apparatus 1 shown in FIG. 1, it goes without saying that the image processing apparatus 1A shown in FIG. 12 may be used.

  In the electronic device 100 shown in FIG. 17, an image acquisition unit 101 as an image acquisition unit and an image output unit 102 as an image output unit are added to the image processing apparatus 1. For example, when the electronic device 100 is a digital camera, the image acquisition unit 101 corresponds to an imaging device (imaging unit), a conversion unit to YUV 422 format, etc. The image output unit 102 is a display unit such as a liquid crystal display, a memory It corresponds to a storage medium such as a card.

  The electronic device 100 illustrated in FIG. 17 causes the image acquisition unit 101 to convert an image captured by the imaging device into the YUV422 format and input the image to the image processing apparatus 1. The image processing apparatus 1 performs distortion correction processing and digital zoom processing as in the above-described embodiment, and the image output unit 102 displays an image subjected to distortion correction processing and digital zoom processing on the display unit.

  The image acquisition unit 101 is not limited to an imaging device or the like. For example, if the image is stored in a storage medium such as a memory card or a hard disk drive, the storage medium may be used. If the image is acquired via a network or the like, a network interface etc. May be

  Similarly, the image output unit 102 is not limited to the display unit. For example, if the image is stored in a storage medium such as a memory card or a hard disk drive, the storage medium may be used. If the image is output via a network or the like, a network interface or the like may be used. It is also good.

  The electronic device 100 is not limited to a photographing device such as a digital camera, but may be a video device, an information device, a communication device, an image forming device, or the like having distortion correction, rotation, etc. .

  Further, the present invention is not limited to the above embodiment. That is, those skilled in the art can carry out various modifications without departing from the gist of the present invention in accordance with conventionally known findings. As long as the configuration of the image processing apparatus of the present invention is provided even by such a modification, it is of course included in the scope of the present invention.

1, 1A Image processing device 11 Coordinate data table (coordinate storage means)
12 Free deformation part (deformation means)
13 memory controller 14 memory 15 enlargement / reduction unit (enlargement / reduction means)
16 Blending section (mixing means)
100 electronic device 101 image acquisition unit (image acquisition means)
102 Image output unit (image output means)
S101 to S107 deformation process

Patent No. 5154361 gazette

Claims (10)

An image processing apparatus including a deformation unit that performs predetermined deformation processing on an input image and outputs an output image,
The deformation means divides the output image into a plurality of areas, and when the number of pixels of a reference area to be referred to in the input image for each of the divided areas is smaller than the number of pixels of the divided areas, subjected to contraction small processing on the deformation processing co the divided region, when the number of pixels of the reference region is large, and characterized by applying expansion increasing processing to the transformation processing co against the partition area Image processing device. An image processing apparatus including a deformation unit that performs predetermined deformation processing on an input image and outputs an output image,
The deformation means divides the output image into a plurality of areas, and the direction of one of the reference areas to be referred to in the input image for each of the divided areas with respect to the number of pixels in one direction of the divided areas. If the number of pixels is small, the aforementioned divided areas subjected to contraction small processing in one direction, when the number of pixels the one direction of the reference region is large, expanded to the one direction of the divided area Process and
When the number of pixels in the other direction of the reference area is smaller than the number of pixels in the other direction orthogonal to the one direction of the divided area, the reduction process is performed in the other direction of the divided area. When the number of pixels in the other direction of the reference area is large, the enlargement process is performed in the other direction of the divided area;
An image processing apparatus characterized by The image processing apparatus further includes coordinate storage means in which the divided area is in a rectangular shape and the coordinate position of the input image corresponding to at least four vertices of the rectangle is stored.
The image processing apparatus according to claim 1, wherein the deformation unit performs the enlargement process or the reduction process on the divided area based on the coordinate position stored in the coordinate storage unit. It further comprises mixing means for mixing a plurality of images,
In the divided area to be mixed by the mixing means, the deformation means may perform the enlargement process or the enlargement process in accordance with the largest magnification among the enlargement or reduction rates respectively calculated for the divided areas of the respective output images. The image processing apparatus according to any one of claims 1 to 3, wherein a reduction process is performed.   5. The image processing apparatus according to any one of claims 1 to 4, wherein the transformation means limits the enlargement process on subsequent input images when processing of one of the input images has passed a predetermined period or more. An image processing apparatus according to claim 1.   The image processing apparatus according to any one of claims 1 to 5, further comprising enlargement / reduction means for performing enlargement processing or reduction processing on the processed image in the deformation means.   An electronic apparatus comprising the image processing apparatus according to any one of claims 1 to 6.   An image acquisition unit configured to acquire an image, an image processing apparatus according to any one of claims 1 to 6, and an image output unit configured to output an image processed by the image processing apparatus. Electronic equipment characterized by An image processing method including a deformation step of performing predetermined deformation processing on an input image and outputting an output image,
In the deformation step, the output image is divided into a plurality of areas, and when the number of pixels of a reference area to be referred to in the input image for each of the divided areas is smaller than the number of pixels of the divided areas, subjected to contraction small processing on the deformation processing co the divided region, when the number of pixels of the reference region is large, and characterized by applying expansion increasing processing to the transformation processing co against the partition area Image processing method. An image processing method including a deformation step of performing predetermined deformation processing on an input image and outputting an output image,
In the deformation step, the output image is divided into a plurality of areas, and the direction of one of the reference areas to be referred to in the input image for each of the divided areas with respect to the number of pixels in one direction of the divided areas. If the number of pixels is small, the aforementioned divided areas subjected to contraction small processing in one direction, when the number of pixels the one direction of the reference region is large, expanded to the one direction of the divided area Process and
When the number of pixels in the other direction of the reference area is smaller than the number of pixels in the other direction orthogonal to the one direction of the divided area, the reduction process is performed in the other direction of the divided area. When the number of pixels in the other direction of the reference area is large, the enlargement process is performed in the other direction of the divided area;
An image processing method characterized in that.

Images