JP4686200B2 - Image conversion method and apparatus - Google Patents

Description

  The present invention relates to an image processing technique, and more particularly to a technique for realizing image conversion such as image enlargement, illumination conversion, and viewpoint conversion.

  With the digitization of image devices and networks, any image device can be connected, and the degree of freedom of image exchange is increasing. An environment in which users can freely handle images without being restricted by differences in systems has been developed. For example, a user can output an image taken with a digital still camera to a printer, publish it on a network, or view it on a home television.

  On the other hand, the system side needs to support various image formats, and image format conversion is naturally required to be sophisticated. For example, image size conversion frequently occurs, and an up converter (a conversion device that increases the number of images and lines) and a down converter (a device that reduces the number of pixels and lines) are required. For example, when printing on A4 paper (297 mm x 210 mm) at a resolution of 600 dpi, an original of 7128 pixels x 5040 lines is required, but since the resolution of many digital still cameras is lower than this, an up converter is required. . On the other hand, since the final output form of the image published on the network is not determined, it is necessary to convert it to a corresponding image size every time an output device is determined. Also, in the home television, digital terrestrial services have been started, and conventional standard television images and HD (High Definition) television images are mixed, and image size conversion is frequently used.

  In image enlargement, it is necessary to newly create image data that did not exist at the time of acquisition, and various methods have already been proposed. For example, as a general method, there is a method using interpolation such as a bilinear method or a bicubic method (Non-Patent Document 1). However, when interpolation is used, only intermediate values of sampling data can be generated, so that sharpness such as edges tends to deteriorate and the image tends to be blurred. Therefore, a technique is disclosed in which an interpolated image is used as an initial enlarged image, and then an edge portion is extracted to emphasize only the edge (Patent Document 1, Non-Patent Document 2). However, it is difficult to separate the edge portion from the noise, and the noise is enhanced together with the enhancement of the edge portion, which tends to cause image quality deterioration.

  Therefore, there is a learning method as a method for enlarging an image while suppressing deterioration in image quality. That is, a high-resolution image corresponding to the enlarged image is taken in advance by a high-definition camera or the like, and a low-resolution image is created from the high-definition image. The low-resolution image is generated by a method of sub-sampling through a low-pass filter. Many pairs of such low resolution images and high resolution images are prepared, and the relationship is learned as an image enlargement method. Therefore, in the learning method, the above-described enhancement processing does not exist, and therefore, image enlargement with relatively little image quality deterioration can be realized.

  As an example of the learning method, for example, a technique is disclosed in which learning is performed by a statistical method assuming that the relationship between luminance values with adjacent pixels is determined by a Markov process (Non-Patent Document 3). In addition, a feature vector is obtained for each pixel in a conversion pair from a low resolution image to a high resolution image, and an enlarged image is generated based on the degree of coincidence with the feature vector of the input pixel and consistency with surrounding pixels. A technique is disclosed (Non-Patent Document 4).

The learning method is used not only for image enlargement but also for conversion of the illumination direction, for example (Non-Patent Document 5). In Non-Patent Document 5, illumination is applied to a plurality of objects having different textures (such as irregularities and patterns on the surface of the object) from a plurality of different directions, learning data is created, and the illumination direction is converted while maintaining a texture. The technology is disclosed.
US Pat. No. 5,717,789 (FIG. 5) Shinya Araya, “Meikyou 3D Computer Graphics”, Kyoritsu Shuppan, pp. 144-145, September 25, 2003 Nakashizuka et al., “Higher image resolution in multi-scale luminance gradient plane”, IEICE Transactions D-II Vol. J81-D-II No. 10 pp. 2249-2258, October 1998 Freeman et al., “Learning Low-Level Vision”, International Journal of Computer Vision 40 (1), pp. 25-47, 2000 Hertzmann et al., “Image Analogies”, SIGGRAPH 2001 Proceedings, pp. 327-340, 2001 Malik et al., “Representing and Recognizing the Visual Appearance of Materials using Three-dimensional Textons”, International Journal of Computer Vision 43 (1), pp. 29-44, 2001

  However, the prior art has the following problems.

  In the learning method described above, the learning data for image conversion is fixed. For this reason, when an image that is not assumed at the time of learning is input, learning data appropriate for image conversion of the image cannot always be acquired. As a result, the image quality of the converted image is not guaranteed, and there is a possibility that the image quality is deteriorated. In addition, there is no indication in the above-mentioned prior art documents about a coping method when an image that is not assumed at the time of learning is input.

  In view of the above problems, the present invention has an object to suppress image quality degradation in a converted image even when an image that is not assumed at the time of learning is input in image conversion using a learning method. And

  According to the present invention, as image conversion, an input image feature vector obtained by performing image feature analysis on a first image as a conversion source, obtaining a first image feature vector, and learning in advance according to predetermined image conversion A second image feature vector after a predetermined image conversion is obtained based on the first image feature vector by referring to the replacement data representing the correspondence between the group and the output image feature vector group. A second image after conversion is generated from the image feature vector.

  Further, according to the present invention, it is determined whether or not an image feature vector similar to the first image feature vector is included in the input image feature vector group included in the replacement data at the time of image feature vector conversion. If not, a second image feature vector is generated by interpolation using a plurality of image feature vectors selected from the output image feature vector group included in the replacement data.

  According to the present invention, with reference to the replacement data obtained by learning in advance, the first image feature vector obtained from the first image of the conversion source is used as the basis of the second image after conversion. Convert to image feature vector. In the conversion, when the image feature vector similar to the first image feature vector is not included in the input image feature vector group included in the replacement data, the output image feature included in the replacement data A second image feature vector is generated by interpolation using a plurality of image feature vectors selected from the vector group. That is, for the first image feature vector, if there is no image feature vector that is sufficiently similar in performing the image conversion in the replacement data learned in advance, an interpolation operation using a plurality of image feature vectors is performed. A second image feature vector is generated. For this reason, even when an image that is not assumed at the time of learning is input, a more appropriate second image feature vector is selected than simply selecting from the output image feature vector group included in the learning data. Can be generated. Therefore, it is possible to suppress deterioration in image quality in the converted second image.

  In the present invention, it is determined whether or not an image feature vector similar to the first image feature vector is included in the input image feature vector group included in the replacement data during the image feature vector conversion. If not included, a new pair of an input image feature vector and an output image feature vector corresponding to a predetermined conversion is generated and added to the replacement data.

  According to the present invention, with reference to the replacement data obtained by learning in advance, the first image feature vector obtained from the first image of the conversion source is used as the basis of the second image after conversion. Convert to image feature vector. In the conversion, when the image feature vector similar to the first image feature vector is not included in the input image feature vector group included in the replacement data, the input image corresponding to the predetermined conversion A new pair of feature vector and output image feature vector is generated and added to the replacement data. That is, for the first image feature vector, if there is no image feature vector that is sufficiently similar for performing image conversion in the replacement data learned in advance, the input image feature vector and output corresponding to the predetermined conversion are output. A new pair with the image feature vector is added to the replacement data. For this reason, even when an image that is not assumed at the time of learning is input, the learning data for image conversion is updated accordingly. Therefore, it is possible to suppress deterioration in image quality in the converted second image.

  For example, in the case of image enlargement, a first image feature vector and an image feature vector of an image obtained by photographing the subject of the first image at high resolution may be added as a pair. Further, in the case of illumination conversion or viewpoint conversion, the first image feature vector and the image feature of an image taken by illuminating the subject of the first image from another direction, or an image taken from another viewpoint Add a pair of vectors.

  According to the present invention, in image conversion using image feature vector conversion, interpolation of image feature vectors included in learning data is used even when an image that is not assumed at the time of learning is input. Thus, more appropriate image feature vector conversion can be realized. Further, learning data can be updated by adding a conversion pair of image feature vectors according to the input image. Therefore, image quality deterioration in the converted image can be suppressed.

In the first aspect of the present invention, as a method of performing predetermined image conversion on the first image and generating a second image, image feature analysis is performed on the first image, and the first image feature vector is obtained. An image feature analysis step for obtaining the first image, and replacement data representing a correspondence relationship between the input image feature vector group and the output image feature vector group obtained by learning in advance according to the predetermined image conversion, An image feature vector conversion step for obtaining a second image feature vector after the predetermined image conversion based on an image feature vector, and an image generation step for generating the second image from the second image feature vector The image feature vector converting step includes an image feature vector similar to the first image feature vector in the input image feature vector group included in the replacement data. A first step for determining whether or not the input image feature vector is included, and when it is determined in the first step that the input image feature vector and the output image feature vector are not included in the first step, And a second step of adding to the replacement data, and each of the input image feature vectors and each output image feature vector is associated with each type in the replacement data . The step of 2 determines the first type related to the first image by interpolation using the types respectively associated with the input image feature vector group included in the replacement data; and Obtaining the second type from the type according to the content of the predetermined image conversion, and based on the second type, the predetermined type for the first image And a step of acquiring an additional image to be a candidate of the image after the image conversion, and an image feature vector of the first image feature vector and the additional images in pairs, providing a shall be added to the replacement data To do.

In the second aspect of the present invention, the predetermined image transformation, the viewpoint direction or lighting direction is perspective transformation or illumination conversion predetermined angle changing, the kind, the first embodiment is an angle indicating the view direction or lighting direction An image conversion method is provided.

In the third aspect of the present invention, as a device that performs predetermined image conversion on the first image and generates the second image, image feature analysis is performed on the first image, and the first image feature vector is obtained. An image feature analysis unit that obtains the image data, and replacement data that represents a correspondence relationship between the input image feature vector group and the output image feature vector group obtained by learning in advance according to the predetermined image conversion. Referring to the first image feature vector, an image feature vector conversion unit that obtains a second image feature vector after the predetermined image conversion, and from the second image feature vector, the second image feature vector An image generation unit that generates an image, and the image feature vector conversion unit includes an image feature vector similar to the first image feature vector in the input image feature vector group included in the replacement data. A first means for determining whether or not the input image feature vector and the output image feature vector corresponding to the predetermined conversion are determined to be not included by the first means. And a second means for adding to the replacement data, and in the replacement data, each input image feature vector and each output image feature vector are represented in association with each other, and The second means obtains the first type of the first image by interpolation using the types respectively associated with the input image feature vector group included in the replacement data; A step of obtaining a second type from the first type according to the content of the predetermined image conversion, and after the predetermined image conversion for the first image based on the second type The candidate of the image, perform the acquiring additional images, to provide what is added to the replacement data and the image feature vector of the first image feature vector and the additional images in pairs.

In the fourth aspect of the present invention, the predetermined image conversion is viewpoint conversion or illumination conversion that changes a viewpoint direction or illumination direction by a predetermined angle, and the type is an angle that indicates the viewpoint direction or illumination direction. An image conversion apparatus is provided.

  Embodiments of the present invention will be described below with reference to the drawings.

(First embodiment)
FIG. 1 is a block diagram showing an image conversion apparatus according to the first embodiment of the present invention. With the configuration in FIG. 1, an image conversion method for performing a predetermined image conversion such as image enlargement on the input image IIN as the first image and generating the output image IOUT as the second image is realized. . The input image feature vector database 102, vector quantization unit 103, vector quantization error calculation unit 104, selector 105, image feature vector number replacement table 106, output image feature vector database 107, and image feature vector interpolation operation unit 108 The image feature vector conversion unit of the invention is configured.

  In the configuration of FIG. 1, replacement data representing a correspondence relationship between an input image feature vector group and an output image feature vector group is prepared by learning in advance according to predetermined image conversion. This replacement data includes an input image feature vector database 102, an output image feature vector database 107, and an image feature vector number replacement table 106. The input image feature vector database 102 stores in advance image feature vectors of a plurality of images assumed as input images. Each image feature vector is numbered. Similarly, the output image feature vector database 107 stores image feature vectors of a plurality of images assumed as output images with numbers. The image feature vector number replacement table 106 describes the correspondence between the input image feature vector group and the output image feature vector group by associating the image feature vector numbers with each other.

  In consideration of interpolation processing described later, the data in the image feature vector number replacement table 106 is preferably arranged so that input image feature vectors having consecutive numbers are adjacent to each other in the feature amount space. Further, by describing the correspondence between the input image feature vector group and the output image feature vector group using the image feature vector number, the data capacity of the image feature vector number replacement table 106 can be reduced.

  In FIG. 1, an image feature analysis unit 101 performs image feature analysis on an input image IIN, and generates an input image feature vector IINFV as a first image feature vector. The image feature analysis here is performed using spatial frequency analysis such as wavelet transform as shown in FIG. At this time, the image feature is represented by a multi-resolution expression. In the example of FIG. 2, wavelet transform outputs HL, LH, HH, and LL are obtained for each of n times of scaling, and these are combined for each layer, thereby obtaining a (3n + 1) -dimensional vector as an input image feature vector IINFV. . Since the image feature vector IINFV is required for each pixel, the LL image is set to have the same size in each scale. Of course, the image feature analysis and expression method of the present invention is not limited to this, and any method can be applied.

  The vector quantization unit 103 refers to the input image feature vector database 102, extracts an image feature vector most similar to the input image feature vector IINFV, and outputs it as an input image quantization feature vector IINQFV as a third image feature vector To do. In addition, the number assigned to the input image quantization feature vector IINQFV is output as the input image feature vector number IINFVN. The input image quantization feature vector IINQFV is provided to the vector quantization error calculation unit 104, and the input image feature vector number IINFVN is provided to the image feature vector number replacement table 106.

  Here, since the vector quantization unit 103 assigns the continuous input image feature vector IINFV to the discrete input image feature vector number IINFVN, a vector quantization error occurs. The vector quantization error calculation unit 104 calculates a difference between the input image quantization feature vector IINQFV and the original input image feature vector IINFV, that is, a vector quantization error VQE. The calculated vector quantization error VQE is given to the selector 105, and is used to select the output image feature vector IOUTFV as the second image feature vector, as will be described later.

  On the other hand, in the image feature vector number replacement table 106, one input-side node is selected according to the given input image feature vector number IINFVN. In FIG. 1, as an example, it is assumed that the node having the image feature vector number 2 is selected. At this time, the image feature vector number (20 in the example of FIG. 1) recorded on the output side of the same node is output as the candidate output image feature vector number CaIOUTFVN. The first candidate output image feature vector CaIOUTFV is given to the selector 105 by referring to the output image feature vector database 107.

  Also, the output numbers of several nodes (upper and lower in the example of FIG. 1) around the node selected by the input image feature vector number IINFVN (5, 20, and 7 in the example of FIG. 1). Is output as a candidate output image feature vector number group CbIOUTFVNs. Then, by referring to the output image feature vector database 107, it is converted into a candidate output image feature vector group CbIOUTFVs and given to the image feature vector interpolation calculation unit 108. The image feature vector interpolation calculation unit 108 generates one second candidate output image feature vector CbIOUTFV by interpolation calculation based on the candidate output image feature vector group CbIOUTFVs and supplies the second candidate output image feature vector CbIOUTFV to the selector 105.

  Here, the reason why the interpolation calculation is used is to avoid an increase in the degree of image quality degradation due to a mixture of large and small errors. The quantization error relating to the second candidate output image feature vector CbIOUTFV is an average error. This is to keep it at a minimum.

  The interpolation method in the image feature vector interpolation calculation unit 108 is arbitrary, but here, the weighted average of the candidate output image feature vector group CbIOUTFVs is obtained using the magnitude of the quantization error as a weighting coefficient. In this case, the vector quantization unit 103 also outputs the image feature vector SQFV corresponding to the upper and lower numbers of the input image feature vector number IINFVN to the vector quantization error calculation unit 104. The vector quantization error calculation unit 104 calculates a vector quantization error with respect to the input image feature vector IINFV for the given image feature vector SQFV, and provides the vector feature error to the image feature vector interpolation calculation unit 108.

The interpolation calculation is performed, for example, as (Equation 1).
Here, the vector F represents the second candidate output image feature vector CbIOUTFV to be obtained, and the vector Gi (i = 1 to j) represents the original candidate output image feature vector group CbIOUTFVs. In the example of FIG. Further, wi is a weighting coefficient and is given by (Equation 2).
Here, ei is a vector quantization error of the vector Gi. As can be seen from (Expression 2), the smaller the vector quantum error from the input image feature vector IINFV in the original candidate output image feature vector group CbIOUTFVs, the more the result of the calculation of the second candidate output image feature vector CbIOUTFV. , Contribute more.

  The selector 105 selects one of the first candidate output image feature vector CaIOUTFV and the second candidate output image feature vector CbIOUTFV as the output image feature vector IOUTFV by threshold processing for the vector quantization error VQE.

  That is, when the vector quantization error VQE is larger than a predetermined threshold (for example, 0.1), it is determined that an appropriate image feature vector cannot be extracted from the input image feature vector IINFV, and is obtained by interpolation calculation. The selected second candidate output image feature vector CbIOUTFV is selected. On the other hand, when the vector quantization error VQE is smaller than or equal to the predetermined threshold, the first candidate output image feature vector CaIOUTFV corresponding to the image feature vector most similar to the input image feature vector IINFV is selected.

  It can be said that it is determined whether the image feature vector similar to the input image feature vector IINFV is included in the input image feature vector group included in the replacement data by the threshold processing described above.

  The output image feature vector IOUTFV output from the selector 105 is converted into a luminance image in an output image generation unit 109 as an image generation unit, and is output as an output image IOUT.

  FIG. 3 is a flowchart showing the operation of the image conversion apparatus of FIG. 1, that is, the image conversion method according to the present embodiment. Note that the image conversion method according to the present embodiment can be realized by causing a computer to execute a program for realizing the method.

  First, in pre-processing S00, replacement data is prepared, and an input image feature vector database 102, an output image feature vector database 107, and an image feature vector number replacement table 106 are created. Details of the preprocessing S00 will be described later.

  In step S1, the image feature analysis unit 101 performs image feature analysis on the input image IIN. In step S <b> 2, vector quantization that assigns the input image feature vector IINFV to the closest one of the plurality of image feature vectors stored in the input image vector database 102 is performed. In step S3, a vector quantization error VQE that is a difference between the input image feature vector IINFV and the input image quantization feature vector IINVQFV is calculated.

  In step S4, the vector quantization error VQE is compared with a predetermined threshold value. If the vector quantization error VQE is larger than the threshold value, a candidate output image feature vector number group CbIOUTFVNs to be interpolated is acquired in step S10. Then, in step S11, the candidate output image feature vector number group CbIOUTFVNs is converted into the candidate output image feature vector group CbIOUTFVs with reference to the output image feature vector database 107, and in step S12, the interpolation operation as described above is performed. In S13, an output image IOUT is generated from the output image feature vector IOUTFV.

  On the other hand, when the vector quantization error VQE is smaller than or equal to the predetermined threshold value in step S4, the input image feature vector number IINFVN is replaced with the candidate output image feature vector number CaIOUTFVN in step S20. In step S21, the output image feature vector database 107 is referenced to convert it into a first candidate output image feature vector CaIOUTFV. In step S22, this is used as an output image feature vector IOUTFV to generate an output image IOUT.

  In the threshold determination in step S4, when the vector quantization error VQE matches the threshold, the process proceeds to step S20. However, the process may proceed to step S10.

<Preparation of replacement data>
The preprocessing S00, that is, the preparation of replacement data will be described with reference to FIGS. In FIG. 4, it is assumed that the predetermined image conversion is image enlargement. As shown in FIG. 4, the image feature vector of the low resolution image is stored in the input image feature vector database 102, and the image feature vector of the high resolution image is stored in the output image feature vector database 107. In the example of FIG. 4, the image feature vector is exemplified as a 10-dimensional vector obtained when the wavelet transform of FIG. The image feature vector number replacement table 106 is constructed from the correspondence between the same pixels of the low resolution image and the high resolution image.

  Note that the storage order of the replacement data is such that the input image feature vectors having consecutive numbers are adjacent to each other in the feature amount space in consideration of a case where a plurality of image feature vectors are interpolated. Is preferred.

  FIG. 5 shows three image feature vectors IFV1, IFV2, and IFV3 adjacent to each other. The feature amount space FS is an n-dimensional space composed of n types of feature amounts. When looking at IFV1, the shortest vector distance is IFV2, there is no other feature vector between IFV1 and IFV2 in the feature space FS, and IFV1 and IFV2 are adjacent to each other. . Similarly, when looking at IFV2 as the center, IFV3 has the shortest vector distance except IFV1, and there is another feature vector between IFV2 and IFV3 except IFV1 in feature space FS. The IFV2 and IFV3 are adjacent to each other. In this manner, adjacent image feature vectors are sequentially obtained from one image feature vector, and the replacement data is stored in the image feature vector number replacement table 106 in this order.

  In the above-described interpolation calculation, in order to improve the interpolation accuracy, an output image feature vector used for interpolation should be obtained from a node as close as possible to the input image feature vector in the feature amount space. Therefore, for example, if the replacement data is stored in the image feature vector number replacement table 106 in the order of the image feature vectors IFV1, IFV2, and IFV3 adjacent to each other in the feature amount space, the input image feature vector becomes the triangle A1 in FIG. When entering, an output image feature vector suitable for the interpolation operation can be easily extracted from successive nodes.

  The same applies to image conversion other than image enlargement, for example, illumination conversion and viewpoint conversion. That is, in the case of illumination conversion, two sets of images with different illumination directions may be prepared, and the image feature vector of each pixel may be registered in the image feature vector number replacement table 106. In the case of viewpoint conversion, two sets of images with different viewpoint directions may be prepared, and the image feature vector of each pixel may be registered in the image feature vector number replacement table 106.

  Further, although FIG. 4 has been described assuming that one set of replacement data is prepared for simplicity, a plurality of sets may be used.

<Update replacement data>
In this embodiment, replacement data is also updated. That is, when an appropriate input image feature vector is not included in the input image feature vector group, a new pair of the input image feature vector and the output image feature vector is added to the replacement data.

  FIG. 6 is a flowchart showing replacement data update processing in this embodiment, and FIG. 7 is a block diagram showing a configuration related to replacement data update processing. The difference unit 201, threshold determination unit 202, and data change unit 203 of FIG. 7 are included in the image feature vector conversion unit of the present invention.

  The process of FIG. 6 starts when YES is determined in step S4 of FIG. 3, that is, when it is determined that the vector quantization error is larger than the threshold value. In FIG. 3, when YES is determined in step S4, an image is generated by interpolating image feature vectors in steps S10 to S13. On the other hand, the process of FIG. 6 is executed in parallel with steps S10 to S13, and supplements the missing image feature vector conversion pairs. Therefore, when the processing of FIG. 6 is executed, image quality deterioration of image conversion for the current input image is suppressed.

  Here, the replacement data is updated together with the image generation by the interpolation calculation. However, the replacement data may be updated without performing the image generation by the interpolation calculation. Instead of updating the data, only image generation by interpolation calculation may be performed.

  First, in step S31, an additional candidate image INC that provides an output image feature vector that becomes a conversion pair with the input image feature vector IINFV is acquired. When image enlargement is considered as the predetermined image conversion, the additional candidate image INC becomes a high-resolution image, and therefore the same subject SUB as the input image IIN may be captured at a high resolution. The method for acquiring the high-resolution image is arbitrary, but as shown in FIG. 8, for example, in the case of the multifocal lens camera 401, the high-resolution image IMH is optically obtained by changing the focal length with respect to the subject SUB1. In the case of the single focus lens camera 402, the photographer approaches the subject SUB1 and shoots, or the subject SUB1 approaching the photographer is photographed to obtain a high resolution image IMH.

  Next, in step S32, reverse conversion of the content of predetermined image conversion is executed on the additional candidate image INC to obtain an inverse conversion input image IININV. For example, when the enlargement ratio in image enlargement is four times, image reduction of 1/4 is performed on the additional candidate image INC to obtain an inversely converted input image IININV.

  In step S33, the image feature analysis unit 101 performs image feature analysis on the inversely transformed input image IININV, and the difference unit 201 obtains a difference between the obtained additional candidate image feature vector INCFV and the input image feature vector IINFV. Further, the threshold determination unit 202 determines whether or not the difference is larger than a predetermined threshold. This threshold determination is to determine whether or not the additional candidate image INC is appropriate as an image after image conversion.

  When the difference is larger than the threshold value (YES in S33), it is determined that the additional candidate image INC is not appropriate as the converted image of the input image INN, the process returns to step S31, and the additional candidate image is acquired again. S32 and 33 are executed. On the other hand, when the difference is smaller than or equal to the threshold value, it is determined that the additional candidate image INC is appropriate as an image after image conversion, and the process proceeds to step S34. When the difference matches the threshold value, the process may return to step S31.

  In step S34, the data changing unit 203 adds a conversion pair of the input image feature vector IINFV and the additional candidate image feature vector INCFV to the replacement data. The position at which the conversion pair is added is determined based on the vector quantization error of the already registered input image feature vector group with respect to the input image feature vector IINFV. That is, a new transform pair is added between the node having the smallest vector quantization error and the second smallest node (between the image feature vector numbers 2 and 3 in FIG. 7). The number on the input side is incremented by “1” after the added node. Further, the number on the output side of the added node is “q + 1”.

  In FIG. 7, the order 801 with the smallest vector quantization error is described in the image feature vector number replacement table 106, but this is for convenience of explanation and is not particularly necessary when performing replacement.

  By such processing, the replacement pairs of image feature vector conversion pairs for the current input image IIN are supplemented to the replacement data. As a result, in the subsequent processing, even when an image equivalent to the input image IIN is input, the vector quantization error falls within a predetermined threshold.

  Here, the replacement data update processing has been described by taking the image enlargement as an example, but even if the image conversion is other than the image enlargement, it can be applied to any kind of image conversion as long as the reverse conversion is possible. is there.

(Second Embodiment)
In the second embodiment of the present invention, replacement data is updated by a method different from that of the first embodiment. That is, the type of the input image is specified, the type of the output image is determined according to the content of the image conversion, an additional image corresponding to the type of the output image is acquired, and the conversion pairs of the insufficient image feature vectors are supplemented. Other configurations and processes are the same as those in the first embodiment.

  FIG. 9 shows an example of replacement data premised on the present embodiment. In FIG. 9, the predetermined image conversion is a viewpoint conversion in which the viewpoint direction is changed by a predetermined angle (90 degrees here). In the image feature vector number replacement table 106, each input image feature vector and each output image feature vector are associated with an angle indicating the viewpoint direction as a type.

  The conversion source image IS has a vertical stripe, but when the viewpoint direction is rotated by 90 degrees, a horizontal stripe is formed as in the converted image ID. The input image feature vector database 102 and the output image feature vector database 107 have p types of image feature vector pairs whose viewpoint directions differ by 90 degrees. In the image feature vector number replacement table 106, 360-degree degrees of viewpoint angles obtained by dividing 360 degrees into p types are recorded as types.

  In the present invention, the “type” added to the image feature vector is determined according to the content of the image conversion. For example, in the viewpoint conversion, an angle indicating the viewpoint direction is a type, and in the illumination conversion, an angle indicating the illumination direction is a type. Note that the subject and the lighting environment are the same.

  FIG. 10 is a flowchart showing replacement data update processing in this embodiment. The process of FIG. 10 starts when YES is determined in step S4 of FIG. 3, that is, when it is determined that the vector quantization error is larger than the threshold value. The process of FIG. 10 is executed in parallel with steps S10 to S13 of FIG. 3, and supplements the missing image feature vector conversion pairs. Therefore, when the processing of FIG. 10 is executed, image quality deterioration of image conversion for the current input image is suppressed. It should be noted that the replacement data in this embodiment may be updated without generating an image by interpolation calculation.

  In step S41, the first type related to the input image IIN is determined. The type is determined by obtaining an image feature vector located in the vicinity of the input image feature vector IINFV and interpolating these.

Now, it is assumed that No. 2 is obtained as the number of the image feature quantity vector most similar to the input image feature vector IINFV. At this time, the image feature vectors of No. 2 and above and below, that is, No. 1 and No. 3 are acquired, and the type of the input image is determined from (Equation 3).
Here, c is the type of input image to be obtained, cj is the type of each image feature vector, m is the number of image feature vectors used for interpolation (here, 3), and wj is a weighting coefficient. The weighting coefficient wj is given by (Equation 4).
Here, ej is the distance between each image feature vector and the input image feature vector IINFV, and the smaller the ej, the greater the contribution to the type calculation. Here, since the image feature vector of number 2 is assigned by vector quantization of the input image feature vector IINFV, the distance between the input image feature vector IINFV and the image feature quantity vector of number 2 is shown in FIG. D2 is the shortest, and the contribution ratio in the interpolation is the largest. The next shortest is the distance D1, and the distance D3 is the longest. Therefore, the image feature vector number 3 has the smallest contribution in interpolation.

  In step S42, the type of output image is determined from the type of input image and the content of image conversion. In the case of FIG. 9, since the content of the image conversion is conversion of the viewpoint direction at an angle of 90 degrees, the type of the output image is an angle obtained by adding 90 degrees to the angle obtained as the type of the input image.

  In step S43, an additional image corresponding to the type of output image is acquired. In the case of FIG. 9, since the type of the output image is an angle obtained by adding 90 degrees to the type of the input image, the subject is photographed from a direction rotated 90 degrees from the viewpoint direction of the input image, and an additional image is obtained.

  In step S44, a conversion pair of the input image feature vector IINFV and the image feature vector of the additional image is added to the replacement data.

  By such processing, the replacement pair of the image feature vector conversion pair for the current input image IIN is supplemented. As a result, in the subsequent processing, even when an image equivalent to the input image IIN is input, the vector quantization error falls within a predetermined threshold.

  In the above example, the image conversion is the viewpoint conversion. However, even if the image conversion is other than this, the conversion is applicable if the conversion source image and the converted image can be acquired together with the image type. Is possible.

  FIG. 12 shows another example of replacement data in this embodiment. In FIG. 12, the predetermined image conversion is image enlargement in which the image is enlarged by a predetermined magnification (here, 8 times). For example, assuming that an image of 512 pixels × 512 lines has an image magnification of 1 ×, 256 pixels × 256 lines in which the number of pixels and the number of lines are halved is an image magnification of ½ ×, and the number of pixels and the number of lines is 4 ×. Thus, 2048 pixels × 2048 lines has an image magnification of 4 times. In the image feature vector number replacement table 106, each input image feature vector and each output image feature vector are represented in association with the magnification of the image as a type. Specifically, an image magnification obtained by dividing 1/4 to 4 times into p types in the same magnification relationship is recorded as the type. The storage order is ascending order of magnification, but may be descending order of magnification.

To obtain the image magnification 4 by multiplying the image magnification ¼ by the same magnification (p−1) times,
The magnification which becomes is good. For example, when p = 5, (Equation 5) is 2, so that 4 (= 1/4 × 2 × 2 × 2 × 2) is obtained by multiplying the image magnification ¼ by 2 four times. In general, to obtain the image magnification n by multiplying the image magnification m by the magnification α (p−1) times, the magnification α is
It becomes.

  Since the content of the image conversion is 8 times enlargement, the type of the output image is an image magnification obtained by multiplying the image magnification given as the type of the input image by 8. Here, since an image of 512 pixels × 512 lines is set to an image magnification of 1 ×, an image of 128 pixels × 128 lines corresponding to an image magnification of 1/4 × is 1024 pixels corresponding to an image magnification of 2 × by 8 × magnification. It is converted into an image of × 1024 lines.

  Similarly to FIG. 9, replacement data as in FIG. 12 can also be updated using interpolation calculation using an image feature vector located in the vicinity of the input image feature vector IINFV according to the flowchart of FIG. 10. it can.

  In FIG. 12, the equal-magnification relationship is used when giving the type of the input image, but an equality relationship may be used. The “same size” relationship means that the “division value” of the adjacent upper and lower nodes is the same in all combinations, and the “equal division” relationship means that the “difference value” of the adjacent upper and lower nodes is the same. Means equal in all combinations.

(Third embodiment)
FIG. 13 is a time chart showing an image conversion method according to the third embodiment of the present invention. Here, it is assumed that the content of the image conversion is image enlargement and the image is captured while changing the shooting magnification with a video camera.

  The video camera has the configuration shown in FIGS. 1 and 7, and operates according to the flowcharts shown in FIGS. That is, when the video camera executes the image enlargement of the learning method with the configuration of FIG. 1 and an input image feature vector for which appropriate replacement data is not prepared is generated, and the replacement data for image expansion is insufficient In accordance with the flow of FIG. 3, an image feature vector is obtained by interpolation. In addition, with the configuration of FIG. 7, in accordance with the flow of FIG. 6, a conversion pair of insufficient image feature vectors is acquired and added to the replacement data.

  For this reason, even when the photographer increases the shooting magnification and zooms in, an enlarged image can be generated by the image enlargement process of the learning method. At this time, since the optical zoom does not move and shooting can be continued at a wide angle of view, the narrow angle of view due to zoom-in does not occur, and there is less possibility that the attention point will deviate from the angle of view and spill out. can get.

  FIG. 13A is a graph showing the change over time of the shooting magnification determined by the photographer and the shooting magnification of the optical zoom of the video camera. The vertical axis represents shooting magnification, and the horizontal axis represents time. The broken line is the photographing magnification determined by the photographer, and the solid line is the photographing magnification of the optical zoom. The control system of the video camera can easily recognize that the photographing magnification of the optical zoom matches the photographing magnification designated by the photographer by comparing the photographing magnifications. When the shooting magnification increases, the zoom-in state is entered, and when the shooting magnification decreases, the zoom-out state is entered.

  FIG. 13B is a graph showing the change over time of the vector quantization error corresponding to FIG. 13A, and the vertical axis is the vector quantization error. The illustrated threshold value corresponds to the predetermined threshold value used in step S4 of FIG. FIG. 13C is a graph showing the operation of the video camera corresponding to FIG. As operations, image feature vector interpolation (S10 to S13 in FIG. 3), image feature vector replacement (S20 to S22 in FIG. 3), video recording by optical zoom, image acquisition (S31 to S33 in FIG. 6), and data addition ( FIG. 6 shows S34).

  At time t1, the photographer increases the shooting magnification and instructs to zoom in. However, since the image enlargement is executed by the image conversion process, the magnification of the optical zoom is not changed. Further, since the vector quantization error is smaller than the threshold, the selector 105 in FIG. 1 selects the first candidate output image feature vector CaIOUTFV as the output image feature vector IOUTFV. That is, “image feature vector replacement” is executed.

  However, since an error 1 exceeding the threshold occurs at time t2, in order to acquire a high resolution image by optical zoom, the imaging magnification of the optical zoom is changed to the imaging magnification determined by the photographer. From time t2 to time t3 when the optical zoom reaches the photographing magnification determined by the photographer, the selector 105 in FIG. 1 selects the second candidate output image feature vector CbIOUTFV as the output image feature vector IOUTFV. That is, “image feature vector interpolation” is executed. After time t3, since an image with a shooting magnification determined by the photographer has been acquired by optical zoom, image feature vector interpolation is not performed, and a captured image by optical zoom is used as an output image.

  On the other hand, in order to perform additional update of replacement data, “image acquisition” (acquisition of high resolution image and reduction in resolution thereof) is executed from time t2, and addition of appropriate learning data is started. Since appropriate learning data is found at time t4 (NO in step S33 in FIG. 6), “data addition” is executed, and the conversion pair for error 1 is added to the replacement data. As a result, image enlargement with respect to the input image at time t1 can be executed by the learning method, and the magnification of the optical zoom returns to the original 1 and is zoomed out. Since the photographer continues to maintain the zoom-in state, the image enlargement is executed by the image conversion process, and the video imaged at the wide angle of view is recorded. The above example is a case where the vector quantization error does not exceed the threshold until “data addition” is completed.

  Next, it is assumed that the vector quantization error continuously exceeds the threshold value, and error 2, error 3, and error 4 are continuously generated. Since an error 2 occurs at time t5, the optical zoom shifts to the shooting magnification determined by the photographer, and the shift is completed at time t7. For this reason, the “image feature vector interpolation” started at time t5 is interrupted at time t7, and thereafter, until the time t12 when the replacement data is updated, the imaging magnification of the optical zoom is set to the imaging magnification determined by the photographer. Set and record high resolution image by optical zoom.

  From time t7, “data addition” for error 3 has been executed, but since learning data for error 2 has not yet been found, “image acquisition” remains on. An error 4 occurred at time t9 while no learning data with error 2 was found, and it was necessary to additionally update the learning data for the two errors. “Data addition” for error 2 is executed from time t10, and “data addition” for error 4 is executed from time t12, so that the image can be enlarged by image conversion. Therefore, from time t13 when the “data addition” is completed, the optical zoom starts zooming out and returns to the photographing magnification of 1 again.

  Further, although an error 5 occurs at time t14, since the magnification determined by the photographer is 1, it is not necessary to move the optical zoom. Therefore, it is not necessary to execute “image feature vector interpolation”, and “image feature vector replacement” is executed. On the other hand, since learning data needs to be supplemented, “image acquisition” is started. Since the photographer increased the photographing magnification at time t15 while the additional update of the learning data for the error 5 has not been achieved, the optical zoom follows this. Since learning data to be additionally updated is found at time t16, “image acquisition” is turned OFF, and “data addition” for error 5 is executed. Image enlargement can be performed by image conversion after time t17 when the addition of the learning data is completed. For this reason, even if the photographing magnification determined by the photographer continues to increase, the optical zoom zooms out and returns to the photographing magnification of 1.

  Hereinafter, configuration examples for realizing the present invention will be exemplified.

(First configuration example)
FIG. 14 is a diagram showing a first configuration example, which is an example of a configuration for performing image conversion according to the present invention using a personal computer. Here, image enlargement is assumed as an example of image conversion. The resolution of the camera 501 is lower than the resolution of the display 502, and an enlarged image is created by an image conversion program loaded in the main memory 503 in order to make full use of the display capability of the display 502. The low resolution image captured by the camera 501 is recorded in the image memory 504. The external storage device 505 is prepared in advance with an input image feature vector database 102, an output image feature vector database 107, and an image feature vector number replacement table 106 as shown in FIG. It can be referred from.

  The operations of the image conversion program, the contents of the input image feature vector database 102, the output image feature vector database 107, and the image feature vector number replacement table 106, the creation method, and the like are as described in the first and second embodiments. .

  The image conversion program in the main memory 503 reads a low resolution image in the image memory 504 via the memory bus 508, converts it into a high resolution image in accordance with the resolution of the display 502, and again converts the video memory 509 via the memory bus 508. Forward to. The high resolution image transferred to the video memory 509 is displayed on the display 502.

  When performing viewpoint conversion as image conversion, the viewpoint conversion is performed by an image conversion program loaded in the main memory 503 in order to change the viewpoint direction and display the subject captured by the camera 501 on the display 502. The image conversion program reads an original image from the image memory 504 via the memory bus 508, converts the original image according to the content of the viewpoint conversion instruction, and transfers it again to the video memory 509 via the memory bus 508. The viewpoint conversion image transferred to the video memory 509 is displayed on the display 502.

  Note that the present invention is not limited to the configuration of FIG. 14 and can have various configurations. For example, the input image feature vector database 102, the output image feature vector database 107, and the image feature vector number substitution table 106 may be acquired from an external storage device connected to another personal computer via the network 510. The image acquired by the camera 501 may be acquired via the network 510.

(Second configuration example)
FIG. 15 is a diagram showing a second configuration example, which is an example of a configuration for performing image conversion according to the present invention using a server client system. Here, image enlargement is assumed as an example of image conversion. The resolution of the camera 601 is lower than the resolution of the display 602, and image conversion is executed in the server client system in order to make the best use of the display capability of the display 602. In the server 603, the image feature analysis unit 608 corresponds to the image feature analysis unit 101 in FIG. 1, and the image feature vector conversion unit 604 includes the input image feature vector database 102, the vector quantization unit 103, the vector in FIG. Quantization error calculation unit 104, selector 105, image feature vector number replacement table 106 and output image feature vector database 107, difference unit 201, threshold value determination unit 202, and data change unit 203 shown in FIG. The input image IIN is enlarged and output as an output image feature vector IOUTFV.

  The output image feature vector IOUTFV is transmitted to the client 606 via the network 605. The client 606 converts the received output image feature vector IOUTFV into an image by the image generation unit 607, and displays the obtained enlarged image on the display 602.

  Further, when performing viewpoint conversion as image conversion, image conversion is executed in order to display a subject photographed by the camera 601 on the display 602 while changing the viewpoint direction. In the server 603, the image feature vector conversion unit 604 converts the viewpoint direction of the input image IIN and outputs it as an output image feature vector IOUTFV. The output image feature vector IOUTFV is transmitted to the client 606 via the network 605. The client 606 converts the received output image feature vector IOUTFV into an image by the image generation unit 607 and displays the obtained viewpoint change image on the display 602.

  In addition, this invention is not restrained by the structure of FIG. 15, and can take various structures. For example, the camera 601 may be a part of the client 606.

  In the present invention, since various image conversions such as image enlargement, illumination conversion, and viewpoint conversion can be performed freely, for example, it is used in the video entertainment field for recording a scene in front of the eyes such as sports, sightseeing, and commemorative photography. be able to. In the field of culture and art, it can be used to provide a highly flexible digital archive system that is not limited by the subject or shooting location.

1 is a block diagram illustrating an image conversion apparatus according to a first embodiment of the present invention. It is a figure showing the image feature analysis using a wavelet transform. It is a flowchart which shows the image conversion method which concerns on the 1st Embodiment of this invention. It is a figure shown about the preparation of the replacement data for image conversion. It is a conceptual diagram which shows the image feature vector in the feature-value space. It is a flowchart which shows the update process of the replacement data in the 1st Embodiment of this invention. It is a block diagram which shows the structure regarding the update of replacement data. It is a figure which shows the example of the method of acquiring a high resolution image. It is an example of the replacement data which the 2nd Embodiment of this invention presupposes. It is a flowchart which shows the update process of the replacement data in the 2nd Embodiment of this invention. It is a figure which shows notionally the distance of image feature vectors in the feature-value space. It is another example of the replacement data which the 2nd Embodiment of this invention presupposes. It is a time chart which shows the image conversion method which concerns on the 3rd Embodiment of this invention. It is a 1st example of composition which realizes the present invention, and is a figure showing composition using a personal computer. It is a 2nd structural example which implement | achieves this invention, and is a figure which shows the structure using a server client system.

101 Image Feature Analysis Unit 102 Input Image Feature Vector Database 103 Vector Quantization Unit 104 Vector Quantization Error Calculation Unit 105 Selector 106 Image Feature Vector Number Replacement Table 107 Output Image Feature Vector Database 108 Image Feature Vector Interpolation Operation Unit 109 Output Image Generation Unit 201 Difference unit 202 Threshold determination unit 203 Data change unit IIN Input image (first image)
IINFV input image feature vector (first image feature vector)
IINFVN input image feature vector number IINVQFV input image quantization feature vector (third image feature vector)
CaIOUTFVN Candidate output image feature vector number CbIOUTFVNs Candidate output image feature vector number group CbIOUTFVs Candidate output image feature vector group CaIOUTFV First candidate output image feature vector CbIOUTFV Second candidate b output image feature vector VQE Vector quantization error IOUTFV Output image feature Vector (second image feature vector)
IOUT output image (second image)

Claims (4)

A method of performing a predetermined image conversion on a first image and generating a second image,
An image feature analysis step for performing an image feature analysis on the first image to obtain a first image feature vector;
With reference to replacement data representing a correspondence relationship between the input image feature vector group and the output image feature vector group obtained by learning in advance according to the predetermined image conversion, based on the first image feature vector, An image feature vector conversion step for obtaining a second image feature vector after the predetermined image conversion;
An image generation step of generating the second image from the second image feature vector,
The image feature vector conversion step includes:
A first step of determining whether or not an input image feature vector group included in the replacement data includes an image feature vector similar to the first image feature vector;
In the first step, when it is determined that it is not included, a second pair of an input image feature vector and an output image feature vector corresponding to the predetermined conversion is generated and added to the replacement data. With steps,
In the replacement data, each input image feature vector and each output image feature vector are each associated with a type,
The second step includes
Obtaining a first type of the first image by interpolation using types associated with the input image feature vector group included in the replacement data; and
Obtaining a second type from the first type according to the content of the predetermined image conversion;
Obtaining an additional image based on the second type, which is a candidate for the image after the predetermined image conversion for the first image,
An image conversion method, wherein the first image feature vector and the image feature vector of the additional image are added to the replacement data as a pair. In claim 1 ,
The predetermined image conversion is a viewpoint conversion or illumination conversion that changes a viewpoint direction or an illumination direction by a predetermined angle,
The image conversion method characterized in that the type is an angle indicating a viewpoint direction or an illumination direction. An apparatus that performs predetermined image conversion on a first image and generates a second image,
An image feature analysis unit that performs image feature analysis on the first image and obtains a first image feature vector;
It has replacement data representing a correspondence relationship between an input image feature vector group and an output image feature vector group obtained by learning in advance according to the predetermined image conversion. The first image is referred to by referring to the replacement data An image feature vector conversion unit for obtaining a second image feature vector after the predetermined image conversion based on a feature vector;
An image generation unit that generates the second image from the second image feature vector,
The image feature vector conversion unit includes:
First means for determining whether or not an input image feature vector group included in the replacement data includes an image feature vector similar to the first image feature vector;
A second pair for generating a new pair of an input image feature vector and an output image feature vector according to the predetermined transformation and adding the new pair to the replacement data when it is determined by the first means not to be included. der those with the means is,
In the replacement data, each input image feature vector and each output image feature vector are each associated with a type,
The second means includes
Obtaining a first type of the first image by interpolation using types associated with the input image feature vector group included in the replacement data; and
Obtaining a second type from the first type according to the content of the predetermined image conversion;
Obtaining an additional image that is a candidate for the image after the predetermined image conversion for the first image based on the second type;
The image conversion apparatus characterized in that the first image feature vector and the image feature vector of the additional image are paired and added to the replacement data . In claim 3,
The predetermined image conversion is a viewpoint conversion or illumination conversion that changes a viewpoint direction or an illumination direction by a predetermined angle,
The type is an angle indicating a viewpoint direction or an illumination direction.
An image conversion apparatus characterized by that.