JP2003153159A - Method and device for processing image and computer program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an encoding and decoding technology for realizing high compressibility and high image quality. SOLUTION: An image inputting part 12 acquires an image showing a facial expression when a person says a word as a key frame in each voice KF. A voice recognizing part 18 recognizes each voice included in voice data S acquired by a voice inputting part 16. An arranging part 20 decides an arrangement of key frames KF so as to make voices to be in the order of each of recognized voices. A processor 22 acquires corresponding point information C between adjacent key frames KF in the decided arrangement. A stream generating part 24 generates an encoded data stream CI in a form including a key frame KF corresponding to the each of recognized voices and the corresponding point information C.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image processing technique, and more particularly to an encoding and decoding method and apparatus for smoothly reproducing a moving image on the receiving side when transmitting and receiving image data via a network. .

[0002]

2. Description of the Related Art With the explosive spread of mobile phones, the exchange of e-mails and still image data using mobile phone terminals has become popular. In the future, delivery of high quality moving image data using mobile phone terminals is desired.

[0003]

The distribution of moving image data via a network cannot be considered without a compression coding technique. In particular, the commercial value of a mobile phone is its light weight, long battery life, low hardware cost, and light operability, and it is reluctant to download heavy image data over a long period of time. Further, the CPU power for processing such heavy image data is also disadvantageous in terms of power consumption.

Therefore, as a method for realizing the compression coding in the mobile phone terminal, a technique that enables both higher compression rate and image quality is desired.

The present invention has been made in view of such a situation, and an object thereof is to provide a good moving picture compression technique.

[0006]

The image coding technique and the image decoding technique of the present invention will be described below for the case of generating an intermediate image from two still images and generating this as a moving image. Specifically, it relates to a moving image compression technique.

[0007]

One aspect of the present invention relates to an image processing method. In this method, a terminal capable of displaying video acquires data that can be converted into audio, acquires an image corresponding to the audio after conversion as a key frame, and intermediates based on the acquired key frame. The method includes the steps of generating an image and generating a moving image, and displaying the moving image in synchronization with the sound. The step of generating the intermediate image may utilize a multi-resolution singularity filter,
A method such as optical flow or motion vector may be used.

Here, "data that can be converted into voice" means
Generally, voice data is assumed, but not limited to this, text data may be used. A predetermined voice is associated with the key frame. For example, the voice "a" may correspond to the facial expression when the voice is pronounced as a key frame. A human face is generally used as the facial expression, but the present invention is not limited to this, and may be an animal, for example, a pet such as a dog or cat. The correspondence between the voice and the facial expression may be arbitrary. In particular, in the case of animals, it is difficult to associate a specific voice with a facial expression, so a specific voice may be associated with an arbitrary facial expression.

[0009] Further, even if different sounds are generated, the sounds may be associated with the same key frame if the expressions of their pronunciations are substantially the same. For example, if the facial expressions for pronouncing the voice of "a" and the voice of "ka" are similar, "a" and "ka" are associated with the same key frame.

As a result, it is possible to make a call while viewing the image of the other party as a moving image such as a videophone as long as it is voice data. Also, any number of voices can be associated with one key frame. When many voices are associated with one key frame, fewer key frames need to be prepared.

Although a videophone requires a larger band than a normal voice-only call, when a videophone is realized by a mobile phone, it is not possible to charge in proportion to the band occupation, and communication is not possible. There is a problem in that it is troublesome for the business operator and that the line is inconvenient for the user. Therefore, the key frame to be prepared is, for example, 5,
By suppressing the number to a small number such as 6, it is possible to realize a videophone with a mobile phone without being aware of the above-mentioned problems, although it is a pseudo. The number of intermediate images depends on the processing capability of the receiving device, and if the capability is low, only the key frame may be displayed without intentionally generating an intermediate image.

The method further includes the step of saving the key frame, wherein the step of acquiring the image corresponding to the sound as the key frame acquires and saves the image prior to the step of acquiring the data convertible to the sound. By setting this, if a situation occurs in which a moving image is generated from a key frame, the amount of communication data may be suppressed by using the saved key frame. When often communicating with a specific party, it is inefficient to acquire a key frame for each call. Therefore, in a general call, the other party can be specified. Therefore, if the key frame associated with the specified party is already saved, the saved key frame is selected and an intermediate image is generated to create a video. Is displayed. Further, not only for incoming calls but also for outgoing calls, if the other party is already registered, the key frame may be used to generate a moving image during a call.

The key frame need not be an image of the facial expression of the other party of the call, but an image of another target may be used. For example, a video may be displayed using images of athletes and actors. It may be obtained from a given site on the network.

Another aspect of the present invention also relates to an image processing method. This method is a method of notifying that by a moving image and a predetermined voice when information is received on a terminal capable of displaying a video, and a step of acquiring an image corresponding to the voice as a key frame. , And a step of generating an intermediate image based on the acquired key frame to generate a moving image, and a step of displaying the generated moving image in synchronization with the sound.

When receiving an e-mail or a call, a moving image may be displayed together with a voice saying "There is an incoming call from Mr. A" to notify the user. An image may be attached to an email in advance as a key frame, and an intermediate image may be created based on the image to display a moving image.

Another aspect of the present invention relates to an image distribution method. This method includes the steps of holding an image of facial expressions associated with voice at a predetermined site on the network, and delivering the image to the user in response to the user's request. These images may be still images or moving images, and this distribution may be paid or free.

Yet another aspect of the present invention relates to an image processing method. In this method, an image obtained by capturing the facial expression of the user when uttering a specific sound is held in association with the specific sound, and when the user utters a word or sentence relating to the continuation of the specific sound, the word or An image corresponding to a specific sound forming a sentence is displayed continuously and after smoothing processing. The smoothing process may be morphing.

Another aspect of the present invention also relates to an image processing method. This method, from a predetermined site on the network,
A step of acquiring and storing an image to which a sound is associated as a key frame, a step of acquiring data that can be converted into a sound, and a key frame corresponding to the sound after the data conversion is selected from the saved key frames. And a step of generating an intermediate image by using a multi-resolution singularity filter based on those key frames to generate a moving image, and a step of synchronizing and outputting the moving image and the sound.

Another aspect of the present invention relates to an image processing apparatus. This device includes a data input unit that acquires data that can be converted into voice, a voice conversion unit that converts that data into voice, an image acquisition unit that acquires a key frame corresponding to the voice after conversion, voice and a key. A correspondence unit that associates frames, an intermediate image generation unit that generates an intermediate image by using a multi-resolution singularity filter based on a key frame to generate a moving image, an audio output unit that outputs audio, and the moving image And an image output unit for outputting in synchronization with voice. It may further include a key frame storage unit that stores the acquired key frame.

The image acquisition unit may acquire a key frame from a predetermined site on the network and store the key frame in the key frame storage unit. Good. The image acquisition unit further determines whether the key frame has been acquired,
It is also possible to acquire only key frames that have not been acquired yet and store them in the key frame storage unit. The image acquisition unit may have a function of selecting a desired key frame from the stored key frames, or an image selection unit having this function may be separately provided.

Another aspect of the invention relates to a computer program. This program includes a step of acquiring data that can be converted into audio on a terminal capable of displaying video,
A step of acquiring an image corresponding to the converted sound as a key frame, a step of generating an intermediate image based on the acquired key frame to generate a moving image, and a step of displaying the moving image in synchronization with the sound. Let the computer run.

Another aspect of the present invention also relates to a computer program. This program is a program for notifying the user of the arrival of information by a moving image and a predetermined voice when a terminal capable of displaying a video receives a step of acquiring an image corresponding to the voice as a key frame. , Causing the computer to execute a step of generating an intermediate image based on the acquired key frame to generate a moving image and a step of displaying the moving image in synchronization with the sound.

It is also possible to arbitrarily replace the above components and processing contents, replace part or all of the expressions between the method and the apparatus, or add the expressions, or change the expressions to a computer program, a recording medium, or the like. The present invention is effective.

[0024]

BEST MODE FOR CARRYING OUT THE INVENTION First, a multiresolutional singularity filter technique used in the embodiments and an image matching process using the technique will be described in detail as a "base technique". These techniques are the ones that the applicant has already obtained in Japanese Patent No. 2927350, and are optimal for combination with the present invention. However, the image matching technique that can be adopted in the embodiment is not limited to this. From FIG. 18 onward, the image processing technology using the base technology will be specifically described. [Background of prerequisite technology] First, the elemental technology of the prerequisite technology will be described in detail in [1], and the processing procedure will be specifically described in [2]. In addition, we will report the results of the experiment in [3].

[1] Details of Elemental Technology [1.1] Introduction A new multi-resolution filter called a singular point filter is introduced to accurately calculate matching between images. No prior knowledge of objects is required. The calculation of matching between images is calculated at each resolution while going through the hierarchy of resolutions. In that case, the hierarchy of resolution is traced in order from a coarse level to a fine level. The parameters required for the calculation are fully automatic set by a dynamic calculation similar to the human visual system. It is not necessary to manually specify the corresponding points between images.

The base technology can be applied to, for example, completely automatic morphing, object recognition, stereoscopic photogrammetry, volume rendering, and generation of smooth moving images from a small number of frames. When used for morphing, a given image can be transformed automatically. When used for volume rendering, it is possible to accurately reconstruct an intermediate image between cross sections. The same applies when the distance between the cross sections is large and the shape of the cross section changes significantly.

[1.2] Hierarchical Singularity Filter Hierarchy The multi-resolutional singularity filter according to the prerequisite technique can reduce the resolution of an image while preserving the brightness and position of each singularity contained in the image. Here, the width of the image is N and the height is M. For simplicity, N = M = 2 ⁿ
(N is a natural number). Also, the interval [0, N] ⊂R
Is described as I. Let p be the pixel of the image at (i, j)
_It is described as _{(i, j)} (i, jεI).

Here, a multi-resolution hierarchy is introduced. hierarchy
The digitized image group is generated by a multi-resolution filter. Many
The double resolution filter is a two-dimensional search for the original image.
Search for singularities and extract the detected singularities
Generate another image with a lower resolution than the original image.
Here, the size of each image at the m-th level is 2^m× 2 ^m
(0 ≦ m ≦ n). There are four types of singularity filters:
Construct a new hierarchical image recursively downward from n
It

[0029]

[Equation 1] However, here

[Equation 2] And Hereinafter, these four images will be referred to as sub-images. min _{x ≦ t ≦ x + 1} , max
_{When x ≦ t ≦ x + 1} is described as α and β, the sub-images can be described as follows.

[0030] P^{(M, 0)}= Α (x) α (y) p^{(M + 1,0)} P^{(M, 1)}= Α (x) β (y) p^{(M + 1,1)} P^{(M, 2)}= Β (x) α (y) p^{(M + 1, 2}) P^{(M, 3)}= Β (x) β (y) p^{(M + 1,3)} That is, they are like the tensor product of α and β
Conceivable. Each sub-image corresponds to a singular point.
As is clear from these equations, the singularity filter
For each block of 2 × 2 pixels
Detect singularity. At that time, two people of each block
The maximum pixel value or the minimum pixel in the direction, that is, in the vertical and horizontal directions
Search for points with values. As a pixel value, it is bright in the base technology.
Degree, but various numerical values related to images are used.
You can Maximum pixels in both directions
The value pixel is the maximum for both the maximum point and the two directions.
Pixels with a small pixel value are the minimum point and one of the two directions.
Is the maximum pixel value, and the other is the minimum pixel value.
Pixels having a value are detected as saddle points.

The singularity filter represents the image of the block (here, 4 pixels) by the image of the singularity (here, 1 pixel) detected inside each block.
Reduce the image resolution. From the theoretical point of view of the singularity, α (x) α (y) preserves the minimum point and β (x) β
(Y) preserves the maximum points, α (x) β (y) and β
(X) α (y) stores the saddle point.

First, a singularity filter process is separately applied to a starting point (source) image and an ending point (destination) image to be matched, and a series of image groups, that is, a starting point hierarchical image and an ending point hierarchical image are generated. Keep it. The start point hierarchical image and the end point hierarchical image are generated in four types corresponding to the types of singular points.

After that, the matching between the starting point hierarchical image and the ending point hierarchical image is achieved in a series of resolution levels. First, the minimum point is matched using p ^{(m, 0)} . Then, based on the result, saddle points are matched using p ^{(m, 1)} and other saddle points are matched using p ^{( m, 2)} . And finally p ^{(m, 3)}
The maximum points are matched using.

FIG. 1C and FIG. 1D are respectively shown in FIG.
The sub-image p ^{(5, 0)} of (a) and FIG. 1 (b ⁾ is shown. Similarly, in FIG. 1 (e) and FIG. 1 (f), p ^(5,1) ,
1 (g) and 1 (h) show p ^(5,2) , and FIGS. 1 (i) and 1 (j) show p ^(5,3) , respectively. As can be seen from these figures, the sub-image facilitates matching of characteristic portions of the image. First, p ^(5,0) makes the eyes clear. This is because the eyes are the minimum points of brightness in the face. The mouth becomes clear according to p ^(5,1) . This is because the mouth has low brightness in the lateral direction. According to p ^(5,2) , the vertical lines on both sides of the neck become clear. Finally, p ^{(5,3) defines} the brightest points on the ears and cheeks. This is because these are maximum points of brightness.

Since the feature of the image can be extracted by the singularity filter, for example, the feature of the image photographed by the camera,
By comparing the characteristics of several objects recorded in advance, it is possible to identify the subject reflected in the camera.

[1.3] Calculation of mapping between images The pixel at the position (i, j) of the starting point image is p ⁽ⁿ⁾ _{(i, j)}
Similarly, the pixel at the position (k, l) of the end point image is changed to q
^(N) _{Described in (k, l)} . Let i, j, k, lεI. The energy of mapping between images (described later) is defined. This energy is determined by the difference between the brightness of the pixel of the start point image and the brightness of the corresponding pixel of the end point image, and the smoothness of the mapping. First, p ^{(m, 0)} and q with the smallest energy
Mapping between ^{(m, 0)} f ^{(m, 0)} : p ^{(m, 0)} → q
^{(M, 0)} is calculated. Based on f ^{(m, 0 )} , the mapping f ^{(m , 1)} between p ^{(m, 1} ) and q ^{(m, 1)} having the minimum energy is calculated. This procedure is p ^{(m, 3)}
And q ^{(m, 3)} map f ^{( m, 3)} until the calculation is completed. Each map f ^{(m, i)} (i = 0, 1, 2,
...) is called a submapping. For the convenience of calculating f ^{(m, i)} , the order of i can be rearranged as follows. The reason why the rearrangement is necessary will be described later.

[0037]

[Equation 3] Here, σ (i) ε {0, 1, 2, 3}.

[1.3.1] When the matching between the bijection start point image and the end point image is represented by a map, the map should satisfy the bijection condition between both images. This is because there is no conceptual superiority or inferiority between the two images, and pixels of each other should be connected in a bijective and injective manner. However, unlike the usual case, the mapping to be constructed here is a bijection digital version. In the base technology, pixels are specified by grid points.

Start point sub-image (provided for the start point image
From sub-image to end-point sub-image (provided for end-point image
The mapping to the sub-image is f^{(M, s)}: I / 2^nm× I
/ 2 ^nm→ I / 2^nm× I / 2^nm(S = 0,
1, ...). Where f
^{(M, s)}(I, j) = (k, l) is p of the starting point image
^{(M, s)} _{(I, j)}Is the end image q^{(M, s)}
_{(K, l)}Means to be mapped to. For simplicity
And when f (i, j) = (k, l) holds, the pixel q
_{(K, l)}Q_{f (i , J)}Write.

The definition of bijection is important when data is discrete, such as pixels (grid points) handled in the base technology. Here, it is defined as follows (i, i ', j, j', k,
l is an integer). First, each square region represented by R in the plane of the starting point image,

[Equation 4] (I = 0, ..., 2 ^m −1, j = 0, ..., 2 ^m −
1). Here, the direction of each side (edge) of R is determined as follows.

[0041]

[Equation 5] This square must be mapped by the map f to a quadrilateral in the end image plane. a quadrilateral represented by f ^{(m, s)} (R),

[Equation 6] Must satisfy the following bijection conditions.

1. The edges of the quadrilateral f ^{(m, s)} (R) do not intersect each other. 2. The directions of the edges of f ^{(m, s)} (R) are equal to those of R (clockwise in the case of FIG. 2). 3. Contraction map (retraction: retrac) as relaxation condition
tions).

This is because the mapping that completely satisfies the bijection condition is only the unit mapping unless some relaxation condition is set. Here ^{f (m, s)} the length of one edge of the (R) is 0, i.e. ^{f (m, s)} (R) may become a triangle. However, it must not be a figure whose area is 0, that is, one point or one line segment. Figure 2 (R)
2A and FIG. 2D satisfy the bijection condition when the original is a quadrangle, FIG. 2B, FIG. 2C, and FIG.
(E) is not satisfied.

In an actual implementation, the following conditions may be further imposed in order to easily guarantee that the mapping is surjective. That is, each pixel on the boundary of the start point image is mapped to the pixel occupying the same position in the end point image. That is, f (i, j) = (i,
j) (where i = 0, i = 2 ^m- 1, j = 0, j = 2 ^m
-1 on the four lines). Hereinafter, this condition is also referred to as “additional condition”.

[1.3.2] Energy of mapping [1.3.2.1] The energy of the cost mapping f with respect to the luminance of the pixel is defined. The goal is to find the map that minimizes the energy. The energy is mainly determined by the difference between the brightness of the pixel of the starting point image and the brightness of the pixel of the corresponding end point image. That is, the energy C ^{(m, s)} _{(i, j) at} the point (i, j) of the map f ^{(m, s} ₎
Is determined by the following equation.

[0046]

[Equation 7] Here, V (p ^{(m, s)} _{(i, j)} ) and V (q
^{(M, s)} _{f (i, j)} ) is the pixel p ^{(m, s)}
The luminance of _{(i, j)} and q ^{(m, s)} _{f (i, j)} . The total energy C ^{(m, s)} of f is one evaluation formula for evaluating matching.
^It can be defined by the sum of ^{(m, s) and} _{(i, j)} .

[0047]

[Equation 8] [1.3.2.2] Cost of Pixel Position for Smooth Mapping In order to obtain a smooth mapping, another energy Df for the mapping is introduced. This energy is independent of the brightness of the pixel, p ^{(m, s)} _{(i, j)} and q ^{(m , s)}
Determined by the position of _{f (i, j)} (i = 0, ..., 2 ^m
-1, j = 0, ..., 2 ^m -1). Energy D ^{(m, s )} _{(i, j)} of the map f ^{(m, s) at} the point (i _{, j)}
Is defined by the following equation.

[0048]

[Equation 9] However, the coefficient parameter η is a real number of 0 or more, and

[Equation 10]

[Equation 11] And here,

[Equation 12] And f (i ', for i'<0 and j '<0
j ') is set to 0. E ₀ is (i, j) and f (i,
It depends on the distance of j). E ₀ prevents pixels from being mapped to pixels that are too far away. However, E ₀ is replaced later with another energy function. E ₁ guarantees the smoothness of the map. E ₁ represents the distance between the displacement of p _{(i, j) and} the displacement of its neighbors. Based on the above consideration, energy D _f, which is another evaluation expression for evaluating matching, is determined by the following expression.

[0049]

[Equation 13] [1.3.2.3] Total energy of mapping Total energy of mapping, that is, a comprehensive evaluation formula related to the integration of a plurality of evaluation formulas is defined by λC ^{( m, s)} _f + D ^{(m, s)} _f . Here, the coefficient parameter λ is a real number of 0 or more. The purpose is to detect the state where the comprehensive evaluation formula has an extreme value, that is, to find the map that gives the minimum energy shown by the following formula.

[0050]

[Equation 14] Note that for λ = 0 and η = 0 the map is a unit map (ie all i = 0, ..., 2 ^m
F for −1 and j = 0, ..., 2 ^m −1
^{(M 2 , s)} (i, j) = (i, j)). As will be described later, in the base technology, since the case of λ = 0 and η = 0 is evaluated first, the mapping can be gradually transformed from the unit mapping. Temporarily change the position of λ in the comprehensive evaluation formula to C
^{If (m, s)} _f + λD ^{(m , s)} _f is defined,
When λ = 0 and η = 0, the comprehensive evaluation formula is C ^{(m, s)} _f
Then, the pixels that are originally unrelated to each other are associated with each other simply because the brightness is close, and the mapping becomes meaningless. Even if the map is transformed based on such a meaningless map, it makes no sense at all. Therefore, consideration is given to how to give the coefficient parameter so that the unit map is selected as the best map at the start of the evaluation.

The optical flow is the same as in this base technology.
Consider differences in pixel brightness and smoothness. However, optical flow cannot be used for image conversion. This is because only the local movement of the object is considered.
A global correspondence can be detected by using the singularity filter according to the base technology.

[1.3.3] Determining Map by Introducing Multi-Resolution Map f that gives the minimum energy and satisfies the bijection condition
_min is obtained using a multi-resolution hierarchy. Compute the mapping between the starting and ending sub-images at each resolution level. Starting from the top (coarse level) of the hierarchy of resolutions, the mapping for each resolution level is determined, taking into account the mappings of other levels. The number of mapping candidates at each level is limited by using a higher, or coarser level of mapping. More specifically, when determining the mapping at a certain level, the mapping obtained at one coarser level is imposed as a kind of constraint condition.

First,

[Equation 15] , P ^{(m-1, s)} _{(i ', j')} , q
^{(M-1, s)} _{(i ', j')} are respectively p ^{(m, s)}
_{(I, j)} and q ^{(m, s)} are called parents of _{(i, j)} . [X] is the maximum integer that does not exceed x. Also, p ^{(m, s )} _{(i, j)} , q ^{(m, s)}
Let _{(i, j)} be p ^{(m-1, s)} _{(i ', j ')} , q ^{(m-1, s), respectively.}
_It is called a child of _{(i ', j')} . Function parent
(I, j) is defined by the following equation.

[0054]

[Equation 16] The mapping f ^{(m, s)} between p ^{(m, s)} _{(i, j)} and q ^{(m, s)} _{(k, l} ⁾ is determined by performing the energy calculation and finding the minimum one. To be done. f ^{( m, s)}
The value of (i, j) = (k, l) is f ^{(m-1, s)} (m =
By using 1, 2, ..., N), it is determined as follows. First, the condition that q ^{(m, s)} _{(k, l)} must be inside the following quadrangle is used to narrow down maps with high reality among maps that satisfy the bijective condition.

[0055]

[Equation 17] However, here

[Equation 18] Is. The quadrilateral determined in this way is p ^{(m, s)}
We call it an inherited quadrilateral of _{(i, j)} . In the inherited quadrilateral, find the pixel that minimizes the energy.

FIG. 3 shows the above procedure. In the figure, the pixels A, B, C, and D of the start point image are A ′, B ′, C ′, and C ′ of the end point image at the m−1th level, respectively.
It is projected to D '. The pixel p ^{(m, s)} _{(i, j)} is a pixel q existing inside the inherited quadrangle A′B′C′D ′.
⁽ M ^{, s)} must be mapped to _{f (m) (i, j)} . From the above consideration, the mapping from the m-1th level mapping to the mth level mapping is performed.

The energy E ₀ defined above is replaced by the following equation in order to calculate the submapping f ^{(m, 0} ) at the m-th level.

[0058]

[Formula 19] Further, the following equation is used to calculate the submapping f ^{(m, s)} .

[0059]

[Equation 20] This gives a map that keeps the energy of all submaps low. According to Expression 20, the submappings corresponding to different singular points are associated within the same level so that the submappings have a high degree of similarity. Formula 19 is f
^{(M, s)} shows the distance between (i, j) and the position of the point where (i, j) is to be projected when it is considered as a part of the m-1th level pixel.

If there is no pixel satisfying the bijection condition inside the inherited quadrilateral A'B'C'D ', the following measures are taken. First, a pixel whose distance from the boundary line of A′B′C′D ′ is L (initially L = 1) is examined. Among them, if the one with the smallest energy satisfies the bijection condition, this is selected as the value of f ^{(m, s)} (i, j). L is increased until such a point is found or L reaches its upper limit L ^(m) max. L ^(m) max is fixed for each level m.
If such a point is not found at all, the third condition of bijection is temporarily ignored, and a mapping such that the area of the transformed quadrangle becomes zero is accepted, and f ^{(m, s)} (i , J). If none of the conditions is found, then the bijective first and second conditions are removed.

The approximation method using multi-resolution is indispensable for determining the global correspondence between images while avoiding that the mapping is affected by the image details. It is impossible to find the correspondence between pixels that are far apart without using an approximation method based on multiresolution. In that case, the size of the image must be limited to an extremely small size, and only an image with a small change can be handled. Furthermore, since smoothness is required for the normal mapping, it is difficult to find the correspondence between such pixels. This is because the energy of mapping from a pixel with a distance to a pixel is high. According to the approximation method using multi-resolution, it is possible to find an appropriate correspondence relationship between such pixels. This is because those distances are small at the upper level (coarse level) of the hierarchy of resolution.

[1.4] Automatic determination of optimum parameter value One of the main drawbacks of the existing matching technology is the difficulty of parameter adjustment. In most cases, adjustment of parameters is done manually, and it is extremely difficult to select the optimum value. With the method according to the base technology, the optimum parameter value can be determined completely automatically.

The system according to the base technology comprises two parameters, λ and η. In short, λ is the weight of the difference in the luminance of the pixels, and η is the rigidity of the mapping. The initial values of these parameters are 0. First, η is fixed to 0 and λ is gradually increased from 0. When the value of λ is increased and the value of the comprehensive evaluation expression (Expression 14) is minimized, the value of C ^{(m, s)} _f for each submapping generally decreases. This basically means that the two images must match better. However, when λ exceeds the optimum value, the following phenomenon occurs.

1. Pixels that should not be corresponded to each other are erroneously corresponded to each other simply because they have similar brightness. 2. As a result, the correspondence between pixels becomes strange,
The map begins to collapse.

3. As a result, in Equation 14, D
^{(M, s)} _f tends to increase rapidly. 4. As a result, the value of Equation 14 tends to increase rapidly, so that _f ^{(m, s)} _f is suppressed so as to suppress the rapid increase.
^{(M, s)} changes, resulting in an increase in C ^{(m, s)} _f . Therefore, the threshold value at which C ^{(m, s)} _f turns from decreasing to increasing is detected while maintaining the state that Expression 14 takes the minimum value while increasing λ, and sets λ to the optimum value at η = 0. Next, increase η little by little and increase C ^{(m, s)}
The behavior of _f is inspected, and η is automatically determined by the method described later. Λ is also determined corresponding to the η.

This method resembles the operation of the focus mechanism of the human visual system. In the human visual system, images of both eyes are matched while moving one eye. When the object is clearly recognizable, its eyes are fixed.

[1.4.1] Dynamic determination of λ λ is increased from 0 by a predetermined step size, and the submapping is evaluated each time the value of λ changes. As in Equation 14, the total energy is defined by λC ^{(m, s)} _f + D ^{(m, s)} _f . D ^{(m, s)} _{f in} Expression 9 represents the smoothness, and theoretically has the minimum value in the case of the unit mapping, and E ₀ and E ₁ increase as the mapping becomes distorted. Since E ₁ is an integer, the minimum step size of D ^{(m, s)} _f is 1. Therefore, the current change (decrease ₎ of λC ^{(m, s)} _{(i, j} )
If is not greater than 1, then the total energy cannot be reduced by changing the mapping. Because D ^{(m, s)} _f increases by 1 or more as the mapping changes, λ
This is because the total energy does not decrease unless C ^{(m, s)} _{(i, j)} decreases by 1 or more.

Under this condition, it is shown that C ^{(m, s)} _{(i, j)} decreases in the normal case as λ increases.
The histogram of C ^{(m, s)} _{(i, j} ) is described as h (l). h (l) is energy C ^{(m, s)} _{(i, j)}
Is the number of pixels with l ² . Since λl ² ≧ 1 holds, consider the case of l ² = 1 / λ, for example. When λ slightly changes from λ ₁ to λ ₂ ,

[Equation 21] A pixels shown by

[Equation 22] It changes to a more stable state with the energy of. Here, it is assumed that the energies of these pixels are all zero. In this expression, the value of C ^{(m, s)} _f is

[Equation 23] Show that only changes, and as a result,

[Equation 24] Is established. Since h (l)> 0, usually C
^{(M, s)} _f decreases. However, when λ tries to exceed the optimum value, the above-mentioned phenomenon, that is, the increase of C ^{(m, s)} _f occurs. The optimum value of λ is determined by detecting this phenomenon.

When H (h> 0) and k are constants,

[Equation 25] Assuming that

[Equation 26] Holds. At this time, if k ≠ −3,

[Equation 27] Becomes This is a general expression for C ^{(m, s)} _f (C is a constant).

When detecting the optimum value of λ, the number of pixels that violates the bijection condition may be checked for safety. Here, when determining the mapping of each pixel, it is assumed that the probability of breaking the bijection condition is p ₀ . in this case,

[Equation 28] Therefore, the number of pixels that violates the bijection condition increases at the rate of the following equation.

[0071]

[Equation 29] Therefore,

[Equation 30] Is a constant. If h (l) = Hl ^k is assumed,
For example,

[Equation 31] Becomes a constant. However, when λ exceeds the optimum value, the above value increases rapidly. By detecting this phenomenon, B ₀ λ
^{3/2 +} value of ^k / ^{2/2 m} examines whether exceeds the abnormal value _{B 0thres,} it is possible to determine the optimal value of lambda. Similarly, by checking whether or not the value of B ₁ λ ^{3/2 + k /} _2/2 ^m exceeds the abnormal value B _1thres , the increase rate B _{1 of} pixels that violates the third condition of bijection is confirmed. To do. The reason for introducing the factor 2 ^m will be described later. This system is not sensitive to these two thresholds. These thresholds can be used to detect undue distortions of the map that are missed by observing the energy C ^{(m, s)} _f .

In the experiment, when calculating the submapping f ^{(m, s)} , if λ exceeds 0.1, the calculation of f ^{(m, s)} is stopped and the calculation of f ^{(m, s + 1)} is started. did. λ> 0.
When 1, the difference of only “3” in the luminance 255 level of the pixel affected the calculation of the submapping, and λ> 0.1.
This was because it was difficult to get the correct result when.

[1.4.2] Histogram h (l) C^{(M, s)} _fThe inspection depends on the histogram h (l)
Absent. H (l) during bijection and the inspection of its third condition
Can be affected by. Actually (λ, C^{(M, s)} _f)
When lotted, k is usually around 1. In the experiment, k = 1
Using B₀λ ^TwoAnd B₁λ^TwoWas inspected. If k is true
If the value of is less than 1, then B₀λ^TwoAnd B₁λ^TwoBecomes a constant
Without factor λ^{(1-k) / 2}Gradually increase according to
To do. If h (l) is a constant, for example, the factor is λ
^1/2Is. However, such a difference is a threshold B
_0thresCan be absorbed by setting
You can

Here, the center of the starting point image is (x
₀ , y ₀ ) and a circular object of radius r.

[Equation 32] On the other hand, the end point image has the center (x ₁ , y ₁ ),
It is assumed that the object has a radius r.

[Expression 33] Here, it is assumed that c (x) has a form of c (x) = x ^k . If the centers (x ₀ , y ₀ ) and (x ₁ , y ₁ ) are far enough,
The histogram h (l) has the form:

[Equation 34] When k = 1, the image shows an object with a sharp border embedded in the background. This object has a dark center and becomes brighter toward the edges. When k = -1, the image represents an object with ambiguous boundaries. This object is brightest in the center and darkens towards the periphery. The general object does not lose its generality, even though it is considered to be somewhere in between these two types of objects. Therefore, k can cover most cases with −1 ≦ k ≦ 1, and it is guaranteed that Equation 27 is generally a decreasing function.

It should be noted that r is affected by the resolution of the image, that is, r is proportional to 2 m, as can be seen from equation 34. For this reason [1.4.1]
Introduced a factor of 2 m.

[1.4.3] The dynamic determination parameter η of η can be automatically determined by the same method. First, η = 0, and the final mapping f at the finest resolution is obtained.
Calculate ⁽ⁿ⁾ and energy C ⁽ⁿ⁾ _f . Subsequently, η is increased by a certain value Δη, and again the final map f ⁽ⁿ⁾ and the energy C ⁽ⁿ⁾ _{f at} the finest resolution are obtained.
Recalculate. This process is continued until the optimum value is obtained.
η indicates the rigidity of the map. Because it is the weight of the following formula.

[0077]

[Equation 35] When η is 0, D ⁽ⁿ⁾ _f is determined independently of the immediately preceding submapping, and the current submapping is elastically deformed and excessively distorted. On the other hand, when η is a very large value, D
^(N) _f is almost completely determined by the immediately preceding submapping. At this time, the submapping is very rigid, and the pixels are projected on the same place. As a result, the map becomes a unit map. When the value of η gradually increases from 0, C ⁽ⁿ⁾ _f gradually decreases as described later. However, if the value of η exceeds the optimum value,
As shown in, energy begins to increase. In the figure, the X axis is η and the Y axis is C _f .

With this method, an optimum value of η that minimizes C ⁽ⁿ⁾ _f can be obtained. However, as a result of various factors affecting the calculation as compared with the case of λ, C ⁽ⁿ⁾ _f changes with a small fluctuation. In the case of λ, the submapping is recalculated once each time the input changes by a small amount.
This is because in the case of, all submappings are recalculated. Therefore, it is not possible to immediately determine whether or not the obtained value of C ⁽ⁿ⁾ _f is the minimum. When a candidate for the minimum value is found, it is necessary to search for the true minimum value by setting a finer interval.

[1.5] The range of f ^{(m, s)} can be expanded to R × R in order to increase the degree of freedom when determining the correspondence between super-sampling pixels (R is a real number). set). In this case, the brightness of the pixel of the end point image is interpolated,

[Equation 36] An f ^{(m, s)} with a luminance at is provided. That is, super sampling is performed. In the experiment, f
^{(M, s)} is allowed to take integer and half-integer values,

[Equation 37] Is

[Equation 38] Given by.

[1.6] Normalization of Pixel Luminance of Each Image When the start point image and the end point image include objects that are very different, it is difficult to use the original pixel luminance as it is for the calculation of mapping. This is because the energy C ^{(m, s)} _f related to the brightness becomes too large because the difference in the brightness is large, and correct evaluation is difficult.

For example, consider a case where a human face and a cat face are matched. The cat's face is covered in hair, with very bright and very dark pixels. in this case,
To calculate the submapping between two faces, the subimage is first normalized. That is, the brightness of the darkest pixel is set to 0 and the brightness of the brightest pixel is set to 255, and the brightness of other pixels is obtained by linear interpolation.

[1.7] Implementation An inductive method is used in which the calculation proceeds linearly according to the scanning of the starting point image. First, the value of f ^{(m, s)} is determined for the uppermost leftmost pixel (i, j) = (0,0). Next, increase each i by 1
^{(M, s)} Determine the value of (i, j). When the value of i reaches the width of the image, the value of j is increased by 1 and i is returned to 0.
Thereafter, as the start point image is scanned, f ^{(m, s)} (i,
j) is decided. When the pixel correspondences are determined for all points, one mapping f ^{(m, s)} is determined.

Corresponding point q with respect to a certain p _{(i, j)
If f (i, j)} is determined, then the corresponding point qf _{(i, j + 1)} of p _{(i, j + 1)} is determined. At this time, q
_{Since the position of f (i, j + 1 )} satisfies the bijection condition,
Limited by the position of _{qf (i, j)} . Therefore, in this system, the higher the priority is, the higher the corresponding point is determined. When (0,0) always has the highest priority, an extra bias is added to the final mapping required. In the base technology, in order to avoid this state, f
^{(M, s)} is determined by the following method.

First, when (s mod 4) is 0, (0,
Start from 0) and gradually increase i and j while deciding. When (s mod 4) is 1, the right end point of the top row is set as the starting point, i is decreased, and j is increased. When (s mod 4) is 2, the right end point of the bottom row is set as the starting point, and i and j are decreased and determined. (S
When mod 4) is 3, the leftmost point of the bottom line is set as the starting point, and i is increased and j is decreased. Since the concept of submapping, that is, the parameter s, does not exist in the nth level having the smallest resolution, s = 0 and s =
Two directions were calculated consecutively as being 2.

In an actual implementation, a penalty is given to a candidate that violates a bijection condition, so that f ^{(m, s)} ( i, j) (m = 0, ..., N) values were chosen. Candidate energies that violate the third condition D (k,
l) is multiplied by φ, while candidates that violate the first or second conditions are multiplied by ψ. This time φ = 2, ψ = 100000
Was used.

In order to check the above-mentioned bijection condition, the following test was conducted in determining (k, l) = f ^{(m , s)} (i, j) as an actual procedure. That is, f
^{(M, s) For} each lattice point (k, l) included in the inherited quadrangle of (i, j), it is confirmed whether the z component of the outer product of the following equation is 0 or more.

[0087]

[Formula 39] However, here

[Formula 40]

[Formula 41] (Here, the vector is a three-dimensional vector, and the z axis is defined in the orthogonal right-handed coordinate system). If W is negative, then D ^{(m, s)} _{(k, l )} has ψ
Give a penalty by multiplying by and try to avoid choosing as much as possible.

FIGS. 5A and 5B show the reason for inspecting this condition. FIG. 5A shows a candidate with no penalty, and FIG. 5B shows a candidate with a penalty. Map f ^{(m, s)} for adjacent pixel (i, j + 1 ⁾
When determining (i, j + 1), if the z component of W is negative, no pixel satisfies the bijection condition on the starting image plane. This is because q ^{(m, s)} _{(k, l)} crosses the boundary line between adjacent quadrilaterals.

[1.7.1] In the submapping order implementation, σ (0) = 0, σ (1) = 1, σ (2) = 2, σ when the resolution level is even.
(3) = 3, σ (4) = 0 is used, and when odd, σ
(0) = 3, σ (1) = 2, σ (2) = 1, σ (3) =
0 and σ (4) = 3 were used. This appropriately shuffled the submap. In addition, originally there are four types of submappings,
s is any one of 0-3. However, actually s = 4
Was performed. The reason will be described later.

[1.8] Interpolation calculation After the mapping between the start and end images is determined, the corresponding
The brightness of the matching pixel is interpolated. In the experiment, tri-linear
Interpolation was used. Square p in the starting image plane
_{(I, j)}p_{(I + 1, j)}p_{(I, j + 1)}p
_{(I + 1, j + 1)}Is a quadrangle q on the end image plane
_{f (i, j)}q_{f (i + 1, j)}q_{f (i, j + 1)}q
_{f (i +1, j + 1)}Suppose it is projected to. Easy
Therefore, the distance between the images is set to 1. Distance from the starting image plane
Pixels r (x,
y, t) (0≤x≤N-1, 0≤y≤M-1) is
Required as required. First, the position of pixel r (x, y, t)
(However, x, y, tεR) is calculated by the following equation.

[0091]

[Equation 42] Subsequently, the brightness of the pixel at r (x, y, t) is determined using the following equation.

[0092]

[Equation 43] Here, dx and dy are parameters and change from 0 to 1.

[1.9] Mapping when a constraint condition is imposed Up to now, the determination of the map when no constraint condition exists has been described. However, when the correspondence relationship between the specific pixels of the start point image and the end point image is defined in advance, the mapping can be determined under the constraint condition.

The basic idea is that the starting point image is roughly transformed by a rough mapping in which a specific pixel of the starting point image is moved to a specific pixel of the ending point image, and then the mapping f is accurately calculated.

First, a rough mapping is determined in which a specific pixel of the starting point image is projected onto a specific pixel of the ending point image, and other pixels of the starting point image are projected onto appropriate positions. That is, a pixel close to a specific pixel is a map that is projected in the vicinity of the place where the specific pixel is projected. Here, the rough mapping of the m-th level is described as F ^(m) .

The rough map F is determined by the following procedure. First, the mapping is specified for some pixels. N _s pixels for the starting image,

[Equation 44] When specifying, determine the following values.

[Equation 45] The displacement amounts of the other pixels in the starting point image are p _{(ih, jh)} (h
= 0, ..., _Ns −1) is an average obtained by weighting the displacement. That is, the pixel p _{(i, j)} is projected onto the following pixels of the end point image.

[0097]

[Equation 46] However, here

[Equation 47]

[Equation 48] And

Subsequently, the energy D ^{(m, s)} _{(i, j)} of the mapping f is changed so that the candidate mapping f close to F ^(m) has less energy. To be exact, D
^{(M , s)} _{(i, j)} is

[Equation 49] Is. However,

[Equation 50] And κ and ρ ≧ 0. Finally, f is completely determined by the automatic map calculation process described above.

Where f ^{(m, s)} (i, j) is F
^(M) When they are sufficiently close to (i, j), that is, their distance is

[Equation 51] It should be noted that E ₂ ^{(m, s)} _{(i, j)} becomes 0 when it is within. The reason I defined it that way is
This is because as long as each f ^{(m, s)} (i, j) is sufficiently close to F ^(m) (i, j), its value is automatically determined so as to settle at an appropriate position in the end point image. For this reason, the starting point image is automatically mapped to match the ending point image without having to specify the exact correspondence in detail.

[2] The flow of processing by each elemental technique of the concrete processing procedure [1] will be described. Figure 6
Is a flowchart showing the overall procedure of the base technology. As shown in the figure, first, a process using a multi-resolution singularity filter is performed (S1), and then a start point image and an end point image are matched (S2). However, S2 is not essential,
Processing such as image recognition may be performed based on the characteristics of the image obtained in S1.

FIG. 7 is a flow chart showing the details of S1 in FIG. Here, it is assumed that the start point image and the end point image are matched in S2. Therefore, first, the starting point image is hierarchized by the singular point filter (S1
0), obtaining a series of starting point hierarchical images. Subsequently, the end point image is layered by the same method (S11) to obtain a series of end point hierarchical images. However, the order of S10 and S11 is arbitrary, and the starting point hierarchical image and the ending point hierarchical image can be generated in parallel.

FIG. 8 is a flow chart showing the details of S10 of FIG. The size of the original starting point image is 2 ⁿ × 2 ⁿ . Since the starting point hierarchical image is created in ascending order of resolution, the parameter m indicating the resolution level to be processed is set to n (S100). Next, the m-th level images p ^{(m, 0)} , p ^{(m, 1)} ,
Singular points are detected from p ^{(m, 2 )} and p ^{(m, 3)} using a singular point filter (S101), and the images at the ^{(m-1) th} level p ^(m-1,0) and p ^(m- ), respectively. ^1,1) , p
^{(M-1 , 2)} and p ^{(m-1, 3)} are generated (S10 ^).
2). Since m = n here, p ^{(m, 0)} = p
^{(M, 1)} = p ^{(m, 2)} = p ^{(m, 3)} = p ⁽ⁿ⁾ , and four types of sub-images are generated from one starting point image.

FIG. 9 shows a part of the m-th level image and the m-th level image.
The correspondence relationship of a part of a 1-level image is shown. Numerical values in the figure indicate the brightness of each pixel. In the figure, p ^{(m, s)} is p
^{(M, 0) ~p (m} , 3) intended to symbolize the four ^images, when generating the ^{images p (m-1,0),} p
Consider ^{(m, s)} to be p ^{(m, 0)} . [1.2]
According to the rule shown in, p ^(m-1,0) is, for example, "3" out of the four pixels included in the block in which luminance is entered in the figure, and p ^(m-1,1) is "8". , P
^{(M-1 , 2)} is "6" and p ^{(m-1, 3)} is "10"
, And replaces this block with the acquired single pixel. Therefore, the size of the sub-image at the (m-1) th level is 2 ^m-1 × 2 ^m-1 .

Next, decrement m (S1 in FIG. 8).
03), and confirm that m is not negative (S10
4) Returning to S101, a sub-image with coarse resolution is generated next. As a result of this repeated processing, S = 0 ends when m = 0, that is, when the 0th level sub-image is generated. The size of the 0th level sub-image is 1 × 1.

FIG. 10 illustrates the starting point hierarchical image generated in S10 for the case of n = 3. Only the first starting point image is common to the four series, and thereafter, the sub-images are independently generated according to the types of singular points. The process of FIG. 8 is common to S11 of FIG. 7, and the end point hierarchical image is also generated through the same procedure. With the above, S in FIG.
The process according to 1 is completed.

In the base technology, in order to proceed to S2 in FIG.
Prepare for the hatching evaluation. Figure 11 shows the procedure
There is. As shown in the figure, first, multiple evaluation formulas are set.
(S30). Regarding the pixels introduced in [1.3.2.1]
Energy C^{(M, s)} _fAnd [1.3.2.2]
Energy D related to the smoothness of the entered map^{(M, s)}
_fIs that. Next, we integrate these evaluation formulas and
A total evaluation formula is established (S31). Derived in [1.3.2.3]
Total energy input λC^{(M, s)} _f+ D^{(M, s)} _f
Is that, using η introduced in [1.3.2.2]
If     ΣΣ (λC^{(M, s)} _{(I, j)}+ ΗE₀ ^{(M, s)} _{(I, j)}+ E₁ ^{(M , S)} _{(I, j)} ) (Equation 52) Becomes However, the sum is 0 and 1 for i and j, respectively.
... 2^mCalculate with -1. Preparation for matching evaluation
Is set.

FIG. 12 is a flow chart showing the details of S2 in FIG. As described in [1], the matching between the starting point hierarchical image and the ending point hierarchical image is performed between images having the same resolution level. In order to achieve good global matching between images, matching is calculated in order of coarse resolution. Since the starting point hierarchical image and the ending point hierarchical image are generated using the singular point filter, the position and brightness of the singular point are clearly preserved even at the coarse level of resolution, and the result of global matching is better than before. It will be very good.

As shown in FIG. 12, first, the coefficient parameter η is set to 0 and the level parameter m is set to 0 (S20). Subsequently, matching is calculated between each of the four sub-images at the m-th level in the start point hierarchical image and the four sub-images at the m-th level in the end point hierarchical image, satisfying the bijective conditions, and 4 types of submappings f ^{(m, s)} (s = 0,1,2,3) that minimize
1). The bijection condition is checked using the inherited quadrilateral described in [1.3.3]. At this time, as shown in Expressions 17 and 18, the sub-mappings at the m-th level are constrained to those at the (m-1) -th level, so that matching at a coarser resolution level is sequentially used. This is a vertical reference between different levels. Note that m = 0 now and there is no coarser level, but this exceptional processing will be described later with reference to FIG.

On the other hand, the horizontal reference within the same level is also
Done. As in Expression 20 of [1.3.3], f
^{(M, 3)}Is f^{(M, 2)}And f^{(M, 2)}Is f
^{(M, 1)}And f ^{(M, 1)}Is f^{(M, 0)}To that
It is decided to be similar. The reason is that the type of singularity is
Even if they are different, they are originally the same start image and end image.
Situation in which the submappings are completely different because they are included
Is unnatural. As you can see from Equation 20,
The closer the images are to each other, the smaller the energy
Goodness is considered good.

Since there is no submapping that can be referred to at the same level for f ^{(m, 0) to} be determined first, one coarse level is referred to as shown in Expression 19. However, in the experiment, after obtaining up to f ^{(m, 3),} the procedure was taken to update f ^{(m, 0)} once with this as a constraint condition. This is equivalent to substituting s = 4 into Equation 20 and making f ^{(m, 4)} a new f ^{(m, 0)} . f ^{(m, 0)}
This is to avoid the tendency that the degree of association between f and ⁽ f ^{(m, 3))} becomes too low, and this measure improved the experimental results. In addition to this measure, in the experiment, the shuffling of the submapping shown in [1.7.1] was also performed. This is also to keep close the degree of association between submappings that are originally determined for each type of singularity. Further, the point of changing the position of the starting point according to the value of s in order to avoid the deflection depending on the starting point of the processing is as described in [1.7].

FIG. 13 is a diagram showing how the submapping is determined at the 0th level. At level 0, each sub-image consists of only one pixel, so four sub-maps
^{All f (0, s )} are automatically determined as unit maps. Figure 1
FIG. 4 is a diagram showing how a submapping is determined at the first level. At the first level, each sub-image consists of 4 pixels. In the figure, these four pixels are shown by solid lines.
Now, when searching the corresponding point of the point x of p ^{(1, s)} in q ^{(1, s)} , the following procedure is performed.

1. The upper left point a, the upper right point b, the lower left point c, and the lower right point d of the point x are obtained at the first level resolution. 2. A pixel to which the points a to d belong at one coarse level, that is, the 0th level is searched for. In the case of FIG. 14, points a to d belong to pixels A to D, respectively. However, the pixels A to C are virtual pixels that do not originally exist. 3. Plot the corresponding points A ′ to D ′ of the pixels A to D, which have already been obtained at the 0th level, in q ^{(1 , s)} . Pixels A ′ to C ′ are virtual pixels, and are pixels A to C, respectively.
Shall be in the same position as. 4. The corresponding point a ′ of the point a in the pixel A is considered to be in the pixel A ′, and the point a ′ is plotted. At this time, the position of the point a in the pixel A (lower right in this case) and the point a ′
Are assumed to occupy the same position in pixel A '. Plot corresponding points b ′ to d ′ in the same way as in 5.4,
Make an inherited quadrilateral at points a'-d '. 6. Find the corresponding point x ′ of the point x so that the energy is minimized in the inherited quadrilateral. The candidates of the corresponding point x ′ may be limited to those in which the center of the pixel is included in the inherited quadrangle, for example. In the case of FIG. 14, all four pixels are candidates.

The above is the procedure for determining the corresponding point of the point x. Similar processing is performed for all other points to determine the submapping. At the second level and above, it is considered that the shape of the inherited quadrangle gradually collapses, so that a situation occurs in which the pixels A ′ to D ′ are spaced apart as shown in FIG. 3.

Thus, if four sub-maps of a certain m-th level are determined, m is incremented (S2 in FIG. 12).
2) Make sure that m does not exceed n (S2
3) and returns to S21. Hereinafter, each time the process returns to S21, the sub-mapping of the finer resolution level is obtained, and when the process finally returns to S21, the n-th level mapping f ⁽ⁿ⁾ is determined. Since this mapping is fixed for η = 0, f
^(N) Write (η = 0).

Next, the maps for different η should be obtained.
Shift η by Δη and clear m to zero (S2
4). New η is the predetermined search cutoff value η_maxBeyond
Confirm that it is not (S25), return to S21,
map f with respect to η^(N)Calculate (η = Δη). This place
The process is repeated, and in S21, f^(N)(Η = iΔη) (i =
0, 1, ...) η is η_maxWhen exceeded
Proceeding to S26, the optimum η = η by the method described later._optDecide
And f^(N)(Η = η_opt) Is finally a map f ^(N)
And

FIG. 15 is a flow chart showing the details of S21 of FIG. This flowchart determines the submapping at the m-th level for a certain η. When determining the submappings, the precondition technique independently determines the optimum λ for each submapping.

As shown in the figure, first, s and λ are cleared to zero.
(S210). Next, for λ at that time (and
And implicitly for η) the submapping f that minimizes the energy
^{(M, s)}(S211), and this is f
^{(M, s)}Write (λ = 0). Also the mappings for different λ
To obtain, shift λ by Δλ, and set a new λ
Search cutoff value λ_maxConfirm that it does not exceed (S
213), the process returns to S211 and f is repeated in the subsequent processing.
^{(M, s)}(Λ = iΔλ) (i = 0,1, ...)
It λ is λ _maxWhen it exceeds, the process proceeds to S214 and the optimum
λ = λ_optAnd f^{(M, s )}(Λ = λ_opt)
Finally the map f^{(M, s)}(S214).

Next, in order to find another submapping at the same level, λ is cleared to zero and s is incremented (S215). It is confirmed that s does not exceed 4 (S216), and the process returns to S211. When s = 4, f ^{(m, 3 )} is updated using f ^{(m, 3)} as described above,
Finish the submapping determination at that level.

FIG. 16 shows that f ^{(m, s)} (λ = iΔλ) (i = 0,
It is a figure which shows the behavior of the energy C ^{(m, s)} _f corresponding to 1, ...). As described in [1.4], when λ increases, C ^{(m, s)} _f usually decreases. However, when λ exceeds the optimum value, C ^{(m, s)} _f starts to increase. Therefore, in the base technology, λ when C ^{(m, s)} _f has a minimum value is determined as λ _opt . Even if C ^{(m, s)} _f becomes small again in the range of λ> λ _opt as shown in the figure, the map is already broken at that point and it does not make sense, so pay attention to the first minimum point. Good. λ _opt is independently determined for each submapping, and finally f ⁽ⁿ⁾ is also determined.

On the other hand, FIG. 17 is a diagram showing the behavior of the energy C ⁽ⁿ⁾ _f corresponding to f ⁽ⁿ⁾ (η = iΔη) (i = 0, 1, ...) Obtained while changing η. . Again, when η increases, C ⁽ⁿ⁾ _f usually decreases, but when η exceeds the optimum value, C ⁽ⁿ⁾ _f starts to increase. So C
^(N) Determine η when _f takes a minimum value as η _opt .
It can be considered that FIG. 17 is an enlarged view of the vicinity of zero on the horizontal axis of FIG. Once η _opt is determined, f ⁽ⁿ⁾ can be finally determined.

As described above, according to the base technology, various advantages can be obtained. First, since it is not necessary to detect edges, it is possible to solve the problems of the edge detection type prior art. Also,
A priori knowledge of the objects contained in the image is not required, and the corresponding points are automatically detected. The singular point filter can maintain the brightness and position of the singular point even at a coarse resolution level, and is extremely advantageous for object recognition, feature extraction, and image matching. As a result, it is possible to construct an image processing system that significantly reduces manual work.

The following modified techniques can be considered for the base technology. (1) In the base technology, the parameters are automatically determined when matching is performed between the starting point hierarchical image and the ending point hierarchical image, but this method does not perform matching between hierarchical images but ordinary two images. Available in all cases.

For example, between two images, the energy E ₀ related to the difference in pixel brightness and the energy E ₁ related to the positional shift of pixels are used as evaluation expressions, and the linear sum E _{t ot} = αE ₀ + E _{1 of these is used.} Is a comprehensive evaluation formula. Paying attention to the vicinity of the extreme value of this comprehensive evaluation formula, α is automatically determined. That is, a map that minimizes E _tot is obtained for various α. Among these mappings, α when E ₁ has a minimum value with respect to α is determined as the optimum parameter. The map corresponding to that parameter is finally regarded as the optimum matching between both images.

In addition to this, there are various methods for setting the evaluation formula, and for example, 1 / E ₁ and 1 / E ₂ which take a larger value as the evaluation result becomes better may be adopted. . The total evaluation formula does not necessarily have to be a linear sum, and n
The sum of multiplications (n = 2, 1/2, -1, -2, etc.), a polynomial, an arbitrary function, etc. may be appropriately selected.

The parameter may be α only, two cases of η and λ as in the premised technique, or more cases. If the parameter is 3 or more, change it one by one and decide.

(2) In the base technology, after the mapping is determined so that the value of the comprehensive evaluation formula is minimized, one evaluation formula C ^{(m, s)} _f constituting the comprehensive evaluation formula is minimized. The points were detected and the parameters were determined. However, instead of such a two-step process, it may be effective to simply determine the parameters so that the minimum value of the comprehensive evaluation formula is minimized depending on the situation. In that case, for example, αE ₀ + βE ₁ may be used as a comprehensive evaluation formula, and a constraint condition of α + β = 1 may be provided to treat each evaluation formula equally. This is because the essence of automatic parameter determination is to determine the parameters so that the energy is minimized.

(3) In the base technology, four types of sub-images relating to four types of singular points are generated at each resolution level. However, naturally, one, two, and three types among the four types may be selectively used. For example, if there is only one bright point in the image, a corresponding effect should be obtained even if a hierarchical image is generated only with f ^{(m, 3)} related to the maximum point.
In that case, different submappings at the same level are not necessary, which has the effect of reducing the amount of calculation regarding s.

(4) In this base technology, when the level advances by one with the singularity filter, the number of pixels becomes 1/4. For example, it is possible to use 3 × 3 as one block and search for a singular point in the block. In that case, the pixel becomes 1/9 when the level advances by one.

(5) When the start point image and the end point image are in color, they are first converted into a black and white image and the mapping is calculated.
The color image at the starting point is converted using the mapping obtained as a result. As another method, a submapping may be calculated for each of the RGB components.

[Embodiment Regarding Image Coding and Decoding] In the base technology, matching points are generated by matching key frames, and an intermediate frame is generated based on this matching point information. Therefore, this technique can be used for compression of moving images, and in actuality, it has been actually confirmed in the experiment that both the image quality and the compression rate exceeding MPEG are compatible. Further, since the intermediate frame between the key frames can be accurately generated based on the key frame and the corresponding point information, it can be used to generate a smooth moving image from a plurality of still images. In the present embodiment, a technique will be described in which, between terminals that transmit and receive image data together with audio data via a network, a smooth moving image can be reproduced on the receiving side even if the communication amount of image data is reduced. Hereinafter, the image encoding and decoding techniques using the prerequisite technique will be described.

FIG. 18 is a diagram showing the structure of the image coding apparatus 10 according to the embodiment of the present invention. The image encoding device 10 is a transmission-side terminal in a videophone, a video conference, or the like. The image encoding device 10 includes an image input unit 12,
The image storage unit 14, the voice input unit 16, the voice recognition unit 18, the array unit 20, the processor 22, the stream generation unit 24, and the communication unit 26 are included.

The image input unit 12 acquires, as a key frame KF, an image showing a facial expression when a person speaks a word for each sound. The image input unit 12 may acquire a key frame KF as a still image for each sound, or may have a function of acquiring a moving image and determining the key frame KF from the acquired moving image. Further, the image at this time may be an image of a character instead of the actual image of the user.

The image holding unit 14 holds the key frame KF acquired by the image input unit 12 in association with each sound. In addition, the image holding unit 14 may group different sounds and associate them with the same key frame KF when the expressions when uttering those sounds are similar.

As one method of performing this grouping, sounds that will have similar facial expressions when uttered are grouped in advance, and the image input unit 12 is made to acquire one facial expression image for each group. Good. As another method, a unit for determining the degree of similarity of the acquired images may be provided, and those determined to be similar may be grouped and held.

The voice input unit 16 inputs the voice data S. The voice recognition unit 18 is a known voice recognition unit that recognizes each sound included in the voice data S. The voice recognition unit 18 further has means for detecting the timing at which each sound is uttered in the voice data S. The timing here is, for example, a time interval at which each sound is uttered. The voice recognition unit 18 further determines the key frame KF according to the timing.
A means for specifying the number Ni of intermediate frames to be artificially generated is provided between them.

The arranging section 20 arranges the key frames KF in the order of the recognized sounds. The processor 22 performs matching between the two key frames KF on a pixel-by-pixel basis based on the singularity or the like based on the base technology or another arbitrary technology. As a result, the processor 22 generates corresponding point information C between the arranged key frames KF. However, the processor 22 may be omitted, and the corresponding point information C itself may be acquired from another location and input to the stream generation unit 24.

The stream generation unit 24 generates the coded data stream CI based on the arranged key frames KF, the corresponding point information C, and the number Ni of intermediate frames to be generated between each key frame KF. The communication unit 26 outputs the encoded data stream CI together with the audio data to the image decoding device 50 described later via a network (not shown).

An encoding procedure with the above configuration will be shown.

First, the key frame KF is input from the storage or camera via the image input unit 12. The image encoding device 10 gives an instruction for the user to input the key frame KF. For example, say "a". Is displayed, and an image showing the facial expression when the user utters the sound "A" is acquired. At this time, the image encoding device 10 may simultaneously acquire the voice information of the user. In that case, the voice recognition unit 18 can recognize the sound included in the voice data S based on the voice information of the user himself, and thus the recognition accuracy can be improved.

FIG. 19 is a diagram showing the internal structure of the image holding unit 14. The image holding unit 14 has a frame number column, a key frame column, and a sound column. The key frame KF, which is an image showing the facial expression when the user utters each sound, is held in association with the frame number and the sound. For example, the key frame KF of the frame number A1 is an image showing a facial expression when the user utters “A”. In the present embodiment, a key frame is associated with each sound, but if the facial expressions when the user utters "i" and "shi" are similar, frame number B
The sound of “shi” may be associated with one key frame and held. By grouping in this way, the number of required keyframes KF can be reduced, so that the amount of communication can be further reduced.

The acquisition and holding of the above key frame KF are processed in advance before the input of the audio data S. Next, the input of the voice data S will be described. The voice data S is
It may be input when the user calls the user of the image decoding device 50.

The voice input unit 16 inputs the voice uttered by the user as the voice data S. For example, when the user utters "Hello", the voice recognition unit 18 uses the voice data S
Recognize that the sound "is good, good, good" is included. When the same sound is duplicated, as in this case, the arrangement unit 20 arranges the frame numbers of the respective key frames KF instead of arranging the key frames KF themselves.

Also, the voice recognition unit 18 determines the number N of intermediate frames to be generated between the adjacent key frames KF in accordance with the time interval at which each sound included in the voice data S is generated.
Calculate i. When the time interval between adjacent key frames KF is ΔT and the number of display frames per second of the display device is n, the number of intermediate frames Ni to be generated is Ni = nΔT.

In the stream generation unit 24, the array of the key frame KF and the frame number from the array unit 20 and the corresponding point information C between the processor 22 and the key frame KF are stored.
The number Ni of intermediate frames is input from the voice recognition unit 18. The stream generation unit 24 incorporates these data in an appropriate order and generates an encoded data stream CI.

FIG. 20 is a diagram showing the structure of the coded data stream CI generated by the stream generation unit 24. The coded data stream CI includes a key frame KF (E7) when "mo" is uttered, a key frame KF (B3) when "shi" is uttered, corresponding point information C and a frame between these key frames KF. Sequence information H indicating the sequence of numbers is included.

FIG. 21 shows the format of the array information H. The sequence is "E7, B3, E7, B3". In addition, the number Ni of intermediate frames to be inserted between the frame numbers is included between each frame number.

The communication section 26 receives the audio data S from the audio input section 16 and the coded data stream CI from the stream generation section 24, and transmits them to the image decoding apparatus 50. The above is the processing on the encoding side. The image encoding device 10 can provide a moving image that can be easily reproduced on the decoding side.

FIG. 22 is a diagram showing the structure of the image decoding device 50 according to the embodiment of the present invention. Image decoding device 50
Is a terminal on the receiving side in a videophone or a video conference. The image decoding device 50 includes a communication unit 52, a stream input unit 54, a stream classification unit 56, an audio input unit 58,
Synchronization adjustment unit 60, intermediate image generation unit 62, image output unit 6
4 and a voice output unit 66.

The communication section 52 receives data from the image coding apparatus 10 via the network. The stream input unit 54 acquires the encoded data stream CI. The stream classification unit 56 decomposes the input coded data stream CI into its constituent elements, and includes a key frame KF included in the data stream CI, frame number array information H, corresponding point information C between the key frames KF, and The number Ni of intermediate frames is detected. The voice input unit 58 acquires the voice data S.

The intermediate image generator 62 receives the key frame KF, the array information H, and the corresponding point information C from the stream classification unit 56.
Also, the number of intermediate frames Ni is received, and Ni intermediate frames are generated between each key frame KF by the interpolation processing described in the base technology.

The synchronization adjusting unit 60 adjusts the synchronization of reproduction of the audio data S and the moving image so that each key frame KF is output in synchronization with the output timing of each sound in the audio data S. The audio output unit 66 outputs the audio data S. The image output unit 64 outputs the key frame KF and the intermediate frame as a moving image.

The present invention has been described above based on the embodiments. Note that the present invention is not limited to this embodiment, and various modifications thereof are also effective as aspects of the present invention.

For example, in the embodiment, the key frame KF is registered in advance, and the voice recognition technique is used to extract the key frame KF corresponding to each sound included in the input voice data S to obtain the encoded data stream. Although the CI is generated, the key frames KF may be sequentially acquired at a predetermined timing at the same time when the audio data S is input. In this case, the acquisition of the key frame KF may be performed substantially in real time with the input of the audio data S. In this way, it is not necessary to perform voice recognition, so the process becomes simpler.

This example will be described below with reference to FIGS. 18 and 22. The image input section 12 has a key frame KF.
However, the voice data S is input to the voice input unit 16 in substantially real time. The processor 22 sequentially generates corresponding point information C between the input key frames KF. The stream generation unit 24 generates the encoded data stream CI in a format including the key frame KF and the corresponding point information C.

The image input unit 12 may have means for detecting a change in the input image, and may acquire the key frame KF at the timing when the facial expression of the person talking is changed. Further, the image encoding device 10 may have a means for detecting the timing at which the person speaks a different sound, and may acquire the key frame KF at that timing.

The image decoding device 50 generates an intermediate frame between the key frames KF based on the coded data stream CI generated as described above, and synchronizes with the output timing of the audio data S to generate the key frame. The KF and the intermediate frame are output as a moving image.

Further, in the present embodiment, the corresponding points information C between the frames is generated after the key frames KF are arranged by the arranging section 20, but the corresponding point information C between all the key frames is generated in advance. It is held in the image holding unit 14,
It may be taken out if necessary. In this way, it is not necessary to generate corresponding point information every time, and the encoded stream CI can be efficiently input to the audio data S.
Can be generated.

Further, as shown in FIG. 23, the image decoding device 5
0, a key frame storage unit 7 that stores the key frame KF transmitted from the image encoding device 10 in association with each sound.
4 and the key frame storage unit 74 when generating the intermediate image.
An image selecting section 72 for selecting the key frame KF from the above may be provided. Further, the image decoding apparatus 50 may be provided with a determination unit 76 that notifies the image encoding apparatus 10 of the acquired key frame KF and urges the image encoding apparatus 10 to transmit only the unacquired key frame KF. The determination unit 76 may be provided in the image encoding device 10, and only the untransmitted key frame KF may be transmitted to the image decoding device 50. By doing this, all keyframes KF
Is stored in the key frame storage unit 74, the moving image can be reproduced by using the key frame KF stored in the key frame storage unit 74 only by transmitting the array information H from the image encoding device 10. .

Further, the image of the character is stored as a key frame in the image decoding device 50, the voice data S and the arrangement information H are transmitted from the transmitting side, and the moving image of the character is outputted in synchronization with the voice. Good.

Also, in the present embodiment, the image decoding device 50
Is configured to receive the arrangement information H at the same time as the audio data S, but is not limited to this, only the audio data S is received, a voice recognition function is provided on the image decoding device 50 side, and the image data is stored in the image decoding device 50 in advance. A moving image may be generated from a certain key frame KF and output in synchronization with audio. As a result, even if there is no image encoding function for transmitting an image to the transmitting side, or even if no image is transmitted from the transmitting side, a call using a moving image can be realized.

FIG. 24 is a diagram showing the configuration of a modified example of the image decoding apparatus 50 for realizing this. Hereinafter, the same configurations as those in FIG. 22 will be assigned the same reference numerals and the description thereof will be appropriately omitted.
In the new configuration in FIG. 24, the image decoding device 50 replaces the stream input unit 54 and the stream classification unit 56 with the image input unit 12, the voice recognition unit 18, the key frame storage unit 74, the image selection unit 72, and the specification. It has a part 70.

The image input unit 12 obtains images associated with voices as a key frame KF group. This may be an image of the above-mentioned character, an image of an actor, or the like. The key frame storage unit 74 stores the acquired key frame KF group in association with the sender. The specifying unit 70, for example, when a call is received from the sender side,
The sender is specified and notified to the image selection unit 72. The image selection unit 72 displays the key frame K associated with the sender.
The F group is selected from the key frame storage unit 74.

The key frame storage unit 74 may store the key frame KF in association with the telephone number of the sender, for example. In this case, the identifying unit 70 identifies the telephone number of the sender by using the caller number notification function or the like. As a result, the image output unit 64 can output a moving image based on the key frame KF associated with the sender. This method can be used not only during a call but also as an incoming signal. For example, the image decoding device 50 may receive a telephone call from the sender side and generate and display a moving image together with a voice such as “A call is received from Mr. A”.

The voice recognition unit 18 recognizes the sound included in the voice data S, and the image selection unit 72 determines the key frame KF corresponding to the sound based on the instruction from the synchronization adjustment unit 60 in the key frame storage unit 74. It is acquired from the key frame KF group associated with the sender. The intermediate image generation unit 62 generates an intermediate image based on the acquired key frame KF,
The image output unit 64 outputs a moving image. At that time, the synchronization adjustment unit 60 synchronizes with the sound to be output.

Although the data input to the voice input unit 58 is voice data here, it may be any data that can be converted into voice, and may be text data received by e-mail or the like.

Further, the function of the image coding apparatus 10 may be provided in a server on the network so that a moving image can be displayed on the receiving terminal side based on the audio data S input from the conventional transmitting terminal. .

FIG. 25 is a diagram showing the structure of a network system 180 to which the communication terminal 120 and the key frame providing server 110 are connected. The key frame providing server 110 has the function of the image encoding device 10. The communication terminal 120 includes the image decoding device 50. When displaying a moving image in synchronization with voice, the user of the communication terminal 120 acquires in advance the key frame KF required to generate the moving image from the key frame providing server 110 and stores it. Here, the image used as the key frame KF may be acquired from the sender as an attached file of an electronic mail, or may be acquired from a recording medium such as a memory card.

In the above description, the image decoding device 50 and the communication terminal 120 including it are not distinguished. Further, the communication terminal 120 may have the function of the image encoding device 10.

In the present embodiment, the still image is stored as the key frame KF and the moving image is generated as appropriate, but the present invention is not limited to this, and the moving image may be stored and used. In particular word to frequent, and even sentences, for example videos, such as greetings and proper nouns, such as "Hello" may be stored. This is particularly effective when synchronizing a moving image with text data.

Further, in the embodiment, the matching technique which is a prerequisite technique is used, but the existing technique such as optical flow or block matching may be used.

Although not specified, the audio input unit 16 of the image encoding device 10 performs compression encoding of audio by a method such as MPEG layer 3 and transmits the compressed audio data to the image decoding device 50. Good.

[Brief description of drawings]

1A and 1B are images obtained by applying an averaging filter to the faces of two persons, and FIG.
(C) and FIG. 1 (d) are images of p ^{(5, 0)} required by the base technology for the faces of the two persons, and FIGS. 1 (e) and 1 (f) are the faces of the two persons. Images of p ^(5,1) required by the base technology, Fig. 1 (g) and Fig. 1 (h)
Is the p required by the base technology for the faces of two people.
The images of ^{(5, 2)} , Fig. 1 (i) and Fig. 1 (j) are p ^{(5, 3)} obtained by the base technology for the faces of two persons.
3 is a photograph of a halftone image in which each of the images is displayed on the display.

2 (R) is a diagram showing an original quadrangle, FIG.
(A), FIG. 2 (B), FIG. 2 (C), FIG. 2 (D), FIG.
(E) is a figure which shows each inherited quadrangle.

FIG. 3 is a diagram showing a relationship between a start point image and an end point image and a relationship between an mth level and an m−1th level by using an inherited quadrangle.

FIG. 4 is a diagram showing a relationship between a parameter η and energy C _f .

5 (a) and 5 (b) are diagrams showing how to determine from a cross product calculation whether or not a mapping regarding a certain point satisfies a bijection condition.

FIG. 6 is a flowchart showing the overall procedure of the base technology.

FIG. 7 is a flowchart showing details of S1 in FIG.

FIG. 8 is a flowchart showing details of S10 of FIG.

FIG. 9 is a diagram showing a correspondence relationship between a part of an m-th level image and a part of an m-1th level image.

FIG. 10 is a diagram showing a starting point hierarchical image generated by the base technology.

FIG. 11 is a diagram showing a procedure of preparation for matching evaluation before proceeding to S2 of FIG.

FIG. 12 is a flowchart showing details of S2 in FIG.

FIG. 13 is a diagram showing how a submapping is determined at the 0th level.

FIG. 14 is a diagram showing how a submapping is determined at the first level.

FIG. 15 is a flowchart showing details of S21 in FIG.

FIG. 16 is a diagram showing a behavior of energy C ^{(m, s)} _f corresponding to f ^{(m, s)} (λ = iΔλ) obtained by changing λ for a certain f ^{(m, s)} .

FIG. 17 shows f ⁽ⁿ⁾ (η = determined by changing η
energy C corresponding to iΔη) (i = 0, 1, ...)
It is a figure which shows the behavior of ⁽ⁿ⁾ _f .

FIG. 18 is a diagram showing a configuration of an image encoding device according to an embodiment of the present invention.

FIG. 19 is a diagram showing an internal configuration of an image holding unit.

[Fig. 20] Fig. 20 is a diagram illustrating the configuration of an encoded data stream generated by a stream generation unit.

FIG. 21 shows a format of sequence information.

FIG. 22 is a diagram showing a configuration of an image decoding device according to an embodiment of the present invention.

FIG. 23 is a diagram showing the configuration of a modified example of the image decoding device according to the embodiment of the present invention.

FIG. 24 is a diagram showing the configuration of a modified example of the image decoding device according to the embodiment of the present invention.

FIG. 25 is a diagram showing a network system in which a communication terminal and a key frame providing server according to an embodiment of the present invention are connected.

[Explanation of symbols]

10 image encoding device, 12 image input unit, 14
Image holding unit, 16 voice input unit, 18 voice recognition unit,
20 array unit, 22 processor, 24 stream generation unit, 26 communication unit, 50 image decoding device,
52 communication unit, 54 stream input unit, 56
Stream classification unit, 58 audio input unit, 60 synchronization adjustment unit, 62 intermediate image generation unit, 64 image output unit, 70 identification unit, 72 image selection unit, 74 key frame storage unit, 76 determination unit, 110 key frame provision server.

─────────────────────────────────────────────────── ─── Continuation of front page (51) Int.Cl. ⁷ Identification code FI theme code (reference) H04N 5/92 G10L 3/00 E 5/93 SF term (reference) 5B057 AA20 CA02 CA08 CA12 CA16 CB02 CB08 CB12 CB16 CC01 CE08 CE20 CG07 5C023 AA32 CA03 DA04 5C053 FA07 GB00 JA07 JA12 LA14 5D045 AB00 AB26

Claims (17)

[Claims] 1. A terminal capable of displaying video, a step of acquiring data that can be converted into sound, a step of acquiring an image corresponding to the converted sound as a key frame, and based on the acquired key frame An image processing method comprising: a step of generating an intermediate image to generate a moving image; and a step of displaying the moving image in synchronization with the sound. 2. A method of informing a moving image and a predetermined voice when an incoming information is received in a terminal capable of displaying a video, and obtaining an image corresponding to the voice as a key frame. An image processing method comprising: a step of generating an intermediate image based on the acquired key frame to generate a moving image; and a step of displaying the moving image in synchronization with the sound. 3. The image processing method according to claim 1, wherein the key frame is an image showing a facial expression when producing a sound. 4. The image processing method according to claim 3, wherein different sounds are associated with the same key frame when the facial expressions of the pronunciations are substantially the same. 5. The method further comprises the step of saving the key frame, wherein the step of acquiring the image corresponding to the sound as a key frame acquires and saves the image prior to the step of acquiring the data convertible to the sound. The image according to any one of claims 1 to 4, characterized in that when the moving image is generated from the key frame, the amount of communication data is suppressed by using the stored key frame. Processing method. 6. The image processing method according to claim 5, wherein in the step of saving the key frame, only key frames that are not acquired in the step of acquiring the key frame are selected and saved. 7. The method further comprises: a step of identifying a sender of the data that can be converted into the voice, and a step of selecting a key frame corresponding to the identified party from the stored key frames. The image processing method according to claim 5, wherein 8. An image obtained by photographing a user's facial expression when a specific sound is uttered is stored in association with the specific sound,
When the user utters a word or a sentence related to the continuation of the specific sound, an image corresponding to the specific sound constituting the word or the sentence is displayed continuously and after smoothing processing. Processing method. 9. The image processing method according to claim 8, wherein the smoothing process is morphing. 10. A step of storing an image associated with voice as a key frame in a predetermined site on a network, a step of acquiring the key frame from the site and storing the key frame, and converting data convertible into voice. A step of acquiring, a step of selecting a key frame corresponding to the voice after the conversion of the data from the stored key frames, a step of generating an intermediate image based on the key frames to generate a moving image, And a step of synchronizing and outputting the moving image and the sound, the image processing method. 11. A data input unit that acquires data that can be converted into voice, a voice conversion unit that converts the data into voice, an image acquisition unit that acquires a key frame corresponding to the voice after conversion, and the voice. With a corresponding key frame, an intermediate image generation unit that generates an intermediate image based on the key frame and generates a moving image, an audio output unit that outputs audio, and an output that synchronizes the moving image with the audio. An image processing device comprising: 12. A key frame storage unit for storing the acquired key frame, wherein the key frame is acquired and stored prior to acquisition of data that can be converted into voice, and the image acquisition unit stores a moving image. The image processing apparatus according to claim 11, wherein the image processing apparatus selects and acquires from a key frame from a key frame storage unit when generating. 13. The image acquisition unit acquires the key frame from a predetermined site on a network and stores the key frame in the key frame storage unit.
The image processing device according to item 1. 14. The image acquisition unit determines whether or not the key frame has been acquired, and notifies the communication partner of that fact, thereby urging the communication partner to transmit only the unacquired key frame. The image processing device according to claim 12, wherein 15. A terminal capable of displaying video, the step of acquiring data that can be converted into audio, the step of acquiring an image corresponding to the audio after conversion as a key frame, and based on the acquired key frame. A computer program that causes a computer to execute a step of generating an intermediate image to generate a moving image, and a step of displaying the moving image in synchronization with the sound. 16. A program capable of notifying a moving image and a predetermined voice when an incoming information is received in a terminal capable of displaying a video, and acquiring an image corresponding to the voice as a key frame. And a step of generating an intermediate image based on the acquired key frame to generate a moving image, and a step of displaying the moving image in synchronization with the voice, the computer program characterized by the above-mentioned. . 17. A step of holding an image of a facial expression of a user when uttering a specific sound in association with the specific sound, and holding the word when the user utters a word or a sentence related to the continuation of the specific sound. A computer program characterized by causing a computer to execute a step of displaying an image corresponding to a specific sound forming a sentence continuously and after smoothing processing.