JP2002094881A - Picture and sound interpolation processor and its method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a picture and sound interpolation processor and method capable of quickly and inexpensively generating a pseudo picture or pseudo sound whose quality is satisfactory indicating the age change or change over aging of an object to be imaged or a sound source. SOLUTION: This picture interpolation processor 1 for generating the pseudo picture of an object to be imaged in designated time information from real pictures I1,... of the same object to be imaged whose image pickup time is different is provided with a characteristic part recognizing means 7 for recognizing the characteristic parts of the real pictures, a characteristic part relating means 3 for relating the characteristic parts among a plurality of real pictures I1,..., an interpolating means 4 for executing interpolation by using the time information as independent variables and the pixel values of the characteristic parts as the dependent variables, and for outputting interpolated curves, and a pseudo picture generating means for calculating the pixel values in the designated time information by using the interpolated curves, and for outputting the pseudo picture of the object to be imaged.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an interpolation processing apparatus for generating a realistic pseudo image or a pseudo sound using a real image actually picked up or a sound actually recorded.

[0002]

2. Description of the Related Art A technique called morphing, which continuously changes an image A to another image B as one of digital special video effects used in a movie or the like, has been known. Thereby, for example, as shown in FIG. 19, the appearance of the person (female) 100 is continuously changed from that of the person (male) 101 (from left to right), and
It is possible to generate a special effect that can generate the person images 102 to 104 in the middle of the image 101 and continuously change the appearance of a human to that of an animal. Also, as another example, several images showing the facial expressions of animals are prepared, and an intermediate image between the images is generated in a simulated manner, as if the animals were talking, or the facial expressions of the animals. There is also a morphing technique that realizes a special effect as if the sound changes continuously.

On the other hand, voice morphing is also known. With this kind of audio morphing technology, for example,
The sounds A and B are sampled as digital sounds, and an intermediate waveform of the sounds A and B is generated by interpolation based on the original waveforms (wave tables) of the sounds A and B, and the sounds A and B are generated with time. To the sound B continuously. This kind of effect can be realized by using, for example, an electronic musical instrument (sampler) that can reproduce a recorded analog voice waveform at an arbitrary pitch.

[0004]

However, even though the conventional image morphing technique produces an intermediate image between two types of images A and B which are not related to each other, the aging and age of the object to be imaged are changed. It is not predictable of change. Conventionally, in criminal investigations and the like, there has been a case where a special graphic software is used to create a current or future pseudo image of the subject based on a previously captured image of the subject. Since each part of the image is designated one by one and the type and degree of change of each part is specified to create a pseudo image, the accuracy of the pseudo image largely depends on the skill and intuition of the operator,
In addition, there is a problem that a great deal of labor and cost are required to produce a large number of pseudo images because the processing takes time. Therefore, the conventional technology has not been able to adequately cope with an increasing demand for special video effects such as morphing in recent movies and television programs.

Further, the conventional voice morphing technique also has a problem that the sound quality of the pseudo voice is low.
For this reason, for example, a sound synchronized with a moving image by morphing tends to have low realism, and it is easy to require a lot of labor and cost to give the sound a real feeling.

SUMMARY OF THE INVENTION In view of the above problems, the present invention is intended to solve the problems described above with respect to an image and a sound that quickly and inexpensively produce a high-quality pseudo image or a pseudo sound showing the age change and the secular change of the object or sound source. The present invention is to provide an interpolation processing device and an interpolation processing method thereof.

[0007]

According to a first aspect of the present invention, there is provided an image interpolating process for generating a pseudo image of a subject using a plurality of real image data of the subject. A method, comprising: an image storage unit that stores a plurality of real image data of the same object having different imaging times; and pseudo image data in designated time information based on the plurality of real image data selected from the image storage unit. And an image interpolating unit that calculates the value by interpolation.

According to a second aspect of the present invention, there is provided the image interpolation processing apparatus according to the first aspect, wherein the image interpolating unit automatically recognizes a characteristic portion of a plurality of real image data selected from the image storing unit. Characteristic unit recognizing means, and characteristic part associating means for associating the characteristic parts with each other among the plurality of real image data.

According to a third aspect of the present invention, in the image interpolation processing apparatus according to the second aspect, the image interpolating unit sets the time information as an independent variable for a pixel value in the characteristic unit, And an interpolation unit that calculates an interpolation curve by executing interpolation with the dependent variable as a dependent variable, and calculates a pixel value at the specified time information using the interpolation curve, and generates and outputs a pseudo image of the object to be captured. Image generation means;
Is further provided.

The invention according to claim 4 is the invention according to claims 1 to 3.
The image interpolation processing device according to any one of claims 1 to 3, wherein an alternative image storage unit storing an alternative image corresponding to the time information, and an alternative image storage device storing the alternative image corresponding to the designated time information with reference to the alternative image storage unit. Image synthesizing means for searching for a substitute image to be performed and applying the substitute image to the pseudo image generated by the pseudo image generating means.

Next, a fifth aspect of the present invention is an image interpolation processing method for generating a pseudo image of a subject using a plurality of real image data of the subject. Recording a plurality of real image data of the same object to be photographed, and (b) calculating pseudo image data at specified time information by interpolation based on the real image data recorded in the step (a). And the following.

The invention according to claim 6 is the image interpolation processing method according to claim 5, wherein the step (b) is performed.
Further comprises: (b-1) automatically recognizing a characteristic portion of the real image data, and (b-2) correlating the characteristic portion among a plurality of the real image data. Things.

The invention according to claim 7 is the image interpolation processing method according to claim 6, wherein the step (b) is performed.
(B-3) calculating an interpolation curve by executing interpolation using the time information as an independent variable and the pixel value as a dependent variable for the pixel value in the characteristic portion;
(B-4) calculating a pixel value at the specified time information using the interpolation curve, and generating a pseudo image of the object to be imaged.

The invention according to claim 8 is the invention according to claims 5-7.
The image interpolation processing method according to any one of claims 1 to 3, wherein after the step (b), (c) a substitute image corresponding to the time information is added to the pseudo image data generated in the step (b). And a step of applying

A ninth aspect of the present invention is a sound interpolation processing apparatus for generating a pseudo sound of a sound source using a plurality of real sound waveform data, wherein the same sound source is generated in a plurality of time zones separated by time. A real sound waveform storage unit that stores a plurality of real sound waveform data obtained from the real sound waveform data unit, and a real sound waveform interpolation unit that calculates pseudo sound waveform data at specified time information by interpolation based on the plurality of real sound waveform data selected from the real sound waveform storage unit. And a unit.

According to a tenth aspect of the present invention, in the sound interpolation processing apparatus according to the ninth aspect, the real sound waveform interpolating unit weights and averages the real sound waveform data selected from the real sound waveform storage unit, This is obtained by calculating pseudo sound waveform data at designated time information.

According to an eleventh aspect of the present invention, in the sound interpolation processing apparatus according to the ninth aspect, the real sound waveform interpolating unit stores the time information of the real sound waveform data selected from the real sound waveform storage unit as an independent variable. A pseudo-sound generating means for calculating an interpolation curve by executing interpolation using the strength of the actual sound waveform as a dependent variable, and calculating pseudo sound waveform data at the specified time information using the interpolation curve. It is provided.

According to a twelfth aspect of the present invention, in the sound interpolation processing apparatus according to the ninth aspect, the real sound waveform interpolation unit calculates a frequency characteristic distribution of the real sound waveform data selected from the real sound waveform storage unit. Frequency characteristic distribution calculating means,
A feature component extraction unit that extracts a feature component that is a frequency emphasis component from the frequency characteristic distribution, a feature component interpolation unit that calculates a pseudo feature component in the designated time information by interpolating a plurality of the feature components, A characteristic component removing unit that generates and outputs actual sound waveform data excluding the characteristic component; and outputs the pseudo acoustic waveform data by adding the pseudo characteristic component to the actual sound waveform data output from the characteristic component removing unit. Characteristic component adding means.

The invention according to claim 13 is the invention according to claim 12
The sound interpolation processing device according to claim 1, further comprising: a correction data storage unit that stores correction data for correcting the characteristic component, wherein the characteristic component interpolation unit refers to the correction rule storage unit to determine a sound quality of the sound source. Acquisition of correction data corresponding to
The characteristic component is corrected based on the correction data.

The invention according to claim 14 is the invention according to claims 9 to
13. The sound interpolation processing device according to any one of 13),
A function of synchronizing the pseudo sound waveform data with a pseudo image generated by the image interpolation processing device according to any one of claims 1 to 4.

Next, the invention according to claim 15 is a sound interpolation processing method for generating a pseudo sound of a sound source using a plurality of real sound waveform data obtained from a sound source, wherein (d) a plurality of time zones Recording a plurality of real sound waveform data obtained from the same sound source, and (e) calculating pseudo sound waveform data at specified time information by interpolation based on the real sound waveform data recorded in the step (d). And the following.

The invention according to claim 16 is the invention according to claim 15.
The sound interpolation processing method according to claim 1, wherein said step (e)
In the above, the actual sound waveform data recorded in the step (d) is weighted and averaged, and the pseudo sound wave data at the designated time information is calculated by interpolation.

The invention according to claim 17 is based on claim 15.
The sound interpolation processing method according to claim 1, wherein said step (e)
(E-1) calculating an interpolation curve by executing interpolation using the time information of the actual sound waveform data as an independent variable and the intensity of the actual sound waveform as a dependent variable; and (e-2).
Calculating the pseudo sound waveform data at the designated time information using the interpolation curve.

The invention according to claim 18 is directed to claim 15
The sound interpolation processing method according to claim 1, wherein said step (e)
(E-3) calculating a frequency characteristic distribution of the actual sound waveform data; (e-4) extracting a characteristic component that is a frequency emphasis component from the frequency characteristic distribution; Calculating a pseudo feature component in the designated time information by interpolating a plurality of feature components; (e-6) generating and outputting real sound waveform data excluding the feature component; 7) calculating the pseudo sound waveform data by adding the pseudo feature component calculated in the step (e-5) to the actual sound wave data output in the step (e-6). Further provisions.

The invention according to claim 19 is the invention according to claim 18
(F) correcting the characteristic component of step (e-4) or (e-7) according to correction data corresponding to the sound quality of the sound source;
Is further provided.

The invention according to claim 20 is based on claim 1
The sound interpolation processing method according to any one of claims 5 to 19, wherein the pseudo sound waveform data is added to a pseudo image generated by the image interpolation processing method according to any one of claims 5 to 8. Synchronize.

[0027]

<First Embodiment> FIG. 1 is a functional block diagram showing a schematic configuration of an image interpolation processing apparatus 1 according to a first embodiment of the present invention.

Image interpolation processing device 1 according to the present embodiment
Is provided with a real image database 2 that stores actual image (real image) data of the object to be imaged. The real image database 2 stores real image data I ₁ , I ₂ ,
..., I _n each time information (imaging time) T _1, T _2,
..., it is stored in labeled the T _n. In the example shown,
Time information T ₁ is January 1, 1950, T ₂ is 1960
Month 1 day, T ₃ January 7, 1980, ..., T _n 2000
January / January. Note that the time information may indicate the date and time of imaging, or may indicate the age and elapsed time of the object to be imaged. The real image database 2 may directly store the captured real image data, or may capture and store existing photographs and image data. Further, in the present embodiment, a person image will be described as an example. However, in the present invention, the object to be imaged is not limited to a person, but may be another living thing, plant, building, or the like.

The image interpolation processing apparatus 1 further includes a feature recognition unit 7 for automatically recognizing features such as the eyes and nose of the object displayed on the plurality of real image data, and real image data I ₁ , I _2, ..., a feature association unit 3 to correspond to each other feature between I _n, provided with an interpolation means 4 for performing interpolation between the plurality of actual image data, the interpolation means And a pseudo image generating means 5 for generating a pseudo image of the object to be picked up using the interpolation curve calculated in step 4.

The processing procedure of the image interpolation processing device 1 having such a configuration will be described in detail below with reference to the flowchart of FIG. First, a plurality of real image data of a required shooting date is selected from among the real image data T ₁ ,..., T _n stored in the real image database 2 (ST1).

Next, the characteristic unit associating means 3 executes a process of associating the background of the actual image data thus selected and the characteristic unit of the object to be imaged (ST2). FIG.
As shown in FIG. 2, the selected real images are an image A (photographed at the age of 30 in 1980) and an image C (2
000 years and 50 years old) is associated after the features (eyes, nose, lips, eyebrows, ears, etc.) of both images A and C are recognized. Note that clothes, hairstyles, and the like do not easily show age changes and are not easily compared, and thus may be excluded from the feature portion association processing. Such an association process may be performed by an operator operating a pointing device such as a mouse or a stylus using image processing software to specify and recognize common features of both images one by one. Alternatively, the characteristic portions may be automatically recognized using the pattern recognition technology and may be associated with each other.

When the characteristic part recognizing means 7 has a pattern recognition function, the characteristic part recognizing means 7 is used to distinguish the object to be displayed and the background image displayed on each of the selected real image data. Recognize the contours of the imaged object in a pattern, and show physical characteristics indicating the age change of the imaged object, for example, nose muscles, bare skin,
Characteristic parts such as hair are extracted by pattern matching with a reference model prepared in advance, which simulates those characteristic parts. As a result, the processing time is shortened, and a pseudo image of the object to be imaged can be created in a short time and efficiently. For example, a specific description will be given with reference to FIG. 3. In the feature part recognizing means 7, the outline 10 a of the imaging object 10 displayed in the image A and the outline 12 a of the imaging object 12 in the image C are recognized. Objects 10 and 12 to be imaged and background images 11 and 13 are identified. Next, after the features such as the eyes and the nose of the object to be imaged 10 and the features such as the eyes and the nose of the object to be imaged 12 are recognized, each feature of both the objects to be imaged 10 and 12 It is associated in units.

Next, the processing shifts to step ST3.
The pixel value (red
(R), luminance of pixel composed of green (G) and blue (B)
Value) is interpolated. The graph of FIG.
As an example, time information T is an independent variable, and pixel value is
The time information T for the pixel a of the image A is set as the dependent variable.
_AThe point indicating the pixel value of_A, Time for pixel c of image C
Information T_CThe point indicating the pixel value of_C, Two points S_A,
S_CInterpolated curve f_i(T) (i: i-th pixel value
Is shown). In the example shown, the interpolation music
Line f_i(T) is 2 points S_A, S_CIt is a straight line that connects them,
Known interpolations such as least squares and Lagrange's interpolation formula
The interpolation curve may be formed using a means. like this
Interpolation curve f_iUsing (T), two points S_A, S_CTime between
Information T_BThe point P at_BAnd the two points S_A,
S_CTime information T outside the range between_DThe point P at_DIndicated by
Pixel values can be determined.

As shown in the graph of FIG. 4B, when there are a plurality of real images and there are a plurality of points S _A , S _B , and S _C indicating the pixel values associated with the real images, The points S _A , S _B ,
What is necessary is just to form an interpolation curve f _i (T) connecting S _C with a straight line. By performing such linear interpolation, for example, the time information T
It is possible to quickly calculate the pixel value indicated by a point P _C in _C. Instead of using a linear interpolation curve f _i (T), in the case of an interpolation method that takes all points into consideration, such as a least square method, an interpolation curve g as illustrated in FIG. _i (T) is formed, and although it takes slightly more processing time than in the case of linear interpolation, the specified pixel value can be calculated more accurately.

Next, the process proceeds to step ST4, in which the pseudo image generating means 5 calculates pixel values in the designated time information for all the pixels using the interpolation curve created in step ST3. Generates a pseudo image and outputs it to the display device 6, and the display device 6 displays the pseudo image (ST5).

In this way, as illustrated in FIG. 3, the object 10 at the age of 30 (1980) displayed on the image A and the image at the age of 50 (2000) displayed on the image C are displayed.
The image-capturing object 14 at the age of 40 (1990) displayed in the image B and the image-capturing object 16 at the age of 55 (2005) displayed in the image D are generated from the image-capturing object 12 Can be displayed. In addition, by continuously generating a pseudo image between the image A and the image B along the time axis, the appearance of the same person in the display device 6 from the image A to the image B or in the opposite direction is displayed. It is possible to display a state in which the age changes gradually. Therefore,
According to the first embodiment, it is possible to create a pseudo image at desired time information with high accuracy and in a short time so that there is a sense of realism by using real images of the same object having different imaging times. It is.

Second Embodiment Next, a second embodiment of the present invention will be described. FIG. 5 is a functional block diagram illustrating a schematic configuration of the image interpolation processing device according to the second embodiment. In FIG. 5, blocks denoted by the same reference numerals as those described in the first embodiment have substantially the same functions, and a detailed description thereof will be omitted.

An image interpolation processing device 2 according to the second embodiment
0 is provided with an alternative image database 21 storing alternative images and an image synthesizing means 22 for synthesizing the alternative images with the pseudo image in addition to the configuration of the image interpolation processing apparatus 1 according to the first embodiment. I have.

The processing procedure of the image interpolation processing device 20 is as follows.
This will be described below with reference to the flowchart of FIG. First, the steps ST1 to ST described in the first embodiment are performed.
4 (FIG. 2). That is, real images having different shooting dates and times (shooting times) are selected from the real image database 2 (ST1), and the characteristic unit associating means 3
The characteristic portions of the plurality of real images selected in (1) are associated between the real images (on the time axis) (ST2), and interpolation processing between the characteristic portions associated in step ST2 is executed by the interpolation means 4 (ST3). Then, a pseudo image is generated by the pseudo image generating means 5 using the interpolation curve (S
T4).

Then, the processing shifts to step ST10, where the image synthesizing means 22 refers to the substitute image database 21 and corresponds to the age of the pseudo image output in step ST4, and performs interpolation by the interpolation means 4. Search for an alternative image that can replace a difficult-to-find feature. As the substitute image, the background and clothes of the age corresponding to the age of the pseudo image,
Hairstyle and the like. Also, if the imaged object displayed in the sample real image is all black hair, it is difficult to generate a pseudo image of the imaged object with white hair by interpolation, so it depends on the age of the pseudo image. It is desirable to use an alternative image of gray hair. When the photographing time of the sample real image and the designated time of the pseudo image are far from each other, it is desirable to use the substitute image so as to supplement the age change.

Next, in step ST11, an image synthesizing process of the pseudo image and the substitute image is executed by replacing the characteristic portion which is difficult to interpolate by the interpolation means 4 with the substitute image.
For example, in the image C shown in FIG. 3, the background image 15 of the object 14 can be replaced with another background image corresponding to the age, or the clothing of the object 14 can be changed. Finally,
The synthesized pseudo image output from the image synthesis means 22 is:
It is displayed on the display device 6 (ST12).

As described above, according to the present embodiment, the use of the substitute image database 21 allows the substitute image to be applied to a portion that is difficult to interpolate. Can be generated. In addition, it is possible to reproduce a moving image by executing high-precision morphing using the pseudo image.

<Third Embodiment> When reproducing a moving image or performing morphing using a pseudo image created by the image interpolation processing apparatus according to the above embodiment, a pseudo sound synchronized with the pseudo image is performed. (Hereinafter, it is called a pseudo sound.) By reproducing the sound, the sense of reality can be further enhanced. FIG. 7 is a functional block diagram showing a sound interpolation processing device 30 according to Embodiment 3 of the present invention for generating such a pseudo sound.

The sound interpolation processing device 30 according to the third embodiment
Are the actual sound waveform data W ₁ , W ₂ , W of the sound output from the sound source.
_3, ..., a real waveform storing section 31 which stores W _n. The real sound waveform data is quantized by a PCM (Pulse-Code Modulation) method or the like, for a fixed time, at a sampling frequency of about several tens of kHz and a resolution of about 12 to 20 bits at a sampling frequency of about several tens of kHz. Sampling. The actual sound waveform data W ₁ ,..., W _n are stored in the actual sound waveform storage unit 31 in a state in which time information (sampling time) at the time of sampling the actual sound waveform is associated. In the illustrated example, the time information T ₁ of the actual sound waveform data W ₁ is January 1, 1945, T ₂ is January 1, 1960, T ₃ is January 1, 1980,..., T _n is 2000. January 1st. still,
As in the case of the actual image data, the time information may indicate the date and time of image capturing, or may indicate the age or elapsed time of the sound source. Further, the recorded actual sound waveform data may be directly stored in the actual sound waveform storage unit 31, or existing sound data may be captured and stored.

The sound interpolation processing device 30 includes an actual sound waveform averaging means 32 for averaging the actual sound waveform data stored in the actual sound waveform storage unit 31 and outputting the averaged sound data to the acoustic device 33.

The processing procedure of the sound interpolation processing device 30 will be described below in detail with reference to the flowchart of FIG.
First, time information (Ts) designated from the outside and a plurality of real sound waveform data W ₁ ,... From the real sound waveform storage unit 31 are input to the real sound waveform averaging means 32 (ST20).

Next, the actual sound waveform averaging means 32 generates a weighted and averaged actual sound waveform based on the time information Ts, and outputs it to the acoustic device 33 (ST21). That is, when the actual sound waveform corresponding to the time information T is w _i (T) and the weight coefficient corresponding to the time information T _i is M _i , the time information Ts
Is between the time information T _k and T _{k + 1} (k: 1 to n−1), the actual sound waveform w _s (t) corresponding to the time information Ts is expressed by the following equation.

W _s (t) = M _k × w _k (t) + M _{k + 1} × w
_{k + 1} (t) (however, M _k + M _{k + 1} = 1) Here, as shown in the xy graph (0 ≦ x ≦ 1) of FIG. _When the points of _k and x = 1 correspond to T _{k + 1} and the points of x = x _s to Ts, M _k = 1−x,
It suffices to set M _{k + 1} = x.

Therefore, in step ST21, the actual sound waveform averaging means 32 calculates which time information of the actually measured actual sound waveform data is within the input time information Ts, and calculates the time information Ts based on the above equation. Is calculated, the pseudo sound of the actual sound waveform is reproduced by the acoustic device 33 (ST22). For example, when the input time information Ts is January 1, 1950, the time information Ts is the time information T ₁ of the actual sound waveform data W ₁ and W ₂ (Jan. 1945).
1) and T ₂ (Jan. 1, 1960), x _s corresponding to the input time information Ts shown in FIG.
Is x _s = (1950-1945) / (1960-1945) = 1/3, and the weighting factor is M ₁ = 1 /
3, M ₂ = ２. Therefore, the input time information T
The actual sound waveform corresponding to s is w _s (t) = (１ ／) w
It is expressed by _{1 (t) + (2/3)} w 2 (t).

As described above, in the third embodiment, the sound generated by the sound source in the non-sampled time information can be generated and reproduced in a pseudo manner based on a plurality of real sound waveform data of the same sampled sound source. Can be obtained. Further, by combining the sound interpolation processing device 30 according to the third embodiment with the image interpolation processing devices according to the first and second embodiments, the sound interpolation processing device 30 is extremely synchronized with the moving image output from these image interpolation processing devices. It is also possible to realize realistic sound morphing and special sound effects.

<Fourth Embodiment> FIG. 10 is a functional block diagram showing a schematic configuration of a sound interpolation processing device 40 according to a fourth embodiment of the present invention. Note that, in FIG. 10, blocks denoted by the same reference numerals as those described above have substantially the same functions, and a detailed description thereof will be omitted.

The sound interpolation processing device 40 according to the fourth embodiment
Is a real sound waveform storage unit 31 that is the same as that described in the third embodiment, a real sound waveform interpolation unit 41 that performs an interpolation process on the axis of time information of the plurality of real sound waveform data, A pseudo-sound generating means for generating a pseudo-sound based on the actual sound waveform data interpolated at 41.

The processing procedure of the sound interpolation processing device 40 will be described in detail below with reference to the flowchart of FIG. 11. First, the real sound waveform interpolation means 41
1, a plurality of actual sound waveform data W ₁ ,..., W _n in different time zones sampled from the same sound source are acquired (S
T30).

Next, the actual sound waveform interpolation means 41, their actual sound waveform data W _1, ..., the actual sound waveform of _{W n w 1 (t),} ...,
w _n (t) is interpolated in the axial direction of the time information T to form an interpolation curve or interpolation surface (ST31). That is, the actual sound waveform interpolating means 41 determines that the sampling time T when the actual sound waveform data is sampled is T = T _j (j =
, 1, 2,...), And the interpolation curve F
_j (T) (j = 1, 2,...) is calculated. As the interpolation method, a known interpolation means such as a least squares method or a Lagrange's interpolation method may be used.

Next, the process proceeds to step ST32, where the pseudo sound generating means 42 receives the input of the time information Ts,
The interpolation curve F calculated by the actual sound waveform interpolation means 41
_{Using j} (T), a pseudo-sound at time information T = Ts is generated and output to the audio device 33 so that the pseudo-sound is synchronized with the moving image by morphing or the like, and the pseudo-sound is reproduced by the audio device 33 ( ST33). For example, when performing interpolation processing on the actual sound waveform data W ₁ to _W-3 in a real-waveform interpolation means 41, the interpolation curve or using an interpolation curved surface thereof actual sound waveform data time information T ₁ (1 January 1945),
T ₂ (January 1, 1960), T ₃ (January 7, 1980)
Days) (1945 to 1980)
Year), but outside that range (less than 1945 and 19
(Over 80 years).

As described above, according to the fourth embodiment, similarly to the third embodiment, based on the actual sound waveform data from a plurality of the same sound sources having different sampled dates and times, the date and time that are not sampled are determined. , It is possible to simulate the sound of the same sound source in, and create a morphing of a sound with a realistic feeling and a special sound effect. Further, in the third embodiment, since the interpolation curve of the actual sound waveform w _i (t) is created on the axis of the time information T, not only within the range of the time information of the sampled actual sound waveform data but also outside the range. It is possible to calculate a pseudo sound waveform and reproduce the pseudo sound.

<Fifth Embodiment> Next, FIG. 12 is a functional block diagram showing a schematic configuration of a sound interpolation processing device 60 according to a fifth embodiment of the present invention, and FIG.
11 is a flowchart illustrating a processing procedure according to the first embodiment.

As shown in the third embodiment, the sound interpolation processing device 60 stores the actual sound waveform storage unit 31 storing a plurality of actual sound waveform data W ₁ ,..., W _n having different time zones (recording times). Frequency characteristic distribution calculating means 69 for calculating the frequency characteristic distribution of the actual sound waveform data; further, a characteristic component extracting means 61 for extracting a characteristic component of the frequency characteristic distribution; and a predetermined component based on the designated time information Ts. , A characteristic component interpolating means 64 for converting a characteristic component according to the rule, a characteristic component removing means 62 for removing the characteristic component from the corresponding real sound waveform data, and a real sound waveform data W ₁ ′,.
'And waveform synthesizing means 63 for synthesizing the actual sound waveform data W ₁ excluding the feature component' W _n, ..., characteristic components added for adding feature components after conversion to each of the W _n 'in the characteristic component interpolating means 64 Means 66. Characteristic component adding means 66
The sound signal output from is reproduced by the acoustic device 33.

The sound interpolation processing device 6 having such a configuration
0 will be described in detail below with reference to the flowchart of FIG. First, the frequency characteristic distribution calculating means 69
The actual sound wave data W ₁ ,.
_n (n ≧ 2) is acquired (ST50), and FFT (Fast Four
ier Transform; by high-speed Fourier transform), their actual sound waveform data W _1, ..., the frequency characteristics of W _n, respectively distributions F _1, ..., is converted to F _n (ST51). For example, FIG.
The actual sound waveform W _k shown in FIG. 4A is converted into a frequency characteristic distribution F _k as shown in FIG. Such frequency characteristic distribution F _k, a plurality of characteristic components 70 and 70 resulting highlighted in a specific frequency domain, ... are included, a source-specific components that characterize the sound quality. Generally, characteristic components of musical instruments and voices are called formants. Speech formants are largely determined by the frequency of the sound emitted by the vocal organs of the palate, etc., while male formants are in a relatively low frequency range, while female formants are in a relatively high frequency range. It is common to do.

Next, the characteristic component is extracted from the above-mentioned frequency characteristic distributions F ₁ ,..., F _n by the characteristic component extracting means 61 (ST 52), and a value representing each characteristic component (the reference frequency f
₀ ) is the characteristic component removal means 62 and the characteristic component interpolation means 6
4 is output. The characteristic component is FFT or LPC
(Linear Prediction Coding) can be extracted by known cepstrum processing using analysis or the like.

Next, the characteristic component removing means 62 converts the characteristic component of the frequency characteristic from the original actual sound waveform data W ₁ ,..., W _n into an IIR (Infinite Imp
ulseResponse) filter and FIR (Finite Impulse Res)
The characteristic component is flattened using a digital filter such as a filter (ST53). Thus the actual sound waveform data W ₁ obtained ', ..., W _n' is waveform synthesis means 6
3 is output.

Next, the waveform synthesizing means 63 interpolates the actual sound waveforms W ₁ ′,..., W _n ′ from which the characteristic components have been removed from the frequency characteristic distribution, and generates a pseudo-waveform containing no characteristic components in the time information Ts. A real sound waveform W _c 'is synthesized (ST54). As the interpolation processing, averaging processing using the same weighting factor as described in the third embodiment,
Interpolation processing using the same interpolation curve as described above may be applied.

In parallel with the processing of steps ST53 and ST54, the characteristic component extracted by the characteristic component extraction means 61 is interpolated by the characteristic component interpolation means 64 to generate a pseudo characteristic component in the time information Ts. (ST5
5). As the interpolation processing, an averaging processing using the same weight coefficient as described in the third embodiment or an interpolation processing using an interpolation curve similar to the one described in the fourth embodiment can be applied. I just need.

Further, the characteristic component interpolation means 64 executes the above-mentioned correction processing in accordance with the rules (correction data) stored in the correction rule database 65. Correction rule database 65
Stores various control parameters. Specifically, when using an IIR filter, the center frequency f ₀ , Q value, level (unit: dB) and the like are used. When using an FIR filter, a coefficient group of each tap according to the score (order) is used. Is a control parameter, which is a numerical value for determining a new feature component. In order to determine these control parameters, for example, a rule that defines a range of control parameters according to the classification of a man, a woman, a child, or an elderly person, a fluctuation range that follows a reference frequency of an input audio signal, and the like is stored in the correction rule database 65. Is stored.

For example, in the frequency characteristic of a sampled male voice, when there are characteristic components (peaks) at 2 kHz and 3 kHz, if a reference frequency of 1 kHz is input, each level at the center frequency 2 kHz and 3 kHz of the characteristic component is input. Can be controlled to be increased by a predetermined enhancement amount. Further, for example, the control can be performed such that the characteristic component is shifted so that the peak of the characteristic component is located at 2.1 kHz and 2.8 kHz, which are the overtones of 700 Hz. Further, when the input sound has an overtone of the reference frequency, or when the input sound has a relatively low frequency such as 200 Hz, the frequency of the characteristic component may be slightly changed. Furthermore, a rule may be provided so that the frequency of the overtone is correlatedly changed with a change in the reference frequency input to the characteristic component interpolation means 64. For example, the peak frequency of the characteristic component can be changed at a tracking rate of several tens% of the rate of change of the reference frequency. As described above, since the characteristic components can be corrected so as to correspond to the sound quality of the sound source, it is possible to create a realistic pseudo sound.

Next, the process proceeds to step ST56,
In the characteristic component adding means 66, the above-mentioned step ST54 is performed.
The characteristic components of the plurality of real sound waveforms interpolated in step ST55 are added to the composite waveform calculated in step ST55.

As a circuit configuration of the characteristic component adding means 66, there is a circuit in which three state variable filters 71 ₁ , 71 ₂ and 71 ₃ are connected in series as exemplified in FIG. In FIG. 15, reference symbol A1 indicates an adder, D1 indicates a delay unit, and M1 indicates a multiplier. Each filter 71 ₁ , 7
1 _2, 71 _3, respectively to control the frequency characteristic according to the three peak frequencies F1, F2, F3 as a BPF, respectively (center frequency, Q value, level) of the set of parameters (F1, Q1, L1) , (F2, Q2, L2), (F
When (3, Q3, L3) is specified, an output Y is obtained for the input signal X. By using the state variable filter 71 _{1-71 3} such 3 group in series, FIG. 16
As shown in FIG. 5, the above three sets of parameters (F1, Q1,
L1), (F2, Q2, L2), (F3, Q3, L3)
Can be obtained. For example, the reference frequency 1 kHz of the feature component is input from the feature component extraction unit 61 to the feature component interpolation unit 64, and the feature component interpolation unit 64 refers to the correction rule database 65 and adds +6 dB for the overtone 2 kHz and 3 kHz for the 3 kHz. +2
When the rule of setting the level to +8 dB for dB and 4 kHz is applied, the state variable filter 71 ₁
Set of control parameters transmitted to the -71 _3, respectively (2kHz, 2.0,6.0dB), (3kHz , 2.
5, 2.0 dB), (4 kHz, 4.0, 8.0 dB)
Becomes

Note that an FIR filter may be used instead of the state variable filter illustrated in FIG. In this case, generally, the entire frequency characteristic can be controlled with a frequency resolution corresponding to a score of about 64th to 1024th order.

Finally, the actual sound waveform data generated by the characteristic component adding means 66 is output to the audio equipment 33, and the pseudo sound is reproduced (ST56).
The acoustic device 33 is, for example, a PWM (Pulse Width).
Modulation) sound source, FM (Frequency Modulatoin) sound source, PCM (Pulse Code Modulation) sampling sound source, DSP (Digital Signal Processing) sound source, DL
An S (Down Loadable Sampling) sound source or the like may be used together.

As described above, according to the sound interpolation processing device 60 according to the fifth embodiment, the characteristic component is extracted from the frequency characteristic distribution of the actual sound waveform data by the characteristic component extraction means 61, and the characteristic component is extracted by the characteristic component interpolation means 64. Since the interpolation process based on the desired time information Ts is performed on the characteristic component, it is possible to generate and reproduce a highly accurate and realistic pseudo sound. In addition, since the characteristic component interpolation means 64 executes the correction process in accordance with the rule with reference to the correction rule database 65, the pseudo sound of the time information, which is difficult to interpolate based on the sampled actual sound waveform data, is generated in a more realistic manner. Can be generated and reproduced.

<Modification of the Fifth Embodiment> As shown in FIG. 17, the waveform output from the characteristic component adding means 66 interposed between the characteristic component adding It is desirable to further provide a waveform converting means 67 for converting based on Ts. The waveform converting means 67 reads a parameter (correction data) corresponding to the predetermined time information Ts with reference to the conversion parameter database 68, changes the amplitude of the waveform output from the characteristic component adding means 66, Has a function of expanding and contracting at a predetermined pitch conversion rate. As a result, for example, in order to represent a vocal cord change (for example, a voice change) due to a change in age, an actual sound waveform recorded at 1 kHz illustrated in FIG. 18A is compressed on the time axis, and FIG. 2) can be generated, or the waveform of FIG. 3A can be expanded to generate a 500 Hz reproduced waveform of FIG. 3C. In this case, the pitch conversion rate is an example of a parameter stored in the conversion parameter database 68.
Generally, the conversion range of the sound waveform is to improve the sound quality,
It is desirable to suppress it to about plus or minus one octave (pitch conversion rate: 1/2 to 2 times).

While the various embodiments of the present invention have been described above, the image interpolation processing apparatuses according to the first and second embodiments and the sound interpolation processing apparatuses according to the third to sixth embodiments are combined to form an image. Simultaneous reproduction of the moving image generated by the interpolation processing device and generation of the pseudo sound by the sound interpolation processing device in synchronization with the moving image enables realization of a special video and audio effect with a sense of reality.

[0073]

As described above, according to the image interpolation processing apparatus and the interpolation processing method according to the first and second aspects of the present invention, it is possible to obtain a desired image based on a real image of an object already captured. A pseudo image of time information (age and age) can be created with a sense of reality. For this reason, for example, using two real images of the object to be imaged when they are 20 years old and 40 years old,
It is possible to create a pseudo image of the object to be imaged at the age of 50 or 50 years.

According to the second and sixth aspects, since the characteristic portion of the object to be displayed displayed in the real image can be automatically recognized, the processing time for associating the characteristic portion between the real images can be shortened. A pseudo image of the object to be imaged can be generated efficiently in a short time. In addition, it is possible to create a pseudo image in which the age of an object to be imaged changes by morphing in a short time and with high accuracy.

According to the third and seventh aspects, it is possible to create a more realistic and accurate pseudo image.

According to the fourth and eighth aspects, a substitute image can be applied to a portion that is difficult to interpolate using a sampled real image, so that a more realistic pseudo image can be created. . For this reason, the shape and color of the hair and beard corresponding to the age, the properties of the skin, and the like can be replaced with those of the substitute image, and the realism can be increased.

According to the sound interpolation processing apparatus and the interpolation processing method according to the ninth aspect of the present invention,
By using sounds recorded in multiple time zones separated by time from the same sound source, it is possible to create a realistic pseudo sound in the desired time information (age, elapsed time, date, etc.) in a short time Becomes

According to the tenth and sixteenth aspects, it is possible to calculate the pseudo sound waveform data with high accuracy by averaging the actual sound wave data.

According to the eleventh and seventeenth aspects, it is possible to more accurately generate and reproduce a pseudo sound by using an interpolation curve.

According to the twelfth and eighteenth aspects, since interpolation is performed between a plurality of feature components extracted from the frequency characteristic distribution, it is possible to obtain an extremely realistic pseudo sound.

According to the thirteenth and nineteenth aspects, since the characteristic component can be corrected so as to correspond to the sound quality of the sound source, an actual sound waveform in time information that is difficult to interpolate is created with a sense of realism. It becomes possible to reproduce. For this reason, for example, when the sound source is a man, a woman, a child, or an elderly person, the interpolated characteristic component creates a pseudo sound corresponding to the sound quality of each sound source, or the sampled real sound waveform data is entirely vocalized. In the case where the sound is before the sound change, a pseudo sound after the sound change can be created based on the correction data relating to the sound change.

According to the fourteenth and twentieth aspects, since a sound synchronized with the pseudo image can be created, the special video effect can be further enhanced.

[Brief description of the drawings]

FIG. 1 is a block diagram illustrating a schematic configuration of an image interpolation processing device according to a first embodiment of the present invention.

FIG. 2 is a flowchart illustrating a processing procedure performed by the image interpolation processing device according to the first embodiment;

FIG. 3 is a schematic diagram showing a real image and a pseudo image of the imaged object having different ages.

FIG. 4A is a graph showing an interpolation curve of two pixel values on a time axis, and FIG. 4B is a graph showing an interpolation curve of three pixel values on a time axis.

FIG. 5 is a block diagram illustrating a schematic configuration of an image interpolation processing device according to a second embodiment of the present invention.

FIG. 6 is a flowchart illustrating a processing procedure performed by the image interpolation processing apparatus according to the second embodiment;

FIG. 7 is a functional block diagram illustrating a schematic configuration of a sound interpolation processing device according to a third embodiment of the present invention.

FIG. 8 is a flowchart illustrating a processing procedure performed by the sound interpolation processing device according to the third embodiment;

FIG. 9 is an explanatory diagram showing a graph for calculating a weight coefficient according to the third embodiment.

FIG. 10 is a functional block diagram showing a schematic configuration of a sound interpolation processing device according to Embodiment 4 of the present invention.

FIG. 11 is a flowchart showing a processing procedure by the sound interpolation processing device according to the fourth embodiment.

FIG. 12 is a functional block diagram illustrating a schematic configuration of a sound interpolation processing device according to a fifth embodiment of the present invention.

FIG. 13 is a flowchart showing a processing procedure by the sound interpolation processing device according to the fifth embodiment.

FIG. 14A is a schematic diagram illustrating an example of an actual sound waveform.
(B) is a schematic diagram showing the frequency characteristic distribution.

FIG. 15 is a circuit diagram of an IIR filter used in a characteristic component adding unit of the sound interpolation processing device according to the fifth embodiment.

16 is a schematic diagram showing a waveform example using the IIR filter of FIG.

FIG. 17 is a functional block diagram illustrating a main configuration of a modification of the fifth embodiment.

18A is a diagram showing an actual sound waveform, and FIG.
(C) is a diagram in which the actual sound waveform shown in (a) is compressed along the time axis, and FIG.

FIG. 19 is a diagram showing a conventional example of image morphing.

[Explanation of symbols]

DESCRIPTION OF SYMBOLS 1 Image interpolation processing apparatus 2 Real image database 3 Characteristic part associating means 4 Interpolating means 5 Pseudo image generating means 6 Display device 7 Characteristic part recognizing means 10, 12, 14, 16 Subject 11, 13, 15, 17 Background image 20 Image Interpolation processing device 21 Alternative image database 22 Image synthesis means 30 Sound interpolation processing device 31 Real sound waveform storage unit 32 Real sound waveform averaging means 33 Audio equipment 40 Sound interpolation processing device 41 Real sound waveform interpolation means 42 Pseudo sound generation means 60 Sound interpolation processing device 61 feature component extraction means 62 feature component removal means 63 waveform synthesis means 64 feature component interpolation means 65 correction rule database 66 feature component addition means 67 waveform conversion means 68 conversion parameter database 69 frequency characteristic distribution calculation means 71 _{1 to} 71 ₃ state variable type filter

Continued on front page F-term (reference) 5B050 AA08 BA08 BA10 BA12 DA07 EA04 EA13 EA18 EA19 EA24 FA02 FA10 5B057 AA20 BA02 CD11 CE08 DA16 DA17 DB02 DC01 DC32 DC36 5C023 AA01 AA11 AA21 AA34 AA38 BA01 DA08 CA01 AB

Claims (20)

[Claims] An image interpolation processing method for generating a pseudo image of a subject using a plurality of real image data of the subject, wherein the real image data of the same subject having different imaging times is obtained. An image storage unit that stores a plurality of images, and an image interpolation unit that calculates pseudo image data at specified time information by interpolation based on a plurality of real image data selected from the image storage unit. Interpolation processing device. 2. The image interpolation processing apparatus according to claim 1, wherein the image interpolation unit automatically recognizes a plurality of real image data selected from the image storage unit. An image interpolating apparatus further comprising: a characteristic unit associating unit for associating the characteristic units with each other among the plurality of real image data. 3. The image interpolation processing apparatus according to claim 2, wherein the image interpolator uses the time information as an independent variable for the pixel value in the characteristic part, and uses the pixel value as a dependent variable. And a pseudo-image generation unit that calculates a pixel value at the specified time information using the interpolation curve, and generates and outputs a pseudo image of the object. An image interpolation processing device. 4. The image interpolation processing apparatus according to claim 1, wherein: a substitute image storage unit storing a substitute image corresponding to the time information; and An image synthesizing unit further comprising: an image synthesizing unit that searches for a substitute image corresponding to the designated time information by referring to the substitute image, and applies the substitute image to the pseudo image generated by the pseudo image generating unit. 5. An image interpolation method for generating a pseudo image of a subject using a plurality of real image data of the subject, comprising: (a) realization of the same subject having different imaging times; Recording a plurality of image data; and (b) calculating, by interpolation, pseudo image data at specified time information based on the actual image data recorded in the step (a). Image interpolation processing method. 6. The image interpolation processing method according to claim 5, wherein the step (b) comprises: (b-1) automatically recognizing a characteristic portion of the real image data; and (b-2). Associating the features with each other among a plurality of the real image data. 7. The image interpolation processing method according to claim 6, wherein the step (b) comprises: (b-3) making the time information an independent variable with respect to a pixel value in the characteristic portion; (B-4) calculating a pixel value at the specified time information using the interpolation curve to generate a pseudo image of the object to be captured; And an image interpolation processing method further comprising: 8. The image interpolation processing method according to claim 5, wherein after the step (b),
(C) applying an alternative image corresponding to the time information to the pseudo image data generated in the step (b). 9. A sound interpolation processing apparatus for generating a pseudo sound of a sound source using a plurality of real sound waveform data, wherein the plurality of real sound waveform data acquired from the same sound source in a plurality of time zones separated by time is provided. An actual sound waveform storage unit to be stored, and based on a plurality of actual sound waveform data selected from the actual sound waveform storage unit, an actual sound waveform interpolation unit that calculates pseudo sound waveform data at specified time information by interpolation, Sound interpolation processing device. 10. The sound interpolation processing apparatus according to claim 9, wherein the actual sound waveform interpolation unit weights and averages the actual sound waveform data selected from the actual sound waveform storage unit, and generates a pseudo sound wave in the designated time information. A sound interpolation processing device that calculates shape data. 11. The sound interpolation processing apparatus according to claim 9, wherein the actual sound waveform interpolator uses time information of the actual sound waveform data selected from the actual sound waveform storage unit as an independent variable, and obtains the strength of the actual sound waveform. A sound interpolation processing apparatus further comprising: a pseudo sound generation unit that calculates an interpolation curve by performing interpolation using the interpolation variable as a dependent variable, and calculates pseudo sound waveform data at the specified time information using the interpolation curve. 12. The sound interpolation processing apparatus according to claim 9, wherein the actual sound waveform interpolation unit calculates a frequency characteristic distribution of the actual sound waveform data selected from the actual sound waveform storage unit. A feature component extraction unit that extracts a feature component that is a frequency emphasis component from the frequency characteristic distribution, and a feature component interpolation unit that interpolates a plurality of the feature components to calculate a pseudo feature component in the designated time information. A feature component removing unit that generates and outputs real sound waveform data excluding the feature component; and adding the pseudo feature component to the real sound waveform data output from the feature component removing unit to generate the pseudo sound waveform data. And a feature component adding means for outputting. 13. The sound interpolation processing device according to claim 12, further comprising: a correction data storage unit that stores correction data for correcting the characteristic component, wherein the characteristic component interpolation unit stores the correction rule. , A correction data corresponding to the sound quality of the sound source, and a sound interpolation processing device for correcting the characteristic component based on the correction data. 14. A sound interpolation processing apparatus according to claim 9, wherein the pseudo-image is generated by the image interpolation processing apparatus according to any one of claims 1 to 4. A sound interpolation processing device having a function of synchronizing the pseudo-sound waveform data. 15. A sound interpolation method for generating a pseudo sound of a sound source using a plurality of real sound waveform data obtained from a sound source, wherein (d) a plurality of real sounds obtained from the same sound source in a plurality of time zones. Recording waveform data; (e)
Calculating the pseudo sound waveform data at the designated time information by interpolation based on the actual sound wave data recorded in the step (d). 16. The sound interpolation processing method according to claim 15, wherein, in the step (e), the actual sound waveform data recorded in the step (d) is weighted and averaged.
A sound interpolation method for calculating pseudo sound waveform data at the designated time information by interpolation. 17. The sound interpolation processing method according to claim 15, wherein in the step (e), (e-1) the time information of the actual sound waveform data is set as an independent variable, and the intensity of the actual sound waveform is determined. Performing interpolation using the dependent variable to calculate an interpolation curve; and (e-2) calculating pseudo sound waveform data at the designated time information using the interpolation curve. Interpolation processing method. 18. The sound interpolation method according to claim 15, wherein said step (e) comprises: (e-3) calculating a frequency characteristic distribution of said actual sound waveform data;
-4) extracting a feature component that is a frequency emphasis component from the frequency characteristic distribution, and (e-5) calculating a pseudo feature component in the designated time information by interpolating the plurality of feature components. (E-6) generating and outputting actual sound waveform data excluding the characteristic component; and (e-7) adding the actual sound waveform data output in step (e-6) to the step (e- 5) calculating the pseudo sound waveform data by adding the pseudo feature component calculated in 5). 19. The sound interpolation method according to claim 18, wherein (f) said step (e-4) or (e-
And 7) correcting the characteristic component according to correction data corresponding to the sound quality of the sound source. 20. The sound interpolation processing method according to claim 15, wherein the sound interpolation processing method comprises the steps of:
A sound interpolation processing method for synchronizing the pseudo sound waveform data with a pseudo image generated by the image interpolation processing method described in the section.