JPH0888853A - Medium processing system - Google Patents

Abstract

PURPOSE: To efficiently acquire the essence of information by respectively limiting the information to be an object, acquiring the information and integrating the information based on the attributes of the acquired information. CONSTITUTION: One or plural limitation input devices 3a-3n are provided for an optional information source 1 as an information object and limited information I1 is acquired by the limitation input device 3. The acquired information I3 is appropriately processed in a processor 5, inputted to an output device 7 as processing information I5 and outputted from the output device 7 as video images and sound further. The limitation input device 3 is constituted of plural cameras and they are, for instance, a fixed camera, the camera moving in a horizontal direction, the camera moving in a vertical direction, the camera rotating around an axis connecting the camera and the object and the camera performing repetitive movement within a certain fixed range, etc. Also, the processor 5 is provided with a movement amount setting circuit and an encoder.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a media processing system in an information processing system for carrying out processing such as communication, broadcasting or storage of information including image information.

[0002]

2. Description of the Related Art Conventionally, in order to efficiently compress and code moving image data having a large amount of information, a method of efficiently reducing redundancy by utilizing correlation in the time axis direction or the space axis direction has been used. Has been. Of these, "motion compensation" is used as a method for reducing redundancy in the time axis direction.
(Hereinafter referred to as MC: Motion Compensation).
The MC measures the amount of movement of the moving object (the amount of movement of the object from the previous frame to the position of the current frame) between the current frame and the previous frame, and determines its direction and size. This is a method of generating a vector (motion vector) to be represented and then encoding and transmitting or storing the motion vector. On the reproducing side, based on this motion vector information, the moving object of the previous frame is moved by that amount to fit the moving object at the position of the current frame to create the current frame.

FIG. 6 shows an example of a conventional motion information acquisition device. There is one camera, and the information from this camera 181 is taken into the processing device 182. In the processing device 182, the motion information is detected using the conventional MC method based on the information of the previous frame and the currently captured current frame, and the result is transmitted to the decoding side.

However, since the magnitude of the movement of the moving body or the moving area is not clear in advance, conventionally, wide-range movement detection is performed assuming a range capable of covering all the movement amounts. Therefore, the amount of calculation increases, and it is necessary to assign codes to the motion vector information indicating all the detection ranges, so that the code amount of the motion vector also increases. Further, when there are moving objects or moving parts having different movements within one screen, it is necessary to code all the motion information, resulting in a further increase in the code amount.

Further, in the conventional media processing system,
As a method of acquiring image information as shown in FIG. 12, an image from one viewpoint, which is usually obtained from one camera as the input device 271, has been acquired as image information. This method has been widely used since it is the simplest and natural means for obtaining an image in which the condition of a subject and its background is captured from an objective viewpoint from the camera side.

However, as another important image information, there is an image of the subjective viewpoint from the user side of the camera. Normally, the camera user takes a picture of the subject by looking at the camera's monitor, so it is difficult to distinguish between the objective viewpoint image and the subjective viewpoint image, but the camera user does not recognize himself as the subject. In this case, the image of the objective viewpoint is the image of the user himself / herself taken by the camera, and the image of the subjective viewpoint is the environmental image seen from the user side. In future media processing systems, it is considered that information will be transmitted from individuals in many cases, so there is a demand for camera users to convey their own images (objective viewpoint images) to the viewers and It is thought that both the demands for sharing emotions by transmitting images viewed from the viewpoint (images from the subjective viewpoint) will be strengthened.
In the conventional media processing system, such a future 2
There is a problem that one requirement cannot be satisfied at the same time.

Furthermore, there is a problem that the conventional system lacks a method of displaying both a user image and an image from the user's viewpoint as one image in a natural form. is there.

On the other hand, depending on the situation of the user who sends the information, it is desired to keep his / her own environmental image information concealed, but a monotonous environmental image is not preferable, and it is desired to change it in a visually effective form from the sending side. It is expected that the demand will be stronger in the future. One of the conventionally used methods for concealing environmental images is to send no environmental image. However, this method has a problem in that the image receiving side displays an image having no change in the background, so that the system can provide only the image information that is not realistic to the user.

Further, conventionally, when a landscape or the like is imaged by a camera, data of pixels spatially arranged at equal intervals and at equal resolution is output as image data of the landscape. And
In general, due to the fine processing technology of CCD, the amount of calculation, and the amount of data, sufficient resolution cannot be obtained for the entire screen.

Therefore, studies are underway to obtain one high-resolution image by using a plurality of low-resolution images obtained by a low-resolution camera. (For example, "Study on high resolution image acquisition from multiple low resolution images" Miyaji, Hanamura, Tominaga,
1993 IEICE Spring Conference D-355. ) Low-resolution images may be taken with a single camera while being temporally shifted, or may be taken with a plurality of cameras at the same time. Here, FIG. 23 shows a block diagram when a plurality of cameras are used. First, the entire landscape is simultaneously photographed using the first camera 371 and the second camera 372. The two cameras are low-resolution cameras, but one sampling point Sa is the other sampling point Sb as shown in FIG.
Adjust the direction of the camera so that it is in between and shoot. The data synthesizer 375 then outputs the two low resolution image signals 3
By synthesizing 73 and 374 as shown in FIG. 24, a high resolution image signal 376 is synthesized and output.

However, in this method, even if two cameras are used, the pixel density of each portion of the entire image is uniformly doubled, and further resolution cannot be improved unless the number of cameras is increased. In other words, if you want to see a certain important part of the screen in great detail, you need many cameras.

Further, in the conventional videophone, even if the position, orientation, shape, etc. of the subject change from moment to moment, the camera state such as the camera position, orientation, and reception sensitivity is fixed and can be left alone. If you leave it, the subject will stick out of the screen or you will shoot the subject backwards. As for voice input, if the user moves and moves out of the area where the microphone has high directivity, good voice input cannot be performed. Further, when presenting an image to the user on the display, it may be difficult to see the display when the user moves.

At present, research and development of systems for transmitting / storing media information such as images / sounds are being actively conducted. In the conventional media transmission / storage device, each media has a single input means, or
A method is used in which one of the information input from a plurality of input means is selected to form one information, and this information is used as one information for processing such as compression encoding and transmission / accumulation.

FIG. 31 shows a block diagram of a conventional device for inputting an image / sound, compressing the image / sound, and transmitting / storing it. Further, FIG. 32 shows a block diagram of a conventional device corresponding to the device of FIG. 31, which reproduces an image / sound from transmitted / stored information.

In the input and transmission / accumulation device shown in FIG. 31, images and sounds to be transmitted / accumulated are inputted from one or more cameras 501a and microphones 501b, respectively. When there are a plurality of cameras / microphones, the image selector / synthesizer 502a and the sound selector / synthesizer 50 respectively.
In 2b, one image and sound are selected / synthesized. The processing in the image selecting / synthesizing unit 502a and the sound selecting / synthesizing unit 502b is processing for selecting one of the images and sounds from a plurality of input means, or processing such as filtering for each and adding. The output image and the sound are both composed of one image and sound. If there is only one input means, selection / combination is not performed, and the input image and sound become the image and sound as they are.
This image and sound are encoded by encoders 503a and 503, respectively.
In b, the data is compressed and encoded to a small amount of information. The compression-encoded coded sequence is multiplexed by a multiplexer 504 and transmitted / stored.

On the other hand, in the reproducing apparatus shown in FIG.
The coded sequence transmitted / stored from the storage device is separated in the demultiplexer 506 into an image coded sequence and an audio coded sequence, and the image decoder 507a and the sound decoder 50 are respectively separated.
In 7b, the reproduced image and reproduced sound are reproduced.

In the apparatus of FIG. 31, the image encoder 50
The encoding method in 3a is, for example, ISO / M.
Image coding methods such as PEG1 and MPEG2 and ITU-T
H. 261, H.H. A method combining motion compensation and discrete cosine transform such as 262, a method combining motion compensation with subband coding, pyramid coding, or the like, or a model paste mainly for coding a face image Encoding or the like is used. Among these coding methods, the method that combines motion compensation and discrete cosine transform, subband, pyramid, etc. can obtain a certain coding efficiency for any image input, but especially for humans. As far as the face part is limited, the model paste coding method specialized for it is inferior in terms of coding efficiency.

On the other hand, in the model paste coding method, the coding efficiency is good for a part such as a person's face that matches the model assumed in the coding method in the image, but other background parts etc. The coding efficiency is extremely low. For this reason, the face part and the other part are separated, the face part is coded by the model paste coding, and the other part is coded by motion compensation + discrete cosine transform.

Therefore, the total coding efficiency does not necessarily become high when a lot of information is included in the parts other than the face. In addition, it is generally very difficult to accurately separate the face and other parts from the input image, and it is often the case that part of the background is included in the part cut out as a face, and vice versa. Was leading to a decline in.

As a coding method in the sound coder 503b, for example, ISO MPEG1, MPEG
The audio encoding method of No. 2 and the encoding method such as CELP are used. These conventional coding methods are mainly intended to obtain good coding quality for a specific field such as music or human voice, and the coding efficiency is not necessarily required for sounds outside the expected field. Not good.

[0021] For example, the CELP system is mainly intended to code speech with high efficiency, and good coding quality cannot be obtained for sounds other than speech, such as music and background sounds. Therefore, in a conventional device that synthesizes a single sound and then encodes it like the media input device shown in FIG. 31, the encoded input contains various sounds such as a human voice and a background sound in a mixed manner. In other words, although the sound (speech) part that conforms to the coding method used in the coding device 503b can be coded with high efficiency, the coding efficiency of other sounds is not good. There was a point.

Further, in the media reproducing apparatus shown in FIG. 32, both the image decoder 507a and the sound decoder 507b have the same encoding method as that used in the image encoder 503a and the sound encoder 503b in FIG. 31, respectively. Correspondingly, decoding is performed. The image and sound reproduced in this way are reproduced as one image and sound, respectively. Therefore, the reproduced image and sound are separated into individual elements (for example, human face and background, human voice and background sound), and some components are replaced with others ( For example, you can change the background to another image,
In order to change the voice to the voice of another person), it is necessary to perform separation processing on the reproduced image and sound.

It is difficult to accurately perform such separation processing from an image / sound in which all the elements are combined into one image. Furthermore, the reproduced image / sound that has been compression-encoded at the time of transmission / accumulation contains encoding distortion, and the separation accuracy is further reduced as compared with the case where the input image / sound is directly separated.

As described above, the media input transmission / accumulation device according to the prior art has only a single input means for each medium, or selects / combines from a plurality of input means to select each medium. Each had one piece of information created. The created information is a mixture of information composed of multiple components. For this reason, when a compression coding method specialized for a certain component being input is used, there arises a problem that the component has high coding efficiency but the other components have low coding efficiency.

In order to avoid this, it is necessary to decompose into individual constituent elements and use a compression coding method suitable for each of them, but it is difficult to perform decomposition into elements with high precision,
As a whole, high coding efficiency could not be obtained. In addition, it is necessary to devote a large amount of information to a portion such as a background portion where it is not necessary to faithfully transmit / store the input information as it is, and it is necessary to allocate a large amount of information. I couldn't get it.

Further, in the conventional media reproducing apparatus, there is a problem that it is difficult to perform processing such as disassembling the reproduced media into constituent elements or partially replacing them.

Further, conventionally, when transmitting or accumulating moving image data having a large amount of information, a method of utilizing the correlation in the time axis direction or the spatial axis direction to effectively reduce the redundancy is used. . Especially when the target image is specified to some extent like a face image, a common model (wireframe model) is used by the encoder and the decoder, and eye and mouth movements are extracted from one input image. There is a method of transmitting the change amount.

This conventional method will be described with reference to the block diagram shown in FIG. First, a face image as an information source is captured via the camera 631. Using the face image obtained as an input, the change amount detecting means 632 detects a part in which motion has occurred and how much motion has occurred in each part while comparing it with a three-dimensional wire frame model, and uses them as motion information. Output. The natural image synthesizing unit 633 decodes the motion part and the motion amount using the motion information, changes the wire frame model according to the information, and creates a natural image so that each area of the wire frame model looks natural. The corresponding portion is mapped to generate a composite image, which is displayed on the display device 634.

In the case of this method, since a common wire frame model is used, a moving image can be realized with a small amount of information, but since a natural image is transformed to generate a decoded image, it is inevitably unnatural. There is a problem that occurs.

Further, in the conventional information processing mainly using a face image of a videophone or the like, in almost all cases, only the image taken by the camera located in front of the subject is processed. Although this method has the advantage that the device can be simplified, it causes distortion.

Further, as a method for detecting the movement of a face, generally, an image is subdivided into blocks, and blocks having a small error are searched from images having a temporally front-and-back relationship, and the blocks are searched before and after the movement. The block matching method is adopted in which each of the blocks is regarded as a block, and a motion vector that is a locus of movement and an error between blocks are obtained.

This state will be described with reference to FIG. First, it is assumed that there is a face image shown in FIG. It is assumed that, after a predetermined time has elapsed, the entire face is moved in parallel in the horizontal direction and the image is changed to an image with the mouth widened, as shown in FIG. When attention is paid to blocks 675 and 676 in the face image shown in FIG. 50B, FIG.
The blocks shown in are the blocks 67 in the face image.
Since it corresponds to 1 and 679, respectively, FIG.
As shown in, the motion vectors V _a and V _b are respectively obtained.

By the way, most of the face portion is the motion vector V.
_Since it is moving in the direction of _a , it is better to represent the motion vector in a format in which the motion of the entire face is V _a and the local motion of each part is added to it, for example, when a face image is encoded. It is convenient in terms of coding efficiency because the variance of the size can be suppressed. Also, in the case of encoding using a face model or application to a man-machine interface, it is more natural and rational to express the face movement separately for the entire face and local movements of each part.

However, it is difficult to obtain the motion vector having the above-mentioned form by block matching. One of the reasons is that only one image can be obtained by one sampling. Therefore, 1
It is difficult to understand in detail from the movement of the entire face to the local movement only with the image information of one sheet, and therefore it is necessary to obtain a plurality of multifaceted image information.

[0035]

As described above, conventionally, since the MC method that covers the expected motion well is required, the amount of calculation required for motion detection increases, and
There is a problem that the code amount of the motion vector information itself increases. Further, when there are moving objects or moving parts with different movements within one screen, it is necessary to code all the motion information, which further increases the code amount.

Further, in the conventional media processing system,
It is not possible to display the user image and the environmental image viewed from the user side on one screen at the same time, and to make it a realistic image in a natural form. Also, the environmental image of the user is hidden due to the concealment of the environmental image. There is a problem in that the user cannot change it in a visually effective form.

According to the present invention, the user conveys the image of himself (the image of the objective viewpoint) to the viewer of the image, and at the same time, conveys the image seen from his own viewpoint (the image of the subjective viewpoint) to share the impression. It is an object of the present invention to provide a media processing system that can be visualized and can display the surrounding situation including oneself as one image with a natural and realistic feeling by using both images and at the same time for information transmission. An object of the present invention is to provide a media processing system capable of visually changing the environmental image of the user while concealing the environmental image of the user who is the sender when used.

Further, in the conventional media processing system, there is a problem that the image data can be obtained only at a low resolution although it is an important part, and the subject cannot be imaged in an optimum state. .

Further, in the conventional media processing system,
The number of media input means is one for each medium, or the processing is performed after selecting / combining from the input from a plurality of input means to make one. For this reason,
There is a problem that it is difficult to decompose the input information into constituent elements and perform compression encoding suitable for each. Further, in the media reproducing apparatus using the conventional technology, since the information reproduced is one for each medium,
There is a problem that it is difficult to disassemble each component or replace it partially.

Further, in the conventional media processing system,
Although there is a method of transmitting information using a common model in the encoder / decoder in order to improve the encoding efficiency, a synthetic image is generated from the deformation / processing of a natural image, so it is inevitable There was a problem that unnaturalness would occur.

The present invention has been made in view of the above circumstances. Based on a high-definition artificial image, the region of the artificial image corresponding to the region in which the motion is detected is updated temporally to obtain a natural image. The object is to generate a moving composite image and to generate a composite image using a line drawing instead of an artificial image.

Further, in the conventional media processing system,
By means of a plurality of means, particularly for a media processing device such as a man-machine interface, etc., which individually and accurately obtains information regarding the entire face and movements of each part, and performs efficient encoding and accurate processing, It is an object to provide an input means and a processing method thereof.

The present invention has been made in view of the above problems, and is a medium capable of capturing information in a multidimensional manner and efficiently obtaining the essence of information by a process suitable for the characteristics of the information. The purpose is to provide a processing system.

[0044]

In order to achieve the above object, the first invention of the present application is to provide a plurality of input means for respectively limiting target information to acquire information, and attributes of the acquired acquisition information. The gist of the present invention is to have a processing means for integrating the acquired information based on the above and an output means for outputting the processing information obtained by the integration by the processing means.

In the second invention of the present application, a plurality of input means for respectively limiting target information to acquire information, and the acquired information excluding less important information from the acquired information are acquired. The gist of the present invention is to have a processing means for integrating the above and an output means for outputting the processing information obtained by the integration by this processing means.

The third invention of the present application is a plurality of image information input means for respectively acquiring image information related to the information object, and a change with time of the information object included in the acquired image information. The gist of the present invention is to have a detection means for detecting the above and an output means for outputting the temporal change information obtained by this detection means.

According to the present invention, a plurality of cameras are provided, and different motions are given to each of the cameras so that the motion of the moving object that best represents the motion of each moving object or moving object in the screen to be encoded. A method for adaptively using information is realized. That is, the motion vector of each moving object or moving part is expressed using the information from the camera that is moving so that the amount of generated information of the motion vector by the MC is minimized.

Furthermore, an actual "moving" object or "moving"
By operating the camera by operating it from the outside according to the movement of the part, for example, looking into the camera, a person moves the camera in accordance with the movement of the target object or part Allows generation of corresponding motion vectors. When there are a plurality of movements, a plurality of cameras can be assigned to different movements one by one and operated.

Further, by controlling the movement of the camera based on the movement of each moving body or moving portion actually measured, that is, the camera is moved in a direction in which the actually encoded movement information is canceled. It is also possible to reduce the amount of generated information of the motion vector by automatically moving.

Further, in the fourth invention of the present application, one or a plurality of first information inputting means for respectively obtaining information including image information related to the information object, and the first information inputting means are operated. Second information input means for obtaining information including image information for the information subject, first information obtained by the first information input means, and second information obtained by the second information input means The gist is to have a synthesizing means for synthesizing the information of 1.

Desirably, in a media processing device having a plurality of input means for limiting the information object to obtain different limited information, means for processing the obtained information, and means for outputting the processing information, A first input means provided on the front surface of the device for obtaining the image information of the user, and a second fixed or movable device provided in a different direction from the front surface of the device.
And a means for displaying the environment image obtained from the second input means on all or part of the background of the user image.

Further, it is used in a media processing system having a plurality of input means for limiting different information objects to obtain different limited information, means for processing the obtained information, and means for outputting the processing information. Person's image information, coded environmental sound information or voice information, means for obtaining environmental image information from the code, and environmental image information on all or part of the background of the user image. It is preferable to have means for displaying an environmental image obtained by using it.

The present invention is a media processing system having a plurality of input means for limiting different information objects to obtain different limited information, means for processing the obtained information, and means for outputting the processing information. A first input means for obtaining the image information of the user, a second input means for obtaining the environment image seen from the user side, and a second input means for the whole or part of the background of the user image. And a means for displaying an environmental image obtained from the above.

Furthermore, the gist of this media processing system is to have means for setting the size or range in which the user image occupies the display screen. Further, the gist of this media processing system is that it has means for displaying only the user image by inverting the mirror image when displaying the environmental image obtained from the second input means. Furthermore, the gist of this media processing system is that it has means for displaying only the image from the first input means or the image from the second input means. Further, the media processing device of the present invention has a plurality of input means for limiting the information object to obtain different limited information, a means for processing the obtained information, and a means for outputting the processing information. The device includes a first input means provided on the front surface of the device to obtain image information of the user, and a fixed or movable second input means provided in a different direction from the front surface of the device. The gist is to have means for displaying an environmental image obtained from the second input means on all or part of the background of the image.

Further, the gist of this media processing device is to have means for setting the size or range of the user image on the display screen. Further, the gist of this media processing device is to have means for displaying only the user image by mirror-inversion when displaying the environmental image obtained from the second input means. Furthermore, the gist of this media processing device is that it has means for displaying only the image from the first input means or the image from the second input means.

Further, the media processing system of the present invention comprises a plurality of input means for limiting the information object to obtain different limited information, a means for processing the obtained information, and a means for outputting the processing information. In the media processing system, a means for obtaining the image information of the user, a means for obtaining encoded voice or environmental sound information, a means for obtaining the environmental image information from the code, and all or one of the backgrounds of the user image. The gist is to have means for displaying an environmental image obtained by using the environmental image information in the section.

As means for obtaining coded sound information and means for obtaining environmental image information from coding, there is provided means for changing the environmental image based on the code, using the pitch frequency extracted from the sound as code information. And The gist of the present invention is to have means for changing the environmental image based on the LPC coefficient extracted from the sound, as means for obtaining the coded sound information and means for obtaining the environmental image information from the coding.

In the fifth invention of the present application, one or a plurality of first image information inputting means for individually obtaining image information related to the information object, and the first image information inputting means. Between the second image information input means for obtaining image information relating to an arbitrary area included in the first image information and the first pixel value data obtained by the first image information input means, And a means process for inserting the second pixel value data acquired by the second image information input means.

In the present invention, a media processing system having a plurality of input means for limiting the information object to obtain different limited information, means for processing the obtained information, and means for outputting the processing information. In at least two image pickup means as input means, the image pickup range of the first image pickup means includes the image pickup range of the second image pickup means. The gist is that one image data and the second image data from the second image pickup means are combined and processed as one integrated image data.

Further, in a media processing system having a plurality of input means for limiting different information objects to obtain different limited information, means for processing the obtained information, and means for outputting the processing information, At least one image pickup means as means, a correction signal generation means for generating an input state correction signal in accordance with image data from the image pickup means, and a means for correcting the state of the image pickup means or other input means by the input state correction signal. Having it is the gist.

The correction signal generating means detects a face area from the image data and corrects the orientation or magnification of the image pickup means so that the face area has a predetermined position or size on the screen. The point is to generate a signal. Further, the input means has a voice input means, the correction signal generation means detects a face area from the image data, and the directivity of the voice input means is increased in a portion corresponding to the mouth of the face area. The gist is to generate a correction signal for correction as described above. Also, having a light emitting means,
In the processing means or the correction signal generating means,
The gist of the present invention is to detect reflected light of the light emitted from the light emitting means from the image data and detect a predetermined area based on the detection result.

Further, it has an image display means, and the correction signal generation means detects a face area from the image data,
It is characterized in that a correction signal for correcting the orientation of the image display means is generated in the direction of the eyes of the face area.
Further, the image pickup means is separated from the media processing system, and has a support means for supporting the image pickup means and a horizontal holding means for a screen input from the image pickup means.

Further, the sixth invention of the present application is: first input means for acquiring information relating to an information object; candidate accumulating means for accumulating candidates for replacing a part of the information object; and information object input Input condition acquisition means for acquiring the input condition in, and second input means for acquiring the information related to the information object,
Based on the input condition acquired by the input condition acquisition unit and the input information from the first input unit, it becomes a source of replacement from at least information from one of the candidate accumulating unit and the second input unit. The gist is to have a replacement unit that selects information and replaces a part of the information target input with replacement information.

In order to solve the above-mentioned problems, in the present invention,
What is claimed is: 1. A media input device comprising: a plurality of input means for limiting different information objects to obtain different limited information; a means for processing the obtained information; and a means for outputting processing information. The input condition includes means for accumulating candidates for replacing a part, means for obtaining an input condition for inputting an information object, and means for obtaining information input from another input means different from the input means. And the information from the candidate accumulating means and / or the information from the other input means based on the input information from the input means, and based on the input condition and the input information. The gist is to modify the selected replacement information and replace a part of the information object input with the selected modified replacement information.

Further, means for reproducing the replaced information based on the information object input condition and the information object replacement information from the media input device, and the candidates of information to be replaced are accumulated. It is characterized in that it has means, means for reproducing the information input by the media input means based on the information target input information, and means for synthesizing the reproduced replacement information and the reproduced information.

Further, it comprises at least one or more of the media input device, the media reproducing device, and means for providing information to be a source of the replacement, and information to be a source of replacement in the media input means. Is provided from the stored information in the input device and / or the replacement information providing means and / or other media input means,
It is a gist that the information that is the basis for reproducing the replacement information in the media reproducing device is provided from the accumulated information in the reproducing device and / or the replacement information providing means and / or another media input means.

In the seventh invention of the present application, a plurality of image information input means for respectively acquiring image information of the information object, an acoustic information input means for acquiring acoustic information of the information object, and the plurality of image information input means. A state acquisition means for acquiring the state of each information object from the acquired information obtained by the image information input means and the acoustic information input means, and an artificial image corresponding to the state of the information object acquired by this state acquisition means. The gist is to have a correction means for correcting each part.

The present invention comprises at least two or more cameras, said camera capturing a part of an information source to obtain desired limited information, means for obtaining each state from each obtained information, and artificial The gist is to have means for moving each part of the image according to the above-mentioned state. Further, the present invention includes at least two or more cameras and one acoustic microphone, the camera obtains desired limited information by photographing a part of the information source, and obtains each state from each obtained information. And a means for obtaining a voice state from voice information obtained from the acoustic microphone, a means for correcting the state using the voice state, and a means for moving each part of the artificial image in accordance with the state. To do.

Further, according to the present invention, in the state correcting means, the state of the acquired information from the acoustic microphone is output when the background noise level is low among the state of the acquired information from the camera and the state of the audio information, and the background noise level is output. The point is to output the state of the acquired information from the camera when is high. Further, the present invention comprises at least two or more cameras, said camera capturing a part of an information source to obtain desired limited information, means for obtaining each line drawing from each obtained information, and the line drawing. The gist is to have means for integrating and displaying.

Furthermore, the present invention comprises means for at least two or more cameras and one acoustic microphone, said camera taking a part of an information source to obtain desired limited information.
And a means for obtaining each line drawing from each acquired information, a means for obtaining a voice state from voice information obtained from the acoustic microphone, a means for correcting the line drawing using the voice state, and an integrated display of the line drawings. The gist is to have means for doing so. Further, according to the present invention, the line drawing correction means outputs a line drawing of the acquired information from the acoustic microphone when the background noise level is low among the states of the information acquired from the camera and the state of the audio information, and when the background noise level is high. The point is to output a line drawing of the information acquired from the camera.

Further, the eighth invention of the present application is a first image input means for acquiring image information of an information object, and the first image input means.
Second image input means for acquiring image information of a part or the whole of the image area acquired by the image input means of
Extracting means for extracting a temporal change of at least one of the image information acquired by the image input means and the second image input means, and the first image according to the temporal change extracted by the extracting means. The replacement means for replacing a part of the image information acquired by the input means with the image information acquired by the second image input means, and the replacement of the image information by this replacement means are acquired by the first image input means. The gist of the present invention is to have a control means for controlling a part of the information object included in the image information thus generated, which corresponds to the overall motion of the information object.

The present invention is a device for inputting an image of the entire face including a background, detecting and analyzing the movement of the entire face and each local movement in the face such as eyes and mouth, and encoding the information. Alternatively, it is used in a media processing device such as a man-machine interface for analyzing the behavior of a person to be photographed, and has a plurality of image input means as the input means, and the image input means includes one main image input means and the one image input means. There is another input means subordinate to the main image input means, the main image input means inputs the image of the entire face, and the subordinate input means inputs the main input such as around the eyes and mouth and the background. The means for inputting a part or all of the image obtained by the means, and the means for performing the processing are extraction of a temporal change of the face image and a part of the whole face image obtained by the main input means, that is, the eye. Place the surrounding area of the mouth and mouth with the image obtained by the subordinate input means. The means for performing the processing also extracts the overall movement of the face, and the subordinate input means can follow the movement of the eyes and mouth around the movement of the entire face. The gist is to have a means for controlling the input means for performing.

Further, the gist of the present invention is to use an input device to be worn on the head as the input means, and to detect the position of the entire face, the motion, or the image around the eyes and mouth. Also, having a device for obtaining an image of light having a wavelength in the infrared region as an input means, extracting the face region from the image of the entire face including the background, or the position of the pupil or oral cavity, the nasal cavity from the image of the entire face. The point is to detect.

[0074]

In the media processing system according to the first aspect of the present invention, when the plurality of input means acquire the information by limiting the target information, the processing means acquires the information based on the attribute of the acquired acquisition information. Information is integrated and output from the output means.

In the media processing system according to the second aspect of the present invention, when the plurality of input means each obtain limited information by limiting the target information, the less important information is removed from the obtained obtained information. The acquired information is integrated by the processing means and output from the output means.

In the media processing system of the third invention of the present application, the image information relating to the information object is limitedly acquired by the plurality of image information inputting means, and the time-dependent change of the information object included in the acquired image information. Is detected by the detection means,
This time-dependent change information is output.

According to the present invention, it is possible to adaptively use the information from the camera of the motion that best expresses the motion of each moving object or moving part in the screen to be encoded. It is not necessary to represent all actual motions as motion vectors, and it is possible to reduce the amount of information generated by MCs about motion vectors. Furthermore, since the motion amount detection range is automatically expanded by the camera movement, it is possible to limit the detection range when calculating the actual motion vector, and the code assigned to the motion vector is also reduced. It is also possible to reduce the amount of vector information. It is also possible to further reduce the amount of information generated in the motion vector by moving the camera or automatically controlling the motion based on the actual motion of each moving object or moving part obtained. Become.

In the media processing system according to the fourth aspect of the present invention, each piece of information including image information relating to the information target is acquired by the one or more first information input means. Information including image information is acquired by operating the first information input means, for example, the operator as the information target by the second information input means. Furthermore, the first information obtained by the first information input means and the second information obtained by the second information input means are combined by the combining means.

In the media processing system according to the present invention, when the user of the system transmits information, the image of the user (the image of the objective viewpoint) and the environmental image (the image of the subjective viewpoint) viewed by the user are naturally displayed. In order to obtain one realistic image, the first input means obtains the image information of the user, and the second input means obtains the environmental image viewed from the user side.

Next, by displaying the environment image obtained from the second input means on the background portion of the user image, the user image and this environment image can be naturally combined.

Furthermore, by providing means for setting the size or range of the user image occupying the display screen, it becomes possible to freely use which of the user image and the environmental image is desired to be displayed. .

Further, when displaying the environment image obtained from the second input means, by providing means for displaying only the user image by mirror-inversion, the direction of the object pointed to by the user looking at the actual environment can be determined. The direction of the object pointed by the image of the user can be matched in the displayed screen.

Further, the media processing device of the present invention is provided with the first device provided for obtaining the image information of the user on the front surface of the device.
Since the input means and the second input means, which is fixed or movable and is provided in a different direction from the front surface of the apparatus, are provided, the user can use the front surface of the apparatus with his or her front side facing toward the user. However, the environmental image viewed from the user side can be acquired without any interference.

Further, the media processing system of the present invention obtains the encoded environmental sound information or voice information,
By converting this code into image information by a preset method and displaying it as the environment image on the background of the user image, the environment image can be changed at a timing that matches the change in the sound from the user side. it can. Further, when the pitch frequency extracted from the voice or the environmental sound is used as the code information, it is possible to bring about a change that the environmental image information is made periodic based on the pitch frequency. As another code information, when the background image is filtered based on the extracted LPC coefficient information and the image quality is changed, it is possible to give the environmental image a change in image quality corresponding to the change obtained from the sound. Therefore, it is possible to provide a media processing system that displays a concealed background image that conveys the atmosphere of changes in the sound on the sender side.

In the media processing system according to the fifth aspect of the present invention, one or a plurality of first image information input means each obtain limited image information of an information object. This first
The second image information input means acquires image information relating to an arbitrary area included in the first image information acquired by the image information input means. Further, the second pixel value data acquired by the second image information input unit is inserted between the first pixel value data acquired by the first image information input unit.

In such a configuration, when the first image pickup means picks up the entire screen and the second image pickup means picks up an important part, the processing means performs the other important part. It is possible to process the image data as integrated image data having a higher resolution than the portion. Further, in the correction signal generation means, an input state correction signal is generated by the image data from the image pickup means, and the input state correction signal is used to correct the state of the image pickup means or other input means to an optimum state every moment. Is possible. In the correction signal generation means, a face area is detected from the image data, and a correction signal for correcting the orientation or magnification of the image pickup means is generated so that the face area has a predetermined position or size on the screen. It is possible to image the face without protruding from the imaging range. Further, in the correction signal generating means, a face area is detected from the image data, and a correction signal for correcting the directivity of the voice input means is generated in a portion corresponding to the mouth of the face area to generate a voice. It becomes possible to input well. The processing means or the correction signal generating means detects the reflected light of the light emitted from the light emitting means from the image data, and detects the position of the glasses, the position of the eyes, etc. based on the detection result. It becomes possible to do.

Further, the correction signal generating means detects a face area from the image data and generates a correction signal for correcting the orientation of the image display means in the direction of the eyes of the face area, whereby the user is moved. Also, the orientation of the image display means is always kept in a direction that is easy for the user to see. Further, the support means for supporting the image pickup means and the image pickup means may be required to be tilted and supported depending on the external situation. Even in that case, the screen horizontal holding means keeps the screen horizontal.

In the media processing system of the sixth invention of the present application, the information related to the information object is acquired by the first input means and the second input means. Further, candidates for replacing a part of the information target are stored in the candidate storage means. on the other hand,
Information serving as a source of replacement from at least information from one of the candidate accumulating means and the second input means based on the input condition acquired by the input condition acquiring means and the input information from the first input means. To replace part of the information target input with replacement information.

In the media input device according to the present invention, in the media such as the input image / sound, if the input conditions are known, a replaceable part is searched for and separated based on the information from the database access or other input means. Processing is performed. Then, when transmitting / accumulating, instead of transmitting the image / sound or the like as it is to the replaceable portion, the input condition information is transmitted / accumulated, and at the time of reproduction, database access or A method of creating and synthesizing a replacement image / sound based on information from other input means is used.

For example, in a portion of the image such as a human portion, which requires high fidelity, the input image is compressed and coded as it is, and transmitted / stored. The input condition information is transmitted / stored instead of using the image as it is. Therefore, the amount of information for transmission / accumulation of the background portion can be significantly reduced as compared with the conventional transmission / accumulation device. In addition, even for the part where the image / sound is compression-coded as it is, each input means can be used by limiting the input target such as a face and a person, and compared with the case where the whole is treated as one input, it becomes a constituent element. It is easy to separate, and high compression efficiency can be obtained by performing compression encoding suitable for each component. Further, since the information to be transmitted / stored is separated for each component such as a person and a background, it is easy to perform a process of replacing a certain component with another one during reproduction.

In the media processing system of the seventh invention of the present application, the image information of the information object is limited and acquired by the plurality of image information input means, and the acoustic information input means acquires the acoustic information of the information object. The status of each information object is acquired by the status acquisition means from the acquired information acquired by these information input means. The correction means corrects each part of the artificial image in accordance with the state of the information target acquired by the state acquisition means.

As a result, according to the present invention, since the image can be combined from the limited information, the moving image can be expressed with very little information. Moreover, since a moving image is generated based on a high-definition artificial image, it is possible to generate a high-quality moving image without distortion. In addition, since a desired artificial image can be selected from the database, various synthetic images can be obtained for the same information source. Furthermore, since the movement of the composite image is corrected by the audio information, it is possible to obtain a composite image with no discomfort.

In the media processing system of the eighth invention of the present application, the image information of a part or the whole of the image area of the image information related to the information object acquired by the first image input means is the second image input means. Earned by. At least one temporal change of the image information acquired by the first image input means and the second image input means is extracted by the extraction means. The replacement unit replaces a part of the image information acquired by the first image input unit with the image information acquired by the second image input unit according to the change with time extracted by the extraction unit. The replacement of the image information by the replacement means is controlled by the control means so as to correspond to the movement of a part of the information object included in the image information acquired by the first image input means To be done.

In the present invention, the movement of the entire face is detected from the image of the entire face, and the images around the eyes and mouth are separately input while following the movement of the entire face. In the images around the eyes and mouth obtained in this way, the movements of the entire face are canceled by the follow-up operation, so that these local movements can be easily captured. Therefore, it is possible to separate the motion of the entire face from the local motion around the eyes and the mouth accurately, and obtain a clear quality image with less blurring. As a result, for example, the amount of information about the movement of the face can be suppressed when the face image is encoded. Especially when the face of the photographed person is modeled and encoded in the form of a wireframe model, etc., the motion of the entire face and the motion of each local area can be detected and analyzed separately, so that the model can be moved naturally. Become.

Further, since the motion of the person to be photographed can be accurately analyzed, it is possible to further ensure appropriate processing in the man-machine interface. In addition, by using an input device that is worn on the head, the device detects the position and movement of the face, and also obtains images of the area around the eyes and mouth, to detect the position and movement of the entire face, eyes and mouth. It is possible to more accurately obtain information on the local movement of the periphery. Further, it has a device for obtaining an image of light having a wavelength in the infrared region as an input means, and extracts the area of the face, which is a heating element, from the image of the entire face including the background, or does not generate heat from the image of the entire face. It is possible to detect the positions of the pupil, the oral cavity, and the nasal cavity more accurately.

[0096]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An embodiment of the present invention will be described below with reference to the drawings. FIG. 1 is a block diagram showing the configuration of a media processing system according to the present invention. As shown in Figure 1,
For any information source 1 as an information object, one or a plurality of limited input devices 3a to 3n are provided, and the limited input device 3 acquires the limited information I1 and the acquired information I3.
Is input to the processing device 5. This acquisition information I3 is
The information is appropriately processed in the processing device 5, is input to the output device 7 as the processing information I5, and is further output from the output device 7 as an image, a sound or the like.

Next, a first embodiment according to the present invention will be described. First, referring to FIG. 2, the limited input device 11
0 is composed of a plurality of cameras 101, ..., 105, .. These are, for example, a fixed camera 101, a camera 102 that moves in the horizontal direction, and a camera 10 that moves in the vertical direction.
3, a camera 104 that rotates around an axis that connects the camera and the subject, or a camera 105 that repeatedly moves within a certain range. The acquired information fetched from these is sent to the processing device 140, and after being subjected to processing such as compression, it is transmitted to the decoding side via the transmission path. The decoder 150 on the decoding side decodes the information on the basis of the transmitted information, forms reproduction information, and outputs the reproduction information.

On the other hand, the processing device 140 has a motion amount setting circuit 130 and an encoder 120. Movement amount setting circuit 1
A signal for controlling the motion of the limited input device 110 is output from the 30 and at the same time, the motion information is output from the encoder 120.
Transferred to. In this example, these movements can be manually set independently by the external operation board 131, but it is also possible for a human to operate the board by looking into the camera and following the movement of the moving object. Is.
In addition, it does not matter if a person directly looks into the camera and moves the camera.

FIG. 3 is a diagram showing the inside of the encoder 120 in detail, and FIG. 4 is a diagram explaining the motion compensation (hereinafter simply referred to as MC) actually performed by this encoder. The image information from the plurality of cameras 101 to 101 of the limited input device 110 is input to the input information switching circuit 121, and the motion information of each camera is represented as camera movement 1, camera movement 2, camera movement 3, ... Motion compensation circuit 123
Is taken into. In the motion compensation circuit 123, the screen is divided into a plurality of areas, and each area has a MC similar to the conventional one.
At this time, MC is performed between the information (plurality of current frame information) fetched from each camera and the information (previous frame information) in the frame memory 124. Then, the current frame information with the smallest amount of MC information, that is, the camera movement that takes in the current frame information that can be created from the previous frame with the least movement (specifically, the camera movement 1 and the camera movement 2 in FIG. 3). , Camera movement 3, ...) As an initial motion vector, and is output to the motion amount setting circuit 130 as motion amount information. Details of the MC portion will be described with reference to FIG. In the previous frame A in the frame memory, the moving object a is in the position of FIG. 4 (a),
The next frame (current frame to be encoded) B is shown in FIG.
Moving in the direction of arrow b in (a) or (b), FIG.
It is assumed that the user has moved to a. At this time, the motion vector becomes b itself in the figure, and conventionally, this vector was encoded as information. Here, if the camera itself moves following the moving object as indicated by d in FIG. 4C, if the camera originally did not move, the current frame B in FIG. 4C is displayed.
The moving object, which should be in the position c, remains in the left side position in the current frame C after the movement.

At this time, if the positional relationship of a in the frame A is the positional relationship of f in the frame C in FIG. 4C, the movement of the moving object in the frame is as shown in FIG. ) Is indicated by an arrow e. This takes a smaller value as the camera movement e is closer to the actual movement b of the moving object. The final motion vector is represented by (d + e).

The above operation is performed for each "moving" object or "moving" area. That is, since the cameras whose motions are referred to are naturally different for the regions of different motions, the motion of the camera corresponding to the above d is calculated for each region.

Referring again to FIG. 3, the limited input device 1
The signal from 10 is e
Is switched by the input information switching circuit 121 so as to be the smallest. That is, the signals from the limited input device 110 are sequentially fetched into the motion compensation circuit 123, and the motion of the camera at that time is used as the initial motion vector in the frame memory 12.
4 and the smaller motion vector e from here
Is calculated in the motion compensation circuit 123. The motion of the camera when this e becomes the smallest is adopted as the true initial vector.

At this time, if the motion of the camera is set in advance and the decoding side can understand it, it is not necessary to transmit the motion of the camera, but if it is changed on the operation board 131 at any time, or if the human actually does the camera. If is moving, the motion of the camera finally selected as the initial vector (any of camera motion 1, camera motion 2, camera motion 3 ... In FIG. 3) is transferred to the decoding side. There is a need to.

For the actual MC, the MC error signal is taken by the subtracter 127 with (d + e) as the true motion vector, but the information taken in from the input information switching circuit 121 as the current frame signal at this time is the fixed camera 10.
It is the information taken in from 1 (that is, the state of B in FIG. 4C), and it means that b is created by the above (d + e). Needless to say, at this time, if the same prediction regarding the motion amount can be performed during transmission and reception, it is not necessary to transmit the information regarding the motion amount.

The MC error signal is compression-encoded by the encoding circuit 122 and sent to the decoding side, and at the same time, the decoder 125
After being decoded in (1), it is added to the previous frame signal used for MC in the adder circuit 126, and a reproduction signal is created and stored in the frame memory 124.

FIG. 5 is a diagram showing the encoding side of another example of the first embodiment. It is assumed that the processing device 171 is performing the same processing as that shown in FIGS. 2 and 3 using the information captured from the plurality of cameras 161 and 162. At this time, since the actual movement of the moving object becomes clear in the processing device (b in the above example), if there is a correlation in the moving direction of the moving object, the movement of each camera is calculated based on this movement. It becomes possible to control automatically. As an example, the motion control circuit 1
72 is installed as shown in the figure, and the actually measured motion is classified into the number of cameras by a method similar to the vector quantization, and the respective motions are transmitted to each camera. Since each camera moves according to the instruction from the motion control circuit 172, if the motion has a time correlation, the small motion vector e in FIG. 4C can be automatically reduced.

Next, a second embodiment according to the present invention will be described. FIG. 7 is a diagram showing the basic configuration of the media processing system of the present invention. In this figure, the processing unit 203 processes two types of image information obtained from the limited input device 202 for obtaining the image of the user and the limited input device 201 for obtaining the environmental image viewed from the user side. An example of displaying the image obtained by the output device 204 is shown. The processing unit 203 performs a combining process of two pieces of image information so that the input environment image becomes the background portion of the input user image.

As an example of the synthesizing method, there is a method of overwriting the environment image with a pure user image obtained by separating and removing an unnecessary background image incidental to the user image. There are various known methods for determining the area of the user image portion, but description thereof will be omitted here.

Another example of the second embodiment in which the present invention is applied to a media processing device will be described below. FIG. 8 shows an example of the configuration of the media processing device of the invented method. The input device 206 is a camera provided to obtain image information of the user on the front surface of the device. On the other hand, the input device 205 is a fixed or movable camera provided in a direction different from the front surface of the device so that the environmental image viewed from the user side can be acquired without obstruction. Here, for simplification of description, it is assumed that the input device 205 is provided on the back surface of the device, but if the input device can achieve the purpose of inputting an image viewed from the user side, the input device 205 may be installed on the upper part of the device or the user's body. It doesn't matter if it is installed.

The processing unit 207 performs processing for displaying the environment image in the background of the user image in a natural form. Here, the processing unit 207 performs background separation removal processing 208, image size setting processing 209, display range setting processing 210, and mirror image determination processing 211. The size setting process 209 and the range setting process 210 are for setting the size and the display position when the user image is finally displayed. Here, the size setting process and the range setting process 210 can be set from outside the apparatus.

The mirror image inversion processing 211 is, for example, when the media processing device of the present invention is used as a communication terminal, when the user talks with the other party about an object in the environmental image, the direction and display screen of the object pointed by the user. In particular, it is necessary to match the direction of the object pointed by the user's image.
This will be described with reference to FIG. In FIG. 9A, the user is in a real environment (landscape) while using the media processing device 220.
Shows that you are pointing at something (the moon in this example). At this time, the environmental image captured by the camera 221 on the rear side of the device is as shown in FIG. 9B, and the user image captured by the camera 222 on the front side of the device is as shown in FIG. 9C. When the two images are combined, in order to allow the user to point in the direction of the moon, the user image in FIG. 9 (c) is mirror-inverted and combined with the environment image in FIG. 9 (b). Good. The composite image thus obtained is shown in FIG.
It shows in (d).

The image selector 212 inputs the environment image from the input device 205, the user image from the input device 206, and the user image without the background from the processing unit 207, and according to the screen designation information, the user image and the background image are displayed. One of a non-user image, an environment image viewed from the user side, and a composite image in which an environment image viewed from the user side is incorporated in the background of the user image is output to the output device 213.

At this time, the person may be made into a mirror image and the environment image may be left unchanged, or the person may be left as it is and the environment image may be made into a mirror image.

Next, another example of the second embodiment applied to the media processing system of the present invention will be described. FIG. 10 shows the configuration of another example of the media processing system according to the second embodiment. Here, an encoding / decoding system will be described in which the information input on the transmitting side is encoded and transmitted, and the information transmitted on the receiving side is decoded and output.

First, the transmission side of the system will be described. FIG.
At 0, the input device 221 is a microphone for inputting voice or environmental sound. The input device 222 is a camera for capturing an image of the user. Sound information coding unit 23
At 0, the input sound is analyzed by the pitch analysis unit 231 or the LPC analysis unit 232, and the pitch frequency information or the LPC coefficient information obtained therefrom is encoded by a method such as encoding. The sound encoding unit 250 encodes the input sound and outputs it. The image encoding unit 240 separates and removes the image of the background portion of the user from the input image to conceal the environmental information, and encodes and outputs the used image. The multiplexing unit 251 multiplexes the sound code information, the coded image information, and the coded sound information, and outputs the multiplexed sound code information to the transmission path.

Next, the receiving side of the system will be described. The demultiplexing unit 252 demultiplexes the received data to separate the information into sound code information, encoded image information, and encoded sound information. The image generation unit 253 generates the environmental image information by a preset method using the sound code information. As an example, when a pitch frequency extracted from a voice or an environmental sound is used for the code information, a periodic image can be generated in time based on the pitch frequency that changes in time. In addition, a more complicated environmental image can be generated by changing the position at which the image is cyclically changed in time. As another method, it is possible to provide an environmental image in which colors are variously changed by setting the correspondence between the pitch frequency and the color of the environmental image.

When the LPC coefficient information extracted as chord information is used as another sound information, the image quality can be changed by applying a filter based on the LPC coefficient, so that the image quality can be varied in time. It is possible to generate a changing environment image. As yet another method, by setting the correspondence between the LPC coefficient information and the color of the environmental image, it is possible to provide the environmental image with various colors.

With such a method, an environment image that coincides with the change in the environment on the transmission side can be displayed on the background, so that the monotony of the concealed background image can be greatly reduced. In addition, when a display window corresponding to the frame of the painting is displayed on a part of the background and an environmental image is displayed in this window,
It is possible to effectively display an environmental image generated from sound as so-called art, which has a kind of artistry.

Next, still another example of the second embodiment in which the present invention is applied to the media processing system will be described.
FIG. 11 shows the configuration of the media processing system according to the second embodiment, which is a modification of the above-described embodiment. 11, the same parts as those in FIG. 10 are designated by common reference numerals and the description thereof will be omitted. In this embodiment, in addition to the microphone 221, a sound source or a sound source terminal 260 for supplying an environmental sound for obtaining a sound code is provided.
By separating the sound source for obtaining the sound code from the microphone in this way, it is possible to generate a variety of environmental images using favorite music and prevent the music from interfering with the call.

A third embodiment of the present invention will be described below with reference to the drawings. FIG. 13 is a diagram showing an example of the third embodiment. The camera 301 captures a panoramic view of the landscape, for example, trees and flowers, and the panoramic image data is sent to the processing device 303. At the same time, the camera 302 images only a part of the landscape, here flowers, and the flower image data is also sent to the processing device.
The panoramic image data is shown in FIG. This image is shown in Figure 1.
It is expressed by a group of pixels arranged discretely as shown in FIG. (Actually, it is not as rough as in FIG. 14B, but here it is composed of 25 pixels to make the diagram easier to see.) Here, if the pixel density is not sufficient, small parts such as tree leaves and flower patterns It is blurred and does not appear in the panoramic image data.

On the other hand, the flower image data is shown in FIG. This data is also composed of the same number of pixels as the panoramic image data as shown in FIG. 15B, but the shooting range is limited to flowers, so no trees are shown, but the detailed shape and The handle is clearly visible.

In the processing device 303, the panoramic image data and the flower image data are combined to create the combined image data shown in FIG. 16 (a). Since FIG. 16 (a) is a combination of FIG. 14 (a) and FIG. 15 (a), both tree and flower are shown in FIG. 16 (a), and the pattern of the flower is clear. . This pixel arrangement is shown in FIG. The composite image data is input to the display 304. When the number of pixels of the display 304 is sufficiently large as shown in FIG. 16C, pixel parts are interpolated for the tree part to interpolate pixel data not included in the combined image data, and the flower part is combined. The high-density image data is used as it is. In this way, if one wants to pay particular attention to flowers while overviewing the entire landscape, it is possible to obtain a high-resolution image of the flower portion by combining two inexpensive cameras with a small number of pixels.

If an image encoder is used instead of the display 304, it is possible to maintain a high resolution for the part of interest and to express the other parts with a small amount of information, so that the whole part can be represented. It can be compressed to a code amount smaller than that transmitted at high resolution.

FIG. 17 is a block diagram showing another example in which the third embodiment is used for a videophone or the like. Camera 311
From the image data is sent to the correction device 313 and the processing device 314. Also, voice data is sent from the microphone 312 to the processing device 313.

In the correction device 313, first, as shown in FIG. 18, the face area 321 is detected from the image data. here,
The lower left corner of the image data 320 is coordinate (0,0), and the lower left corner of the face area is coordinate (k, l). In addition, the image data 32
0 width is M, height is N, face area width is K, height is L
And The camera is oriented so that the face is in the center of the screen ((k + K) / 2)-(M /
2), in the vertical direction ((l + L) / 2)-(N /
2) Send a signal to the camera 311 to move only. Also, a signal for multiplying the camera magnification by J / L is sent to the camera 311 so that the face has a predetermined size (height J) on the screen. The camera 311 faces as instructed and the magnification is corrected.

As for the microphone 312 as well, the microphone 31 and the microphone 312 are detected based on the current orientations of the camera 311 and the microphone 312 and the detection result of the face area so that the sensitivity becomes higher in the direction of the user's mouth.
Change the direction to find the correction angle of the direction of 2. The position of the mouth is, for example, in FIG. 18, coordinates ((k + K) / 2, (1 + L) /
Decide as 4).

If there is a display, the coordinates ((k + K) / 2, in FIG.
Be oriented in the direction of (l + L) / 2).

FIG. 19 is a block diagram showing another example for area detection of the third embodiment. Light of a predetermined color is emitted from the light emitting device 331 to the front surface, and at the same time, the camera 332 is illuminated.
, The image data is captured, and the image data is processed by the processing device 334.
And sent to the correction device 333. When a person's face is included in the image data, the light emitted from the light emitting device 331 reflected by the eye portion such as the eyeglasses or the eyeballs is included. Therefore, the correction device 333 finds a portion of the irradiation light from the image data and determines the portion as the eye region. Alternatively, as the signal to be detected, an image pattern that changes spatially or a time-varying pattern may be used instead of the color. The light emitting device 331 may also serve as a display. In that case,
The color or pattern in the image to be displayed may be appropriately selected and used as a signal to be detected.

FIG. 20 is an external view showing an embodiment in which the image pickup means of the third embodiment is separated from the media processing system. The camera 342 is housed in a spherical container 341. The container 341 is connected to the first support rod 3 via the connecting portion 343.
It is connected to 44. Further, the first support rod 344 is connected to the second support rod 345 via the connecting portion 346.
The connection unit 343 is variable in the longitude direction and the connection unit 346 is variable in the latitude direction. A movable joint is provided at a substantially middle portion of the second support rod 345, which can be bent in any direction, and is usually semi-fixed. A clip 347 is connected to the second support rod 345. The clip 347 is for attaching the camera 342 to a wall, a pillar, a personal computer, etc. near the user. Instead of or together with this clip 347, it can be used in environments such as suction cups, magnets, hooks, pins, and velcro. It can be used together.

An example of a sectional view of the contents of the container 341 is shown in FIG.
Shown in The housing 352 of the camera 342 is made of a magnetic material such as a magnet, and the weight 353 is fixed to the bottom portion when the imaging screen is horizontal. A frame body 355 in which a large number of electromagnets 354 are arranged is provided around the housing 352. When the power of the imaging device is turned on, the electromagnet 354 is also energized, and the housing 35
If the N poles or the S poles face each other on the surfaces of the 2 and the electromagnet 354 that face each other, the housing 352 slightly floats. At this time, the electromagnet 354 may be a permanent magnet. Since the camera 342 includes the weight 353, the imaging screen is always kept horizontal by gravity regardless of the imaging direction of the camera 342.

For example, suppose that a clip 347 is attached to a rearview mirror to take a picture of one's face while driving a car. In that case, the camera 342 generally tilts. Therefore, if the connection portion 343 and the connection portion 346 are manually adjusted, the camera 342 can be directed toward one's face. Then, in accordance therewith, the image pickup screen of the electromagnetically floating camera 342 automatically becomes a horizontal screen due to gravity, so that there is no need to separately perform horizontal adjustment. Further, even if the vehicle shakes a little, it is not transmitted to the camera 342, and a good image without shaking can be input.

FIG. 22 shows another example of the third embodiment. The camera 361, the first support rod 363, the second support rod 365, and the third support rod 367 are connected via connection portions 362, 364, 366, respectively, as shown in the figure.
In this embodiment, the connecting portion 366 is used for the longitudinal direction and the connecting portion 362 is used.
Adjust the latitude direction with. Then, the connection portion 364 is weakly tightened so that the first support rod 363 and the camera 361 can be hung down. Then, the image pickup screen of the camera 361 is always kept horizontal due to gravity.

Next, a media input device will be shown as an example of the fourth embodiment according to the present invention. This embodiment is an apparatus for transmitting or recording and storing images and sounds.
In this device, regarding the part of the input image / sound that can be replaced with the image / sound stored in the database,
It is characterized in that the amount of information related to transmission / storage is significantly reduced by transmitting / storing the database index and the like.

Referring to FIG. 25, an image and a sound to be transmitted / stored are input from the camera 401 and the microphone 402, respectively. The position detector 403 is means for detecting the present location of the media input device. This may be detected based on the geomagnetism, or GPS (Global)
Positioning System (a global positioning system) may be used. Temperature / humidity sensor 40
Reference numeral 4 is a means for detecting weather conditions such as temperature, humidity, and air volume. The timer 405 is a means for obtaining the current date and time. The azimuth meter 406 is an azimuth sensor for detecting the direction of the image / sound captured by the camera 401 and the microphone 402 from the apparatus body. The image / sound database 409 stores images / sounds that are easy to replace based on information such as position and time in the input image / sound like the background portion. The image / sound database 409 may store images / sounds at any points in the world, and if the place where the input device is used is limited, the images / sounds in the limited place are stored.
Sound may be accumulated. The transformation processor 410 performs transformation processing on the image / sound selected from the image / sound database 409 as necessary.

The background image / sound determiner 408 determines which of the images stored in the image / sound database 409 is to be used and how to modify the image. The background image / sound determiner 408 includes a position detector 403,
Temperature / humidity sensor 404, timer 405 and compass 40
6 is connected, and the index of the database corresponding to the current position, direction, weather condition, date and time inputted from here is determined. Here, if there is a highly accurate match in the database with the actual background, this is used as it is, but if not, it is determined what kind of transformation should be applied to the image / sound in the database. .

For example, when only the images in the northwest and north directions are stored in the database even though the camera is oriented in the north-northwest direction, a composition considering rotation from these two-direction images is performed. Go and create north-northwest images. In addition, the database contains the position of the sun, the seasons such as clouds,
Images that change according to time and weather conditions are not included, seasons,
The position of the sun, the shape and extent of the cloud (the position and brightness of the stars in the case of night), the illuminance, based on the time of day, current location, direction, and weather conditions.
It is also possible to calculate the degree of swaying of trees, create an image of the sun, clouds, etc. corresponding to this, and combine it with the image in the database. In the case of a place where wild birds inhabit, the bird's bark accumulated in the database is selected as the background sound, and this is processed by changing the bark's loudness based on the season and weather conditions. Is also good. The position detector 40
3. Since information detected by the temperature / humidity sensor 404, the timer 405, and the azimuth meter 406 includes an error, a background image / sound that is slightly different from the actual one may be created. To prevent this, the camera 401 and the microphone 40
2 is also connected to the background image / sound determiner 408, and based on the input images / sounds from these, corrections are added to the determination and processing of the database index and the determination of the transformation method to create a more accurate background image / sound. be able to.

The background image / sound signal created by the camera 401, the microphone 402 and the deformation processor 410 is the background extractor 4.
The image / sound background is separated and extracted. For background separation / extraction, for example, a portion of the input image / sound having a small residual difference from the created background signal may be set as the background, and the other portions may be set as non-background. Alternatively, an important part such as a person image or a human voice may be set as a non-background, and other parts that are not so significant may be separated as a background.

The area determined as non-background by the background extractor 407 is subjected to high efficiency coding and transmitted / stored as non-background information. For this high-efficiency encoding, for example, an encoding method such as motion compensation + DCT for an image and ADPCM for a sound may be used. Alternatively, a highly efficient coding method suitable for objects that have been determined to be important, such as a human image or non-voice, for example, model paste coding for a human face, CELP for a human voice, etc. Encode using. On the other hand, with respect to the portion determined to be the background, the index of the background database and the parameter of the background deformation are targets for transmission / storage. The transformation parameter may be a parameter that directly represents the transformation method (for example, the filter coefficient when filtering is performed), but if the amount of information is too large, the current location that is the basis for determining the transformation, Information such as time and weather conditions may be encoded and transmitted / stored. These are multiplexed by the multiplexer 411 and transmitted / stored.

FIG. 26 shows an example of generation of a background image from an input image and background / non-background separation. The sun image 425, the cloud image 423, and the tree image 426 are combined in the deformation processor 410 with the background image 420 selected from the image / sound database 409 and subjected to conversion of illuminance and the like. The solar image 423 shows the season, time, weather conditions, current location,
It is an image calculated based on the azimuth. The cloud image 425 may be detected as a difference between the input image and the background image 420. In this case, the cloud image 425 is encoded by a general encoding method such as motion compensation + DCT as in the non-background portion, and is transmitted as non-background information. /accumulate. Alternatively, when the fidelity with the input image is not required so much, the images may be created and combined based on the weather conditions. In this case, since the cloud image can be created based on the information about the background, it is not necessary to transmit / store it as non-background information. The tree image 426 is the one whose source is contained in the database, but is deformed such as fluctuation in consideration of weather conditions and the like. When detecting an important image such as a person, the person portion 421 is separated,
Encoding suitable for this, such as model paste encoding, is performed. Further, an image like the animal 422, which is not stored in the database and is not so important as the person, but which should be transmitted / stored, is generically encoded with lower accuracy than the person portion 421.

FIG. 27 is a block diagram of an apparatus for reproducing an image / sound from the information transmitted / stored in FIG. First,
The demultiplexer 431 separates the non-background information, background database index, and deformation parameter. The non-background information is input to the non-background image / sound creator 434, and the image / sound of the non-background portion is reproduced. Non-background image / sound creator 43
The reproduction processing in No. 4 is performed by a method corresponding to the non-background encoder 412 in FIG. These pieces of information are input to the non-background image generator 432 to reproduce the background image / sound. image/
The image / sound corresponding to the background database index is extracted from the sound database 433, and the image / sound corresponding to the deformation parameter is subjected to deformation / processing. This transformation / machining is the same as the processing of the transformation processor 410 of FIG. The reproduced background image / sound and non-background image / sound are combined by the combiner 435 and output as a reproduced image and reproduced sound.

FIG. 28 is a block diagram of another example of the fourth embodiment according to the present invention. This device also aims at transmitting / accumulating input images / sounds, and reduces the amount of information relating to transmission / accumulation by replacing the background with a database or the like. Also, there are multiple cameras and microphones for inputting images / sounds, and not only the internal database of the device but also external databases, street corner cameras / microphones and other similar inputs as images / sounds that are the basis of the background replacement. Images / sound from the device are also used.

Cameras 441-443 and microphone 444-
Images and sounds to be transmitted / stored are input from 446, respectively. Cameras 441-443 and microphone 4
Each of 44 to 446 is assigned a role of inputting the whole image / sound and one of inputting a central person, object, voice and the like. Also, the person camera 4
Reference numeral 41 adjusts the shooting direction, range, focal length, etc. for the purpose of capturing the entire main person, and inputs the whole person image. Further, the face camera 442 captures and inputs the face portion of the person. The entire view is input from the whole camera 443.

Since the person camera 441 and the face camera 442 are provided separately from the panoramic camera 443 as described above, the person image and the face image, which are more important information, can have higher resolution than other portions. Also, subsequent processing can be different from other parts. Microphone 444
~ 446 are also divided into roles, and the voice microphone 444 outputs the voice of the central person to the background microphones 445, 44.
6 is installed for the purpose of capturing the background sound. The voice microphone 444 may be a handheld microphone held by a central person, or may be attached to a part of the body or clothes to transmit voice to the device body using radio waves or infrared rays. You may equip it and use it.
Further, the microphones 445 and 446 may respectively capture sounds in different directions such as leftward and rightward directions or forward and backward directions.
Two microphones may be built in 5,446 to make a stereo sound (2 channels or 4 channels).

Current location detector 447, weather sensor 448,
The timer 449 and the compass 450 are respectively a position detector 403, a temperature / humidity sensor 404, and a timer 40 shown in FIG.
5 and azimuth meter 406, and is means for detecting the current location of the device, weather conditions, the current date and time, and the direction viewed from the device to be input. The image / sound database 453 is a database of images / sounds that are the basis of partial replacement of input images / sounds.

Position detector 447, temperature / humidity sensor 44
8, timer 449, azimuth meter 450, and overall camera 44
3. Background microphones 445 and 446 are background image / sound creator 4.
A background image / sound is created here, but the difference from the embodiment shown in FIG. 25 is that not only the database 453 in the device as the background image / sound element, but also other data as necessary. Database 463a, street corner camera / microphone 464a and images from other similar input devices 462 /
I am using sound. There may be multiple external databases, street corner cameras / microphones, and other input devices, each of which is a wired or wireless network 4
It is connected via 61.

What kind of background image is created by the background image / sound creator 452 will be described with reference to FIG. 29 of the input image. When FIG. 29 is input from the panoramic camera 443, the background image here is a portion other than the person image 471a. Of these, the portions such as the sky and the building where the image can be obtained by database access based on the current location, the direction in which the camera is facing, the date and time, and the weather conditions are replaced with the database image. here,
If the target image is not found in the internal database 453 of the apparatus, the image of the external database 463a is used.

The images retrieved from the database are corrected according to the conditions such as weather conditions and time. For example,
When it is fine during the daytime, the sky is made blue and the overall illuminance is made brighter, and the shadow of the building is attached in a direction and concentration suitable for it, and when it is cloudy or rainy, the overall illuminance is made dark and the shadows are made light. The solar image is calculated and composed according to the weather conditions, seasons, and time. The camera 44 is used when determining the above correction and combining methods.
A higher-precision background image may be created by also referring to the input from 3.

Things like moving cars in the background are not in the database. To accurately represent such a portion, an input image from the street corner camera 464 may be used. The area captured by the street corner camera is the area enclosed by the broken line in the area 474. This portion is subjected to high-efficiency coding if necessary by the street corner camera / microphone device 464 and transmitted through the network. When the high-efficiency encoding is performed, it is decoded and reproduced as an image, and then the area 474 is replaced with the image. Or, if you don't need that much precision for a car-like part and you only need to know the approximate traffic volume,
If the image from the street corner camera is not replaced, only the vehicle type and traffic volume parameters are transmitted / stored, and an appropriate amount of vehicle image is generated and synthesized according to this when reproducing. Can be significantly reduced.

The person 471b in the background is not included in the database and is not captured by the street corner camera, but wants to obtain a highly accurate image as necessary. In this case, if another input device 462 captures this person with high accuracy, an image from that person may be input and used. If high-efficiency coding is being performed when the image of the person 471b is transmitted by another input device, a process of decoding this is performed and the reproduced image is combined with the background. The input image from the street corner camera or another input device may have a different image capturing direction from the image captured by the overall camera 443, or the image capturing range may be wider than necessary. In such a case, the image pickup direction is converted and the necessary portion is cut out to synthesize it with the background. The conversion of the shooting direction may be performed, for example, by modeling the image into a three-dimensional model and obtaining an image when the viewpoint is changed in consideration of the current location and the shooting direction of the present device, the street corner camera and other devices.

The background image / sound created by the background image / sound creator 452 is sent to the non-background image / sound extractor 451 for background / non-background separation. The processing on the image here will be described by taking the image of FIG. 29 as an example. Central person 4
71a is also imaged by the person camera 441 and the face camera 442. Inputs from these are defined as a region 472 and a region 473 surrounded by broken lines in FIG.

Since the background portion around this is included in addition to the person and the face, this background portion is removed. This separation may be performed, for example, by removing, as a background, a portion where the input from the camera and the portions in the regions 472 and 473 of the generated background image match with high accuracy. By input matching from the person and face images cut out in this way and the panoramic camera 443, the central person 471a and other portions can be separated. The panorama camera 4 is the part that is determined to be other than a person.
Matching is performed between the input from 43 and the created background image, and the highly matched portion is first determined as the background.

Next, with respect to the portion having a large error in matching, it is judged whether or not the portion is important information.
For example, the airship shown in FIG. 29 does not exist in the database, is not included in the background image because it was not captured by the street corner camera or other input device, and is an image suitable for representing the situation, Encoding processing is performed after it is determined that the background is not present. Non-important noise components and the like are not judged to be non-background, and no encoding processing is performed.

On the other hand, regarding the sound, the input from the voice microphone 444 is used as a voice signal and the background microphones 445 and 446 are used.
The input from is used as the background sound, and the background sound may be replaced by the sound of the database or the street corner microphone. However, the background sound may also be included in the input from the voice microphone, which may be inconvenient for the subsequent processing such as encoding.
In this case, the non-background image / sound extractor 451 performs the background sound removal processing included in the voice microphone input. In this method, for example, the presence / absence of sound may be determined, and when there is no sound, the sound may be erased and the processing such as encoding may not be performed, or the learning identification method using the input from the background microphone as a reference signal, the LMS algorithm. Background noise may be removed by an adaptive noise canceller using the RMS algorithm or the like.

The image / sound determined to be non-background by the non-background image / sound extractor 451 is sent to the encoder 454 and is encoded. Images or sounds of the central person or face may be coded using a coding method suitable for this, for example, images may be coded using model paste coding and sounds may be coded using CELP or the like. On the other hand, FIG.
Things such as the airship in 9 that were not created as a background but are to be transmitted with a certain degree of accuracy are encoded with a lower degree of accuracy than the central person using general encoding.

Reference numeral 455 is an interface with a network and a storage medium. Here, the background image / sound creator 4
In response to a request from 52, an external database, a street corner camera, and other input devices are accessed, and the obtained data is sent back to the background image / sound creator 452. Further, the information about the background image / sound output from the background image / sound creator 452 and the encoded information about the non-background image / sound output from the encoder 454 are multiplexed, and the storage medium 456,
The data is sent to the external storage device 465 or an external playback device described later.

Upon multiplexing, only the minimum information necessary for reproduction is selected. For example, a playback device is connected to the network 466 and the database 46
3. If the street corner camera / microphone 464 and the other input device 462 are also accessible, it is not necessary to transmit the information obtained from these external devices.
For example, referring to the image of FIG. 29, for the sky, the building, and the sun, if the current location of the input device 460, the shooting direction, the weather conditions, the date and time, etc. are known, the images in the internal database of the playback device and the external database 482 are transformed, Since it can be created by performing interpolation and composition, only information such as the current location, shooting direction, weather conditions, date and time, etc. need to be transmitted.
However, when the reproduction is performed in real time, the date and time is the same as that of the input device (or can be corrected even when there is a time difference), and therefore transmission is not necessary. Further, when the positions of the input device and the reproducing device are relatively close to each other, it is considered that the weather conditions are not so different from each other. If the camera is fixed, the current location and shooting direction information may be transmitted only once at the beginning of transmission, or if the camera movement range is narrow, the initial current location and shooting direction are transmitted first. After that, the correction information may be transmitted. Next, broken line 4
A part in 74 and a person 471b in the background can be obtained from the street corner camera 463 or another input device 462 in the case of performing real-time reproduction. You may make it transmit only the information showing whether the deformation | transformation and clipping were performed. On the other hand, when performing non-real time reproduction such as a storage system, real time information cannot be obtained from a street corner camera or other input device. If the storage means is attached to the street corner camera or other input device and it is possible to record the information necessary for the background image generation in the playback device in the storage device, the input device Reference numeral 460 requests the street corner camera and the input device to record necessary information and provide the information in response to a request from the reproducing device. If it is impossible for the street corner camera and other input devices to store information, the information may be stored in the external storage device 481. In this case, it is only necessary to transmit information from the street corner camera or other input device to the external storage device.
The amount of information sent from 0 does not increase. If that is also impossible, the image information of the area 474 and the person 471b portion is also multiplexed at the time of multiplexing by the interface 455. It should be noted that non-background information such as the central person 471a and the airship cannot be obtained from other devices, and therefore must be included in the information to be transmitted / stored.

Further, the interface 455 has a role of an interface for providing a partial image of the background in another input device / an input image which is a source of sound creation. For example, when the other input device A is requested to provide the image of the person 471a, the encoded information regarding the human image among the non-background information is transmitted to the input device A that has made the request. Here, the image / sound information is also transmitted to the image / sound reproduction device described later, and if the person image requested by the input device A is also included in this information sequence, the input device A Therefore, it is not necessary to separately transmit the personal image information, and the input device A may be requested to retrieve the information string for the reproducing device from the network and use only the personal image information out of the information sequence. Device 4
There is no need to newly increase the amount of information transmitted from 60.
Further, it is impossible to answer the information provision request from the input device A considering the total amount of information flowing in the entire network, the amount of information allocated to the input device 460, the allowable processing amount of the input device 460, and the like. If it is determined that this is the case, this request may be rejected, or it may be requested to reduce the quality of the image transmitted to the input device A (that is, reduce the amount of information related to information provision).

FIG. 20 is a block diagram of an apparatus for reproducing the input image / sound. Information necessary for reproduction is provided to the reproduction device 490 via the network 466 and / or the storage medium 456 or both. 491 is an interface with the network 466 and the storage medium 456,
External database 463, street corner camera / microphone 464,
The other input device 462 and the external storage device 465 are requested to provide necessary information, and the input device 460, the external database 463, the street corner camera / microphone 464, and the other input device are requested from the storage medium 456 or via the network. The image / sound information is input from the external storage device 465, decomposed into background information and non-background information, and output to the background image / sound creator 492 and the non-background image / sound reproducer 494, respectively.

The non-background image / sound reproducer reproduces the image / sound of the non-background portion. Since the non-background information is high-efficiency coded, a process for decoding this is performed. This is a decoding process corresponding to the encoder 454 of the input device 460 of FIG.

In the background image / sound creator 492, the background image /
The sound is created. In the example of the image in FIG. 29, the sky is
For a part such as a building or the sun, which is created by modifying the image of the internal database 453 with the input device 460 based on the season, date and time, and weather conditions, select the corresponding image from the database 493 in the playback device. It is transformed into a background image.

However, when the database 453 of the input device and the database 493 of the playback device are different and the image to be selected does not exist in the database 493 in the playback device, similar images in the database 493 in the playback device are displayed. Or the external database 463 is accessed to extract an image that is the same as or similar to the image in the database 453 of the input device, and is appropriately modified to create a background image. For example, the sky does not make a big difference even if the current location is slightly different (for example, the sky is considered to be almost the same in Japan), so select a sky image of a similar area from the database and transform it if necessary. And use it.

If an image of the building exists in the database but the shooting direction is slightly different, the image may be created by performing a modification to correct this. Alternatively, when it is determined that the building exists, but the shape of the building itself is not so important, an appropriate building image may be extracted from the database and applied.
A moving car and a person 471b in the background are images input from a street corner camera and another input device, respectively. For this portion, the corresponding street corner camera and input device are requested to transmit information, and an image of the portion is created based on the transmitted information.

However, when the reproduction is performed in non-real time, the time when the background image is created by the input device 460 is different from the image input from the street corner camera or the other input device at the reproduction time. Cannot be used as is. In this case, if there is a storage medium in the street corner camera and the other input device as described above and the image information at the time when the background is created by the input device 460 is stored therein, the image information is transmitted. To create the background. Alternatively, the storage medium 45 connected to the input device
6 and the external storage device 465 have stored corresponding image information, the image information may be transmitted.
Alternatively, when the transmission information from the input device 460 includes the image information of the corresponding portion as a generic coded sequence, the image can be obtained by performing a decoding process based on it.

If image information is not stored in any of these, a similar image is searched for from the image in the database, or the image currently input from the street corner camera or other input device, and the background image is set. create. For example, for the portion in the region 474 using the image of the street corner camera, the input image from the same street corner camera may be used.
At this time, in order to correct the difference in time, season, weather conditions, etc. between the time of inputting an image with the input device 460 and the reproduction time, the illuminance,
The number of vehicles passing through may be modified and used as a background image.

A non-background image / sound reproduced by the non-background image / sound creator 494 and a background image / sound created by the background image creator 492 are combined by a combiner 495. The non-background information includes information indicating at which position in the whole view the non-background image is composed at the same position as the original input image based on the information.

[0166] Note that, when synthesizing in the synthesizer 495, not only faithful reproduction of the whole-view image / sound, but also processing such as extraction of a part with high precision can be performed if necessary. For example, the background image / the image of the central person without the sound /
If only the sound is required, only the image of the central person and the audio information may be reproduced and taken out. Alternatively, it is also possible to retrieve and reproduce only the face image among them. Also,
With regard to sound, it is possible to change the intensity of the mixture of the voice signal and the background sound according to preference. For example, when it is desired to hear the voice clearly, the voice signal may be increased.

Also, by performing adaptive processing that the background sound is reduced when there is sound and the background sound is increased when there is no sound based on the presence / absence determination information, the voice of the central person can be easily understood and the background situation can be grasped. Easier to do. Further, the background portion may be cut out and used not only in the whole view but only around the central person. On the other hand, when a wider range of background information is required, on the other hand, using the transmitted / accumulated current location, time, weather conditions, etc., the database or street corner camera is searched for the corresponding image and transformed to create a large background image. It is possible to create. Further, when a higher-definition image / sound is required for the background portion, it is possible to close up the portion.

For example, when it is desired to obtain a detailed image of a person in the background of FIG. 29 and the content of conversation, ask the other input device that has captured the person image to transmit the information of the corresponding person image and voice. It is only necessary to request and reproduce the image and sound based on the information.

In the input device of FIGS. 27 and 30,
Since the background and non-background information are separated, if necessary, the background part can be replaced by another image / sound, or parameters such as time and weather conditions can be changed to change the background image / It is possible to easily replace the sound and synthesize the image / sound.

In FIGS. 25 and 27 and FIGS. 28 and 30, an example in which the input device and the reproducing device exist separately has been described, but the input device and the reproducing device may be used in the same housing or connected. . In this case, the database, storage medium, network interface, etc. can be shared. Further, in another embodiment, when another input device 462 and the reproducing device 490 are connected, the image 471b captured from that is not required to be transmitted via the network.

Next, a fifth embodiment according to the present invention will be described with reference to the drawings. FIG. 33 is a block diagram showing the same embodiment. In this example, the configuration is made up of three cameras, and the information source is human face information. In FIG. 33, the information source 5
11 is a camera A 512a, a camera B 512b, and a camera C
It is inputted as different limited information via 512c. In this embodiment, it is assumed that the camera A 512a is installed in the right eye, the camera B 512b is installed in the left eye, and the camera C 512c is installed in an appropriate position for obtaining mouth information.

Right eye information is obtained as an image including the right eye through the camera A 512a. The right eye information is input to the right eye state acquisition means 513a. The right-eye state acquisition unit 513a outputs 1-bit information depending on whether the right eye is open or closed. The determination as to whether the eye is open or closed is performed by, for example, applying an edge enhancement filter to the right eye information, applying a noise removal filter, and then performing an expansion / reduction operation on the output signal to perform line segmentation. Such line segmentation will be referred to as line segmentation processing for simplicity hereinafter. FIG. 34 shows a flow chart of the line segmentation processing. Further, FIG. 35 shows how line segmentation is performed by performing an enlargement / reduction operation. If the size of the largest closed area present in the line segmented image obtained in this way is greater than a reference value, it is determined that the eyes are open, and if it is less than the reference value, the eyes are closed. Determine that

The left-eye information obtained from the camera B 512b is input to the left-eye state obtaining means 513b. The left-eye state obtaining means 513b determines whether the left eye is open or closed by the same method, and the 1-bit information is obtained. Is output. The mouth information acquired by the camera C 512c is input to the mouth condition acquisition unit 513c, and the mouth condition representing the mouth shape is obtained as the output. In the mouth state acquisition means 513c, among the templates included in the template table prepared in advance as the shape of the mouth, the degree of similarity between the template and the mouth line segment information obtained by line-dividing the mouth information. This is achieved by selecting the candidate with the maximum.

Next, an example of a method of selecting a template having the maximum similarity will be described. The horizontal length H of the mouth and the vertical length V of the mouth are measured from the mouth line segment image, and (H
The relationship between V and V is shown in FIG. 39) Et = | H−Ht | + | V−Vt | where t = 0 to T−1 is calculated between these two values and Ht and Vt in the template table. , Et is considered to be the highest in similarity, and the template number t is output as the mouth state (an example of the template table is shown in FIG. 40). The number of templates included in the template table is at least two, including the state in which the mouth is closed, and it is desirable to prepare only the types of phonemes.

In the output image generation means 514, one artificial image is selected in advance from the artificial image database 515, and a small piece is read from the artificial image database based on the right eye state, left eye state and mouth state which are sequentially input, An output image is generated by applying it to each corresponding part of the artificial image and displayed on the display device 516.

Another example of this embodiment will be described below with reference to FIG. Note that the names included in FIG. 37, which have the same names as those in FIG. 33, have the same functions as those described above, so description thereof will be omitted here.

Here, the embodiment shown in FIG. 37 will be described with reference to FIG.
The difference from the embodiment described in 3 is that the acoustic microphone 522 is included in the configuration, and the audio information obtained from the acoustic microphone 522 is transmitted to the speaker 5 via the audio codec 522a.
22c is output, and the mouth state is corrected by the state correction means 527 using the voice state obtained from this voice information. That is, the mouth state is determined only from the mouth information in the previous embodiment, whereas the mouth state is determined from both the mouth information and the voice information in this embodiment.

The voice state can be obtained as an output of the voice state acquisition means 526 which receives voice information. The speech state acquisition means 526 refers to the template table shown in FIG. 40 based on the phoneme information obtained by using the existing phoneme recognition technology, and outputs the template number ts as the speech state.

A method of realizing the state correction means 527 will be described below with reference to FIG. The mouth state t and the voice state ts are given as inputs to the state correcting means 527, and a highly reliable state is selected and output by the control signal. As for the control signal, for example, when the level of background noise is low, the reliability of the voice state is considered to be higher than the reliability of the mouth state.
Control to select the voice state. On the contrary, when the background noise level is high, the mouth state is more reliable than the voice state (because it is highly likely that the phoneme is erroneously recognized due to the background noise), so the mouth state is controlled to be selected. . By doing so, a highly reliable movement of the mouth can be obtained, and thus a more natural composite image can be obtained.

Next, another embodiment of this embodiment will be described with reference to FIG.
It will be described according to 1. Note that each name included in FIG.
Those having the same name as 3 have the same functions as those described above, and therefore the description thereof is omitted here.

Here, the embodiment shown in FIG. 41 will be described with reference to FIG.
The third embodiment is different from the third embodiment in that a synthetic image is generated from a combination of artificial images included in the artificial image database in the previous example, whereas a line image obtained from input information is generated in the third example. The difference is that a combined image is generated depending on the combination of.

Next, the right eye line drawing acquisition means 602 will be described. The right-eye line drawing acquisition unit 602 performs line segmentation processing on the right-eye information to obtain a right-eye line segmented image. The closed area having the largest area existing in the line segmented image obtained in this way is determined to be the right eye, all line segments existing outside the closed area are considered to be noise, and are deleted. This is shown in FIG. The noise-removed right-eye line differentiated image is output as a right-eye line drawing. Similarly, the left-eye drawing acquisition unit 603 can also obtain the left-eye drawing from the left-eye information.

Further, the mouth line drawing acquisition means 604 performs the line segmentation processing on the mouth information to obtain the mouth line segmentation image, determines that the closed region having the largest area existing in the line segmentation image is the mouth, and All line segments existing outside the closed area are considered to be noise and are deleted. The noise-removed mouth segmentation image is output as a mouthpiece.

In the output line drawing creating means 605, an appropriate contour line drawing and hairstyle line drawing are selected in advance from the line drawing database 606 and the sequentially input right eye line drawing, left eye line drawing and mouth line drawing are inserted into corresponding positions of the contour line drawing and output. A line drawing is created and displayed on the display device 607.

Another example of this embodiment will be described below with reference to FIG. Note that the names included in FIG. 43 and having the same names as those in FIG. 41 have the same functions as those described above, so description thereof will be omitted here.

Here, the embodiment shown in FIG. 43 will be described with reference to FIG.
The difference from the embodiment described in 1 is that the acoustic microphone 612 is included in the configuration, and the audio information obtained from the acoustic microphone 612 is transmitted to the speaker 6 via the audio codec 612a.
The point is that the line drawing correction means 618 corrects the mouth line drawing by using the audio state obtained from this audio information while being output from 12b. That is, in the previous embodiment, the mouth line drawing is determined only from the mouth information, but in this embodiment, the mouth line drawing is determined based on both the mouth line drawing and the voice state.

The state correcting means 618 is the same as the state correcting means 527 of the above-mentioned embodiment, and therefore its explanation is omitted here. However, the line drawing correction means 61 of this embodiment
8 is different from the state correction unit 527 of the embodiment in that, when the voice state is selected in 8, the line drawing of the phoneme corresponding to the voice state is selected and output as the mouth state. Also, as a method of realizing the state correction means 618, there is a method of switching according to the level of background noise, but this has been described in the second embodiment described above, so a new description will be omitted here.

Next, a sixth embodiment according to the present invention will be described. FIG. 45 shows an embodiment when the present invention is used for encoding a face image. In the figure, 642 to 644 are cameras, and 641 is a microphone. The camera 642 takes an image of the entire face including the background, the camera 644 takes an image around the eyes, and the camera 642 takes an image around the mouth and the chin. The cameras 644 and 642 can be moved so as to follow the eyes and mouth. The camera 643 may be fixed. The microphone 641 picks up the voice of the person in the picture. The following image information and audio information are obtained from these input devices. That is, audio information is obtained from the image 64 of the entire face, the image 647 around the eyes, the image 648 of the mouth, and the microphone 641 from the cameras 643, 644, and 642, respectively.

The processing device performs the following processing based on these pieces of information. The movement of the entire face is detected by identifying the pupil, nasal cavity, oral cavity, and the like from the image 645 of the entire face and examining the positional relationship. That is, if the distance between the pupils and mouth does not change, it is detected as a parallel movement, and if they expand and contract at the same ratio, they are detected as a movement in the front-back direction. Next, referring to the movement of the entire face, the cameras 644 and 6
Make 42 follow your eyes and mouth. The image 647 around the eyes and the image 648 of the mouth are separately obtained because the eyes and the mouth move frequently in the face and the shape changes greatly. These images 647 and 648 are used to detect the movement around the eyes and around the mouth. FIG. 46 shows the processing at the time of composition. Whole face image 65
Images 647 and 648 around the eyes and the mouth are interpolated at 0.
In order to suppress the distortion of the decoded image due to the interpolation processing, the size of the images 647 and 648 around the eyes and the mouth are adjusted so that the error between the image interpolated in advance and the image x of the whole face during analysis is reduced. Adjust it. It should be noted that not only the eye and mouth portions but also other portions may be photographed by a similar method if necessary in view of image quality.

When a face image is coded, images corresponding to the images around the eyes and the mouth are taken as images 647 and 648, and other parts are taken as images of the whole image 645 at different times, and motion detection is performed for them. Perform processing such as motion compensation.

Also, a motion model, a change in color and the like are coded using a wire frame model as a model of the subject's face or a model close to a real thing in which details of the shape and color of the face and hair are recorded. In this method, in the present embodiment, information on the movement of each part is given as follows. For example, there is rotation as the movement of the entire face, but the rotation can be detected using the position detection of the pupil and the oral cavity from the image of the entire face described above. If the pupil shifts in the height direction, rotation in the face,
If you get closer to the left and right of the screen while the interval is narrowing, rotation around the neck,
Also, if the distance between the pupil and the oral cavity changes and it shifts in the vertical direction, it is detected as a front-back rotation. FIG. 47 shows the above-described movement of the face and changes in the positions of the pupil and the oral cavity. (A),
(B) and (c) show the rotation of the face, the rotation of the neck, and the rotation of the front and back, respectively. Further, detailed analysis information of the local movement of the face such as eyes and mouth is obtained from the images x around the eyes and the mouth. Since the movement around the mouth is complicated, it may be patterned and a pattern suitable for pronunciation may be selected together with voice recognition. There is a more simplified method in which only the syllable and the loudness of the voice are obtained and the mouth is opened and closed accordingly. In addition, a difference often occurs between the model and the actual image, but the difference may be transmitted as necessary.

FIG. 48 shows another embodiment of the image input device for photographing the area around the eyes. The input device 660 has a shape of eyeglasses, and a camera 661 is incorporated in a frame. The lens portion 662 of the spectacles is a half mirror, and the peripheral portion of the eyes reflected in the lens body 662 is a camera body 66 in a horizontal frame.
Take with 1. The image captured by the camera 661 is transmitted by wire or wirelessly as in the previous embodiment. Since the camera is integrated with the face, the function of detecting the position of the eyes and following them is unnecessary. Further, if a gyro or the like is built in this input device, the angle and rotation of the face can be directly detected without performing analysis processing from the face image. This information is used to drive the wireframe model etc. described above.

Next, an example of using the present invention for a man-machine interface will be described. FIG. 49 shows a process of capturing the direction of the line of sight using the image of the entire face and the image of the area around the eyes. First, the image of the entire face 665 is referred to, the accurate position of the eyes is detected by the distance measuring device 664, and then the direction of the line of sight is examined by the image 666 around the eyes to specify what is seen. . If, for example, an autofocus camera capable of controlling the direction and outputting the distance is used as the distance measuring device 664, the position of the eye can be determined from the distance between the eyes and the direction of the camera. This process can generally be used for the operation screen of the device. If a camera for taking a picture of the surroundings of the person to be photographed is separately prepared, it is possible to capture an object viewed by the person to be photographed over a wide range.

If an image from an infrared camera is also used to detect the movement of the entire face, the accuracy can be further improved. Since the temperature of pupils, nasal cavities, oral cavity, etc. is lower than that of the surface of the skin, they are identified using an image from an infrared camera. As a method of collating with the whole face image x of the normal camera, the face image is extracted from the infrared image and the image of the normal camera, and the change in the brightness corresponding to the eyes and the mouth are the same. You can compare them.

[0195]

As described above, according to the present invention, since there are a plurality of means for inputting information, information can be grasped in a multidimensional manner, and the essence of information can be efficiently obtained by processing suitable for the characteristics of the information. It will be possible. When these pieces of multi-dimensional information are integrated, unnecessary information is excluded, and the information is integrated based on the attributes of the mutual information, which facilitates the processing and combining of the information. Also, the amount of information can be efficiently reduced in encoding.

Further, according to the present invention, it is possible to use, as the initial motion vector, the motion that best expresses the motion of the moving object or the moving part in the screen to be actually encoded from among the motions of the plurality of cameras. Since it is not necessary to represent all actual motions as motion vectors, it is possible to reduce the amount of generated motion vector information by MC.
Furthermore, since the motion amount detection range is automatically expanded by the camera movement, it is possible to limit the detection range when calculating the actual motion vector, and the code assigned to the motion vector is also reduced. It is possible to further reduce the vector information.

Further, the media processing system of the present invention is
It is possible to simultaneously provide the image of the user (the image of the objective viewpoint) and the environment image viewed by the user (the image of the subjective viewpoint) as one natural and realistic image. Further, when the background image is concealed, it is possible to provide a concealed background image that does not become monotonous in accordance with the change in the sound on the user side.

Further, according to the present invention, it is possible to improve the resolution of an important portion, and it is possible to automatically follow the change of the photographing environment.

Further, in the media input device according to the present invention, a portion such as a background portion in the input image that can reproduce an approximate image is determined if the input condition is known, and only the input condition information is determined for this portion. By transmitting / accumulating, the amount of information required for transmission / accumulation is greatly reduced. Further, since the input image / sound is decomposed for each component in the information to be transmitted / stored, a process of extracting only a specific component or replacing a certain component with another image / sound in the reproducing apparatus. Can be done easily.

Further, according to the present invention, since an image can be synthesized from limited information, a moving image can be expressed with very little information. Moreover, since a moving image is generated based on a high-definition artificial image, it is possible to generate a high-quality moving image without distortion. In addition, since a desired artificial image can be selected from the database, various synthetic images can be obtained for the same information source. Furthermore, since the movement of the composite image is corrected by the audio information, it is possible to obtain a composite image with no discomfort. Further, it is possible to synthesize an image by a line drawing instead of the artificial image.

Furthermore, according to the present invention, it is possible to accurately obtain information regarding the movement of the entire face and the information regarding the local movement of each part in the face. Therefore, when the face image is encoded, the movement of the face can be efficiently encoded. Further, in the application to model-based coding and man-machine interface, the facial movement can be naturally and reasonably analyzed as described above, so that performance improvement can be realized.

[Brief description of drawings]

FIG. 1 is a block diagram showing a schematic configuration of the present invention.

FIG. 2 is a block diagram showing a configuration of an encoding side and a decoding side of the first example according to the present invention.

FIG. 3 is a block diagram showing in detail an internal configuration of an encoder.

FIG. 4 is a block diagram showing an example of motion compensation.

FIG. 5 is a block diagram showing the configuration of another embodiment according to the present invention.

FIG. 6 is a block diagram showing a conventional configuration.

FIG. 7 is a block diagram showing a media processing system according to a second embodiment of the present invention.

FIG. 8 is a block diagram of a media processing device of another embodiment according to the second embodiment.

FIG. 9 is an explanatory diagram of an example of image combination.

FIG. 10 is a block diagram of a media processing device of another embodiment according to the second embodiment.

FIG. 11 is a block diagram of a media processing device of another embodiment according to the second embodiment.

FIG. 12 is a block diagram showing a conventional configuration.

FIG. 13 is a block diagram showing a configuration of a third exemplary embodiment of the present invention.

FIG. 14 is a diagram showing a first captured image.

FIG. 15 is a diagram showing a second captured image.

FIG. 16 is a diagram showing a composite image.

FIG. 17 is a block diagram showing another embodiment according to the third embodiment.

FIG. 18 is a diagram showing a face area.

FIG. 19 is a block diagram showing still another embodiment according to the third embodiment.

FIG. 20 is an external view showing an embodiment of an image pickup unit.

21 is a diagram showing a horizontal holding mechanism applied in the embodiment shown in FIG.

FIG. 22 is an external view showing still another embodiment according to the third embodiment.

FIG. 23 is a diagram showing a conventional example.

FIG. 24 is a diagram showing a pixel arrangement of a conventional example.

FIG. 25 is a block diagram showing a configuration of a media input device according to a fourth embodiment of the present invention.

FIG. 26 is a diagram illustrating an input image from the media input device of FIG. 25 and background / non-background separation.

27 is a diagram illustrating a configuration of an image reproducing device corresponding to the input device of FIG. 25.

FIG. 28 is a block diagram showing another embodiment of the media input device of the fourth embodiment.

FIG. 29 is a diagram illustrating an input image from the media input device shown in FIG. 28 and background / non-background separation.

30 is a block diagram showing a configuration of an image reproducing device corresponding to the input device shown in FIG. 28.

FIG. 31 is a block diagram showing an example of a media input device according to a conventional technique.

FIG. 32 is a block diagram showing an example of a media reproducing device according to a conventional technique.

FIG. 33 is a block diagram showing a configuration of a fifth exemplary embodiment of the present invention.

FIG. 34 is a flowchart of a process of dividing an input image into lines.

FIG. 35 is a diagram illustrating that line segmentation can be performed by an enlargement / reduction operation.

FIG. 36 is a diagram for explaining how to determine whether the eyes are open or closed.

FIG. 37 is a diagram showing the configuration of another example according to the fifth example of the present invention.

FIG. 38 is a diagram illustrating an example of a method of realizing a state correction unit.

FIG. 39 is a diagram illustrating a relationship between H and V.

FIG. 40 is a diagram showing a template table.

FIG. 41 is a diagram showing the configuration of another example of the fifth embodiment.

[Fig. 42] Fig. 42 is a diagram illustrating the appearance of noise in a line-differentiated image.

FIG. 43 is a diagram showing the configuration of still another example of the fifth example.

FIG. 44 is a block diagram illustrating a conventional method.

[Fig. 45] Fig. 45 is a diagram illustrating an input portion of an image encoding device that is a sixth embodiment according to the present invention.

[Fig. 46] Fig. 46 is a diagram illustrating image processing during decoding in the image encoding device.

[Fig. 47] Fig. 47 is a diagram for describing movement of the face and changes in the positions of the pupil and the oral cavity.

FIG. 48 is a diagram showing another embodiment of the image input device for taking a picture around the eyes.

FIG. 49 is a diagram showing an example of using this embodiment for a man-machine interface.

FIG. 50 is a diagram showing motion detection by a conventional method.

[Explanation of symbols]

 1 information source 3,110,201 limited input device 5,140,303 processing device 7,204,213 output device 120 encoder 130 motion amount setting circuit 150 decoder 203,207 processing unit 205 input device 208 background separation removal Part 209 Size setting part 210 Range setting part 211 Mirror image inverting part 212 Image selector 301, 401, 512 Camera 304 Display 402, 641 Microphone 407 Background extraction part 408 Background image / sound determination part 409 Image / sound database 411 Multiplexing part 513 State Acquisition means 514 Output image creation means 515 Artificial image database 516 Display device 642 Camera that captures the mouth 643 Camera that captures an image of the whole face 644 Camera that captures the area around the eye 645 Image of the entire face 647 Image around the eye 648 Image of the mouth 647 Recovery Image around the eye 648 Decoded image of the mouth 650 Image of the final decoded whole face 660 Input device for image around the eye 661 Camera 662 Lens (half mirror) 663 Camera for taking the entire face 664 Camera around the eyes that has a distance measurement function 665 Image of the entire face 666 Image around the eyes

 ─────────────────────────────────────────────────── ─── Continuation of the front page (72) Yoshihiro Kikuchi Yoshiko Kikuchi 1 Komukai Toshiba-cho, Sachi-ku, Kawasaki-shi, Kanagawa Toshiba Research & Development Center (72) Inventor Masahiro Oshikiri Komukai-Toshiba, Kawasaki-shi, Kanagawa 1 Incorporated Toshiba Research and Development Center (72) Inventor Susumu Kanba 1 Komukai Toshiba-cho, Sachi-ku, Kawasaki-shi, Kanagawa 1 Incorporated Toshiba Research and Development Center

Claims (8)

[Claims] 1. A plurality of input means for respectively limiting target information to acquire information, a processing means for integrating acquisition information based on attributes of the acquired acquisition information, and this processing means. And a means for outputting processing information obtained by being integrated with the media processing system. 2. A plurality of input means for respectively limiting target information and acquiring information, and a processing means for integrating the acquired information by removing information of low importance from the acquired information. A media processing system comprising: and an output unit that outputs processing information obtained by being integrated by the processing unit. 3. A plurality of image information inputting means for respectively acquiring image information related to the information object, and a detecting means for detecting a change with time of the information object included in the acquired acquired image information, A media processing system comprising: an output unit that outputs time-dependent change information obtained by the detection unit. 4. One or a plurality of first information inputting means, each of which obtains information including image information related to the information object, in a limited manner, and an image of an object for operating the first information inputting means as an information object. Second information input means for obtaining information including information, first information and second information obtained by the first information input means
A media processing system comprising: a synthesizing unit for synthesizing the second information acquired by the information inputting unit. 5. One or a plurality of first image information inputting means, each of which obtains image information related to an information object in a limited manner, and the first image information obtained by the first image information inputting means. Second, which acquires image information relating to an arbitrary area included
The second pixel value data acquired by the second image information input means is inserted between the image information input means and the first pixel value data acquired by the first image information input means. A media processing system having means processing. 6. A first input means for acquiring information on an information object, a candidate accumulating means for accumulating candidates for replacing a part of the information object, and an input condition for obtaining an input condition for inputting the information object. Acquisition means, second input means for acquiring information relating to the information object, at least the candidate accumulation based on the input condition acquired by the input condition acquisition means and the input information from the first input means. A media processing system comprising: means for selecting a source of replacement based on information from one of the means and the second input means, and replacing means for replacing a part of the information target input with the replacement information. 7. A plurality of image information input means for respectively acquiring image information of an information target, an acoustic information input means for acquiring acoustic information of the information target, the plurality of image information input means and acoustic information Status acquisition means for acquiring the status of each information object from the acquired information acquired by the input means, and correction means for correcting each part of the artificial image corresponding to the status of the information object acquired by the status acquisition means A media processing system comprising: 8. A first method for acquiring image information of an information object
Image input means, second image input means for acquiring image information of a part or all of the image area acquired by the first image input means, the first image input means and second image Extraction means for extracting at least one temporal change of the image information acquired by the input means, and image information acquired by the first image input means according to the temporal change extracted by the extracting means. Replacement means for partially replacing the image information acquired by the second image input means, and replacement of the image information by the replacement means, an information object included in the image information acquired by the first image input means. And a control means for controlling a part of the information object according to the overall movement of the device so as to correspond to the motion.