JP2009049873A - Information processing apparatus - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To spuriously produce 5.1ch sound information from 2ch input sound signals. <P>SOLUTION: An information processing apparatus comprises: a video frame buffer 100; a sound processing block 200 which inputs right and left 2ch sound signals and produces 4ch sound signals; a sound synthesizing block 300 which weights and synthesizes the 4ch sound signals supplied from the sound processing block to produce 5.1ch surround sound signals; and a sound synthesization control block 500 which includes an image recognizing function and controls a synthesization parameter to be used for synthesizing the 4ch sound signals in the sound synthesizing block 300 based on a result of recognizing image signals synchronized to sound signals. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to an information processing apparatus that records or reproduces and outputs information content including an image signal and an audio signal synchronized with the image signal, and particularly relates to an information processing apparatus that processes audio information corresponding to a surround reproduction environment.

  More specifically, the present invention provides information that provides a sense of reality such as 5.1ch surround by artificially creating 5.1ch of audio information when playing back content shot with a normal 2ch stereo microphone. The present invention relates to a processing apparatus and an information processing apparatus that artificially creates and records audio information for 5.1 ch in a video camera in which only a normal 2 ch stereo microphone is mounted.

  Home video cameras are already widely used. In recent years, an increasing number of digital video cameras digitally encode moving images and audio data and record and manage them as computer files such as DVDs (Digital Versatile Discs) and hard disks. Also, subject recognition processing is possible by combining image recognition technology with a digital camera, and camera work automation technology such as automatic aiming (AF) and automatic exposure (AE) according to the position and size of the subject image is advanced. It has been.

  On the other hand, as a playback system for moving images and image data, a surround playback technique is known in which a plurality of speakers are arranged around a viewing user at an assumed position, which is close to an actual sound source, that is, has a realistic feeling. As an audio data signal system that realizes a surround reproduction environment, for example, AC-3, which is a high-efficiency encoding system for digital multi-channel audio signals developed by Dolby Laboratories in the United States, is known. This surround playback system of AC-3 is, for example, a left channel speaker L arranged at the front left side of the viewer, a center channel speaker C arranged at the front center of the viewer, and the front right side of the viewer. The right channel speaker R and the surround channel speakers Ls and Rs respectively disposed on the left and right sides of the viewer (see FIG. 3), and the number of audio channels Is a 5.1 channel, which is a channel for the super low frequency (0.1 channel) added to the front left and right 2 channels, the front center 1 channel and the rear 2 channels.

  With the widespread use of large-capacity recording media such as DVDs, surround playback environments equipped with 5.1 channels such as home theaters are spreading into the home. In addition, products that can record audio with 5.1 channels have also appeared in home video cameras. When the content shot on the 5.1 channel is played back in the 5.1ch surround environment, the user can enjoy a sense of reality as if he / she was on the spot, which is very beneficial for the user.

  For example, a first plurality of audio signals that include a plurality of microphones and that process audio signals output from the plurality of microphones into a plurality of channels of audio signals, and all channels of audio output from the plurality of microphones. A second audio signal obtained by processing the signal into an audio signal of one channel is simultaneously recorded on the recording medium together with the video signal, and the recording is performed on the first plurality of audio signals and the second audio signal. A video camera has been proposed that can be reproduced independently of the audio signal (see, for example, Patent Document 1).

  In addition, at least four microphones M1, M2, M3, M4,... Mn arranged in the video camera so as to pick up sounds from at least four different directions at the time of shooting, and sounds output from the microphones, respectively. Output signals m 1, m 2, m 3, m 4,... Mn are synthesized, and the audio signal R from the front right, the audio signal L from the left front, the audio signal C from the front center, 3 Voice synthesis means for generating four-channel (Rch, Lch, Cch, Sch) voice signals composed of surround voice signals S from different directions and four-channel voice signals R output by the voice synthesis means , L, C, and S are converted into 2-channel audio data Lt and Rt according to a predetermined arithmetic expression, and the matrix encoder outputs the signal. And a recording device of a video camera having audio data recording means for recording the audio data Lt and Rt of the two channels Ltch and Rtch output from the matrix encoder on a recording medium, and the audio data to be recorded is Although it is the same two channels as before, matrix decoding at the time of reproduction enables multi-channel surround reproduction of four or more channels, and a realistic reproduction sound can be obtained (for example, refer to Patent Document 2). .

  However, for digital home cameras with limited pricing, there are various restrictions such as obtaining a license to support 5.1ch surround, and the multi-channel microphones (from 5 speakers to 5 speakers) There is a problem that it is difficult to arrange five microphones L, C, R, Ls, and Rs). For this reason, there are still many video cameras that can only record with two channels.

JP 2003-18543 A

JP-A-2005-223706

  An object of the present invention is to provide an excellent information processing apparatus capable of suitably processing audio information corresponding to a surround reproduction environment when recording or reproducing and outputting information including a moving image and audio. is there.

  It is a further object of the present invention to obtain a sense of reality such as 5.1ch surround by artificially creating 5.1ch of audio information when playing back content shot with a normal 2ch stereo microphone. An object of the present invention is to provide an excellent information processing apparatus.

  A further object of the present invention is to provide an excellent information processing apparatus capable of creating and recording audio information for 5.1 ch in a pseudo manner in a video camera having only a normal 2 ch stereo microphone mounted thereon. .

The present invention has been made in consideration of the above problems, and is an information processing apparatus that records or reproduces and outputs information content including an image signal and an audio signal synchronized therewith,
The input audio signals L and R consisting of two left and right channels are subjected to signal processing to generate an omnidirectional audio signal C, and further, an audio signal E with a specific effect is generated from the audio signal C to generate four channels. An audio processing block for outputting audio signals L, R, C, E;
The 4-channel audio signals L, R, C, and E output from the audio processing block are weighted and synthesized to correspond to the left-channel audio signal L corresponding to the front left side of the viewer and the front center of the viewer. A surround signal including a center channel audio signal C, a right channel audio signal R corresponding to the front right side of the viewer, and surround channel audio signals Ls and Rs corresponding to the left and right sides of the viewer respectively. A speech synthesis block for generating speech signals;
Image recognition means for recognizing an input image signal synchronized with the audio signal, and a synthesis parameter used when synthesizing the 4-channel audio signals L, R, C, E in the audio synthesis block based on the image recognition result; A speech synthesis control block to be controlled;
It is an information processing apparatus characterized by comprising.

  However, the voice synthesis block weights and synthesizes the four channels of audio signals L, R, C, and E output from the voice processing block to obtain a low-frequency dedicated channel (0.1 channel) for superwoofer driving. An audio signal LFE may be further generated to synthesize and output a 5.1 channel audio signal.

  The audio processing block generates an audio signal E having a specific filter effect applied from the audio signal C by an audio filter. The audio filter specifically includes a bandpass filter that passes only components in a specific frequency band. Consists of filters.

  As a moving image and image data reproduction system, for example, AC-3 developed by Dolby Laboratories in the United States, a plurality of speakers are arranged around the viewing user, and it is close to a real sound source, that is, has a sense of presence. A surround reproduction technique with a 5.1 channel configuration is known. For the user, it is possible to enjoy a sense of presence as if they were there, and the benefits are great.

  However, for home digital cameras with limited pricing, there are various restrictions such as license acquisition to support 5.1 channel surround, and multi-channel microphones can be arranged from the set shape. There is a problem that it is difficult.

  On the other hand, the information processing apparatus according to the present invention uses the image recognition information for pseudo audio information for 5.1 ch when reproducing or recording content shot with a normal 2 ch stereo microphone. Therefore, it is possible to obtain a sense of reality such as 5.1ch surround from AV contents such as those obtained with a video camera having only a two-channel microphone.

  Specifically, first, the audio processing block applies signal processing to the input audio signals L and R consisting of the right two channels to generate an omnidirectional audio signal C, and further applies a specific effect from the audio signal C. A four-channel audio signal L, R, C, E is generated, and then the active synthesis block corresponds to the front center of the viewer from these four-channel audio signals L, R, C, E. Center channel audio signal C, right channel audio signal R corresponding to the front right side of the viewer, and surround channel audio signals Ls and Rs corresponding to the left and right sides of the viewer respectively, A total of 5.1 channels of the audio signal LFE of the low-frequency dedicated channel (0.1 channel) for super woofer driving is synthesized. The speech synthesis control block controls synthesis parameters used when the speech synthesis block synthesizes the 4-channel speech signals L, R, C, and E based on the image recognition result of the input image signal synchronized with the speech signal. To do.

  The voice synthesis block is used when, for example, four voice signals L, R, C, and E are synthesized by the voice synthesis block based on the position and size of the subject in the screen recognized by the image recognition unit. The synthesis parameter to be used may be determined.

  Further, the speech synthesis block may control the speech filter used when creating the speech signal E having a specific effect from the omnidirectional speech signal C in the speech processing block. For example, the characteristics of the audio filter in the audio processing block may be determined based on the number or type of subjects recognized by the image recognition means.

  According to the present invention, when content captured with a normal 2ch stereo microphone is reproduced, 5.1ch sound information is artificially created using image recognition information, such as 5.1ch surround. An excellent information processing apparatus that can provide a sense of reality can be provided.

  Further, according to the present invention, in a video camera in which only a normal 2ch stereo microphone is mounted, excellent information that can create and record audio information for 5.1ch in a pseudo manner using image recognition information. A processing device can be provided.

  Other objects, features, and advantages of the present invention will become apparent from more detailed description based on embodiments of the present invention described later and the accompanying drawings.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

  FIG. 1 schematically shows the configuration of an information processing apparatus according to an embodiment of the present invention. This information processing apparatus, for example, inputs a video signal and a 2ch audio signal from a DVD playback apparatus and outputs the surround playback, or inputs a video signal and an audio signal from a video camera having only a 2ch stereo microphone for surround playback. A process for performing the corresponding recording is executed.

  As shown in FIG. 1, the information processing apparatus includes a video frame buffer 100, an audio processing block 200 that inputs left and right 2ch audio signals to generate a 4ch audio signal, and 4ch supplied from the audio processing block 200. A voice synthesis block 300 that generates a 5.1ch surround sound signal by weighting and synthesizing the voice signals of the audio signal, and a voice synthesis block 300 that has an image recognition function and recognizes an image signal that is synchronized with the voice signal. Is composed of a speech synthesis control block 500 for controlling synthesis parameters used when synthesizing the speech signal.

  The video frame buffer 100 temporarily stores a transmitted video signal for image recognition. The video signal is a playback video signal supplied from a DVD playback device (not shown) or the like, or a video signal shot by a video camera (not shown).

  The audio processing block 200 performs signal processing such as superposition or synthesis of the input left and right audio signals L and R to generate an omnidirectional audio signal C, and further converts the omnidirectional audio signal C into the omnidirectional audio signal C. An audio signal E with a specific effect is produced. Then, the audio processing block 200 outputs the omnidirectional audio signal C and 4ch of the audio signal E obtained by applying a specific effect to the audio signal C together with the left and right 2ch audio signals L and R to the subsequent audio synthesis block 300. To do.

  The audio signal E is an audio signal obtained by extracting only a specific component from the omnidirectional audio signal C through an audio filter. In addition, the audio signal E is somewhat delayed when it passes through the audio filter, but in order to maintain the synchronism of all 4ch audio signals, the other 3ch audio signals L, R, and C are placed on the transmission path. Appropriate delay elements are arranged.

  The audio filter for generating the audio signal E is constituted by, for example, a bandpass filter (BPF), and allows only a line segment of a specific frequency band to pass from the omnidirectional audio signal C. For example, an audio filter can be configured using a bandpass filter that passes only a male voice band.

  Further, the frequency characteristics of the voice filter need not be constant, and the voice synthesis control block 500 having an image recognition function controls the frequency characteristics based on the image recognition result of the image signal input in synchronization with the voice signal. You may make it do. For example, when the image recognition block 550 recognizes an image signal temporarily held in the video frame buffer 100 and recognizes that the (main) subject is a male, for example, only the male voice band is used. An audio filter may be set in the frequency band to pass.

  Further, the voice synthesis control block 500 switches the frequency characteristics of the voice filter in accordance with the number of subjects (target) recognized by the image recognition block 550, so that the voice processing block 200 has a plurality of voice signals E. May be generated. For example, when an adult male and a child are recognized in one screen, two types of audio signals E1 and E2 are generated by extracting the band of the adult male voice and the band of the child voice by two types of band-pass filters, respectively. Then, it is output to the subsequent speech synthesis block 300. In this case, since the audio signal between the audio processing block 200 and the audio synthesis block 300 has a variable number of signals, the audio signal may be exchanged with digital data.

The speech synthesis block 300 corresponds to the left channel audio signal L corresponding to the front left side of the viewer and the front center of the viewer from the 4ch audio signals L, R, C and E output from the audio processing block 200. A total of 5.1 channels including a center channel audio signal C, a right channel audio signal R corresponding to the front right side of the viewer, and surround channel audio signals Ls and Rs respectively corresponding to the left and right sides of the viewer. The surround sound signal is synthesized. Thereby, pseudo 5.1ch recording or pseudo 5.1ch recording can be realized. Specifically, the speech synthesis block 300 is based on each of the four synthesis parameters P _L , P _R , P _C , and P _E that the speech synthesis control block 500 determines based on the image recognition result by the image recognition block 550. Then, the 4ch audio signals L, R, C, and E are weighted and synthesized by the following equation to calculate the 5.1ch audio signals L, R, C, Ls, and Rs, respectively.

  The speech synthesis control block 500 accesses the memory from the signal line 600, analyzes the image in the video frame buffer 100 by the image recognition block 560, and recognizes attribute information (object type, position, The synthesis parameters of the speech synthesis block 300 are changed at any time using the signal line 610 based on the result of creating the size. Also, the speech synthesis control block 500 determines the frequency of the speech filter in the speech processing block 200 via a signal line (not shown) according to the target attribute information (subject type, sex, age, etc.) recognized in the image. The characteristics may be determined (described above).

  In the example shown in FIG. 1, the speech synthesis control block 500 includes a processor 510, a ROM (Read Only Memory) 520, a RAM (Random Access Memory) 530, an input / output interface 540, and a signal line 600. An image recognition block 550 capable of recognizing a video image in the buffer 100 and a bus 560 connecting these to each other.

  The processor 510 executes a process for changing the synthesis parameters used at the time of weight synthesis of the 4ch speech signals L, R, C, and E in the speech synthesis block 300 by executing a predetermined program. It is set in the speech synthesis block 300 via 610. Further, the processor 510 executes a process for changing the characteristics of the sound filter in the sound processing block 200 as needed by executing a predetermined program, and sets the sound processing block 200 via a signal line (not shown).

  The ROM 520 is a memory that holds a program executed by the processor 510, various parameters, and the like, and includes, for example, an EEPROM such as a flash memory. The program stored in the ROM 520 includes the above-described algorithm for changing the synthesis parameter when the audio signal is weight-synthesized in the audio synthesis block 300 and the characteristics of the audio filter in the audio processing block 200. A program for implementing the algorithm is included.

  The RAM 530 is a memory that holds work data and the like necessary for program execution in the processor 510. The RAM 530 includes, for example, a readable / writable memory device such as SRAM (Static RAM) or DRAM (Dynamic RAM). It is used as a memory.

  The input / output interface 540 implements an interface protocol when data is exchanged with an external device (not shown), and is used, for example, for updating a program in the ROM 520.

  The image recognition block 560 accesses the memory via the signal line 600, analyzes the image in the video frame buffer 100, and recognizes attribute information (object type, position, size, etc.) of the recognition target in the image. create. In the image recognition block 560, face recognition for detecting and recognizing a subject is particularly applied. The face recognition process includes, for example, a face detection process that detects the position of a face image and extracts it as a detected face, a face organ detection process that detects the position of a main facial organ from the detected face, and identification of a detected face (person's Specific face) processing. However, the gist of the present invention is not limited to a specific image recognition technique, and will not be described further in this specification.

As described above, the processor 510 executes a predetermined program to weight-synthesize the 4ch audio signals L, R, C, and E in the audio synthesis block 300, thereby generating the 5.1ch audio signals L, R, A process for changing the four synthesis parameters P _L , P _R , P _C , and P _E used when generating C, Ls, and Rs as needed is performed. As an algorithm for changing the synthesis parameter, there is a method of determining the synthesis parameter based on the position and size of the object (subject) in the screen recognized by the image recognition block 560.

Here, when a person (or a target such as a pet such as a dog or a specific mechanical device such as a car) is detected in the screen by the image recognition block 560, the 5.1ch audio signals L, R, C, and Ls are detected. , Rs are respectively determined as shown in Table 1 below, for example, as synthesis parameters P _L , P _R , P _C , and P _E used for generating Rs.

  The variables α, β, γ, and δ included in the above table are determined according to the position of the target detected in the screen. For example, as shown in FIG. 2, assuming that the distance from the center of the screen to the object is a and the distance from the left end of the screen to the center of the screen is l, variables α, β, γ, and δ are determined by the following equations, respectively. be able to. However, the five speakers L, C, R, Ls, and Rs in the figure are assumed to be arranged in the 5.1ch surround reproduction system.

On the other hand, when an object such as a person is not detected in the screen by the image recognition block 560, the synthesis parameters P _L and P _R used to generate the 5.1ch audio signals L, R, C, Ls, and Rs, respectively. , P _C , and P _E are determined as shown in Table 2 below, for example.

  The present invention has been described in detail above with reference to specific embodiments. However, it is obvious that those skilled in the art can make modifications and substitutions of the embodiment without departing from the gist of the present invention.

  The information processing apparatus according to the present invention reproduces content captured with a normal 2ch stereo microphone in a pseudo 5.1ch manner, or in a video camera in which only a normal 2ch stereo microphone is mounted, It can be applied when recording in 1ch.

  In short, the present invention has been disclosed in the form of exemplification, and the description of the present specification should not be interpreted in a limited manner. In order to determine the gist of the present invention, the claims should be taken into consideration.

FIG. 1 is a diagram schematically illustrating a configuration of an information processing apparatus that records or reproduces and outputs information including a moving image and sound according to an embodiment of the present invention. FIG. 2 shows synthesis parameters P _L , P _R , P _C , and the like used when generating 5.1ch audio signals L, R, C, Ls, and Rs when a target such as a person is detected in the screen. It is a figure for demonstrating the calculation formula for determining _PE . FIG. 3 is a diagram schematically showing the configuration of an AC-3 surround playback system.

Explanation of symbols

DESCRIPTION OF SYMBOLS 100 ... Video frame buffer 200 ... Speech processing block 300 ... Speech synthesis block 500 ... Speech synthesis control block 510 ... Processor 520 ... ROM
530 ... RAM
540 ... I / O interface 550 ... Image recognition block 560 ... Bus 600, 610 ... Signal line

Claims (11)

An information processing apparatus for recording or reproducing and outputting information content including an image signal and an audio signal synchronized with the image signal,
The input audio signals L and R consisting of two left and right channels are subjected to signal processing to generate an omnidirectional audio signal C, and further, an audio signal E with a specific effect is generated from the audio signal C to generate four channels. An audio processing block for outputting audio signals L, R, C, E;
The 4-channel audio signals L, R, C, and E output from the audio processing block are weighted and synthesized to correspond to the left-channel audio signal L corresponding to the front left side of the viewer and the front center of the viewer. A surround signal including a center channel audio signal C, a right channel audio signal R corresponding to the front right side of the viewer, and surround channel audio signals Ls and Rs corresponding to the left and right sides of the viewer respectively. A speech synthesis block for generating speech signals;
Image recognition means for recognizing an input image signal synchronized with the audio signal, and a synthesis parameter used when synthesizing the 4-channel audio signals L, R, C, E in the audio synthesis block based on the image recognition result; A speech synthesis control block to be controlled;
An information processing apparatus comprising: The audio processing block generates an audio signal E having a specific filter effect applied from the audio signal C by an audio filter.
The information processing apparatus according to claim 1. The audio filter is composed of a bandpass filter that allows only a specific frequency band component to pass.
The information processing apparatus according to claim 2. The voice synthesis control block controls the frequency characteristics of the voice filter based on an image recognition result of the image recognition means of an image signal input in synchronization with a voice signal.
The information processing apparatus according to claim 3. The speech synthesis control block switches the frequency characteristics of the speech filter according to the number of subjects recognized by the image recognition means, and generates a plurality of speech signals E by the speech synthesis block.
The information processing apparatus according to claim 4. The speech synthesis block weights and synthesizes the 4-channel audio signals L, R, C, and E output from the audio processing block, and the audio signal of the low-frequency dedicated channel (0.1 channel) for driving the superwoofer. Generate more LFE,
The information processing apparatus according to claim 1. The speech synthesis block determines the characteristics of the speech filter in the speech processing block based on the number or type of subjects recognized by the image recognition means.
The information processing apparatus according to claim 1. The speech synthesis block uses synthesis parameters used when the speech synthesis block synthesizes 4-channel audio signals L, R, C, and E based on the position of the subject in the screen recognized by the image recognition unit. decide,
The information processing apparatus according to claim 1. The speech synthesis block is a synthesis parameter used when the speech synthesis block synthesizes 4-channel audio signals L, R, C, and E based on the size of the subject in the screen recognized by the image recognition unit. To decide,
The information processing apparatus according to claim 1. A moving image recording means for recording the surround sound signal generated by the sound synthesis block in synchronization with the input image signal;
The information processing apparatus according to claim 1. It further comprises a moving image reproduction means for reproducing and outputting the surround audio signal generated by the audio synthesis block in synchronization with the input image signal.
The information processing apparatus according to claim 1.

Images