JP2007158396A - Video/audio synchronization transmission apparatus - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a video/audio synchronization transmission apparatus capable of transmitting an audio signal and a video signal of an object image synchronously with each other by clarifying a causal relationship of audio corresponding to the object image on a screen in order that a viewer/listener can particularize a sound source on the screen. <P>SOLUTION: The video/audio synchronization transmission apparatus includes: an identification/position generating means for generating an identifier used to identify the object image for outputting sound from the video signal and positional information for denoting the position of the object image on the screen; an identification information attaching means for attaching the identifier of the object image and the positional information to the video signal; and a synchronizing signal composite means for multiplexing the detected audio signal on the video signal for the object image at the position on the basis of the positional information and providing an output of the resulting signal. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to a video / audio synchronous transmission apparatus for transmitting a subject image and its sound in synchronization when transmitting separately generated video signals and audio signals.

  In recent years, in theaters, movie theaters, and the like, an image based on an image signal is projected and displayed on a large screen. Also, in home AV systems, the increase in screen size is progressing with flat displays and the like. On the other hand, with respect to the sound accompanying such a large screen image, a method using a multi-channel speaker such as stereo or 5.1 channel surround has been applied in order to give a sense of reality. For example, in the 2-channel stereo system, audio signals transmitted in a television program and audio signals recorded on a recording medium are transmitted or recorded separately for each of the two channels, and each channel is output by left and right speakers. This makes it possible to feel the spread of sound on the left and right, albeit planarly. However, if there is a bias in the position of the viewer with respect to the screen, the sound of either the left or right speaker becomes dominant, and it may occur that the spread of the sound is not felt so much. In order to improve this, the 5.1 channel surround system uses three speakers: a speaker placed at the front, right front, left front, right rear, and left rear and a subwoofer speaker for low-frequency output that is usually placed at the front. Realizes a realistic and realistic sound environment.

  However, these methods pursue the spread of sound in the horizontal direction, and do not take much consideration into the spread of sound in the vertical direction, and when used in a video display system with a very large screen, Sound source (position of sound in the screen, position of subject image producing sound, etc.) and actual sound source (speaker) are separated, resulting in unnatural sound reproduction for the video There was a problem that the sense of reality was impaired. To deal with such problems, a plurality of speakers are arranged in a matrix on the back of the screen for video display, and a control means for matching the position of the sound source on the screen with the position of the speakers is provided. There has been proposed a method of providing a certain video display screen (see, for example, Patent Document 1). However, Patent Document 1 discloses in detail how to control the audio signal for specifying the position of the sound source on the screen when reproducing the audio signal recorded on the recording medium. Absent.

  On the other hand, when video and audio signals such as TV programs stored in recording media such as HDDs and DVDs are encoded, a large amount of information is allocated to video scenes and subjects of particular interest on the screen. There is a method for efficiently using the encoded rate and improving the visual image quality. In this method, for example, there is a method of detecting a person as one of the points of interest. However, when there are a plurality of persons, the allocation of information amount to them is dispersed, so that it is difficult to expect improvement in visual image quality. . In such a case, if it is possible to discriminate the subject that is producing the sound, it is considered that the information is useful when the information amount is allocated on the screen.

JP-A-2-224495

  The video and audio playback method in the large-screen video display system disclosed in Patent Document 1 is performed as described above, but has the following problems. Regarding video and audio synchronization methods for television programs and the like, generally, a video signal is extracted as a separate signal through a camera and a sound signal is extracted as a separate signal through a microphone, and is multiplexed on the transmission path as a separate signal. In this case, when playing back on an acoustic system on a large screen, or in encoding control that allocates information efficiently within the screen, a state occurs in which the causal relationship between the subject image on the screen and the sound is not clear, and viewing There is a problem that the person who does it cannot identify the sound source on the screen.

  The present invention has been made to solve the above problems, and in order to enable a viewer to specify a sound source on the screen, the causal relationship between the subject image on the screen and the corresponding sound is clarified. Another object of the present invention is to provide a video / audio synchronous transmission apparatus that enables transmission of a subject image of an audio signal and a video signal in synchronization.

  The video / audio synchronous transmission device according to the present invention includes a video signal created separately at the time of video content production, and an identifier associated with each of the subject images that emit audio appearing on the screen of the video signal. And an audio / synchronous transmission device that synchronizes and transmits the audio signal to the utilization device, and generates an identification signal representing an identifier for identifying a subject image that emits audio from the input video signal, Identification / position generation means for generating position information representing the position of the subject image on the screen, identification information addition means for adding the generated identification signal and position information to the video signal, and identification added to the video signal Based on the position information added to the video signal, an audio signal having an identifier corresponding to the identifier of the subject image is detected from the input audio signal based on the signal. , In which a synchronization signal synthesizing means for outputting as a multiplexed AV signal by multiplexing the detected speech signal to the position of the object image of the video signal.

  According to the present invention, when a video is displayed on a large-screen video display system, an audio signal separated from the AV synthesized signal is given to the acoustic system, and the position information of the sound source on the screen is used as if on the playback screen. This can be used to generate sound corresponding to the subject image from the subject image.

Embodiment 1 FIG.
FIG. 1 is a block diagram showing a functional configuration of a video / audio synchronous transmission apparatus according to Embodiment 1 of the present invention.
First, here, a subject that emits sound when recording a TV program or the like (all those that emit sound of people, animals, etc.) and a sound emitted by the subject are associated with each other. As shown in FIG. 2, for example, when there are three persons as subjects that emit sound, marks a 1, a 2, and a 3 that can be identified in image processing are attached to the actual subjects in advance. For example, a bar code or a light emitter is used for this mark. Accordingly, the video signal 101 obtained from the camera includes a subject image with this mark. In addition, if a mark is identifiable to a portion that emits sound when a moving picture animation is produced by taking a piece with a camera, it can be handled in the same manner as the subject.
On the other hand, as shown in FIG. 1, during the post-recording, audio signals (encoded digital data) 104 to 106 input from microphones provided for each person (subject) are included in the audio signals emitted by each person and each person. Is made to match. That is, the identifier b1 is assigned to the audio signal 104 of the subject with the mark a1, the identifier b2 is assigned to the audio signal 105 of the subject with the mark a2, and the identifier is assigned to the audio signal 106 of the subject with the mark a3. b3 is added in association with each other.

  In FIG. 1, a subject recognition unit (identification / position generation means) 1 recognizes a mark previously attached to an actual subject as described above from a corresponding subject image for an input video signal (analog signal) 101. Thus, the identification signal 102 including an identifier for identifying the subject image corresponding to the mark is generated, and the position information 103 indicating the position of the mark on the screen is generated. The AD conversion / encoding unit (identification information adding means) 10 digitally converts the video signal (analog signal) 101, encodes it by a predetermined method, adds the identification signal 10 and the position information 103, and adds the video signal ( (Digital data) 111 for outputting. Based on the identification signal 102 and the position information 103 input from the subject recognition unit 1, the synchronization signal synthesis unit 2 generates an audio signal having an identifier corresponding to the identifier of the subject image from the input audio signals 104 to 106. Based on the position information detected and added to the video signal, the detected audio signal is multiplexed with the position of the subject image of the video signal 111 and output as an AV multiplexed signal.

Next, the operation will be described.
First, the subject recognition unit 1 identifies a mark attached to an actual subject as shown in FIG. 2 from a video signal (analog signal) 101 input from the camera, and is a subject image corresponding to the mark. An identification signal 102 composed of an identifier representing this is generated. In this case, the identifier of the subject image is generated in association with the identifier of the audio signal input from the microphone. At this time, the subject recognizing unit 1 generates position information (address) 103 that represents the position on the screen where the mark is attached (or the position of the subject image attached with the mark) in pixel units. The video signal (analog signal) 101 input from the camera is digitally converted by the AD conversion / encoding unit 10 and then encoded by a predetermined method, and the identification signal 10 and the position information 103 are added. Are output as video signals (digital signals) 111.

  The video signal 111 and the after-recording audio signals 104 to 106 inputted from the microphone are separately inputted to the synchronization signal synthesis unit 2. First, the synchronization signal synthesizer 2 detects an audio signal having an identifier corresponding to the identifier of the subject image from the audio signals 104 to 106 based on the identification signal 102 added to the video signal 111, and generates a video pixel. Extraction is performed in association with a unit or a plurality of pixel units. Next, the synchronization signal synthesizer 2 multiplexes the audio signal associated with the pixel unit or the plurality of pixel units at the position of the video signal 111 with the mark indicated by the position information 103 corresponding to the identification signal 102, The signal is output as an AV multiplexed signal 107.

  The method of multiplexing the video signal and the audio signal as described above is represented as an image on the screen as shown in FIG. That is, an audio signal having a causal relationship with the subject image is assigned to the position of the subject image on the screen as a sound source of the audio signal in units of pixels or a plurality of pixels, and the audio signal is multiplexed with the video signal. . In this case, the audio signal is not at a fixed rate.

  As described above, according to the first embodiment, the video signal and the audio signal of a television program or the like are not transmitted separately, but the subject image is corresponding to the position of the subject image on the screen serving as a sound source. By assigning audio signals in units of pixels or a plurality of pixels, causal video signals and audio signals are multiplexed in perfect synchronization and transmitted as AV composite signals. Therefore, when a video is displayed on a large-screen video display system, an audio signal separated from the AV composite signal is given to the acoustic system, and the position information (address) of the sound source on the screen is used, as if on the playback screen. This can be used to generate sound corresponding to the subject image from the subject image. Further, when encoding, it is possible to realize efficient allocation of information amount within the screen. Furthermore, when a subject is photographed with a camera, a mark that can be identified for image processing is attached to the subject and the position on the screen of the subject that is the sound source of the audio signal can be determined by recognizing the mark. It is possible to automatically create an AV composite signal.

Embodiment 2. FIG.
FIG. 4 is a block diagram showing a functional configuration of a video / audio synchronous transmission apparatus according to Embodiment 2 of the present invention. In the figure, portions corresponding to those in FIG. 1 are denoted by the same reference numerals, and description thereof will be omitted in principle. The configuration of the second embodiment includes an audio signal multiplexing unit 3 instead of the synchronization signal synthesis unit 2 of the first embodiment.
The audio signal multiplexing unit 3 is a video signal input from the camera based on the identification signal 102 including the identifier of the subject image input from the subject recognition unit (identification / position generation means) 1 and the position information 103 of the subject image. 101 is associated with the audio signals 104 to 106 input from the microphones for after-recording corresponding to the respective subjects, and the identifiers of the subject images that are causally related to the audio signals are multiplexed to generate an audio multiplexed signal. Means for generating and outputting 108. In the second embodiment, the AD conversion / encoding unit 10 may add only the identification signal to the video signal 111.

Next, the operation will be described.
In the audio signal multiplexing unit 3, when the audio signals 104 to 106 from the microphone corresponding to each subject are input, the audio signals 104 to 106 are first based on the identification signal 102 generated by the subject recognition unit 1. An audio signal having an identifier corresponding to the identifier of the subject image is detected from 106. Next, the subject image identifier and the position information (address) 103 of the subject image obtained from the subject identification unit 1 are multiplexed on the detected sound signal and output as a sound multiplexed signal 108. That is, position information indicating where the subject image serving as the sound source is on the screen is multiplexed for each audio signal. The thus generated audio multiplexed signal 108 obtained by multiplexing the position information (address) of each subject image is transmitted as a separate signal from the video signal 111 to the utilization device.

  As described above, according to the second embodiment, the video signal and the audio signal of a television program or the like are transmitted as separate signals as in the prior art, but the position information of the sound source on the screen is included in the audio signal. When displaying video on a large-screen video display system, the audio signal is given to the acoustic system, and the sound source position information detected from the audio multiplexed signal is used as if it were on the playback screen. Can be used to generate sound corresponding to the subject image from the subject image. Further, as in the first embodiment, it is possible to realize efficient information amount allocation in the screen when encoding.

Embodiment 3 FIG.
In the second embodiment, the position information (address) of the subject image on the screen of the subject that is the sound source is multiplexed with the sound signal to generate the sound multiplexed signal. However, when the sound multiplexed signal is generated. In addition, position information (address) may be embedded in the audio signal using a digital watermark technique. In this case, the audio signal multiplexing unit 3 in FIG. 4 has a function of embedding the position information (address) of the subject image on the screen by the digital watermark technique. Also by this, the same effect as in the second embodiment can be obtained.

Embodiment 4 FIG.
FIG. 5 is a block diagram showing a functional configuration of a video / audio synchronous transmission apparatus according to Embodiment 4 of the present invention. In the figure, portions corresponding to those in FIG. 1 are denoted by the same reference numerals, and description thereof will be omitted in principle. The configuration of the fourth embodiment includes a video signal multiplexing unit 4 instead of the synchronization signal synthesis unit 2 of the first embodiment.
Based on the identification signal 102 and position information 103 input from the subject recognition unit 1, the video signal multiplexing unit 4 outputs an audio signal having an identifier corresponding to the identifier of the subject image from the input audio signals 104 to 106. It is a means for detecting and embedding an identifier added to the detected audio signal in the video signal 111 from the identification information adding means as digital watermark data and outputting it as a video multiplexed signal.

Next, the operation will be described.
When the video signal 111 is input to the video signal multiplexing unit 4, the video signal 111 is input from a microphone having an identifier corresponding to the identifier of the subject image among the input audio signals based on the identification signal 102 included therein. Detects a voice signal. Next, an identifier added to the detected audio signal is extracted, embedded in the video signal 111 as digital watermark data, and output as a video multiplexed signal 109. The video multiplexed signal 109 generated by the video signal multiplexing unit 4 and the audio signals 104 to 106 of the microphones are transmitted as separate signals to the utilization device.

  As described above, according to the fourth embodiment, the video signal and the audio signal of a television program or the like are transmitted as separate signals as in the prior art, but the position information of the sound source on the screen is added. Since the identifier of the audio signal corresponding to the sound source is multiplexed and sent to the recorded video signal, when displaying the video on the large-screen video display system, the subject image as the sound source detected from the video multiplexed signal is displayed. Using the position information, a corresponding audio signal can be detected and applied to the acoustic system, and can be used for generating sound corresponding to the subject image from the subject image on the playback screen. Further, when encoding, it is possible to realize efficient allocation of information amount within the screen.

Embodiment 5. FIG.
FIG. 6 is a block diagram showing a functional configuration of a video / audio synchronous transmission apparatus according to Embodiment 5 of the present invention. In the figure, portions corresponding to those in FIG. 1 are denoted by the same reference numerals, and description thereof will be omitted in principle. The configuration of the fifth embodiment includes a subject identification data generation unit (identification / position generation unit) 5 instead of the subject recognition unit 1 of the first embodiment.
In the case of the fifth embodiment, the method of associating the subject that emits sound (all those that emit sound such as people and animals) and the sound emitted by the subject is different from the case of each of the above embodiments. Here, the video of the video signal (analog) 111 input from the camera or the video storage device is usually not provided with marks a1, a2, and a3 that can be identified in image processing for a subject that emits sound. It is an image of.
The subject identification data generation unit 5 includes an identifier for identifying the subject image by reproducing the input video signal 101 on a monitor and bringing the input pen into contact with the subject image serving as a sound source on the screen. A means for generating an identification signal and generating position information 103 representing the position of the subject image on the screen.

Next, the operation will be described.
When the video signal 111 is input from the camera or the video storage device, the subject identification data generation unit 5 reproduces the video signal 111 on the monitor. The program producer brings the input pen into contact with the subject image that is the sound source on the screen. When there are a plurality of subject images, the mode for specifying the subject image of the input pen is switched for each subject image, or the input pens of different modes are used by a plurality of people. As a result, the subject identification data generating unit 5 generates an identifier representing the subject image and generates the identification signal 102. At this time, position information 103 indicating where the subject image is on the screen is generated and input to the AD conversion / encoding unit 10 together with the identification signal 102. The subsequent processing is the same as in the first embodiment.

As described above, according to the fifth embodiment, the normal image signal is used to generate the identification signal and the position information of the subject image serving as the sound source existing on the screen. It is also possible to support voice-dubbing of unmarked video and video content obtained from foreign countries. In addition, as in the above embodiments, when the method of marking a subject itself is used, the mark may not be recognized depending on the perspective and direction of the subject. In this case, since the designation process is performed on the portion of the subject image that appears on the screen, it is accompanied by manual work, but it is possible to assign an identifier that reliably identifies the subject subject.
Note that the same effect can be obtained by applying the subject identification data generation unit 5 to any one of the second to fourth embodiments.

It is a block diagram which shows the function structure of the video / audio synchronous transmission apparatus by Embodiment 1 of this invention. It is explanatory drawing which shows the method of associating the to-be-photographed object and audio | voice based on Embodiment 1 of this invention. It is explanatory drawing which shows the method of multiplexing the video signal of the image specified on the screen concerning Embodiment 1 of this invention, and the audio | voice signal corresponding to it with an image. It is a block diagram which shows the function structure of the video / audio synchronous transmission apparatus by Embodiment 2 of this invention. It is a block diagram which shows the function structure of the video / audio synchronous transmission apparatus by Embodiment 4 of this invention. It is a block diagram which shows the function structure of the video / audio synchronous transmission apparatus by Embodiment 5 of this invention.

Explanation of symbols

  1 subject recognition unit, 2 synchronization signal synthesis unit, 3 audio signal multiplexing unit, 4 video signal multiplexing unit, 5 subject identification data generation unit, 10 AD conversion / encoding unit

Claims (7)

Video signals created separately during video content production and the audio signals with identifiers associated with each of the subject images that emit audio appearing on the screen of the video signal are synchronized and transmitted to the user device A video / audio synchronous transmission device,
An identification / position generating means for generating an identification signal representing an identifier for identifying a subject image that emits sound from an input video signal, and for generating position information representing the position of the subject image on the screen;
Identification information adding means for adding the generated identification signal and position information to the video signal;
Based on the identification signal added to the video signal, an audio signal having an identifier corresponding to the identifier of the subject image is detected from the input audio signal, and based on the position information added to the video signal, A video / audio synchronous transmission apparatus comprising: synchronization signal synthesis means for multiplexing the detected audio signal at the position of the subject image of the video signal and outputting it as an AV multiplexed signal.   2. The video according to claim 1, wherein when the audio signal is multiplexed at the position of the subject image of the video signal, the synchronization signal synthesizing unit multiplexes the audio signal in units of a pixel or a plurality of pixels of the video. -Voice synchronous transmission device. Video signals created separately during video content production and the audio signals with identifiers associated with each of the subject images that emit audio appearing on the screen of the video signal are synchronized and transmitted to the user device A video / audio synchronous transmission device,
Identification / position generating means for generating an identification signal representing an identifier for identifying a subject image that emits sound and position information representing a position of the subject image on the screen from the input video signal;
Identification information adding means for adding the generated identification signal to the video signal;
Based on the identification signal generated by the identification / position generation means, an audio signal having an identifier corresponding to the identifier of the subject image is detected from the input audio signal, and the identification / Audio signal multiplexing means for multiplexing the position information of the subject image generated by the position generation means and outputting as an audio multiplexed signal;
A video / audio synchronous transmission apparatus for separately transmitting a video signal from the identification information adding means and an audio multiplexed signal from the audio signal multiplexing means.   4. The video / audio synchronous transmission apparatus according to claim 3, wherein the audio signal multiplexing means embeds the position information of the subject image in the audio signal as digital watermark data. Video signals created separately during video content production and the audio signals with identifiers associated with each of the subject images that emit audio appearing on the screen of the video signal are synchronized and transmitted to the user device A video / audio synchronous transmission device,
Identification / position generating means for generating an identification signal representing an identifier for identifying a subject image that emits sound and position information representing a position of the subject image on the screen from the input video signal;
Identification information adding means for adding the generated identification signal and position information to the video signal;
Based on the identification signal added to the video signal from the identification information adding means, an audio signal having an identifier corresponding to the identifier of the subject image is detected from the input audio signal and added to the detected audio signal. Video signal multiplexing means for embedding the identifier thus obtained as digital watermark data in the video signal from the identification information adding means and outputting as a video multiplexed signal;
A video / audio synchronous transmission apparatus for separately transmitting a video signal from the video signal multiplexing means and the inputted audio signal.   The identification / position generation means recognizes, from the input video signal, an identification signal comprising an identifier for recognizing a mark previously attached to the actual subject from the corresponding subject image and identifying the subject image corresponding to the mark. 6. The image according to claim 1, wherein the image is generated and subject recognition means for generating position information representing the position of the subject image on the screen from the position of the mark. -Voice synchronous transmission device. The identification / position generation means reproduces an input video signal on a monitor, and an identification signal comprising an identifier for identifying the subject image by bringing the input pen into contact with the subject image serving as a sound source on the screen 6. The video image data according to claim 1, wherein the object identification data generating unit generates position information representing the position of the subject image on the screen. Voice synchronous transmission device.

Images