JP2005510907A - Method and apparatus for forming an audio signal from a video data stream - Google Patents

Abstract

  An audio signal (snd) is formed from a video data stream containing a single image (img) in temporal order, and a motion vector field (vfd) is first obtained from the image data of the single image, for example, MPEG algorithm Is used. Subsequently, at least one feature quantity (chs) of the motion vector (vfd), for example, a dominant motion vector is derived in the analysis module (AAN), and the integration module (SYN) performs acoustics based on the feature quantity (chs). A signal (snd) is formed.

Description

  The present invention relates to the formation of a signal, in particular an acoustic signal, from a video stream containing a single image in temporal order.

  The conversion of motion information into an acoustic signal is known, for example in the form of an acoustic motion detector. For example, the detector takes an image using a video camera and triggers a signal such as an acoustic alarm as soon as the image changes. However, such motion detectors provide only a certain signal. By dividing the image area to be investigated into fields each associated with one different signal, changes in the acoustic signal can only be realized at best. For example, this type of motion detector will naturally not work if the object moving relative to the background is to be monitored.

  Movement-dependent sound formation is also used in the field of unit art. For example, body movements made by humans are interpreted and used to control sound generation. In this way, actors can create musical rhythms and themes through their movements. The interaction of movement and sound is a creative process, which inspires humans to always make a series of additional sounds with new movements. Sound formation is usually performed by a sensor such as a motion detector or light barrier that responds when a person moves, and the sensor signal is processed through the data processor and thus associated with the sensor. A sound is formed. However, these sound formations are based on costly devices that receive visual input via a plurality of recording devices and are fixedly installed during operation.

  An object of the present invention is to show a means of how motion information recorded via one camera can be converted into a sound pattern. Here, the sound to be formed should be variable depending on the type of movement, in particular the direction of movement. Such sound formation should be integrated especially as a function of a mobile phone with a camera attached.

This problem is solved according to the invention by a method as described at the outset with the following steps:
a) using the image data of the preceding and / or subsequent single images to determine a motion vector field from the image data of the single images;
b) Deriving at least one feature quantity from the motion vector field;
c) An acoustic signal is formed depending on one or more feature quantities.

  The method according to the present invention is suitable for an apparatus having a control device having means for obtaining a motion vector field from image data of a single image using image data of a preceding and / or subsequent single image. Is additionally configured to derive at least one feature quantity from the motion vector field and form an acoustic signal depending on a plurality of feature quantities or at least one feature quantity.

  The solution according to the invention forms a sound or acoustic signal that is not only due to the presence of movement but also due to the magnitude and / or direction of the recorded movement. Thereby, the sound to be formed can be configured differently. In the case of monitoring, this means, for example, that this acoustic signal makes it possible to monitor various image regions via acoustic signals, for example, which realize differentiating movements.

  In an advantageous embodiment, the solution according to the invention is realized by a telecommunications terminal device, in particular a mobile telephone. Since the mobile phone is already frequently provided with an image processing function, the implementation of the present invention is particularly suitable for this mobile phone and can be realized in a compact manner. In this case, preferably, the video data stream can be formed by the camera device of the terminal device. Furthermore, the acoustic signal can be output via an audio device of the terminal device, or can be output via a mobile telecommunications connection established using the terminal device.

  In addition, it is advantageous if the motion vector field is determined in step a) using a known MPEG encoder scheme, in particular in the case of mobile telephones.

  In the evaluation of the motion vector field to be simply realized in step b), for example, a distribution is derived using a statistical method, and a statistical characteristic amount is obtained for the derivation, and at least 1 is calculated from this characteristic amount. One feature amount is to be detected.

The invention, together with further advantages, will be described in detail below on the basis of the non-limiting examples shown in the drawings. The drawing shows a schematic diagram,
FIG. 1 shows a front view of a mobile phone according to an embodiment,
FIG. 2 shows a rear view of the mobile phone of FIG.
FIG. 3 shows a block diagram of the mobile phone of FIG.
FIG. 4 shows a block diagram of an MPEG encoder,
FIG. 5 shows an example of a vector field obtained by motion evaluation,
FIG. 6 shows a motion histogram derived from the vector field of FIG.

  As mentioned in advance, the embodiment described below is merely useful as an example, and the present invention described above is not to be construed as being limited to this embodiment.

  1 and 2 show a mobile telephone MOG, which according to the invention converts motion recorded from the surroundings into an acoustic signal (acoustic kaleidoscope). The telephone MOG has a microphone MIC, a speaker LSP, and an input field EIN (for example, a keyboard) for inputting operation commands and a telephone number, as well known as visible features in the casing. An output unit DIS in the form of a screen as described above is provided, and a video image can be displayed on the output unit DIS. The video image img originates in particular from the camera module CAM provided on the back (FIG. 2), which camera module CAM is used for recording images from the surroundings, and these images are used as an image data processing device according to the invention. To supply. Similarly, a compartment CAC for the battery and the SIM card is provided on the back of the mobile phone MOG as is well known.

  The block diagram of FIG. 3 shows the components of the telephone MOG. As is well known, in addition to the input / output elements LSP, MIC, EIN, and the display DIS, an antenna ANN and a transmission / reception device SEE for carrying out a mobile telecommunications function are provided, and the user further inputs via the input unit EIN. A processor PRC is provided as a control device for interpreting the input commands and controlling the device SEE accordingly.

Furthermore, the processor PRC is configured according to the invention to process the image data img of the camera module, from which the sound snd using video coding and motion field analysis as described below. Is formed. The function for forming the sound snd from the motion information of the image img substantially includes the following steps:
a) Encoding of image information img by an encoding module ENC using, for example, an MPEG algorithm for detecting a motion vector field to which it belongs,
b) Analysis of a predetermined feature quantity in the analysis module AAN of the processor system PRC, for example the vector field vfd for the dominant motion vector,
c) In the embodiment shown here, the synthesis module SYN, also implemented in the processor system PRC, forms a sound based on the quantity chs, for example the formation of the sound as a function of the dominant motion vector orientation and value.

  Step a) of encoding the image information is performed using, for example, known MPEG encoding. This step is a standard method for compressing and transmitting a digitized sequence of images and has already been implemented in various ways in mobile phones with cameras for videophone purposes, for example. The essential components in the present method are a motion-evaluation unit BS (see below) for continuous images, a motion compensation unit BK, and a data reduction unit DK connected to a motion-compensated image as a discrete cosine transform unit. It is a conversion part which converts into a frequency domain using (DCT). The principle of the method can be seen in FIG. Details on this are given in “Digital Signal Processing for Multimedia Systems”, Keshab K. Parhi and Takao Nishitani (Hrsg.), Marcel Dekker, Inc., New York, pages 31-37.

In order to derive an audio signal according to the present invention, the result of the motion evaluation unit BS, which is a part of the MPEG system, is required rather than the image compressed by the MPEG system. In addition to the image to be evaluated at time t _n , the input to the motion evaluation unit BS is an image imp at time t _n−1 preceding this image. The image imp is obtained by performing an inverse DCT (iDCT) on the DCT transformed and motion compensated signal of the preceding image, or by being stored intermediately over the period of image exchange using the image storage device IS. can get.

  The image imp is used as a reference image and is divided into a number of blocks bbl, for example 36 pixel blocks of 16 × 16 pixels each as shown in FIG. For each of these pixel blocks bbl, the image imn to be evaluated is checked for the best match in the local neighborhood by the MSE (mean square error) method. In this way, information regarding the displacement vector is obtained for each block bbl. FIG. 5 shows the image img of FIG. 1 as an example, in which the motion vector v calculated for each block bbl is additionally plotted. The images were taken with a car running in the background, and the camera swiveled along with the vehicle when recording a series of images. Based on this turning motion, the motion vector calculated in the vehicle is almost zero, while the surrounding vectors indicate motion. The motion vector field vfd resulting from this processing is an input to the subsequent processing stage.

  The motion estimation described here, which is part of the known MPEG scheme, is simple but nevertheless provides an effective motion analysis. Within the framework of the present invention, other motion analysis methods are also used that provide a motion vector field for this purpose, typically using one or more images that precede or follow in time the image to be examined. be able to.

  In a subsequent step b), one or more feature quantities chs, here for example a dominant motion orientation, are detected from the motion vector field vfd. This detection takes place in the analysis module AAN. In the example considered here, all vector orientations from the motion vector field are entered in the histogram his (see FIG. 6). (For the sake of clarity, the division into 16 direction classes is performed in FIG. 6, but of course the number of classes of the basic quantity can be clearly increased from this number, and the number of blocks bbl and the movement Limited only by the resolution of the rating). The maximum value bmx in the distribution represented by the histogram his indicates the main movement direction in the image. A value of the motion speed is detected for this direction, which is a simple value from all vectors belonging to this main motion direction with a predetermined tolerance (eg two classes each adjacent to the maximum histogram class). Calculated by average value formation. The result of this process is a vector whose main motion in the image is represented by its orientation and value. This vector is a (two-dimensional) feature quantity chs in the example considered here, and a sound is derived in a subsequent processing stage based on this feature quantity chs.

  Of course, in other embodiments of the present invention, the evaluation may be performed in other ways. For example, a histogram can be evaluated that considers the orientation and values of the vectors (= frequency of a two-dimensional basic quantity). The quantity that can be used as the basis of the evaluation is, for example, statistical characteristic data of a distribution such as the most frequently occurring value (maximum value), the second maximum value, the belonging variable, and the higher-order weight.

  In the subsequent step c), sound formation based on the feature value chs is performed in the synthesis module SYN. A sound snd is formed as a function of the orientation and value of the main motion vector detected in advance, and output through the speaker LSP. As an alternative, a sound sequence present as an acoustic signal, which is represented electrically in a manner known per se, can be transmitted to other subscribers via the mobile telecommunication connection of the mobile telephone MOG.

  For example, the value of motion speed during sound formation controls the amount of sound, while on the other hand, various sounds are formed depending on the motion orientation. At this time, for example, each orientation class shown in the histogram his is associated with a pre-stored sound, and this sound is classified according to the pitch and / or tonal characteristics (overtone spectrum). Distinguished from sound. Sounds can be arranged according to pitch, but this is not necessary.

  Of course, you can change the way you sound. That is, in the modification of the present invention, a plurality of maximum values hmx and bmx of the distribution his are superimposed on one sound. In another variant, the histogram his can be fed directly to the synthesizer SYN, which uses this histogram as the upper sound spectrum of the basic sound. Whether the pitch of the basic sound can be kept constant (for example, A, 110 Hz) can be determined according to the method described above.

It is a front view of the mobile telephone by an Example. 2 shows a rear view of the mobile phone of FIG. FIG. 2 shows a block diagram of the mobile phone of FIG. 1 shows a block diagram of an MPEG encoder. An example of the vector field obtained by motion evaluation is shown. Fig. 6 shows a motion histogram derived from the vector field of Fig. 5;

Claims (11)

In a method of forming a signal from a video data stream containing a single image (img) in temporal order:
a) obtaining a motion vector field (vfd) from image data of a single image (img) using image data of a preceding and / or subsequent single image;
b) deriving at least one feature quantity (chs) from the motion vector field (vfd);
c) forming an audio signal (snd) in dependence on one or more of said feature quantities (chs), a method of forming a signal from a video data stream.   The method according to claim 1, carried out in a mobile telecommunication terminal equipment (MOG), for example a mobile telephone.   The method according to claim 2, wherein the video data stream is formed by a camera device (CAM) of the terminal equipment (MOG).   The method according to claim 2 or 3, wherein the acoustic signal is output via an audio device (LSP) of the terminal equipment (MOG).   The method according to claim 2 or 3, wherein the acoustic signal is output via a mobile telecommunications connection established by a terminal equipment (MOG).   The method according to claim 1, wherein in the step a), the motion vector field is obtained using a known MPEG encoder method.   In the step b), a distribution (his) is derived from the motion vector field (vfd), a statistical characteristic amount is obtained with respect to the distribution, and at least one characteristic amount (chs) is detected from the characteristic amount. Item 7. The method according to any one of Items 1 to 6. In an apparatus for forming a signal from a video data stream containing a single image (img) in temporal order,
A control device having means (ENC) for obtaining a motion vector field (vfd) from image data of a single image (img) using a preceding and / or subsequent single image is provided,
The control device additionally derives at least one feature quantity (chs) from the motion vector field (vfd), and depending on a plurality of feature quantities or at least one feature quantity (chs), an acoustic signal (snd) Forming the apparatus.   9. A device according to claim 8, provided in a mobile telecommunications device (MOG), e.g. a mobile phone.   Device according to claim 9, wherein a camera device (CAM) is provided which forms a video data stream supplied to means (ENC) for determining a motion vector field (vfd).   11. Apparatus according to any one of claims 8 to 10, wherein the means (ENC) for determining a motion vector field (vfd) is an MPEG encoder.

Images