TWI636453B - Multimedia data processing device and method - Google Patents

Abstract

A multimedia data processing device and method, comprising an acquiring unit, an image collecting device, an audio collecting device and a processing unit, wherein the acquiring unit is configured to acquire a relative positional relationship between the sound source and the image capturing device; and the image collecting device is used according to the The relative positional relationship is used to collect image data within a predetermined range, and the audio collecting device is configured to collect audio data sent by the sound source within a predetermined range according to the relative positional relationship, and the processing unit is configured to use the image data within the predetermined range that is collected. Correlation with audio data. The invention also provides a multimedia data processing method. The sound source is positioned by acquiring a relative positional relationship between the sound source and the image capture device. The formation of sound information with a sense of orientation will greatly enhance the user's experience.

Description

Multimedia data processing device and method

The invention relates to a multimedia data processing device and method.

Sound and images are usually collected separately when the user is taking a picture. When recording, the video needs to acquire image data and sound data. However, the existing sound data is all sounds that are obtained regardless of the position. When the final sound is played, no sound feeling with a three-dimensional sound is heard. The user experience is poor.

In view of the above, it is necessary to provide a multimedia data processing apparatus and method.

A multimedia data processing method, the method comprising the steps of:

Obtaining a relative positional relationship between the sound source and the image capturing device;

Acquiring image data in a predetermined range and audio data sent by the sound source according to the relative positional relationship; and

The associated image data in the predetermined range is associated with the audio data.

Preferably, the step of acquiring a relative positional relationship between the sound source and the image capturing device comprises:

Acquiring a positioning signal from an output at the sound source; and

And determining a relative positional relationship between the sound source and the image capturing device according to the received positioning signal.

Preferably, the step of acquiring a relative positional relationship between the sound source and the image capturing device specifically includes:

Collecting the angular momentum of the sound source movement through the gyroscope; and

The angular momentum is converted into orientation information corresponding to the sound source to determine a relative positional relationship between the sound source and the image capture device.

Preferably, the step of acquiring a relative positional relationship between the sound source and the image capturing device specifically includes:

Acquiring the displacement and/or acceleration of the image acquisition device by a linear accelerator; and

The displacement and/or acceleration is converted into orientation information corresponding to the image acquisition device to determine a relative positional relationship between the sound source and the image acquisition device.

Preferably, the step of acquiring a relative positional relationship between the sound source and the image capturing device specifically includes:

Acquiring the angular momentum and displacement and/or acceleration of the image acquisition device by the gyroscope and the linear accelerator, and converting the angular momentum and the displacement and/or acceleration of the collected image acquisition device into the first orientation information;

Acquiring the angular momentum and displacement and/or acceleration of the sound source movement through the gyroscope and the linear accelerator, and converting the angular momentum and the displacement and/or acceleration of the collected sound source into the second orientation information;

And determining a relative positional relationship between the sound source and the image capturing device according to the first orientation information and the second orientation information.

Preferably, the method further comprises the steps of:

Determining the perspective of the user viewing the image; and

Playing the audio data corresponding to the viewing angle.

Preferably, the method further comprises the steps of:

Determining the viewing angle and distance of the user viewing the image; and

Volume weighting and direction adjustment are performed according to the angle of view and distance.

A multimedia material processing device, the device comprising:

An acquiring unit, configured to acquire a relative positional relationship between the sound source and the image capturing device;

An image collecting device, configured to collect image data within a predetermined range according to the relative positional relationship;

An audio collection device, configured to collect audio data sent by a sound source within a predetermined range according to the relative positional relationship; and

The processing unit is configured to associate the collected image data in the predetermined range with the audio data.

Preferably, the multimedia data processing device further includes a storage unit, and the storage unit is configured to store the image data and the audio data corresponding to the association.

Preferably, the playing device in the multimedia data processing device is configured to play the multimedia material that establishes the corresponding association, and is used to detect the direction and the viewing angle of the user viewing the image, and correspondingly according to the user's perspective and distance. Volume weighting and direction adjustment.

The above multimedia data processing apparatus and method locates a sound source by acquiring a relative positional relationship between the sound source and the image capturing device. In this way, the formation of a sound material with a sense of orientation can greatly enhance the user's experience.

The technical solutions in the embodiments of the present invention are clearly and completely described in the following with reference to the accompanying drawings in the embodiments of the present invention. It is obvious that the described embodiments are a part of the embodiments of the present invention, but not all embodiments.

In order to make the objects, technical solutions and advantages of the present invention more comprehensible, the multimedia data processing apparatus and method of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

As shown in FIG. 1 and FIG. 2, a preferred embodiment of the present invention provides a multimedia material processing apparatus 100.

The multimedia material processing apparatus 100 is configured to determine orientation information of a plurality of sound sources 10, 12, and 14. In a specific implementation, the sound sources 10, 12, 14 may respectively be sounds of three different characters (such as actors). In this embodiment, the number of the sound sources 10, 12, and 14 is exemplified by three, and may be more or less than three, for example, at least one.

The multimedia data processing apparatus 100 includes an acquisition unit 20, an image collection device 30, an audio collection device 40, a processing unit 50, and a storage unit 60.

The acquiring unit 20 is configured to acquire a relative positional relationship between the sound sources 10, 12, and 14 and the image capturing device 30. The relative positional relationship includes a direction and a distance between the sound sources 10, 12, and 14 and the image capturing device 20.

The image capturing device 30 is configured to collect image data within a predetermined range according to the relative positional relationship. In this embodiment, the image capturing device 30 includes a plurality of imaging lenses for respectively acquiring image data within a predetermined range.

The audio collection device 40 is configured to collect audio data sent by the sound sources 10, 12, and 14 within a predetermined range according to the relative positional relationship.

The processing unit 50 is configured to associate an image data in the collected predetermined range with the audio data.

The storage unit 60 is configured to store the corresponding associated multimedia material.

In a preferred embodiment, when the sound sources 10, 12, 14 and the image processing device 30 are not moved, the obtaining unit 20 outputs a positioning signal to the image processing through the positioning device disposed at each of the sound sources 10, 12, and 14. The device 30 can determine the relative positional relationship between the sound sources 10, 12, 14 and the image processing device 30 based on the received positioning signals. In this embodiment, the positioning device is one of an ultrasonic device, a Global Positioning System (GPS) device, and a Wireless-Fidelity (WiFi) device.

In another preferred embodiment, when the sound source 10, 12, 14 moves and the image processing device 30 does not move, the acquiring unit 20 collects the angular momentum of the sound source 10, 12, 14 by the positioning device, and the angular momentum is The orientation information corresponding to the sound sources 10, 12, 14 is converted to determine the relative positional relationship between the sound source and the image capture device 30. In this embodiment, the positioning device may be a gyroscope.

In another preferred embodiment, when the image capture device 30 moves and the sound source 10, 12, 14 does not move, the acquisition unit 20 acquires the displacement and/or acceleration of the image capture device 30 by the positioning device, and The displacement and/or acceleration is converted to orientation information corresponding to the image capture device 30 to determine the relative positional relationship between the sound sources 10, 12, 14 and the image capture device 30. In this embodiment, the positioning device may be a linear accelerator.

In another preferred embodiment, when the image capture device 30 and the sound source 10, 12, 14 are both moved, the acquisition unit 20 acquires angular momentum and displacement and/or acceleration of the image capture device 30 by the positioning device, and The angular momentum and displacement and/or acceleration of the acquired image acquisition device 30 are converted into first orientation information. Similarly, the acquiring unit 20 collects angular momentum and displacement and/or acceleration of the sound source 10, 12, 14 by the positioning device, and shifts the angular momentum and displacement of the collected sound source 10, 12, 14 and/or The acceleration is converted into the second orientation information, respectively. The processing unit 50 determines a relative positional relationship between the sound sources 10, 12, 14 and the image capturing device 30 based on the first orientation information and the second orientation information.

In a preferred embodiment, the multimedia material stored in the storage unit 60 can be transmitted to a playback device (not shown). The transmission mode can include, but is not limited to, copying through a storage medium or a wireless network. Road transmission and other methods.

When the user uses the playing device, the playing device detects the direction in which the user views, determines the viewing angle of the user viewing the image, and plays the audio data corresponding to the viewing angle according to the acquired associated multimedia material.

In a preferred embodiment, the playback device is further configured to perform volume weighting and direction adjustment according to the angle of view and distance of the user viewing the image.

Specifically, when the user uses a playback device (such as a VR head mounted display), when the front side faces a certain angle of view in the image, the associated sound corresponding to the angle of view will be transmitted from the front of the user. Sounds associated with other perspectives in the image will be transmitted from the user's left rear and right rear. Moreover, the sounds from the various directions are weighted to adjust the volume.

Referring to FIG. 3, the multimedia data processing method includes the following steps:

Step S100: Acquire a relative positional relationship between the sound source and the image collection device, which can be specifically implemented as follows:

When the sound source moves and the image processing device does not move, the angular momentum of the sound source movement is collected by the positioning device, and the angular momentum is converted into the orientation information corresponding to the sound source to determine the relative positional relationship between the sound source and the image capturing device. In a particular embodiment, the positioning device can achieve the acquisition of diagonal momentum by a gyroscope.

When neither the sound source nor the image processing device moves, the positioning device disposed at each sound source outputs a positioning signal to the image processing device, so that the relative positional relationship between the sound source and the image processing device can be determined according to the received positioning signal. In a specific embodiment, the positioning device can achieve positioning of the sound source by one of an ultrasonic device, a global system positioning device, and a wireless fidelity device.

When the image capturing device moves and the sound source does not move, the displacement and/or acceleration of the image capturing device is acquired by the positioning device, and the displacement and/or acceleration is converted into the orientation information corresponding to the image capturing device to determine the sound source and The relative positional relationship between image acquisition devices. In a particular embodiment, the positioning device can be passed through a linear accelerator to achieve displacement and acceleration of the acquisition device.

When the image capturing device and the sound source are both moved, the angular momentum and displacement and/or acceleration of the image capturing device are acquired by the positioning device, and the angular momentum and displacement and/or acceleration of the collected image capturing device are converted into First orientation information. Similarly, the angular momentum and the displacement and/or acceleration of the sound source movement are collected by the positioning device, and the angular momentum and the displacement and/or the acceleration of the collected sound source are respectively converted into the second orientation information. The processing unit determines a relative positional relationship between the sound source and the image capturing device according to the first orientation information and the second orientation information.

Step S102: Acquire image data in a predetermined range and audio data sent by the sound source according to the relative positional relationship.

Step S104: Correlate the associated image data in the predetermined range with the audio data. Specifically, the determined relative positional relationship is combined with the corresponding audio data and image data to generate corresponding associated multimedia materials.

Referring to FIG. 4, the multimedia data playing method includes the following steps:

In step S200, the direction of the user's viewing is detected, and the viewing angle of the user's viewing image is determined, and the audio data corresponding to the viewing angle is played according to the acquired associated multimedia material.

Specifically, when the user views the first view of the image by using the playback device, the associated multimedia data is determined according to the acquired associated multimedia data to determine that the corresponding audio data in the multimedia material is played. When the user views the second view of the image, the associated multimedia material is determined according to the acquired multimedia data to determine the corresponding audio data in the multimedia material. By analogy, when viewing the different views of the image, the user will play the associated multimedia material according to the audio data corresponding to the different viewing angles.

Step S202, performing volume weighting and direction adjustment on the audio data according to different viewing angles and distances of the user viewing the image.

Specifically, when the user uses the playback device and faces the front view to a certain angle of view in the image, the associated sound corresponding to the perspective will be transmitted from the front of the user, and the sound associated with the other perspectives in the image. Will be sent from the user's left rear and right rear. Moreover, the sounds transmitted by the user in various directions are adjusted by the weighting.

The above multimedia data processing apparatus and method locates a sound source by acquiring a relative positional relationship between the sound source and the image capturing device. In this way, the formation of a sound material with a sense of orientation can greatly enhance the user's experience.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. A person skilled in the art should understand that the technical solutions of the present invention may be modified or equivalently substituted without departing from the spirit and scope of the technical solutions of the present invention. All other embodiments obtained by a person of ordinary skill in the art based on the embodiments of the present invention without departing from the inventive scope will fall within the scope of the present invention.

In summary, the present invention has indeed met the requirements of the invention patent, and has filed a patent application according to law. However, the above description is only a preferred embodiment of the present invention, and it is not possible to limit the scope of the patent application of the present invention. Equivalent modifications or variations made by persons skilled in the art in light of the spirit of the invention are intended to be included within the scope of the following claims.

100‧‧‧Multimedia data processing device
10, 12, 14‧‧ ‧ source
20‧‧‧Acquisition unit
30‧‧‧Image acquisition device
40‧‧‧Audio acquisition device
50‧‧‧Processing unit
60‧‧‧ storage unit

1 is a block diagram of a preferred embodiment of a multimedia data processing device.

2 is a schematic diagram of a preferred embodiment of a multimedia data processing device.

3 is a flow chart of a preferred embodiment of a multimedia data processing method.

4 is a flow chart of a preferred embodiment of a method for playing multimedia data.

no

no

Claims (10)

A multimedia data processing method, wherein the method comprises the steps of: acquiring a relative positional relationship between a sound source and an image capturing device; collecting image data within a predetermined range and audio data emitted by the sound source according to the relative positional relationship; And correlating the collected image data with the audio data in the predetermined range. The multimedia data processing method of claim 1, wherein the step of acquiring a relative positional relationship between the sound source and the image capturing device comprises: acquiring a positioning signal from an output at the sound source; and; receiving The resulting positioning signal determines the relative positional relationship between the sound source and the image capture device. The method for processing a multimedia data according to claim 1, wherein the step of acquiring a relative positional relationship between the sound source and the image capturing device comprises: collecting angular momentum of the sound source movement by the gyroscope; and applying the angular momentum Converting to the orientation information corresponding to the sound source to determine the relative positional relationship between the sound source and the image acquisition device. The method for processing a multimedia data according to claim 1, wherein the step of acquiring a relative positional relationship between the sound source and the image capturing device comprises: acquiring a displacement and/or an acceleration of the image capturing device by a linear accelerator; The displacement and/or acceleration is converted into orientation information corresponding to the image acquisition device to determine a relative positional relationship between the sound source and the image acquisition device. The method for processing a multimedia data according to claim 1, wherein the step of acquiring a relative positional relationship between the sound source and the image capturing device comprises: collecting angular momentum and displacement of the image capturing device by using a gyroscope and a linear accelerator; And/or acceleration, and converting the angular momentum and displacement and/or acceleration of the acquired image acquisition device into first orientation information; collecting angular momentum and displacement and/or acceleration of the sound source movement by the gyroscope and the linear accelerator, And converting the angular momentum and the displacement and/or acceleration of the collected sound source into the second orientation information; and determining the relative relationship between the sound source and the image acquisition device according to the first orientation information and the second orientation information. Positional relationship. The multimedia data processing method of claim 1, wherein the method further comprises the steps of: determining a viewing angle of the user viewing the image; and playing the audio material corresponding to the viewing angle. The multimedia data processing method of claim 6, wherein the method further comprises the steps of: determining a viewing angle and a distance at which the user views the image; and performing volume weighting and direction adjustment according to the viewing angle and the distance. A multimedia data processing device, wherein the device comprises: an acquiring unit, configured to acquire a relative positional relationship between the sound source and the image capturing device; and an image collecting device, configured to collect the predetermined range according to the relative positional relationship An image acquisition device, configured to acquire audio data sent by a sound source within a predetermined range according to the relative positional relationship; and a processing unit configured to associate the image data in the collected predetermined range with the audio data Association. The multimedia material processing device of claim 8, wherein the multimedia data processing device further comprises a storage unit, wherein the storage unit is configured to store the corresponding associated image data and audio data. The multimedia data processing device of claim 8, wherein the playback device in the multimedia data processing device is configured to play the corresponding associated multimedia data, and is used to detect a direction in which the user views the image and The angle of view, and the volume weighting and direction adjustment are performed correspondingly according to the user's perspective and distance.