TW201933859A - Electronic device having audio-visual processing function and method thereof - Google Patents

Abstract

An electronic device having audio-visual processing function is provided. The electronic device comprises: a controller for receiving a plurality of images of each direction and a plurality of input audio signals of each direction respectively captured by a plurality of cameras and a plurality of omnidirectional microphones, stitching the plurality of images to generate an omnidirectional image, setting at least one calibration reference point of the omnidirectional image, setting an audio-visual common origin of the omnidirectional image and the input audio signals, calculating directional patterns of left and right channels of the audio-visual common origin, respectively incorporating the input audio signals to generate output audio signals of left and right channels; a display for displaying at least a portion of the omnidirectional image; a speaker for outputting the output audio signals of left and right channels; and wherein when a user rotates the omnidirectional image displayed by the display, the controller calculates an offset angle of the rotation relative to the audio-visual common origin by using the at least one calibration reference point, and adjusting the directional patterns according to the offset angle, and adjusting an non-linear weighting ratio of the input audio signals to adjust the output audio signals to conform to the rotation of the omnidirectional image.

Description

Electronic device with audio and video processing function and method thereof

The present invention relates to electronic devices, and more particularly to electronic devices having audio-visual processing functions and methods therefor.

The current omnidirectional photographic device mainly performs stitching processing on the image portion to achieve the effect of omnidirectional image. When the user rotates the omnidirectional image, the user rotates the head to view the image in the environment. However, the sound in the image does not rotate along with the omnidirectional image, causing the audio and video to be out of sync. Therefore, there is a need for an electronic device having a video processing function and a video and audio processing method, which can take into account the manner in which the sound is changed, and perform dynamic processing in synchronization with the rotation of the image, thereby providing the user with an immersive audio and video effect.

The invention provides an electronic device with audio and video processing function and a video and audio processing method, which can synchronously rotate an omnidirectional image and an audio signal.

An embodiment of the present invention provides an electronic device having a video processing function, including: a controller that receives a plurality of images in each direction captured by a plurality of cameras and a plurality of omnidirectional microphones, and a plurality of input audio signals in each direction, and stitches The plurality of images to generate an omnidirectional image, the omnidirectional image is set At least one calibration reference point, setting the omnidirectional image and a common origin of the audio and video of the input audio, calculating a directional pattern of the left and right channels of the common origin of the video, respectively, and incorporating the input audio to generate a left And an output audio of the right channel; a display for displaying at least a portion of the omnidirectional image; a speaker for outputting the output audio of the left and right channels; and wherein, when a user rotates the display The omnidirectional image, the controller calculates an offset angle of the rotation relative to the common origin of the video by the at least one calibration reference point, and adjusts the directional patterns according to the offset angle, and adjusts the directional image The non-linear weight ratio of the input audio is used to adjust the output audio to conform to the rotation of the omnidirectional image.

Another embodiment of the present invention provides a video processing method for an electronic device including a controller, a display, and a speaker. The method includes: receiving, by the controller, a plurality of cameras and a plurality of omnidirectional microphones respectively Capturing the plurality of images in each direction and the plurality of input audio in each direction, stitching the plurality of images to generate an omnidirectional image, setting at least one calibration reference point of the omnidirectional image, setting the omnidirectional image and the input audio a common origin of a video and audio, calculating a directional pattern of the left and right channels of the common origin of the video, respectively, incorporating the input audio to generate output audio of the left and right channels; and the display is used for displaying At least a portion of the omnidirectional image; the speaker outputs an output audio of the left and right channels; and when a user rotates the omnidirectional image displayed by the display, the controller At least one calibration reference point, calculating an offset angle of the rotation relative to a common origin of the video, adjusting the directional patterns according to the offset angle, and adjusting the inputs The weight ratio of the nonlinear weight information, to adjust the audio output such that it rotates in line with the omnidirectional image.

100‧‧‧Electronic devices

101‧‧‧ camera

102‧‧‧ Omnidirectional microphone

103‧‧‧ Controller

104‧‧‧ display

105‧‧‧Speakers

110‧‧‧Video capture device

120‧‧‧Host

401, 402, 403, 404, 405, 406‧ ‧ steps

A, B, C‧‧‧ calibration reference points

D _L ‧‧‧ directional pattern of the left channel

D _R ‧‧‧ directional pattern of the right channel

X ₁ , X ₂ , X ₃ , X ₄ ‧‧‧ input audio

1A is a schematic diagram of an electronic device 100 having a video processing function according to a first embodiment of the present invention.

1B is a schematic diagram of an electronic device 100 having a video processing function according to a second embodiment of the present invention.

Figure 2A is a schematic diagram showing a portion of an omnidirectional image.

Figure 2B shows a schematic diagram of the correction of the reference point as it moves a portion of the omnidirectional image.

3A is an electronic device 100 according to the first embodiment of the present invention, having four omnidirectional microphones, respectively capturing schematic diagrams of input audio X ₁ , X ₂ , X ₃ , and X ₄ .

Figure 3B shows a schematic diagram of adjusting the directional pattern of the left and right channels after the user turns the omnidirectional image.

Fig. 4 is a flow chart showing a method of processing audio and video according to the first embodiment or the second embodiment of the present invention.

The above and other objects, features and advantages of the present invention will become more <RTIgt;

1A is a schematic diagram of an electronic device 100 having a video processing function according to a first embodiment of the present invention. The electronic device 100 includes a video capture device 110 and a host 120 for providing a user with a view of the omnidirectional video and synchronizing the sound with the rotation of the image. The video capture device 110 may include a camera 101 and an omnidirectional microphone 102 for capturing a plurality of images from various directions and complex input audio in each direction. Host 120 can be a package The mobile device such as a smart phone, a notebook computer, or a tablet computer may be an electronic device such as a personal computer (PC) or a video conferencing system, but the present invention is not limited thereto.

In the first embodiment, the video capture device 110 can be connected to the host 120 through an external connection such as a stand, a self-timer (not shown), and through a USB, High Definition Multimedia Interface (HDMI). ), DisplayPort (DP) and other audio and video transmission interface, or through DVI (Digital Visual Interface) terminal, VGA (Video Graphics Array) terminal, with TRS (Tip Ring Sleeve, TRS) terminal, RCA (Radio Corporation of America, RCA) The audio source line such as a terminal is connected to the host 110 via a wireless method such as infrared, Bluetooth, Wi-Fi, etc., for transmitting the captured plurality of images and the plurality of input audio to the host 120. In other embodiments, the camera 101 and the omnidirectional microphone 102 may be separate devices that are independently connected to the host 120 via the wired or wireless means.

1B is a schematic diagram of an electronic device 100 having a video processing function according to a second embodiment of the present invention. In this embodiment, the same names as those in the first embodiment have the same functions. The main difference between FIG. 1B and FIG. 1A is that the electronic device 100 is provided with the camera 101 and the omnidirectional microphone 102 in a built-in manner. The camera 101 and the omnidirectional microphone 102 may be respectively disposed at an upper position of the electronic device 100, but the present invention is not limited thereto.

In FIGS. 1A and 1B, the camera 101 may have a plurality of cameras, for example, three cameras, which are photographed in each direction to capture a plurality of images in each direction. The omnidirectional microphone 102 can have N omnidirectional microphones, N is a positive integer, and the omnidirectional microphone 102 is Omnidirectional for receiving input audio X ₁ , X ₂ , ..., X _N from different angles. In this implementation, the omnidirectional microphone 102 can be four identical omnidirectional microphones that respectively capture complex input audio in each direction. In the present embodiment, when the camera 101 captures a plurality of images, the electronic device 100 is in a stationary state.

In an embodiment, host 120 includes controller 103, display 104, and speaker 105. The controller 103 can be a microprocessor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), etc., for receiving by the plurality of cameras 101 and the complex omnidirectional The plurality of images in each direction and the complex input audio in each direction are respectively captured by the microphone 102, and then the plurality of images are stitched to generate an omnidirectional image, which is a circular or cylindrical 360-degree image. In other embodiments, the controller 103 can receive an omnidirectional image captured by a single camera without going through a stitching procedure, and the invention is not limited thereto. The controller 103 sets at least one calibration reference point from the omnidirectional image, and the calibration reference point may be at least one fixed or relatively non-movable object, such as a table, a pot, a whiteboard, a lighting, and the like in the omnidirectional image. It is used to calculate the offset angle when the user turns the omnidirectional image by providing a correction reference point. In addition, in order to prevent the person from blocking the correction reference point while walking in the omnidirectional video recording, the controller 103 can set the complex correction reference point to avoid the above situation.

In an embodiment, FIG. 2A is a schematic diagram showing a portion of an omnidirectional image, and the controller 103 sets the cylinder A, the cube B, and the tetrahedron C in the omnidirectional image as correction reference points. Then, the controller 103 sets the omnidirectional image and a common origin of the audio and video of the input audio. In an embodiment, when the electronic device 100 starts capturing images, the controller 103 sets the center position of the screen displayed on the display 104 to the common origin of the video and audio, but may be arbitrarily set by the user, and the present invention is not limited thereto. The controller 103 sets the omnidirectional image and the input audio from each direction to set a common origin of the audio and video, so that the omnidirectional image and the input audio have a common origin on a virtual coordinate, taking the virtual ball coordinate as an example. i.e., the radial distance (radial distance, r), the zenith angle (zenith angle, θ) and the azimuth angle (azimuth angle, φ) are all common origin 0 (r = 0, θ = 0 , φ = 0), with To correct the omnidirectional image and the azimuth angle of the input audio.

The controller 103 also calculates a directional pattern of the left and right channels of the common origin of the video, and the left and right channels refer to a common origin of the user facing the audio and video of the electronic device 100. Corresponding to the two channels of the user's left and right directions, it produces a stereo sound effect. The directional patterns are a sound intensity function D(θ, φ, f) having parameters such as zenith angle, azimuth angle and complex frequency. The complex frequency may be an audible frequency range of the human ear, for example, 20 to 20000 Hz, but the invention is not limited thereto. In calculating the directional patterns of the common origin of the video, the controller 103 uses pre-interval time differences, interaural level differences, and frequency response (Spectral Filter) including the left and right ears. Set the sound model. The controller 103 simulates signals in various directions and various frequencies, the time difference between the left and right ears, the sound pressure difference, and the frequency response caused by the binaural ears, and calculates the left and right channels of the common origin of the video and audio as a reference. The directional pattern of frequency, and this is regarded as the directional pattern based on the common origin of the video, that is, the directed pattern of the left channel D _L (0,0,f) and the directional type of the right channel Sample D _R (0,0,f). Next, referring to FIG. 3A, FIG. 3A is an electronic device 100 according to the first embodiment of the present invention, having four omnidirectional microphones, respectively capturing schematic diagrams of input audio X ₁ , X ₂ , X ₃ , and X ₄ . The controller 103 incorporates the directional patterns of the left and right channels into the input audio to generate the output audio of the left and right channels at the common origin of the video and audio.

Specifically, as shown in FIG. 3A, based on the video to have a common origin pattern D _L (0,0, f) and D _R (0,0, f), respectively, the audio inputs to the X direction _1. After applying different weights to X ₂ , X ₃ , and X ₄ , the output audio beams of the left and right channels at the common origin of the video and audio are synthesized, so that the user can listen to the sound from all directions in the common origin of the video and audio. Signal.

The display device 104 can be a liquid crystal display, an organic light emitting diode display, or the like for displaying at least a part of the omnidirectional image. The display 104 can have a touch function, so that the user can directly operate the electronic device 100 through the touch display. Zoom in, zoom out, or even rotate the omnidirectional image.

The speaker 105 can be a stereo stereo speaker, a stereo earphone, or the like for outputting the output audio of the left and right channels for the user to listen to.

Referring next to Figure 2B, Figure 2B shows a schematic diagram of a calibration reference point moving as a user rotates a portion of the omnidirectional image on display 104. As shown in FIG. 2B, the cylinder A and the cube B move to the lower right corner of the screen, and the controller 103 calculates an offset angle of the rotation relative to the common origin of the video by the at least one correction reference point, the offset The angle includes the zenith angle ( θ ) and the azimuth ( φ ). Furthermore, referring to FIG. 3B, FIG. 3B is a schematic diagram showing the adjustment of the directional patterns of the left and right channels after the user turns the omnidirectional image. The controller 103 adjusts the directional patterns of the left and right channels according to the offset angles ( θ , φ ), calculates the directional patterns of the respective frequencies after the rotation, and obtains D _L (θ, φ, f) and D _R ( θ, φ, f). Correspondingly, in Figure 3B, it can be seen that the directional patterns of the left and right channels are deflected clockwise. Next, for each frequency, dynamically adjusting the weight ratio of the non-linear weight of such audio input, more specifically, using a nonlinear adaptive beamforming (nonlinear adaptive beamformer) technology principle, each microphone audio input X _1, X _2, X ₃ and X ₄ are respectively included to adjust the output audio to conform to the rotation of the omnidirectional image. So far, through the video processing of the controller 103, the dynamic change of the sound is synchronized with the rotation of the image.

4 is a flow chart of a method for processing audio and video according to a first embodiment or a second embodiment of the present invention, for an electronic device including a controller, a display, and a speaker. Referring to FIG. 1A and FIG. 1B of the first embodiment of the present invention, in step 401, the controller 103 receives the plurality of images and directions in each direction captured by the plurality of cameras 101 and the plurality of omnidirectional microphones 102, respectively. The plural input audio. In step 402, the controller 103 stitches the plurality of images to generate an omnidirectional image, and sets at least one calibration reference point of the omnidirectional image. In step 403, the controller 103 sets a common origin of the omnidirectional image and the audio and video of the complex input audio, and calculates a directional pattern of the left and right channels of the common origin of the video, that is, D _L (0 , 0, f) and D _R (0, 0, f), respectively, are included in the input audio to generate output audio of the left and right channels.

In step 404, at least a portion of the omnidirectional image is displayed by the display 104. In step 405, the speaker 105 is used to output the output audio of the left and right channels. In step 406, when a user rotates the omnidirectional image displayed by the display 104, the controller 103 calculates an offset angle ( θ , of the rotation relative to the common origin of the video by the at least one correction reference point. φ ), adjusting the directional patterns of the left and right channels according to the offset angle, obtaining D _L (θ, φ, f) and D _R (θ, φ, f), adjusting the nonlinear weight of the input audio The ratio is used to adjust the output audio to match the rotation of the omnidirectional image.

In addition, in the video processing method of the present invention, the electronic device 100 further includes the cameras 101 and the omnidirectional microphones 102, and the cameras 101 are used to capture the images in various directions. The omnidirectional microphone 102 is configured to capture the input audio in each direction, wherein when the cameras capture the images, the electronic device is in a stationary state.

Further, in step 402, wherein the omnidirectional image is a circular or cylindrical 360 degree image. In step 403, the directional pattern is a sound intensity function having parameters such as zenith angle ( θ ), azimuth angle ( φ ), and complex frequency. It should be noted that, in step 403, when calculating the directional patterns of the common origin of the video, a preset sound model including the time difference, the sound pressure difference, and the frequency response of the left and right ears is used. In step 404, the offset angle includes the zenith angle and the azimuth.

Therefore, the electronic device having the audio-visual processing function of the present invention and the method thereof can take the change of the sound into consideration, and perform dynamic processing in synchronization with the rotation of the omni-directional image, thereby giving the user a more natural experience. And provide users with immersive audio and video effects.

While the present invention has been described in its preferred embodiments, the present invention is not intended to limit the invention, and the present invention may be modified and modified without departing from the spirit and scope of the invention. The scope of protection is subject to the definition of the scope of the patent application.

Claims (11)

An electronic device having a video processing function, comprising: a controller, receiving a plurality of images in each direction captured by a plurality of cameras and a plurality of omnidirectional microphones, and a plurality of input audio signals in each direction, and stitching the plurality of images to generate a plurality of images An omnidirectional image, setting at least one calibration reference point of the omnidirectional image, setting an omnidirectional image and a common origin of the audio and video of the input audio, and calculating a directional pattern of left and right channels of the common origin of the video and audio (directional pattern), respectively incorporating the input audio to generate output audio of the left and right channels; a display for displaying at least a portion of the omnidirectional image; and a speaker for outputting the output of the left and right channels And wherein, when a user rotates the omnidirectional image displayed by the display, the controller calculates an offset angle of the rotation relative to a common origin of the video by the at least one calibration reference point, according to The offset angle adjusts the directional patterns, and adjusts the nonlinear weight ratio of the input audio to adjust the output audio to conform to the full Image rotation. The electronic device of claim 1, further comprising: the cameras for capturing the images in all directions; and the omnidirectional microphones for capturing the input audio in each direction. The electronic device of claim 1 or 2, wherein the offset angle comprises a zenith angle ( θ ) and an azimuth angle ( φ ); the directional pattern is a sound intensity A function having the zenith angle, the azimuth angle, and a complex frequency parameter. The electronic device of claim 1, wherein when the directional patterns of the common origin of the video are calculated, the time including the left and right ears is used. Preset sound model for difference, pitch pressure and frequency response. The electronic device of claim 2, wherein when the cameras capture the images, the electronic device is in a stationary state. The electronic device of claim 1, wherein the omnidirectional image is a circular or cylindrical 360-degree image. An audio and video processing method for an electronic device including a controller, a display and a speaker, the method comprising: receiving, by the controller, a plurality of images in each direction captured by a plurality of cameras and a plurality of omnidirectional microphones And inputting a plurality of audio signals in each direction, stitching the plurality of images to generate an omnidirectional image, setting at least one calibration reference point of the omnidirectional image, setting the omnidirectional image and a common origin of the audio and video of the input audio, and calculating The directional pattern of the left and right channels of the common origin of the video and audio respectively is included in the input audio to generate output audio of the left and right channels; and the display is used to display the omnidirectional image At least a portion of the output audio of the left and right channels is outputted by the speaker; and the controller controls the at least one calibration reference when the user rotates the omnidirectional image displayed by the display Pointing, calculating an offset angle of the rotation relative to the common origin of the video, adjusting the directional patterns according to the offset angle, and adjusting the non-information of the input audio The weight ratio of the right to adjust the audio output such that it rotates in line with the omnidirectional image. The method for processing audio and video according to claim 7, wherein the offset angle comprises a zenith angle ( θ ) and an azimuth angle ( φ ); the directional pattern is a sound intensity function, There is a zenith angle, the azimuth angle and a complex frequency parameter. The method for processing audio and video according to claim 7, further comprising: using the time difference, the sound pressure difference, and the frequency response including the left and right ears when calculating the directional patterns of the common origin of the video and audio. Set the sound model. The method for processing audio and video according to claim 7, wherein when the cameras capture the images, the electronic device is in a stationary state. The method for processing audio and video according to claim 7, wherein the omnidirectional image is a circular or cylindrical 360-degree image.