TWI787799B - Method and device for video and audio processing - Google Patents

Abstract

A method and a device for video and audio processing are provided. The method includes receives a plurality of original video and audio signal for each original video by using a video and audio processing unit; generating one or more output video signal according to the original video by using the video and audio processing unit; adjusting each audio signal according to layout information to obtain a stereo audio signal by using the video and audio processing unit; and outputting the output video signal and the stereo audio signal from the video and audio processing unit to a video and audio playback device.

Description

Video and audio processing method and audio and video processing device thereof

This case relates to an audio-visual processing method and an audio-visual processing device thereof, especially an audio-visual processing method with audio spatialization and an audio-visual processing device thereof.

With the development of science and technology, today's business model and entertainment model are different from those in the past, and the changes in business model and entertainment model also directly affect people's lives. The powerful audio and video playback function has formed a wave of online live broadcasting. A large number of online live broadcasting platforms and online live broadcasting programs have emerged. Users can watch the video of the live broadcaster and listen to the voice of the live broadcaster through electronic products. According to different live broadcast content , people can enjoy many different types of entertainment.

However, when users use electronic products (such as TVs, computers, mobile phones, or tablets, etc.) to play videos and sounds of live programs, the electronic products cannot produce spatialized stereo sound effects according to the speaking position of the live broadcaster. It is impossible to have a sense of presence during the program, which greatly reduces the user experience.

In view of the above problems, the present invention proposes an audio-visual processing device, especially an audio-visual processing device that can make listeners feel more present.

An embodiment provides an audio-visual processing device, which is suitable for connecting with an audio-visual playback device. The video and audio processing device includes a video processing unit, a sound processing unit and an output port. The sound processing unit is connected to the image processing unit. The output port is coupled to the image processing unit and the audio processing unit. The image processing unit receives a plurality of initial images and generates at least one output image signal according to the plurality of initial images. The audio processing unit receives the layout information and the audio signals corresponding to each original video, and adjusts each audio signal into a stereo audio signal according to the layout information. The output port outputs the stereo audio signal and the output video signal to the audio-visual playback device, so that the audio-visual playback device outputs the display images on the plurality of display blocks corresponding to the layout information of the plurality of initial videos by playing the output video signal.

An embodiment provides an audio-spatialized audio-visual processing method, which is applicable to an audio-visual processing device, and the audio-visual processing device is connected to an audio-visual playback device. Here, the audio-visual processing method includes: receiving a plurality of initial videos and an audio signal corresponding to each initial video by the audio-visual processing device; generating at least one output video signal according to the plurality of initial videos by the audio-visual processing device; The layout information adjusts each audio signal into a stereo audio signal; and outputs at least one output video signal and stereo audio signal to the audio-visual playback device through the audio-visual processing device, so that the audio-visual playback device outputs a plurality of initials by playing at least one output video signal A display frame of the video on the plurality of display blocks corresponding to the layout information.

To sum up, according to any embodiment of the audio-visual processing method and the audio-visual processing device of the present application, the audio-visual processing device can adjust the signal parameters of the audio signal of the video correspondingly according to the display position of the video. In some embodiments, the stereo audio signal has a difference in volume, delay, phase, frequency or other signal parameters in any combination compared with the audio signal; or the stereo audio signal corresponds to each channel of the audio-visual playback device, here , when stereo audio signals are output in different channels, they are different in volume, delay, phase, frequency or any combination of other signal parameters. In this way, the audio spatialization effect in the three-dimensional space can be established, so as to provide users with a three-dimensional audio-visual effect.

Please refer to FIG. 1 or FIG. 2 , in some embodiments, the video and audio processing device 1 may be connected between the video and audio capture device 2 and the video and audio playback device 3 . Although the illustration shows two video capture devices 2 (such as video capture devices 21 , 22 ) and two video playback devices 3 , the number of video capture devices 2 and video playback devices 3 in this case is not limited to these numbers.

Taking the audio-visual processing device 1 coupled to the two video-audio capture devices 21 and 22 as an example, the video-audio capture device 21 can shoot the target 41 to generate a video V1, and the video V1 includes the moving image of the target 41 . Wherein, the dynamic image of the target 41 is composed of a plurality of first image frames at a first frame rate. In addition, the audio-video capture device 22 can shoot the object 42 to generate a video V2, and the video V2 includes a moving image of the object 42 . Wherein, the dynamic image of the target 42 is composed of a plurality of second image frames at the second frame rate. Here, the first frame rate and the second frame rate may be the same or different. Wherein, when the audio-visual capture devices 21 and 22 capture the initial videos V1 and V2, the video-video capture devices 21 and 22 can also capture the voice of the target 41 and the voice of the target 42 synchronously, and generate the voice signal A1 of the target 41 and the sound signal A2 of the target 42 .

In one embodiment, the initial videos V1, V2 captured by each video capture device 21, 22 can be a single video stream (that is, stored in a file), or more than two video streams (that is, stored in a file). in two or more files).

Referring to FIG. 1 to FIG. 3 , the video processing device 1 can receive initial videos V1 , V2 and audio signals A1 , A2 of different objects 41 , 42 from different video capture devices 21 , 22 (step S01 ). After receiving the initial video V1, V2 and the audio signal A1, A2, the audio-visual processing device 1 can generate at least one output video signal Vo according to the received initial video V1, V2 (step S02), and adjust each audio signal according to the layout information L1. The audio signals A1, A2 are stereo audio signals S1, S2 (step S03). Then, the audio-visual processing device 1 outputs the output video signal Vo and the stereo audio signals S1, S2 to one or more audio-visual playback devices 3 (step S04), each audio-visual playback device 3 can play the output video signal Vo, and each audio-visual playback device The display screen of 3 corresponds to the initial video V1, V2 on the display block corresponding to the layout information L1.

Here, the layout information L1 may be the configuration state of the display layout of the output screen, that is, defines the number of display blocks included in the output screen and the orientation and/or size of each display block. In the output image, each display block can be used to display the corresponding image frame of the original video from a single image source. Wherein, the output screen may refer to a display screen displayed by the audio-visual playing device 3 . In some embodiments, the output image may also refer to a display image displayed on a display screen (not shown) built in the audio-video processing device 1 .

Moreover, the audio-visual processing device 1 can designate the initial videos V1 and V2 to be displayed at different positions in the display layout of the audio-visual playback device 3 . In some embodiments, the layout information L1 includes layout positions Lo1 and Lo2 respectively corresponding to the original videos V1 and V2 from different sources. Wherein, the configuration positions Lo1 and Lo2 in the layout information L1 correspond one-to-one to the display blocks included in the output screen, and each configuration position Lo1 and Lo2 defines the layout position for the corresponding output block. In some embodiments, each configuration location Lo1, Lo2 corresponds to an image source. For example, in the layout information L1, the source of the image can be represented by a source number, and each configuration position Lo1, Lo2 can use the corresponding source number as an index.

In some embodiments, please refer to FIG. 1 or FIG. 2 , the video and audio processing device 1 includes an input unit 10 , an image processing unit 11 , a sound processing unit 12 and an output port 13 . The input unit 10 is coupled to the image processing unit 11 and the audio processing unit 12 . The image processing unit 11 is coupled to the audio processing unit 12 . The output port 13 is coupled to the image processing unit 11 and the audio processing unit 12 .

The input unit 10 is coupled to one or more video capture devices 21 , 22 , and the output port 13 is coupled to one or more video playback devices 3 . Wherein, the input unit 10 can receive the original video V1, V2 and the audio signal A1, A2 from each audio-visual capture device 21, 22, and then further provide the received initial video V1, V2 to the image processing unit 11, and the received The received audio signals A1 and A2 are provided to the audio processing unit 12 .

Here, the image processing unit 11 can also receive the layout information L1. In one embodiment, the layout information L1 can be obtained by the control signal C1 for setting the layout generated by the user operating the user interface (not shown) of the audio-visual processing device 1 .

In one embodiment, the storage unit can pre-store a variety of preset layout information, and the layout information of these preset preset layouts are respectively used to indicate the display layouts of different display screens (such as picture-in-picture, picture-in-picture, The configuration state of screen juxtaposition, floating screen, screen stacking, etc.) (such as the number of sub-screens, the image source of each sub-screen, the display position of each sub-screen, the size of each sub-screen or any combination thereof, etc.). The user generates a selection signal for a specific display layout to be used by operating the user interface (such as a touch screen or a combination of a display screen and a physical button, etc.), and then the image processing unit 11 reads out from the storage unit according to the selection signal. The corresponding default layout information is used as the layout information L1. In another embodiment, the storage unit may pre-store a preset layout information, which is the layout status of the display layout (ie, default layout) of the display screen displayed by the preset audio-visual playback device 3 . When starting up, the image processing unit 11 can read out the default layout information from the storage unit as the layout information L1. Here, when the user generates a setting signal by operating the user interface, the image processing unit 11 obtains corresponding layout information according to the setting signal to update the currently used layout information L1.

In some embodiments, the image processing unit 11 can set the layout positions of the initial videos V1 and V2 displayed on the audio-visual playback device 3 according to the layout information L1.

For example, take two image sources of the initial video V1 and V2 as an example, and the display layout indicated by the layout information L1 has two sub-picture layout positions as an example. In an embodiment, the display layout can be arranged in a state where the screens are arranged side by side, that is, the display screen is divided into two sub-pictures on the left and the right. In the display layout, as shown in FIG. 4 , the layout position of the initial video V1 and the layout position of the initial video V2 can be respectively a first position on the left and a second position on the right relative to the preset reference point L2 . Wherein, the preset reference point L2 may be the center point of the display layout. For example, the entire layout pair is divided into two left and right blocks on the X-Y plane. The first position can be the left layout position 31 (i.e., the left half block) on the left relative to the preset reference point L2 in the X-Y plane. The position may be the right layout position 32 (ie, the right half block) on the X-Y plane relative to the preset reference point L2. At this time, the image processing unit 11 can set the initial video V1 of the first image source to be displayed at the left layout position 31 and set the initial video V2 of the second image source to be displayed at the right layout position 32 according to the layout information L1. In another embodiment, the display layout may be in a stacked configuration state, that is, the display screen is divided into upper and lower sub-screens. In the display layout, as shown in FIG. 5 , the layout positions of the initial video V1 and the initial video V2 may be a first position above and a second position below relative to the preset reference point L2 respectively. Wherein, the preset reference point L2 may be the center point of the display layout. For example, the entire layout pair is divided into upper and lower blocks on the X-Y plane, the first position may be the upper layout position 33 (that is, the upper half block) located above the preset reference point L2 in the X-Y plane, and the second The position may be the lower layout position 34 (ie, the lower half block) located below relative to the preset reference point L2 in the X-Y plane. At this time, the image processing unit 11 can set the initial video V1 of the first image source to be displayed at the upper layout position 33 according to the layout information L1, and can set the initial video V2 of the second image source to be displayed at the lower layout position according to the layout information L1 34.

In yet another embodiment, the display layout may be a picture-in-picture configuration state, that is, the display screen is divided into two sub-pictures, one large and one small. In the display layout, the layout position of the initial video V1 and the layout position of the initial video V2 can be respectively a first position with a larger layout and a second position with a smaller layout and overlapping with the first position (in terms of front view). At this time, the image processing unit 11 can set the initial video V1 of the first image source to be displayed at the first position according to the layout information L1, and can set the initial video V2 of the second image source to be displayed at the second position according to the layout information L1.

Referring to FIGS. 1 to 3 , the sound processing unit 12 receives the layout information L1 from the image processing unit 11, so that the sound processing unit 12 can adjust the sound signals respectively according to the layout positions Lo1 and Lo2 corresponding to the initial videos V1 and V2 in the layout information L1. A1, A2 are stereo audio signals S1, S2 (step S03).

In some embodiments, the number of output video signals Vo may be multiple, and one-to-one corresponds to the initial video V1, V2. Here, each output video signal Vo is composed of a corresponding initial video V1 or V2. At this time, the output port 13 can also output the stereo audio signals S1 and S2 in the form of multiple audio streams, as shown in FIG. 1 .

In the first embodiment of step S02, the image processing unit 11 may add source tags to the data streams of the initial videos V1, V2 to form the output video signal Vo, so as to use the source tags carried by each initial video V1, V2 to Inform the audio-visual playback device 3 to display the layout positions of each initial video V1, V2. Wherein, the source tag may correspond to or be the same as the data representing the source of the image in the currently used layout information L1. In other words, the image processing unit 11 can integrate the initial videos V1 and V2 and their corresponding positions Lo1 and Lo2 into one stream (that is, output the video signal Vo), and then output the output video signal Vo to the audio-visual playback device 3 . In this way, when receiving the output video signal Vo, the audio-visual playback device 3 can display the image frame of the output video signal Vo at the corresponding display position in the display screen, that is, the positions corresponding to the configuration positions Lo1 and Lo2 of the display screen are presented respectively. The image frames of the initial movie V1, V2.

In the second embodiment of step S02, the image processing unit 11 can associate the source tags of the initial videos V1 and V2 with the data indicating the source of the images in the layout information L1, so as to utilize the sources carried by the original videos V1 and V2 mark to inform the audio-visual playback device 3 of its displayed layout position. For example, before transmitting the output video signal Vo, the image processing unit 11 first transmits the associated information to the audio-visual playback device 3 , and then transmits the output video signal Vo to the audio-visual playback device 3 . Wherein, the associated information is the corresponding relationship between the record source tag and the layout positions Lo1 and Lo2 of the layout information L1. The audio-visual playback device 3 then learns the configuration positions Lo1 and Lo2 of the initial videos V1 and V2 according to the source tags of the original videos V1 and V2, the received associated information and the pre-stored currently used layout information L1, and according to the configuration The positions Lo1 and Lo2 display the initial films V1 and V2 on the corresponding layout positions in the display screen.

In the third embodiment of step S02, the image processing unit 11 may add data indicating its configuration positions Lo1, Lo2 in the data streams of the initial videos V1, V2, so that the audio-visual playback device 3 uses the data indicating its configuration positions Lo1, The information of Lo2 knows the layout position of the initial video V1 and V2. For example, before transmitting each initial video V1, V2, the image processing unit 11 obtains each configuration position Lo1, Lo2 from the layout information L1, and outputs to the two channels of the output port 13 to transmit the initial video V1, V2 respectively. Specifically, the image processing unit 11 first outputs data indicating the locations Lo1 and Lo2 of the initial videos V1 and V2 to the audio-visual playback device 3 through channels corresponding to the initial videos V1 and V2, and then outputs the initial video V1 through this channel. , V2 to the audio-visual playback device 3 .

In some other embodiments, the quantity of the output video signal Vo can be a single one, and the output video signal Vo is composed of initial videos V1 and V2. For example, the output video signal Vo can be an integrated video spliced from the original videos V1 and V2. At this time, the output port 13 can output the stereo audio signals S1 and S2 in the form of a single audio stream, that is, output a single stereo audio stream of the integrated stereo audio signals S1 and S2 (hereinafter referred to as the integrated audio signal So).

In the fourth embodiment of step S02, the image processing unit 11 may first designate the layout positions of the initial video V1 and V2 as the layout positions Lo1 and Lo2 in the layout information L1 in a one-to-one manner, and then receive the initial video V1 , V2 and then splicing the initial video V1, V2 into a single output video signal Vo according to the designated layout positions Lo1, Lo2 (as shown in Figure 3). Specifically, the image processing unit 11 combines the first image frame (that is, the initial video V1) and the second image frame (that is, the initial video V2) at the same playback time point according to the respective corresponding configuration positions Lo1 and Lo2 in the layout information L1. Or cut and combine to form the third image frame (ie output video signal Vo), and then output the third image frame to the audio-visual playback device 3 through the output port 13 . Wherein, a part of the third image frame is the first image frame (complete frame) or a part of the first image frame, and another part of the third image frame is the second image frame (complete frame) or a part of the second image frame.

Moreover, in step S03, after the audio processing unit 12 adjusts the audio signals A1 and A2 into stereo audio signals S1 and S2 according to the layout information L1, it also synthesizes the stereo audio signals S1 and S2 into a single stereo audio string according to the playing time. (hereinafter referred to as the integrated audio signal So), and then output the integrated audio signal So to each audio-visual playback device 3 through the output port 13 . When receiving the output video signal Vo and the integrated audio signal So, the audio-visual playback device 3 will play the output video signal Vo and the integrated audio signal So synchronously.

Then, with reference to FIG. 1 (or FIG. 2 ) and FIG. 3 , the sound processing unit 12 outputs one or more output video signals Vo and stereo audio signals S1 and S2 (that is, individually independent video signals V1 and V2) through the output port 13. The stereo sound signal S1, S2 or the integrated sound signal So after the integration of the stereo sound signal S1, S2) is sent to the audio-visual playback device 3 (step S04), and the audio-visual playback device 3 can synchronously play the initial video V1, V2 of the target 41, 42 And stereo audio signals S1, S2. Here, the playback effect of the stereo audio signal S1 corresponds to the layout position of the initial video V1 displayed on the video playback device 3 . The playback effect of the stereo audio signal S2 corresponds to the layout position of the original video V2 displayed on the audio-visual playback device 3 .

In this way, the user can experience the corresponding stereo audio signals S1, S2 through the different layout positions of the original video V1, V2 based on the audio-visual playback device 3, which can provide the spatialization effect of the user's audio.

In some embodiments, each configuration location Lo1, Lo2 can be coordinate information. In other words, the layout information L1 includes coordinate information indicating the layout positions of the initial videos V1 and V2. Moreover, the layout information L1 further includes coordinate information of the default reference point L2. Herein, the preset reference point L2 refers to the preset position of the viewer of the output image. In some embodiments, assuming that the layout of the display screen is configured in two dimensions (all the configuration positions Lo1 and Lo2 of the output screen shown in Figure 4 and Figure 5 are on the same XY plane), the preset reference point L2 can be The center point of the layout. In some other embodiments, assuming that the layout of the display screen is in a three-dimensional configuration (as shown in FIG. It is the center point of the vertical projection formed by all the layout positions in the layout configuration on the plane where the front layout position is located or on any parallel plane before the front layout position.

In some embodiments of step S03, taking the two initial videos V1 and V2 as an example, the sound processing unit 12 may use the coordinate information indicating the layout position of the initial video V1 in the layout information L1 and the preset reference position (such as preset Assume that the coordinate information of the reference point L2) determines the audio adjustment value (hereinafter referred to as the first audio adjustment value). Moreover, the sound processing unit 12 may determine another audio adjustment value (hereinafter referred to as the second audio adjustment value) according to the coordinate information of the layout information L1 used to indicate the layout position of the initial video V2 and the coordinate information of the preset reference point L2 . Then, the audio processing unit 12 adjusts the audio signal A1 into a stereo audio signal S1 according to the first audio adjustment value, and adjusts the audio signal A2 into a stereo audio signal S2 according to the second audio adjustment value.

In some embodiments of step S03, taking two initial videos V1 and V2 as an example, the sound processing unit 12 calculates the layout position and position of the initial video V1 based on the coordinate information corresponding to the initial video V1 and the coordinate information of the preset reference point L2. Preset the relative distance between the reference points L2, and then determine the audio adjustment value of the audio signal A1 of the initial video V1 according to the relative distance obtained above, and then use the determined audio adjustment value to adjust the corresponding audio signal A1 to generate stereo sound Signal S1. Similarly, the audio processing unit 12 also calculates the relative distance by using the coordinate information corresponding to the initial video V2 and the coordinate information of the preset reference point L2, and then determines the audio adjustment value of the audio signal A2 of the initial video V2, thereby adjusting the audio signal A2 And a stereo audio signal S2 is generated.

In some embodiments, the relative distance may be proportional to the audio adjustment value. That is to say, if the layout position of the initial video V1, V2 is closer to the preset reference point L2, the audio adjustment value generated by the sound processing unit 12 is smaller, that is, the audio processing unit 12 generates the stereo sound corresponding to the initial video V1, V2 The less audio adjustments need to be adjusted for signals S1 and S2. Conversely, if the layout position of the initial video V1, V2 is farther away from the preset reference point L2, the audio adjustment value generated by the sound processing unit 12 is larger, and the corresponding stereo audio signal S1, S2 needs to be adjusted more.

In some embodiments, the coordinate information corresponding to the initial videos V1 and V2 may include specified point coordinate information and edge length information. Wherein, the specified point coordinate information is the position of the specified point defining the layout positions of the initial videos V1 and V2 in the display layout. Wherein, the designated point coordinates of the initial videos V1 and V2 can be set as any point on the layout according to actual needs. The side length information is the side length or size that defines the layout positions of the original videos V1 and V2.

In some embodiments, assuming that the layout positions of the original videos V1 and V2 are rectangles, the side length information may be width information W1 and W2 and length information L3 and L4. Wherein, the width information W1 , W2 and length information L3 , L4 respectively define the lengths of two sides of the initial video V1 , V2 at the corresponding layout positions (as shown in FIG. 4 ).

In one embodiment, the designated point may be the origin. Taking the layout positions of the initial videos V1 and V2 as rectangles as an example, the coordinate information corresponding to the initial videos V1 and V2 may include origin coordinate information O1 and O2, width information W1 and W2, and length information L3 and L4. Wherein, the origin of the layout positions of the initial films V1 and V2 may be any point on the edge of the layout positions of the initial films V1 and V2. Preferably, the origin of the layout positions of the initial videos V1 and V2 may be the apex of the layout positions. Here, the width information W1 , W2 and the length information L3 , L4 may respectively be the lengths of two sides defining the layout positions of the original videos V1 , V2 connecting the origin.

In another embodiment, the designated point may be a center point. Taking the layout positions of the initial videos V1 and V2 as rectangles as an example, the coordinate information corresponding to the initial videos V1 and V2 may include center point coordinate information P1 and P2, width information W1 and W2, and length information L3 and L4. The center point coordinate information P1, P2 is the position of the center point defining the layout position of the initial video V1, V2 in the display layout.

In some embodiments, when the layout of the display screen is two-dimensional (as shown in FIG. 4 and FIG. 5 ), the coordinate information of the specified point and the default reference point L2 may respectively include vertical data and horizontal data. In some embodiments, the longitudinal data and the horizontal data may be X coordinate values and Y coordinate values respectively.

In some embodiments, as shown in FIG. 4 and FIG. 5 , it is taken as an example that the layout of the display screen is located on the X-Y plane. In the layout information L1, the coordinate information corresponding to the initial video V1 may include the origin coordinate information O1 of the layout position of the initial video V1 on the X-Y plane, the width information W1 and the length information L3 relative to the origin coordinate information O1, and correspond to The coordinate information of the initial video V2 may include origin coordinate information O2 on the X-Y plane in the layout position of the initial video V2, width information W2 and length information L4 relative to the origin coordinate information O2. For example, referring to FIG. 4 , it is taken that the layout of the display screen is divided into a left layout position 31 and a right layout position 32 as an example. The origin coordinate information O1 and the origin coordinate information O2 can be the coordinates of the vertex of the left layout position 31 (ie, the origin of the left layout position 31) and the vertex of the right layout position 32 (ie, the vertex of the right layout position 32). origin) coordinates. The origin coordinate information O2 is located to the right of the origin coordinate information O1. The width information W1 and the length information L3 are respectively the lengths of two sides connected to the origin on the left layout position 31 . The width information W2 and the length information L4 are respectively the lengths of two sides connected to the origin on the right layout position 32 . Taking the layout configuration of the display screen as an example, the entire layout pair is divided into an upper layout position 33 and a lower layout position 34 . The origin coordinate information O1 and the origin coordinate information O2 can be the coordinates of the vertex of the upper layout position 33 (ie the origin of the upper layout position 33) and the vertex of the lower layout position 34 (ie the origin of the lower layout position 34) respectively. coordinates, and the origin coordinate information O2 is located below the origin coordinate information O1. The width information W1 and the length information L3 are respectively the lengths of two sides connected to the origin on the upper layout position 33 . The width information W2 and the length information L4 are respectively the lengths of two sides connected to the origin on the lower layout position 34 . It should be clear that although the apex of the display block of each sub-picture is used as the origin of the layout position here as an example, it is not limited to this, and the same relative position of the display block of each sub-picture can be arbitrarily selected in fact. (that is, any point) as the origin of this layout position. Preferably, the origin of each layout position can be any vertex of the display block.

In some embodiments of step S03, the sound processing unit 12 can calculate the layout position of the initial video V1, V2 according to the origin coordinate information O1, O2, width information W1, W2 and length information L3, L4 of the initial video V1, V2 Center point coordinate information P1, P2. Next, the sound processing unit 12 calculates the relative distance between the layout position of the original video V1, V2 and the preset reference point L2 according to the center point coordinate information P1, P2 and the coordinate information of the preset reference point L2 (that is, the center point and the preset reference point L2 absolute distance between reference points L2). Then, the sound processing unit 12 determines the audio adjustment values of the initial video V1, V2 according to the obtained relative distance, and uses the determined audio adjustment values to adjust the corresponding audio signals A1, A2 to generate stereo audio signals S1, S2.

In one embodiment, the sound processing unit 12 can substitute the origin coordinate information O1, width information W1 and length information L3 into the center point calculation formula to calculate the center point coordinate information P1 of the layout position of the initial video V1 (eg, FIG. 4 The X coordinate value and the Y coordinate value of the center point of the left layout position 31 shown). Next, the sound processing unit 12 substitutes the coordinate information of the central point P1 and the coordinate information of the preset reference point L2 into the distance calculation formula to calculate the relative distance between the layout position of the initial video V1 and the preset reference point L2 on the X-Y plane (hereinafter called the first distance). Then, the sound processing unit 12 determines a first audio adjustment value according to the first distance. Similarly, the sound processing unit 12 can substitute the origin coordinate information O2, width information W2 and length information L4 into the center point calculation formula to calculate the center point coordinate information P2 of the layout position of the original video V2 (as shown in FIG. 4 The X coordinate value and the Y coordinate value of the center point of the right layout position 32 of . Next, the sound processing unit 12 substitutes the center point coordinate information P2 and the coordinate information of the preset reference point L2 into the distance calculation formula to calculate the relative distance on the X-Y plane between the layout position of the initial video V2 and the preset reference point L2 (hereinafter referred to as the second distance). Then, the sound processing unit 12 determines a second audio adjustment value according to the second distance.

When the center point coordinate information P1, P2 already exists in the coordinate information, in some embodiments of step S03, the sound processing unit 12 can directly calculate the initial video according to the center point coordinate information P1, P2 and the coordinate information of the preset reference point L2 The relative distance between the layout positions of V1 and V2 and the preset reference point L2. Then, the sound processing unit 12 determines the audio adjustment values of the initial video V1 and V2 according to the obtained relative distance, and uses the determined audio adjustment values to adjust the corresponding audio signals A1 and A2 to generate a stereo audio signal S1 (S2).

In some embodiments, the sound processing unit 12 does not need to actually calculate the relative distance between the layout position of each initial video V1, V2 and the preset reference point L2, but by comparing the configuration positions Lo1, Lo1, Lo2 and preset reference point L2 to obtain the pre-stored audio adjustment value.

In some embodiments of step S03, the sound processing unit 12 can compare the coordinate information of the configuration positions Lo1, Lo2 of the initial video V1, V2 with the preset reference point L2 (for example, comparing the origin coordinate information O1, O2 with the preset reference Coordinate information of point L2) to determine whether the layout position of the initial video V1 and V2 is the first position or the second position, and obtain the audio adjustment value corresponding to the judgment result in the pre-stored audio adjustment value, and then use the obtained audio adjustment value The corresponding audio signals A1, A2 are adjusted to generate stereo audio signals S1, S2.

Taking the two initial films V1 and V2 as examples, when the sound processing unit 12 judges that the layout position of the initial film V1 is the first position and the layout position of the initial film V2 is the second position, the sound processing unit 12 may base on the first position and the second position. The first audio adjustment value is determined by the default correspondence between the audio adjustment values, and the second audio adjustment value is determined according to the default correspondence between the second position and the audio adjustment value. For example, the storage unit stores a correspondence table of various layout positions and corresponding audio adjustment values. The sound processing unit 12 uses the first position and the second position to search for the same layout position in the corresponding table. Then, the sound processing unit 12 reads the audio adjustment value corresponding to the same layout position as the first position from the storage unit as the first audio adjustment value, and reads the audio adjustment value corresponding to the same layout position as the second position from the storage unit. The audio adjustment value is used as the second audio adjustment value. Conversely, when the audio processing unit 12 determines that the layout position of the initial video V1 is the second position and the layout position of the initial video V2 is the first position, the audio processing unit 12 may adjust the value according to the preset value between the second position and the audio. The correspondence determines the first audio adjustment value, and determines the second audio adjustment value according to the preset correspondence between the first position and the audio adjustment value. In this way, the sound processing unit 12 can save the amount of computation for actually calculating the relative distance value.

For example, take the first position as the left layout position 31 shown in FIG. 4 and the second position as the right layout position 32 shown in FIG. 4 as an example. The sound processing unit 12 compares the horizontal data of the origin coordinate information O1 and O2 corresponding to the initial videos V1 and V2 with the horizontal data of the coordinate information of the preset reference point L2. When the horizontal data of the origin coordinate information O1, O2 is greater than the horizontal data of the coordinate information of the preset reference point L2, the sound processing unit 12 determines that the layout position of the initial video V1, V2 is on the right relative to the preset reference point L2 , that is, determine the layout positions of the initial films V1 and V2 as the layout position 32 on the right. Conversely, when the horizontal data of the origin coordinate information O1, O2 is not greater than the horizontal data of the coordinate information of the preset reference point L2, the sound processing unit 12 determines that the layout position of the initial video V1, V2 is located relative to the preset reference point L2 , that is, determine the layout positions of the initial films V1 and V2 as the left layout position 31.

In another example, take the first position as the upper layout position 33 shown in FIG. 5 and the second position as the lower layout position 34 shown in FIG. 5 as an example. The sound processing unit 12 compares the longitudinal data of the origin coordinate information O1 and O2 corresponding to the initial videos V1 and V2 with the longitudinal data of the coordinate information of the preset reference point L2. When the vertical data of the origin coordinate information O1 is greater than the vertical data of the coordinate information of the preset reference point L2, the sound processing unit 12 determines that the layout position of the initial video V1 is above the preset reference point L2, that is, determines the initial The layout position of the movie V1' is the upper layout position 33 . Conversely, when the vertical data of the origin coordinate information O2 is not greater than the vertical data of the coordinate information of the preset reference point L2, the sound processing unit 12 determines that the layout position of the initial video V2 is below the preset reference point L2, that is, Determine the layout position of the initial movie V2 as the lower layout position 34 .

In some embodiments, each audio adjustment value includes adjustment information of at least one signal parameter of the corresponding audio signal A1 (or A2 ). Wherein, the adjustment information may be volume information, delay information, frequency information, or phase information, for example. The volume information is used to adjust the amplitude of the audio signal A1 (or A2). The delay information is used to adjust the delay time of the audio signal A1 (or A2). The frequency information is used to adjust the frequency components of the audio signal A1 (or A2). The phase information is used to adjust the phase of the audio signal A1 (or A2).

In some embodiments, the audio-visual playback device 3 includes at least two audio channels (for example, a left channel and a right channel) and the layout position of the aforementioned initial video V1 is the left layout position 31 and the layout position of the initial video V2 is Take position 32 on the right as an example. In order to create the stereo audio signal S1 with stereo audio effects relative to the X-axis direction, the aforementioned first audio adjustment value may include volume information and/or delay information. In the embodiment of step S03, the sound processing unit 12 can determine the volume information and/or the first audio adjustment value according to the coordinate information of the left layout position 31 (ie, the origin coordinate information O1, the length information L3 and the width information W1). or delayed information. Then, the audio processing unit 12 can attenuate the volume of the audio signal A1 of the initial video V1 according to the volume information of the first audio adjustment value (such as reducing the amplitude of the audio signal A1 by a specific value or a specific ratio according to the volume information), and according to The delay information of the first audio adjustment value adds a delay time to the audio signal A1 to generate a stereo audio signal S1 including the first signal component and the second signal component. Wherein, the attenuated and delayed audio signal A1 becomes the first signal component of the stereo audio signal S1, and the non-attenuated and non-delayed audio signal A1 can become the second signal component of the stereo audio signal S1. Then, the audio processing unit 12 sends the stereo audio signal S1 including the first signal component and the second signal component to the audio-visual playback device 3 . The video playback device 3 plays the first signal component of the stereo audio signal S1 through its right channel, and the video playback device 3 plays the second signal component of the stereo audio signal S1 through its left channel.

Similarly, in order to create the stereo audio signal S2 with stereo audio effects relative to the X-axis direction, the second audio adjustment value may include volume information and/or delay information. In an embodiment of step S03, the sound processing unit 12 can determine the volume information and /or delay information, the sound processing unit 12 can attenuate the volume of the sound signal A2 of the initial video V2 according to the volume information of the second audio adjustment value, and the sound processing unit 12 can further reduce the volume of the sound signal A2 according to the second audio adjustment value. A delay time is added to the audio signal A2 to generate a stereo audio signal S2 including the first signal component and the second signal component. Wherein, the attenuated and delayed audio signal A2 becomes the first signal component of the stereo audio signal S2, and the unattenuated and undelayed audio signal A2 can become the second signal component of the stereo audio signal S2. Then, the audio processing unit 12 sends the stereo audio signal S2 including the first signal component and the second signal component to the audio-visual playback device 3 . The video playback device 3 plays the first signal component of the stereo audio signal S2 through its left channel, and the video playback device 3 plays the second signal component of the stereo audio signal S2 through its right channel.

Therefore, the user of the audio-visual playing device 3 can feel that the main sound of the original video V1 is from the left, and that the main sound of the original video V2 is from the right. That is to say, the stereo audio signal S1 of the initial video V1 can be placed in the layout position of the audio-visual playback device 3 (ie, the left panel position 31) of the original video V1, and the stereo audio signal S2 of the initial video V2 can correspond to the audio-visual position of the original video V2. The layout position of the playback device 3 (ie, the right layout position 32).

In other words, the sound processing unit 12 can also adjust the volume (i.e. the amplitude) and / or delay time.

In some embodiments, the aforementioned audio-visual playback device 3 includes at least two audio channels (hereinafter referred to as the first audio channel and the second audio channel) and the layout position of the initial video V1 is the upper layout position 33 and the layout of the initial video V2 The position is the lower page position 34 as an example. In order to create a stereo sound signal S1 with stereo sound effects relative to the Y-axis direction, the aforementioned first audio adjustment value may include volume information and/or frequency information. In the embodiment of step S03, the sound processing unit 12 may determine the volume information and/or Frequency information, then, the sound processing unit 12 can attenuate the volume of the sound signal A1 of the initial video V1 according to the volume information of the first audio adjustment value, and the sound processing unit 12 can further use the equalization (EQ) method according to the first The frequency information of an audio adjustment value adjusts the frequency components of the audio signal A1 (for example, the audio processing unit 12 can reduce the high frequency components of the audio signal A1), so as to generate the stereo audio signal S1 including the first signal component and the second signal component. Here, the attenuated audio signal A1 with adjusted frequency components becomes the first signal component of the stereo audio signal S1, and the unattenuated audio signal A1 without adjusted frequency components becomes the second signal component of the stereo audio signal S1. Then, the audio processing unit 12 sends the stereo audio signal S1 including the first signal component and the second signal component to the audio-visual playback device 3 . The video playback device 3 plays the first signal component of the stereo audio signal S1 through its first channel, and the video playback device 3 plays the second signal component of the stereo audio signal S1 through its second channel.

Similarly, in order to create the stereo audio signal S2 with stereo audio effects relative to the Y-axis direction, the second audio adjustment value may include volume information and/or frequency information. In the embodiment of step S03, the sound processing unit 12 can determine the volume information and/or Frequency information, then, the sound processing unit 12 can attenuate the volume of the sound signal A2 of the initial video V2 according to the volume information of the second audio adjustment value, and the sound processing unit 12 can further adjust the volume according to the second audio by means of equalization The frequency information of the value adjusts the frequency components of the audio signal A2, so as to generate the stereo audio signal S2 including the first signal component and the second signal component. The attenuated audio signal A2 with adjusted frequency components becomes the first signal component of the stereo audio signal S2, and the unattenuated audio signal A2 without frequency adjustment becomes the second signal component of the stereo audio signal S2. The audio processing unit 12 then sends the stereo audio signal S2 including the first signal component and the second signal component to the audio-visual playback device 3 . The audio-visual playback device 3 plays the first signal component of the stereo audio signal S2 through its second channel, and the audio-visual playback device 3 plays the second signal component of the stereo audio signal S2 through its first channel.

Therefore, the user of the audio-visual playing device 3 can feel that the main sound of the original video V1 is from above, and that the main sound of the original video V2 is from below. That is to say, the stereo sound signal S1 of the initial video V1 can correspond to the layout position of the initial video V1 in the audio-visual playback device 3 (ie, the upper layout position 33 ) and the stereo audio signal S2 of the initial video V2 can correspond to the audio-visual playback of the initial video V2 Layout position of device 3 (ie, lower layout position 34).

In other words, the sound processing unit 12 can also adjust the volume and/or volume of the audio signal A1 (A2) of the initial video V1 (V2) according to the vertical position of the designated point corresponding to each initial video V1, V2 relative to the preset reference point L2. frequency components.

In some embodiments, when the layout of the display screen is in three dimensions (as shown in FIG. 6 ), the coordinate information of the specified point and the coordinate information of the default reference point L2 may include vertical data, horizontal data and depth data respectively. In some embodiments, the vertical data, horizontal data and depth data may be X coordinate values, Y coordinate values and Z coordinate values respectively.

In some embodiments of step S03, the sound processing unit 12 can also calculate the depth distance according to the depth data in the coordinate information of the designated point corresponding to each initial video V1, V2 and the depth data in the coordinate information of the preset reference point L2, And determine the phase information in the audio adjustment value of the audio signals A1, A2 of the initial video V1, V2 according to the calculated depth distance. In other embodiments, the sound processing unit 12 can determine the layout position of the initial video V1, V2 by comparing the coordinate information of the designated point corresponding to the initial video V1, V2 with the coordinate information of the preset reference point L2, and then from the preset The audio adjustment values (including phase information) corresponding to the judgment result are obtained from the set multiple audio adjustment values (including phase information).

In some examples, the layout of the display screen is located on the X-Y-Z plane (as shown in FIG. 6 ) and the display screen is a picture-in-picture (as shown in FIG. 7 ) as an example. Referring to FIG. 6 and FIG. 7 , the layout positions of the initial videos V1 and V2 can be respectively the front layout position 35 and the rear layout position 36 with different depth distances D1 and D2 relative to the preset reference L2 on the Z axis. The layout position of the initial video V1 may be the front layout position 35 , and the layout position of the initial video V2 may be the rear layout position 36 .

In the embodiment of step S03, the sound processing unit 12 can also calculate the depth distance D1 according to the depth data in the center point coordinate information P1 of the front layout position 35 and the depth data in the coordinate information of the preset reference point L2, and the sound The processing unit 12 then determines the phase information in the first audio adjustment value according to the calculated depth distance D1 (for example, the phase information may be proportional to the depth distance D1). Similarly, the sound processing unit 12 can also calculate the depth distance D2 according to the coordinate information P2 of the central point of the rear layout position 36 and the coordinate information of the preset reference point L2, and the sound processing unit 12 then determines the first depth distance D2 according to the calculated depth distance D2. Two phase information in the audio adjustment value (eg, the phase information may be proportional to the depth distance D2).

For example, take the aforementioned audio-visual playback device 3 including the first audio channel and the second audio channel and the layout position of the initial video V1 is the front layout position 35 and the layout position of the initial video V2 is the rear layout position 36 as an example. In order to create the stereo audio signal S1 with stereo audio effects relative to the Z-axis direction, the aforementioned first audio adjustment value may include volume information and phase information. In the embodiment of step S03, the sound processing unit 12 can determine the volume information and phase information of the first audio adjustment value according to the coordinate information of the front layout position 35 (such as origin coordinate information O1, length information L3 and width information W1) . Then, the audio processing unit 12 can attenuate the volume of the audio signal A1 of the initial video V1 according to the volume information of the first audio adjustment value, and the audio processing unit 12 can further adjust the audio signal A1 according to the phase information of the first audio adjustment value. (for example, the sound processing unit 12 can adjust the phase of the sound signal A1 in an anti-phase manner), so as to generate the stereo sound signal S1 including the first signal component and the second signal component. The attenuated and phase-adjusted audio signal A1 becomes the first signal component of the stereo audio signal S1, and the non-attenuated and non-phase-adjusted audio signal A1 becomes the second signal component of the stereo audio signal S1. Then, the audio processing unit 12 sends the stereo audio signal S1 including the first signal component and the second signal component to the audio-visual playback device 3 . The video playback device 3 plays the first signal component of the stereo audio signal S1 through its first channel, and the video playback device 3 plays the second signal component of the stereo audio signal S1 through its second channel.

Similarly, in order to create the stereo audio signal S2 with stereo audio effects relative to the Z-axis direction, the aforementioned second audio adjustment value may include volume information and phase information. In the embodiment of step S03, the sound processing unit 12 can determine the volume information and phase information of the second audio adjustment value according to the coordinate information of the rear layout position 36 (such as the origin coordinate information O2, the length information L4 and the width information W2). . Then, the audio processing unit 12 can attenuate the volume of the audio signal A2 of the initial video V2 according to the volume information of the second audio adjustment value, and the audio processing unit 12 can further adjust the audio signal A2 according to the phase information of the second audio adjustment value. (for example, the sound processing unit 12 can adjust the phase of the sound signal A2 in an anti-phase manner), so as to generate the stereo sound signal S2 including the first signal component and the second signal component. The attenuated and phase-adjusted audio signal A2 becomes the first signal component of the stereo audio signal S2, and the non-attenuated and non-phase-adjusted audio signal A2 becomes the second signal component of the stereo audio signal S2. The audio processing unit 12 then sends the stereo audio signal S2 including the first signal component and the second signal component to the audio-visual playback device 3 . The audio-visual playback device 3 plays the first signal component of the stereo audio signal S2 through its second channel, and the audio-visual playback device 3 plays the second signal component of the stereo audio signal S2 through its first channel. Therefore, the user of the audio-visual playback device 3 can feel that the main sounds of the initial videos V1 and V2 at different depths have different senses of distance.

In another example, the first position is the front layout position 35 shown in FIG. 6, the second position is the rear layout position 36 shown in FIG. Take the rear of point L2 as an example. The sound processing unit 12 can compare the depth data of the origin coordinate information O1, O2 corresponding to the initial video V1, V2 with the depth data of the coordinate information of the preset reference point L2 to obtain the layout positions of the initial video V1, V2. When the depth data corresponding to the initial video V1 is smaller than the depth data corresponding to the coordinate information of the preset reference point L2 and the depth data corresponding to the initial video V2, the sound processing unit 12 determines that the layout position of the initial video V1 is located relative to the preset reference point L2 The final position of the initial video V1 is determined as the rear layout position 36, and then the audio adjustment value corresponding to the rear layout position 36 is obtained from a plurality of preset audio adjustment values. Conversely, when the depth data corresponding to the initial video V1 is smaller than the depth data corresponding to the coordinate information of the preset reference point L2 but greater than the depth data corresponding to the initial video V2, the sound processing unit 12 determines that the layout position of the initial video V1 is located relative to the preset The rear of the reference point L2 and the front relative to the layout position of the initial video V2, that is, determine the layout position of the initial video V1 as the front layout position 35, and then obtain the audio corresponding to the front layout position 35 from a plurality of preset audio adjustment values Adjust the value.

In other words, the audio processing unit 12 can also adjust the volume and/or phase of the audio signals A1, A2 of the initial video V1, V2 according to the depth of the designated point corresponding to the initial video V1, V2 relative to the preset reference point L2.

In one embodiment, the first audio channel may be the left channel and the second audio channel may be the right channel. In another embodiment, the first channel may be an upper channel and the second channel may be a lower channel.

Here, the above-mentioned embodiments are only examples to illustrate how to adjust the audio signal to a stereo audio signal, and it is adjusted for any combination of audio parameters such as volume, delay, phase, frequency or others, so that the stereo audio signal can be more accurate. In line with the current situation of use. For example, the stereo sound signal obtained by adjusting two, three or more sound parameters such as the volume, delay and phase of the sound signal can allow the user to hear the sound coming from the left front, right front, left rear, right rear, Either upper left, upper right, lower left or lower right, etc. In other words, one or more sound parameters can be adjusted according to needs to make the listener more stereoscopic.

It should be clear that, in the above-mentioned embodiments, although each audio-visual capture device 2 respectively outputs the initial video V1 (or V2) and the audio signal A1 (or A2) to the audio-visual processing device 1 for illustration, it does not limit. In other words, in some embodiments, the video capture device 2 can output a single video stream to the video processing device 1, and then perform video processing on the input video stream through the video separation unit (not shown) of the video processing device 1. Separation from audio signals. Wherein, the audio-video separation unit is coupled between the input unit 10 and the image processing unit 11 and between the input unit 10 and the audio processing unit 12 .

Likewise, in some embodiments, the video processing device 1 can output a single video stream to the video playing device 3 . Here, the audio-visual processing device 1 can combine the output video signal Vo and the integrated audio signal So of the original video V1, V2 corresponding to the source mark or the configuration position Lo1, Lo2 into a single audio-visual string through the video-audio synthesis unit (not shown in the figure). The stream is then output to the audio-visual playback device 3 through the output port 13. Here, the audio-video synthesis unit is coupled between the image processing unit 11 and the output port 13 and between the audio processing unit 12 and the output port 13 .

In some embodiments, the aforementioned image processing unit 11 and audio processing unit 12 (and the audio-video synthesis unit and/or the video-audio synthesis unit) may be realized by one or more processing units. Wherein, each processing unit may be any analog and/or digital device based on an operation instruction to operate a signal, but is not limited thereto.

In some embodiments, the aforementioned storage unit may be coupled to the image processing unit 11 or built in the image processing unit 11 . Here, the storage unit can be realized by one or more storage elements. Wherein, the storage element may be, for example, a memory or a register, but is not limited thereto.

In some embodiments, the input unit 10 can be realized by a plurality of connection ports or by one or more network modules. Each connection port can be coupled to an external device of the video processing device 1 (ie, the video capture device 2 ) via a transmission line. Here, the port supports video, video or audio transmission. The network module can conform to any wireless communication protocol or wired communication protocol.

In some embodiments, the output port 13 can be a physical audio-visual connector, or a connector supporting any wireless communication protocol.

In some embodiments, when the audio-visual processing device 1 is coupled to multiple audio-visual playback devices 3, all the audio-visual playback devices 3 can receive the same file from the audio-visual processing device 1, that is, individually receive the same output video signal Vo and stereo sound. Signal S1, S2 (or So).

In some embodiments, the audio-spatialized audio-visual processing method according to the present invention can be implemented by a computer program product, so that the audio-spatialized audio-visual processing method of any embodiment can be completed after the audio-visual processing device 1 is loaded with the program and executed. . In other words, the audio-spatialized audio-visual processing method of any embodiment is implemented on the audio-visual processing device 1 by one or more processing units executing firmware or software algorithms. In some embodiments, the computer program product can be a readable recording medium, and the above programs are stored in the readable recording medium for loading by the audio-visual processing device 1 . In some embodiments, the above-mentioned program itself can be a computer program product, and is transmitted to the audio-visual processing device 1 via wired or wireless means.

To sum up, according to any embodiment of the audio-visual processing method and the audio-visual processing device of the present application, the audio-visual processing device 1 can correspondingly adjust the signal parameters of the audio signal of the video according to the display position of the video. In some embodiments, the stereo sound signal S1, S2 (or So) has a difference in volume, delay, phase, frequency or other combinations compared with the sound signal A1, A2; or the stereo sound signal S1, S2 (or So ) is corresponding to each channel of the audio-visual playback device 3. Here, when the stereo audio signal S1, S2 (or So) is output in different channels, its signal parameters in any combination of volume, delay, phase, frequency or other are different from each other. In this way, the audio spatialization effect in the three-dimensional space can be established, so that users can enjoy stereoscopic video and audio effects.

1: Audio-visual processing device 10: Input unit 11: Image processing unit 12: Sound processing unit 13: Output port 2,21,22: Audio and video capture device 3: Video player 31: Left layout position 32: Right layout position 33: Upper layout position 34: Bottom layout position 35: Front layout position 36: Rear layout position 41: target 42: target V1, V2: Initial movie Vo: output video signal A1, A2: Audio signal C1: Control signal L1: Layout Information L2: preset reference point L3, L4: length information W1, W2: width information S1, S2: Stereo audio signal So: Integrate audio signals O1, O2: origin coordinate information P1, P2: center point coordinate information D1, D2: depth distance S01~S04: Steps

[FIG. 1] is a schematic block diagram of an embodiment of an audio-visual processing device connected to an audio-visual capture device and an audio-visual playback device according to the present application. [ FIG. 2 ] is a schematic block diagram of another embodiment in which the audio-visual processing device is connected to the audio-visual capture device and the audio-visual playback device according to the present application. [Fig. 3] is a flow chart of an embodiment of the video and audio processing method according to the present case. [ FIG. 4 ] is a schematic diagram of an embodiment of a display panel. [Fig. 5] is a schematic diagram of another embodiment of the display panel. [ FIG. 6 ] is a schematic diagram of another embodiment of the display panel. [ FIG. 7 ] is a schematic diagram of a display screen presented on the display panel of FIG. 6 .

1: Audio-visual processing device 10: Input unit 11: Image processing unit 12: Sound processing unit 13: Output port 2,21,22: Audio and video capture device 3: Video player 41,42: target V1, V2: Initial movie A1, A2: Audio signal S1, S2: Stereo audio signal C1: Control signal L1: Layout information

Claims (15)

An audio-spatial audio-visual processing method is applicable to an audio-visual processing device, the audio-visual processing device is connected to an audio-visual playback device, and the audio-visual processing method includes: receiving a plurality of initial videos and an audio signal corresponding to each of the initial videos by the audio-video processing device; generating at least one output video signal according to the plurality of initial videos by the video-audio processing device; adjusting each of the audio signals into a stereo audio signal according to a layout information by the audio-visual processing device; and Outputting the at least one output video signal and the plurality of stereo audio signals to the audio-visual playback device by the audio-visual processing device, so that the audio-visual playback device outputs the layout information with the plurality of initial videos in the layout by playing the at least one output video signal A display frame on the corresponding plurality of display blocks. The audio-spatial audio-visual processing method as described in Claim 1, wherein the quantity of the at least one output video signal is one, and a display layout of the audio-visual playback device includes a plurality of layout positions respectively corresponding to the plurality of display blocks, and borrowing The step of generating the at least one output video signal according to the plurality of initial videos by the audio-visual processing device includes: arranging the plurality of initial videos on the plurality of layout positions by the audio-visual processing device according to the layout information to form the output output video signal. The audio-spatial audio-visual processing method as described in Claim 1, wherein the at least one output video signal includes the plurality of initial videos, and a display layout of the audio-visual playback device includes a plurality of layout positions respectively corresponding to the plurality of display blocks, and The step of the audio-visual playback device outputting the display images on the plurality of display blocks corresponding to the layout information with the plurality of initial videos by playing the at least one output video signal includes: using the audio-visual playback device according to the layout The information shows that the plurality of initial videos are in the plurality of layout positions. The audio-visual processing method for audio spatialization as described in Claim 1, wherein the layout information includes coordinate information corresponding to each of the initial videos, and the audio-visual processing device adjusts each of the audio signals according to the layout information to the The steps of stereo audio signal include: determining an audio adjustment value based on the coordinate information and a default reference point; and The audio signal is adjusted to the stereo audio signal according to the audio adjustment value. The audio-spatial audio-visual processing method as described in Claim 4, wherein the step of determining the audio adjustment value according to the coordinate information and the preset reference point includes: judging a center point corresponding to a layout position of one of the plurality of display blocks according to each of the coordinate information; and The audio adjustment value is determined according to a relative distance between the center point and the preset reference point. The audio-visual processing method of audio spatialization as described in claim 4, wherein the number of the plurality of display blocks is two, and a display layout displayed by the audio-visual playback device includes respectively corresponding to the plurality of display blocks and relative to the preset A first position and a second position of the reference point, and the steps of determining the audio adjustment value according to the coordinate information and the default reference point include: judging a layout position of the initial video as the first position or the second position based on the coordinate information and the default reference point; and The audio adjustment value is determined according to the judging result that the layout position is the first position or the second position. The audio-spatial audio-video processing method as described in Claim 1, wherein the layout information includes a configuration position corresponding to each of the initial videos, and each configuration position is defined by a specified point coordinate information and side length information, the The designated point coordinate information defines the position of a designated point of a layout position of the initial video in a display layout of the audio-visual playback device, and the side length information defines the size of the layout position. The audio-visual processing method for audio spatialization as described in Claim 7, wherein the coordinate information of each designated point includes a horizontal data, a vertical data, a depth data or any combination thereof, and the audio-visual processing device is configured according to the layout The step of information adjusting each of the audio signals to the stereo audio signal includes: determining a volume information, a delay information, a phase information, a frequency information, or any combination thereof based on the horizontal data, the vertical data, the depth data, or any combination thereof; and The stereo sound signal is generated by adjusting the volume, delay time, phase, frequency or other sound parameters of the sound signal according to the volume information, the delay information, the phase information, the frequency information or any combination thereof. An audio-visual processing device, suitable for connecting with an audio-visual playback device, the audio-visual processing device includes: An image processing unit, receiving a plurality of initial images and generating at least one output image signal according to the plurality of initial images; An audio processing unit, connected to the image processing unit, receives a layout information and an audio signal corresponding to each of the original videos, and adjusts each of the audio signals into a stereo audio signal according to the layout information; and An output port, coupled to the image processing unit and the sound processing unit, outputs the stereo audio signal and the at least one output video signal to the audio-visual playback device, so that the audio-visual playback device outputs the at least one output video signal by playing the at least one output video signal. A display frame of the plurality of initial videos on the plurality of display blocks corresponding to the layout information. The audio-visual processing device as described in claim 9, wherein the quantity of the at least one output video signal is one, a display layout displayed by the audio-visual playback device includes a plurality of layout positions respectively corresponding to the plurality of display blocks, and the image processing unit disposing the plurality of initial videos on the plurality of layout positions according to the layout information to form the output video signal for output. The audio-visual processing device as described in Claim 9, wherein the layout information includes coordinate information corresponding to each initial video, and the sound processing unit determines an audio adjustment value according to each coordinate information and a preset reference point, and The stereo audio signal is generated by adjusting the corresponding audio signal according to each audio adjustment value. The audio-visual processing device as described in claim 11, wherein the sound processing unit judges a center point corresponding to a layout position of one of the plurality of display blocks according to each of the coordinate information and according to each of the center points and the preset A relative distance between the reference points is used to determine the audio adjustment value. The audio-visual processing device as described in claim 11, wherein the number of the plurality of display blocks is two, and the display layout displayed by the audio-visual playback device includes images respectively corresponding to the plurality of display blocks and relative to a preset reference position A first position and a second position, and the sound processing unit judges that a layout position of the initial video is the first position or the second position according to the coordinate information and the preset reference point, so as to determine the audio adjustment value. The audio-visual processing device as described in claim 9, wherein the layout information includes a configuration position corresponding to each of the initial videos, and each configuration position is defined by a designated point coordinate information and side length information, and the designated point coordinate information The position of a designated point defining a layout position in a display layout of the audio-visual playback device, and the side length information defines the size of the layout position. The audio-visual processing device as described in claim 14, wherein each of the designated point coordinate information includes a horizontal data, a vertical data, a depth data or any combination thereof, and the sound processing unit is based on the horizontal data, the vertical data, The depth data or any combination thereof determines a volume information, a delay information, a phase information, a frequency information or any combination thereof for adjusting the corresponding sound signal.

Images