CN114422743A - Video stream display method, device, computer equipment and storage medium - Google Patents

Abstract

The embodiment of the present application discloses a video stream display method, device, computer equipment and storage medium; the embodiment of the present application acquires multiple video streams and sound source positions of the current scene, and an image acquisition area corresponding to each of the video streams; The position of the sound source and the image acquisition area are determined from the multi-channel video streams to determine a target video stream; according to the target video stream, a target object is identified, and the target object is an object with lip movements; According to the identification result of the target object, the video stream to be displayed is determined from the multi-channel video streams, and the picture corresponding to the video stream to be displayed is displayed. In the embodiment of the present application, the target video stream used for identifying the speaker is determined by the position of the sound source, which can improve the efficiency of identifying the speaker, and at the same time, the video stream to be displayed can be determined according to the identification result, so that the displayed picture can be focused on the speaker. to present a better meeting picture.

Description

Video stream display method, apparatus, computer equipment and storage medium

technical field

The present application relates to the field of computer technology, and in particular, to a video stream display method, apparatus, computer device, and storage medium.

Background technique

With the development of video technology, more and more occasions will capture and play live pictures in real time through cameras. However, in scenes with multiple people participating, the images captured by the cameras usually fail to highlight the key points in the current scene.

Especially in the scenario of a multi-person conference, there are often multiple different speakers during the conference process. How to focus the displayed image on the speaker and present a better conference image is a problem that needs to be solved urgently.

SUMMARY OF THE INVENTION

Embodiments of the present application provide a video stream display method, apparatus, computer device, and storage medium, which can make the displayed picture focus on the speaker and present a better conference picture.

An embodiment of the present application provides a video stream display method, including: acquiring multiple video streams and sound source positions of a current scene, and an image acquisition area corresponding to each of the video streams; according to the sound source position and the image The acquisition area is to determine the target video stream from the multi-channel video streams; according to the target video stream, identify the target object, and the target object is an object with lip movements; according to the recognition result of the target object, from the Determining a video stream to be displayed among the multiple video streams; and displaying a picture corresponding to the video stream to be displayed.

An embodiment of the present application further provides a video stream display device, including: an acquisition unit for acquiring multiple video streams and sound source positions of a current scene, and an image acquisition area corresponding to each of the video streams; a first determination unit , used to determine the target video stream from the multi-channel video stream according to the sound source position and the image acquisition area; the identification unit is used to identify the target object according to the target video stream, and the target object is an object with lip movements; a second determination unit for determining a video stream to be displayed from the multi-channel video streams according to the recognition result of the target object; a display unit for displaying the video to be displayed The corresponding screen of the stream.

The embodiments of the present application further provide a computer device, including a memory storing a plurality of instructions; the processor loads the instructions from the memory to execute the steps in any of the video stream display methods provided by the embodiments of the present application .

Embodiments of the present application further provide a computer-readable storage medium, where the computer-readable storage medium stores a plurality of instructions, and the instructions are adapted to be loaded by a processor to execute any video provided by the embodiments of the present application A flow shows the steps in a method.

This embodiment of the present application can acquire multiple video streams and sound source positions of the current scene, and an image capture area corresponding to each of the video streams; Determine a target video stream in the stream; identify a target object according to the target video stream, and the target object is an object with lip movements; according to the recognition result of the target object, determine from the multi-channel video streams to be displayed the video stream; display the picture corresponding to the to-be-displayed video stream. In this application, the target video stream used to identify the speaker is determined by the position of the sound source, which can improve the efficiency of identifying the speaker. Better meeting images.

Description of drawings

In order to illustrate the technical solutions in the embodiments of the present application more clearly, the following briefly introduces the drawings that are used in the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of the present application. For those skilled in the art, other drawings can also be obtained from these drawings without creative effort.

1 is a schematic diagram of a scene of a video stream display system provided by an embodiment of the present application;

2 is a schematic flowchart of a method for displaying a video stream provided by an embodiment of the present application;

3 is a schematic structural diagram of a video stream display system provided by an embodiment of the present application;

4 is a schematic flowchart of a data processing module provided by an embodiment of the present application;

5 is a schematic flowchart of a video stream display method provided by another embodiment of the present application;

6 is a schematic structural diagram of a video stream display device provided by an embodiment of the present application;

FIG. 7 is a schematic structural diagram of a computer device provided by an embodiment of the present application.

Detailed ways

The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the protection scope of the present application.

Embodiments of the present application provide a video stream display method, apparatus, computer device, and storage medium.

Specifically, the video stream display apparatus may be integrated in an electronic device, and the electronic device may be a terminal, a server, or the like. The terminal may be a mobile phone, a tablet computer, an intelligent Bluetooth device, a notebook computer, or a personal computer (Personal Computer, PC) and other devices; the server may be a single server, or a server cluster composed of multiple servers.

In some embodiments, the video stream display apparatus may also be integrated into multiple electronic devices, for example, the video stream display apparatus may be integrated into multiple servers, and the video stream display method of the present application may be implemented by multiple servers.

In some embodiments, the server may also be implemented in the form of a terminal.

For example, referring to FIG. 1 , in some embodiments, a scene schematic diagram of a video stream display system is provided, and the image rendering system may include a display data acquisition module 1000 , a server 2000 and a terminal 3000 .

The data acquisition module can acquire multiple video streams and sound source positions of the current scene, and an image acquisition area corresponding to each video stream.

Among them, the server can determine the target video stream from the multiple video streams according to the position of the sound source and the image acquisition area; according to the target video stream, identify the target object, and the target object is an object with lip movements; according to the recognition result of the target object, Determine the video stream to be displayed from the multiple video streams.

The terminal may display a picture corresponding to the video stream to be displayed.

Each of them will be described in detail below. It should be noted that the serial numbers of the following embodiments are not intended to limit the preferred order of the embodiments.

In this embodiment, a video stream-based display method is provided. As shown in FIG. 2 , the specific process of the video stream display method may be as follows:

110. Acquire multiple video streams and sound source positions of the current scene, and an image acquisition area corresponding to each video stream.

The position of the sound source refers to the position where the sound is emitted in the current scene, for example, the position where the speech sound is emitted in the conference scene. You can collect sound by setting a microphone array in the current scene, and calculate the sound source position according to the sound source localization algorithm.

In some embodiments, the multiple video streams include one panoramic video stream and at least one close-up video stream. The image acquisition area refers to the area of the current scene corresponding to the image that can be acquired by the image acquisition device corresponding to the video stream. Among them, the panoramic video stream refers to the video stream containing the panoramic picture of the current scene, and the corresponding image acquisition area is the current scene panorama, which can be collected by a camera with a wide-angle lens, and the close-up video stream refers to the current scene. The corresponding image collection area is the local scene of the current scene, which can be collected by a camera with a telephoto lens.

In some embodiments, the method for acquiring the sound source position may include steps 1.1 to 1.2, as follows:

1.1. Collect the sound information of the current scene;

1.2. Process the collected sound information through the sound source localization algorithm to obtain the sound source position.

Among them, the sound source localization algorithm may adopt TDOA (Time Difference of Arrival, time difference of arrival), GCC-PHAT (Generalized Cross Correlation PHAse Transformation, generalized cross correlation-phase transformation method), and so on.

120. Determine the target video stream from the multiple video streams according to the sound source position and the image acquisition area.

The target video stream is a video stream determined according to the position of the sound source and the relationship between the image acquisition areas. The association relationship may be that the sound source position is located in the image collection area, or the distance between the sound source position and the center of the image collection area is less than a preset distance.

In some embodiments, step 120 may include the step of: determining a target image capture area from the capture areas corresponding to the multiple multi-channel video streams according to the position of the sound source and the relationship between the image capture areas, and selecting the video stream corresponding to the target image capture area Determined as the target video stream.

In some embodiments, due to interference such as reflection, noise, etc., there may be errors in the sound source location determined by sound source localization. Therefore, the possible area where the sound source exists is determined by the sound source location, so as to determine the target corresponding to the area. The video stream increases the accuracy of the acquired image information. Specifically, step 120 may include steps 2.1 to 2.4, as follows:

2.1. Determine the sound source area according to the sound source position;

2.2. For each video stream, determine the overlapping area of the sound source area and the image acquisition area;

2.3. Determine the overlapping area that meets the preset first area size as the target area;

2.4. Determine the video stream corresponding to the target area as the target video stream.

The sound source area refers to the area where the sound source is located, and the area and the image acquisition area are located on the same plane. The sound source area can be determined according to the sound source position and the preset area parameter value. The preset area parameter value can be set according to the current scene or experience. For example, a circle with the sound source position as the center and the preset radius value as the radius will be used. zones as sound source zones, and so on.

In some embodiments, step 2.1 may include the steps of: acquiring a reference point; taking the reference point as a vertex, and a line connecting the reference point and the sound source position as an angle bisector, and determining an included angle that satisfies the preset first angle; The included angle is determined as the sound source area in the area corresponding to the current scene. The reference point may be any boundary point in the current scene. In some embodiments, the reference point may be a location point where sound information of the current scene is collected, for example, a point determined according to a microphone array used to measure the location of the sound source, the point may be any location point on the microphone array, or Can be the midpoint. It should be noted that the reference point, the sound source position, the image acquisition area, the sound source area and the target area are all on the same plane, which can be a horizontal plane. For example, the sound source position is the real sound source position calculated by the sound source localization algorithm. The position point projected onto the horizontal plane.

The preset first area size is a size condition of the area set according to the current scene or experience. It can be a specific value, such as greater than or equal to one-third of the image capture area corresponding to any video stream, or greater than or equal to the area size determined according to the sound source area, such as greater than or equal to half the size of the sound source area one.

In some embodiments, step 2.3 may include the step of: determining an overlapping area with the same size of the sound source area as the target area.

130. Identify a target object according to the target video stream, where the target object is an object with lip movements.

The target object refers to an object with lip movements identified from the image information of the target video stream. Generally speaking, the person with the lip movement is speaking and thus can act as the speaker for the current scene. The lip movement may be the lip movement of a person when speaking determined according to the prior art.

Since the image acquisition areas corresponding to different video streams are different, the target video stream used to identify the speaker can be determined by the position of the sound source, which can reduce the amount of identification data and improve the efficiency of identifying the speaker.

In some embodiments, due to interference such as reflection, noise, etc., there may be errors in the position of the sound source determined by the sound source localization. Therefore, the area used to identify the speaker is determined by the sound source position, so as to determine the area corresponding to the area. The target video stream increases the accuracy of the acquired image information. Step 130 may include steps 3.1 to 3.3, as follows:

3.1. Determine the recognition area according to the location of the sound source;

3.2. According to the recognition area, obtain target image information from the target video stream, and the target image information is the image information corresponding to the recognition area;

3.3. Identify the target object according to the target image information.

Wherein, the identification area refers to the area determined according to the position of the sound source and used to identify the target object, and the area and the image acquisition area are located on the same plane. The position of the sound source is located in the identification area. The identification area can be determined according to the position of the sound source and the preset area parameter values. The preset area parameter values can be set according to the current scene or experience. A circular area with a radius value as a recognition area, and so on. The identification area may also be the sound source area.

In some embodiments, step 3.1 may include the steps of: acquiring a reference point; taking the reference point as a vertex, and a line connecting the reference point and the sound source position as an angle bisector, and determining an included angle that satisfies the preset second angle; The area corresponding to the included angle in the current scene is determined as the recognition area.

The target image information refers to the image information of the recognition area projected into the area of the image captured by the target video stream. Specifically, the coordinate position of the recognition area can be obtained, and the coordinate position can be projected into the coordinate system where the image captured by the target video stream is located to obtain the projected area, and the image information in the area is used as the target image information.

The recognition area that may contain the target object is determined by the position of the sound source, the image information corresponding to the recognition area is obtained from the target video stream through the recognition area, and whether there is a target object is identified from the image information.

In some embodiments, in order to improve the recognition efficiency, step 3.3 may include steps 3.3.1 to 3.3.2, as follows:

3.3.1. When an object with lip movements is identified from the target image information, the object with lip movements is used as the target object;

3.3.2. When the object with lip motion is not recognized from the target image information, expand the recognition area to the preset second area size to recognize the target object.

For example, first set the recognition area as a fan-shaped area with an included angle of 30°, when the target object is not recognized in this area, expand the recognition area to a fan-shaped area with an included angle of 40°, and perform the recognition again. When no target object is recognized, the recognition area is expanded to a fan-shaped area with an included angle of 50°, and so on, until the target object is recognized or the recognition area is expanded to the upper limit.

Since the position of the sound source determined by sound source localization may have errors, the preset recognition area may not recognize the target object when performing lip motion recognition. At this time, by gradually expanding the recognition area, the recognition range can be expanded to This correction of the recognition result and the gradual expansion of the size of the recognition area can also make the area to be recognized each time smaller than the next recognition, so as to obtain the recognition result with the smallest area as much as possible, and improve the recognition efficiency.

In some embodiments, in order to further improve the recognition efficiency, step 3.3.2 may include the step of: when an object with lip movements is not recognized from the target image information, expanding the recognition area to a preset second area size to obtain the enlarged The non-overlapping area between the recognition area and the enlarged area is used as the target recognition area; according to the target recognition area, the target image information is obtained from the target video stream, and the target image information is the image information corresponding to the recognition area; according to the target image information , identify the target object.

140. Determine the video stream to be displayed from the multiple video streams according to the identification result of the target object.

The video stream to be displayed refers to a video stream used to display the current scene. The target object can be displayed in focus through the video stream to be displayed.

In some embodiments, in order to provide a better display effect of the current scene, a display strategy determined according to the target object recognition result is provided, and step 140 may include steps 4.1 to 4.4, as follows:

4.1. When the target object is identified, determine the area to be displayed according to the target object;

4.2. When the target object is not recognized, determine the area to be displayed according to all objects in the target image information;

4.3. Obtain the image capture area corresponding to each video stream;

4.4. Determine the video stream to be displayed according to the area to be displayed and the image acquisition area.

The to-be-displayed area refers to an area to be displayed through the to-be-displayed video stream. When the target object is recognized, the area where the target object is located, such as the sound source area or the recognition area, can be used as the area to be displayed. When the target object is not recognized, the area where all objects in the target image information are located can be used as the area to be displayed. The to-be-displayed area can be on the same plane as the image acquisition area or the image corresponding to the target video stream. When comparing the to-be-displayed area with the image acquisition area and other different plane areas, the to-be-displayed area can be projected onto the image. Compare the plane where the collection area is located.

In some embodiments, when multiple target objects are identified, the area to be displayed is determined according to the multiple target objects. At this time, the to-be-displayed area is an area where multiple target objects are located.

The to-be-displayed area is determined by the recognition result of the target object, and the to-be-displayed area is compared with the image acquisition area to determine the to-be-displayed video stream. For example, by determining the overlapping area between the area to be displayed and each image capturing area, the video stream corresponding to the image capturing area with the largest overlapping area can be used as the video stream to be displayed.

In some embodiments, in order to be able to focus on the speaker and provide a better display effect of the current scene, step 4.4 may include the step of: determining the area size ratio between the area to be displayed and each image acquisition area, and dividing the size of the area to be displayed/image acquisition The video stream corresponding to the image acquisition area with the highest area size ratio is used as the video stream to be displayed. In some implementation manners, in order to prevent the displayed speaker picture from being incomplete, the ratio of the size of the area to be displayed/the size of the image acquisition area is smaller than a preset value, and the value may be 1.

150. Display a picture corresponding to the video stream to be displayed.

In some embodiments, by cropping the display screen and focusing on the speaker, a better display effect of the current scene is provided. Step 150 may include steps 5.1 to 5.3, as follows:

5.1. Obtain the display screen of the video stream to be displayed;

5.2. According to the area to be displayed, trim the display image of the video stream to be displayed, and obtain the trimmed display image;

5.3. Display the cropped display screen.

The cropped display picture is a picture corresponding to the to-be-displayed area in the display picture of the video stream to be displayed.

By trimming the display image of the video stream to be displayed to the image corresponding to the area to be displayed, the speaker can be further focused, so as to provide a better display effect of the current scene.

The video stream display method provided by the embodiment of the present application can be applied in various scenarios where multiple people participate. For example, taking a multi-person conference as an example, obtain the multi-channel video streams and sound source positions of the current scene, and an image capture area corresponding to each video stream; determine the target from the multi-channel video streams according to the sound source position and image capture area Video stream; identify the target object according to the target video stream, and the target object is an object with lip movements; according to the recognition result of the target object, determine the video stream to be displayed from the multiple video streams; display the corresponding video stream to be displayed. screen. Using the solution provided by the embodiment of the present application to determine the target video stream used to identify the speaker through the position of the sound source can improve the efficiency of identifying the speaker. On the person, present a better picture of the meeting.

According to the methods described in the above embodiments, further detailed description will be given below.

In this embodiment, the method of the embodiment of the present application will be described in detail by taking a multi-person conference scenario as an example.

As shown in FIG. 3 , a schematic structural diagram of a video stream display system is provided, and the system includes a data acquisition module, a data processing module and a terminal.

Among them, the data acquisition module is composed of an infrared thermal imager, an ultrasonic module, a dual camera module and an array microphone. The camera module collects information and sends it to the data processing module. details as follows:

The data acquisition module includes two cameras, and the two cameras are respectively a wide-angle lens and a telephoto lens. The wide-angle lens has a large field of view and a wide viewing range, but the distant view is blurred. The telephoto lens has a small field of view and a narrow viewing range, but it can see a clear distant view. When the field of view overlaps and cuts to the camera with the telephoto lens, when the field of view is outside the range of the telephoto lens, it switches to the camera with the wide-angle lens. The dual-camera switching method includes: 1. The dual-camera module includes two types of cameras with a wide-angle lens and a camera with a telephoto lens. The camera with a wide-angle lens has a short focal length and a wide field of view. The camera has a long focal length, a narrow field of view, and fewer pictures, and a larger proportion of objects in the picture. Both the wide-angle lens camera and the telephoto lens camera can output two video streams, one video stream is used for the actual picture presentation, which can be called the preview stream, and the other video stream is used for lip motion detection and face recognition for AI, which can be Called AI Image Streaming. 2. The screen presentation of the terminal can only be one of the preview streams from the two cameras, but the AI image streams of the two cameras can be provided to the image AI thread for lip motion detection and face recognition at the same time. 3. The image AI thread decides to perform lip motion recognition and face recognition on one of the two AI image streams according to the angle information of sound source localization, and then outputs it to the UVC thread to decide which preview stream to cut and crop. , finally showing the face focus effect.

Infrared thermal imager, used to measure the temperature of the target object.

The ultrasonic module is used to detect the distance of the target object in combination with the infrared thermal imager. Since the infrared thermal imager is essentially a camera, the minimum imaging distance of its lens is required. For example, the distance between the measured object and the lens should be greater than 25cm to ensure a clear thermal image. Therefore, the ultrasonic module can be used to detect the distance of the target and prompt the distance requirements of the target.

Matrix microphone module, used for sound source localization to determine the position of the speaker.

Among them, the data processing module includes UVC thread, UAC thread, image AI thread and audio AI thread, and the data processing module obtains the information collected by the camera module and performs data processing. As shown in Figure 4, the workflow of the thread in the data processing module is as follows:

UVC thread is used to collect video stream information of dual cameras. Each camera will output two video streams, one is used for outputting real-time images to the terminal, and the other is used for image AI thread analysis of lip motion analysis and face identify.

The UAC thread is used to collect the audio stream information of the array microphones. It is divided into two types of audio information output. One type of audio information is to directly output the PCM format audio stream data of one of the microphones to the terminal for audio playback. The audio stream data in PCM format collected by all microphones is combined and sent to the audio AI thread for sound source localization.

The image AI thread is used to analyze and process the image information of the two cameras output by the UVC thread, and output the decision to the UVC thread. Focus on speakers. Specifically, the image AI thread obtains two kinds of information, one is the video stream information of the two cameras provided by the UVC thread, and the other is the sound source angle information provided by the audio AI thread. After the image AI thread obtains the sound source angle information, it determines the sound source angle of the current speaker, determines which camera's video stream is obtained according to the field of view of the two cameras to analyze the lip movement, determines the recognition area corresponding to the lip movement, and Face information is recognized. The final feedback UVC thread switches the camera for display, and zooms in and crops to focus on the speaker.

The audio AI thread is used to analyze and process the audio stream data in the PCM format output by the array MIC given by the UAC, perform sound source localization, and send the output sound source angle information to the image AI thread for decision-making.

Among them, the data processing module further includes a policy management module, which is used to obtain the data processed by the data processing module, make scene decisions, and realize speaker tracking, speech subtitle display, and participant sign-in.

The terminal is used for displaying pictures, and the terminal may be a TV (television).

As shown in Figure 5, the specific process of a video stream display method is as follows:

210. The array microphone collects ambient sound in real time.

220. The audio AI thread determines and outputs the sound source angle information according to the collected ambient sound through the sound source localization algorithm.

Before collecting the ambient sound through the display microphone, the following steps may be included: the strategy management module controls the infrared thermal imager and the ultrasonic module to detect the body temperature of the participants. The ultrasonic module activates the distance detection function. When the distance of the target participants reaches the imaging requirements of the infrared thermal imager, the infrared thermal imager starts to detect the body temperature of the target participants. When the temperature exceeds the requirements, they are not allowed to participate in the conference.

The sound source angle refers to the angle between the sound source position and the array microphone. The midpoint of the line segment formed by the array microphone can be used as the vertex, and any one of the sound source position, the midpoint of the line segment formed by the array microphone, and the line segment formed by the array microphone can be used. The angle formed by the vertices is used as the sound source angle.

The array microphone collects the ambient sound in real time and sends it to the UAC thread. After being processed by the UAC thread, it is sent all the way to the terminal for playback, and all the way to the image AI thread for sound source localization.

231. When the audio AI thread does not output the sound source angle information, the image AI thread controls the terminal to display the image captured by the wide-angle camera.

When there is no sound source angle output, it will enter the listening mode. In this mode, the UVC thread will output the image presentation of the wide-angle camera by default. When the image AI thread analyzes one of the AI image streams of the two cameras, there is face recognition. The UVC thread will be instructed to switch to the image presentation of the corresponding camera, and he will simultaneously perform step 210 to collect ambient sounds in real time. If both AI image streams have face recognition, the image of the wide-angle camera will be output preferentially. If there is no face recognition in the two AI image streams, it will not focus, and the image of the wide-angle camera will be output preferentially.

232. When the audio AI thread outputs the sound source angle information, the image AI thread determines the sound source area according to the sound source angle information.

When there is a sound source angle output, the image AI thread will divide the sector by the range of ±15°～±30° of the sound source angle, and this sector is the sound source area.

240. The image AI thread determines the target video stream from the two video streams according to the sound source area.

The image AI thread determines which camera captures the image based on the sound source area. If the sound source area is completely covered by two cameras, the image AI thread prioritizes the image information of the telephoto camera. If the sound source area is within the wide-angle camera , the image AI thread processes the image information of the wide-angle camera.

250. The image AI thread identifies the target object according to the target video stream, and the target object is an object with lip movements.

The image AI thread performs lip motion analysis according to the camera used for identification determined in step 340 and according to the image information captured by the camera, so as to identify the target object.

261. When the target object is recognized, the image AI thread determines the area to be displayed according to the target object.

At the beginning, the area corresponding to ±15° of the sound source angle is used as the sector for recognizing faces. If no person with lip movements is recognized, the area corresponding to ±20° of the sound source angle is used as the sector for recognizing faces. In this way, the size of the sector is increased in increments of 5° each time until a person with lip movements is identified, and the sector at this time is used as the area to be displayed. If there are multiple speakers, the area to be displayed should cover all speakers.

Finally, the UVC thread controls the image information of the output camera, and crops the output image information, so that the user can see the final face focus effect.

262. When the target object is not recognized, the image AI thread determines the area to be displayed according to all objects in the target image information.

270. The image AI thread determines the video stream to be displayed according to the area to be displayed.

280. The image AI thread trims the display image of the video stream to be displayed according to the area to be displayed, and obtains the trimmed display image.

290. The terminal displays the cropped display screen.

When no person with lip movements is recognized, the area corresponding to ±15° of the sound source angle is used as the sector for recognizing faces. If a face is recognized, the area corresponding to ±20° of the sound source angle is used as the recognition area. For the sector of the face, the sector size is increased in increments of 5° each time until the face is recognized, and the sector at this time is used as the area to be displayed. If there are many people in the area to be displayed, the area to be displayed must cover everyone.

Finally, the UVC thread controls the image information of the output camera, and crops the output image information, so that the user can see the final face focus effect.

When no face is recognized, the listening mode is entered, and step 210 is performed to collect ambient sound in real time.

It can be seen from the above that the embodiment of the present application performs dual camera switching by acquiring the angle of the sound source, thereby realizing focusing on the speaker, so that the displayed image can be focused on the speaker, presenting a better conference image.

In order to better implement the above method, an embodiment of the present application further provides a video stream display apparatus, which may be integrated in an electronic device, and the electronic device may be a terminal, a server, or the like. The terminal may be a mobile phone, a tablet computer, an intelligent Bluetooth device, a notebook computer, a personal computer, etc.; the server may be a single server, or a server cluster composed of multiple servers.

For example, in this embodiment, the method of the embodiment of the present application will be described in detail by taking the example that the video stream display device is specifically integrated in the terminal.

For example, as shown in FIG. 6 , the video stream display apparatus may include an acquisition unit 310, a first determination unit 320, an identification unit 330, a second determination unit 340, and a display unit 350, as follows:

(1) Acquisition unit 310

It is used to obtain multiple video streams and sound source positions of the current scene, and each video stream corresponds to an image acquisition area.

In some embodiments, the method for acquiring the sound source position may include steps 6.1 to 6.2, as follows:

6.1. Collect the sound information of the current scene;

6.2. Process the collected sound information through the sound source localization algorithm to obtain the sound source position.

(2) The first determination unit 320

It is used to determine the target video stream from multiple video streams according to the position of the sound source and the image capture area.

In some embodiments, the first determining unit 320 may be specifically used in steps 7.1 to 7.4, as follows:

7.1. Determine the sound source area according to the sound source position;

7.2. For each video stream, determine the overlapping area of the sound source area and the image acquisition area;

7.3. Determine the overlapping area that meets the preset first area size as the target area;

7.4. Determine the video stream corresponding to the target area as the target video stream.

(3) Identification unit 330

It is used to identify the target object according to the target video stream, and the target object is an object with lip movements.

In some embodiments, the identification unit 330 may have steps 8.1 to 8.3, as follows:

8.1. Determine the recognition area according to the position of the sound source;

8.2. According to the recognition area, obtain the target image information from the target video stream, and the target image information is the image information corresponding to the recognition area;

8.3. Identify the target object according to the target image information.

In some embodiments, step 8.3 may include steps 8.3.1 to 8.3.2, as follows:

8.3.1. When an object with lip movements is identified from the target image information, the object with lip movements is used as the target object;

8.3.2. When the object with lip motion is not recognized from the target image information, expand the recognition area to the preset second area size to recognize the target object.

(4) The second determination unit 340

It is used to determine the video stream to be displayed from the multiple video streams according to the recognition result of the target object.

In some embodiments, the second determining unit 340 may be specifically used in steps 9.1 to 9.4, as follows:

9.1. When the target object is identified, determine the area to be displayed according to the target object;

9.2. When the target object is not recognized, determine the area to be displayed according to all objects in the target image information;

9.3. Obtain the image capture area corresponding to each video stream;

9.4. Determine the video stream to be displayed according to the area to be displayed and the image acquisition area.

(5) Display unit 350

It is used to display the picture corresponding to the video stream to be displayed.

In some embodiments, the display unit 350 may be specifically used in steps 10.1 to 10.3, as follows:

10.1. Obtain the display screen of the video stream to be displayed;

10.2. According to the area to be displayed, trim the display image of the video stream to be displayed, and obtain the trimmed display image;

10.3. Display the cropped display screen.

During specific implementation, the above units can be implemented as independent entities, or can be arbitrarily combined to be implemented as the same or several entities. The specific implementation of the above units can refer to the previous method embodiments, which will not be repeated here.

Therefore, the embodiment of the present application can determine the target video stream used to identify the speaker through the sound source position, which can improve the efficiency of identifying the speaker. On the person, present a better picture of the meeting.

Correspondingly, the embodiment of the present application also provides a computer device, the computer device may be a terminal or a server, and the terminal may be a smart phone, a tablet computer, a notebook computer, a touch screen, a game console, a personal computer, a personal digital assistant ( Personal Digital Assistant, PDA) and other terminal equipment.

As shown in FIG. 7 , which is a schematic structural diagram of a computer device provided by an embodiment of the application, the computer device 400 includes a processor 410 having one or more processing cores, and a memory having one or more computer-readable storage media 420 and a computer program stored on memory 420 and executable on the processor. The processor 410 is electrically connected to the memory 420 . Those skilled in the art can understand that the computer device structure shown in the figures does not constitute a limitation on the computer device, and may include more or less components than the one shown, or combine some components, or arrange different components.

The processor 410 is the control center of the computer device 400, and uses various interfaces and lines to connect various parts of the entire computer device 400, by running or loading software programs and/or modules stored in the memory 420, and calling the software programs and/or modules stored in the memory 420. to perform various functions of the computer device 400 and process data, so as to monitor the computer device 400 as a whole.

In the embodiment of the present application, the processor 410 in the computer device 400 loads the instructions corresponding to the processes of one or more application programs into the memory 420 according to the following steps, and the processor 410 executes the instructions stored in the memory. 420 applications to achieve various functions:

Obtain the multi-channel video streams and sound source positions of the current scene, and an image capture area corresponding to each video stream; determine the target video stream from the multi-channel video streams according to the sound source position and image capture area; identify the target video stream according to the target video stream The target object is an object with lip movements; according to the recognition result of the target object, the video stream to be displayed is determined from the multi-channel video streams; the picture corresponding to the video stream to be displayed is displayed.

For the specific implementation of the above operations, reference may be made to the foregoing embodiments, and details are not described herein again.

Optionally, as shown in FIG. 7 , the computer device 400 further includes: a touch display screen 430 , a radio frequency circuit 440 , an audio circuit 450 , an input unit 460 and a power supply 470 . The processor 410 is electrically connected to the touch display screen 430 , the radio frequency circuit 440 , the audio circuit 450 , the input unit 460 and the power supply 470 , respectively. Those skilled in the art can understand that the computer device structure shown in FIG. 7 does not constitute a limitation on the computer device, and may include more or less components than the one shown, or combine some components, or arrange different components.

The touch screen 430 can be used to display a graphical user interface and receive operation instructions generated by a user acting on the graphical user interface. The touch display 430 may include a display panel and a touch panel. Among them, the display panel can be used to display the information input by the user or the information provided to the user and various graphical user interfaces of the computer equipment, and these graphical user interfaces can be composed of graphics, text, icons, videos and any combination thereof. Optionally, the display panel may be configured in the form of a liquid crystal display (LCD, Liquid Crystal Display), an organic light-emitting diode (OLED, Organic Light-Emitting Diode), and the like. The touch panel can be used to collect the user's touch operations on or near it (such as the user's operations on or near the touch panel using a finger, stylus, etc., any suitable object or accessory), and generate corresponding operations instruction, and the operation instruction executes the corresponding program. Optionally, the touch panel may include two parts, a touch detection device and a touch controller. Among them, the touch detection device detects the user's touch orientation, detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into contact coordinates, and then sends it to the touch controller. To the processor 410, and can receive the command sent by the processor 410 and execute it. The touch panel may cover the display panel, and when the touch panel detects a touch operation on or near it, it is transmitted to the processor 410 to determine the type of the touch event, and then the processor 410 provides the display panel according to the type of the touch event. Corresponding visual output. In this embodiment of the present application, the touch panel and the display panel may be integrated into the touch display screen 430 to implement input and output functions. However, in some embodiments, the touch panel and the display panel can be used as two independent components to realize the input and output functions. That is, the touch display screen 430 can also be used as a part of the input unit 460 to realize the input function.

The radio frequency circuit 440 can be used for transmitting and receiving radio frequency signals, so as to establish wireless communication with network equipment or other computer equipment through wireless communication, and send and receive signals with the network equipment or other computer equipment.

The audio circuit 450 may be used to provide an audio interface between the user and the computer equipment through speakers, microphones. The audio circuit 450 can convert the received audio data into an electrical signal and transmit it to the speaker, and the speaker converts it into a sound signal for output; on the other hand, the microphone converts the collected sound signal into an electrical signal, which is converted after being received by the audio circuit 450. In the form of audio data, the audio data is output to the processor 410 for processing, and then sent to, for example, another computer device via the radio frequency circuit 440, or the audio data is output to the memory 420 for further processing. Audio circuitry 450 may also include an ear jack to provide for communication of peripheral headphones with the computer device.

The input unit 460 can be used to receive input numbers, character information or user characteristic information (such as fingerprint, iris, face information, etc.), and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control .

Power supply 470 is used to power various components of computer device 400 . Optionally, the power supply 470 may be logically connected to the processor 410 through a power management system, so that functions such as managing charging, discharging, and power consumption are implemented through the power management system. Power supply 470 may also include one or more DC or AC power sources, recharging systems, power failure detection circuits, power converters or inverters, power status indicators, and any other components.

Although not shown in FIG. 7 , the computer device 400 may further include a camera, a sensor, a Wi-Fi module, a Bluetooth module, and the like, which will not be repeated here.

In the above-mentioned embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference may be made to the relevant descriptions of other embodiments.

It can be seen from the above that the computer device provided in this embodiment can determine the target video stream used to identify the speaker through the position of the sound source, which can improve the efficiency of identifying the speaker. The picture focuses on the speaker, presenting a better picture of the meeting.

Those of ordinary skill in the art can understand that all or part of the steps in the various methods of the above-mentioned embodiments can be completed by instructions, or by instructions that control relevant hardware, and the instructions can be stored in a computer-readable storage medium, and loaded and executed by the processor.

To this end, the embodiments of the present application provide a computer-readable storage medium, in which a plurality of computer programs are stored, and the computer programs can be loaded by a processor to execute any of the video stream display methods provided by the embodiments of the present application. steps in . For example, the computer program may perform the following steps:

Obtain the multi-channel video streams and sound source positions of the current scene, and an image capture area corresponding to each video stream; determine the target video stream from the multi-channel video streams according to the sound source position and image capture area; identify the target video stream according to the target video stream The target object is an object with lip movements; according to the recognition result of the target object, the video stream to be displayed is determined from the multi-channel video streams; the picture corresponding to the video stream to be displayed is displayed.

For the specific implementation of the above operations, reference may be made to the foregoing embodiments, and details are not described herein again.

Wherein, the storage medium may include: a read only memory (ROM, Read Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and the like.

Since the computer program stored in the storage medium can execute the steps in any video stream display method provided by the embodiments of the present application, it is possible to realize the steps of any video stream display method provided by the embodiments of the present application. For the beneficial effects that can be achieved, see the foregoing embodiments for details, which will not be repeated here.

A video stream display method, device, storage medium and computer equipment provided by the embodiments of the present application have been described in detail above. The principles and implementations of the present application are described with specific examples in this article. It is only used to help understand the method of the present application and its core idea; at the same time, for those skilled in the art, according to the idea of the present application, there will be changes in the specific implementation and application scope. The contents of the description should not be construed as limiting the application.

Claims (10)

1. A method for displaying a video stream, comprising:

acquiring a plurality of paths of video streams and a sound source position of a current scene, wherein each video stream corresponds to an image acquisition area;

determining a target video stream from the multiple paths of video streams according to the sound source position and the image acquisition area;

identifying a target object according to the target video stream, wherein the target object is an object with lip action;

determining a video stream to be displayed from the multi-path video stream according to the identification result of the target object;

and displaying the picture corresponding to the video stream to be displayed.

2. The method for displaying a video stream according to claim 1, wherein said determining a target video stream from the plurality of video streams based on the sound source position and the image capturing area comprises:

determining a sound source area according to the sound source position;

for each of the video streams, determining an overlapping area of the sound source area and the image acquisition area;

determining the overlapping area meeting the preset first area size as a target area;

and determining the video stream corresponding to the target area as a target video stream.

3. The method for displaying a video stream according to claim 1, wherein said identifying a target object from said target video stream comprises:

determining an identification area according to the sound source position;

acquiring target image information from the target video stream according to the identification area, wherein the target image information is image information corresponding to the identification area;

and identifying the target object according to the target image information.

4. The method for displaying a video stream according to claim 3, wherein said identifying a target object based on said target image information comprises:

when the object with the lip action is identified from the target image information, taking the object with the lip action as a target object;

and when the object with the lip action is not identified from the target image information, expanding the identification area to a preset second area size so as to identify the target object.

5. The method for displaying a video stream according to claim 1, wherein the determining a video stream to be displayed from the plurality of video streams based on the recognition result of the target object comprises:

when the target object is identified, determining a region to be displayed according to the target object;

when the target object is not identified, determining a region to be displayed according to all objects in the target image information;

acquiring an image acquisition area corresponding to each video stream;

and determining the video stream to be displayed according to the area to be displayed and the image acquisition area.

6. The method for displaying a video stream according to claim 5, wherein the displaying a picture corresponding to the video stream to be displayed comprises:

acquiring a display picture of the video stream to be displayed;

cutting the display picture of the video stream to be displayed according to the area to be displayed to obtain the cut display picture;

and displaying the cut display picture.

7. The video stream display method according to claim 1, wherein the method of acquiring the sound source position includes:

collecting sound information of a current scene;

and processing the collected sound information through a sound source positioning algorithm to obtain the sound source position.

8. A video stream display apparatus, comprising:

the system comprises an acquisition unit, a processing unit and a processing unit, wherein the acquisition unit is used for acquiring a plurality of paths of video streams and a sound source position of a current scene, and each video stream corresponds to an image acquisition area;

a first determining unit, configured to determine a target video stream from the multiple video streams according to the sound source position and the image acquisition area;

the identification unit is used for identifying a target object according to the target video stream, wherein the target object is an object with lip action;

the second determining unit is used for determining the video stream to be displayed from the multi-path video stream according to the identification result of the target object;

and the display unit is used for displaying the picture corresponding to the video stream to be displayed.

9. A computer device comprising a processor and a memory, the memory storing a plurality of instructions; the processor loads instructions from the memory to perform the steps of the video stream display method according to any one of claims 1 to 7.

10. A computer readable storage medium storing instructions adapted to be loaded by a processor to perform the steps of the method of displaying a video stream according to any one of claims 1 to 7.

Images