CN110659623B

CN110659623B - Panoramic picture display method and device based on framing processing and storage medium

Info

Publication number: CN110659623B
Application number: CN201910925947.6A
Authority: CN
Inventors: 熊垚森; 邓辉中; 杨海军
Original assignee: Kandao Technology Co Ltd
Current assignee: Kandao Technology Co Ltd
Priority date: 2019-09-27
Filing date: 2019-09-27
Publication date: 2022-04-08
Anticipated expiration: 2039-09-27
Also published as: CN110659623A; CN114821700A

Abstract

The invention provides a panoramic picture display method based on framing processing, which is used for displaying a panoramic picture, wherein the panoramic picture display method comprises the following steps: if the current picture frame is the detection picture frame, acquiring a user face image to be displayed according to the panoramic picture information and the panoramic sound information of the current picture frame; carrying out display operation on the face image of the user to be displayed; if the current picture frame is the tracking picture frame, acquiring a reference facial image feature point of a user facial image displayed by the previous picture frame; acquiring a corresponding user face image in the tracking picture frame based on the reference face image feature point; and updating the displayed user face image by using the user face image in the current picture frame. The invention carries out user face recognition on the detected picture frame and carries out face image tracking on the tracked picture frame, thereby effectively ensuring the face picture display effect in each frame of picture, avoiding the occurrence of frame skipping and further improving the picture display efficiency.

Description

Panoramic picture display method and device based on framing processing and storage medium

Technical Field

The present invention relates to the field of image processing technologies, and in particular, to a method and an apparatus for displaying a panoramic image based on framing processing, and a corresponding storage medium.

Background

With the development of society, the interaction between people is more and more compact, but the production places of different parts of the same product are more and more dispersed, so that company product leaders often need to contact with employees of different companies in various places for a certain production plan, or to carry out a teleconference and the like.

In the existing teleconference system, two parties in conversation can only see the user who is shooting by the camera, and if the position of the user who talks changes, the position of the user can be refreshed through an image recognition technology, so that tracking shooting of the talking user is realized.

However, the existing image recognition technology has a long time for recognizing the face of the user, and cannot perform effective face recognition on each frame of picture, for example, under the condition that the user moves fast, a frame skipping phenomenon of a user face tracking picture is easily caused, that is, the face picture of the user suddenly jumps from one position to another position, thereby affecting the picture display efficiency of the video conference system.

Therefore, it is necessary to provide a method and an apparatus for displaying a panoramic image based on framing processing to solve the problems of the prior art.

Disclosure of Invention

The embodiment of the invention provides a panoramic picture display method and a panoramic picture display device based on framing processing, which can smoothly display a moving face picture and has higher picture display efficiency, so as to solve the technical problem that the existing panoramic picture display method and device are easy to generate frame skipping phenomenon and further influence the picture display efficiency.

The embodiment of the invention provides a panoramic picture display method based on framing processing, which is used for displaying a panoramic picture, wherein the panoramic picture comprises detection picture frames and corresponding tracking picture frames, and one detection picture frame corresponds to at least one tracking picture frame; and the tracking picture frame is arranged behind the corresponding detection picture frame; the panoramic picture display method comprises the following steps:

if the current picture frame is the detection picture frame, acquiring a user face image to be displayed according to the panoramic picture information and the panoramic sound information of the current picture frame; displaying the user face image to be displayed;

if the current picture frame is the tracking picture frame, acquiring a reference facial image feature point of a user facial image displayed by the previous picture frame; acquiring a corresponding user face image in the tracking picture frame based on the reference face image feature point; and updating the displayed user face image by using the user face image in the current picture frame.

The embodiment of the invention also provides a panoramic picture display device based on framing processing, which is used for displaying the panoramic picture, wherein the panoramic picture comprises detection picture frames and corresponding tracking picture frames, and one detection picture frame corresponds to at least one tracking picture frame; and the tracking picture frame is arranged behind the corresponding detection picture frame; the panorama picture display apparatus includes:

the detection picture frame display module is used for acquiring a user face image to be displayed according to the panoramic picture information and the panoramic sound information of the current picture frame if the current picture frame is the detection picture frame; displaying the user face image to be displayed; and

the tracking picture frame display module is used for acquiring a reference facial image feature point of a user facial image displayed by a previous picture frame if the current picture frame is the tracking picture frame; acquiring a corresponding user face image in the tracking picture frame based on the reference face image feature point; and updating the displayed user face image by using the user face image in the current picture frame.

The present invention also provides a computer-readable storage medium having stored therein processor-executable instructions, the instructions being loaded by one or more processors to perform any of the above-described panoramic image display methods.

Compared with the panoramic picture display method and device in the prior art, the panoramic picture display method and device based on framing processing of the invention can perform user face identification on the detected picture frame and perform face image tracking on the tracked picture frame, thereby effectively ensuring the face picture display effect in each frame of picture, avoiding the occurrence of frame skipping and further improving the picture display efficiency; the technical problem that the frame skipping phenomenon easily occurs in the conventional panoramic picture display method and device, and the picture display efficiency is further influenced is effectively solved.

Drawings

FIG. 1 is a flowchart illustrating a method for displaying a panoramic image based on framing according to a first embodiment of the present invention;

FIG. 2 is a flowchart illustrating a panoramic image displaying method based on framing according to a second embodiment of the present invention;

FIG. 3 is a flowchart illustrating a step S208 of a panoramic image displaying method based on framing processing according to a second embodiment of the present invention;

FIG. 4 is a schematic structural diagram of a panoramic image displaying apparatus according to a first embodiment of the present invention;

FIG. 5 is a schematic structural diagram of a frame display module of a panoramic image display apparatus according to a first embodiment of the present invention;

FIG. 6 is a schematic structural diagram of a tracking frame display module of the first embodiment of the panorama display apparatus according to the present invention;

FIG. 7 is a schematic structural diagram of a panoramic image displaying apparatus according to a second embodiment of the present invention;

FIG. 8 is a schematic structural diagram of a tracking frame display module of a second embodiment of the panorama display apparatus according to the present invention;

fig. 9 is a schematic structural diagram of a first reference human body image feature point obtaining unit of a tracking picture frame displaying module according to a second embodiment of the panoramic picture displaying apparatus of the present invention;

FIG. 10 is a flowchart of an embodiment of a panoramic image displaying method and a panoramic image displaying apparatus according to the present invention;

FIGS. 11a to 11e are schematic views illustrating a panoramic image displaying method and a panoramic image displaying apparatus according to embodiments of the present invention;

fig. 12 is a schematic view of a working environment structure of an electronic device in which the panoramic image display apparatus of the present invention is located.

Detailed Description

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

The panoramic picture display method and the panoramic picture display device are used for the electronic equipment for effectively displaying the panoramic picture. The electronic devices include, but are not limited to, wearable devices, head-worn devices, medical health platforms, personal computers, server computers, hand-held or laptop devices, mobile devices (such as mobile phones, Personal Digital Assistants (PDAs), media players, and the like), multiprocessor systems, consumer electronics, minicomputers, mainframe computers, distributed computing environments that include any of the above systems or devices, and the like. The electronic device is preferably an electronic terminal that receives the panoramic picture and displays the panoramic picture through a display screen, that is, a user can view panoramic picture information shot by the panoramic camera in real time through a fixed terminal or a mobile terminal, such as a user picture of a conference scene.

Referring to fig. 1, fig. 1 is a flowchart illustrating a panoramic image displaying method based on framing processing according to a first embodiment of the present invention. The panoramic picture displaying method of the embodiment can be implemented by using the electronic device, the panoramic picture displayed by the electronic device in the embodiment comprises a detection picture frame and a tracking picture frame, wherein the detection picture frame is a picture frame for realizing the display of the user face image in the panoramic picture by detecting picture pixels of a current picture frame, and the consumption of system resources is higher due to the fact that the comprehensive pixel detection is required to be carried out on the current picture frame, but the accuracy of the display of the user face image is higher; the tracking picture frame is a picture frame for realizing the display of the user face image in the panoramic picture based on the feature point detection of the user face image of the previous picture, and only specific feature points need to be detected, so that the consumption of system resources is low, but the accuracy of the display of the user face image is low due to the continuous loss of the feature points.

The panoramic picture display method of the embodiment ensures the accuracy of the user face image display of the panoramic picture through the detection picture frame of the panoramic picture, and ensures the timeliness of the user face image display of the panoramic picture through the tracking picture frame of the panoramic picture. Therefore, the detection picture frame of the panorama picture of the present embodiment corresponds to at least one tracking picture frame, and the tracking picture frame is disposed behind the corresponding detection picture frame. For example, the first frame of the panorama picture is a detection picture frame, the second to fourth frames may be corresponding tracking picture frames, the fifth frame is a detection picture frame, the sixth to eighth frames may be corresponding tracking picture frames, that is, the 4n +1 th frame is a detection picture frame, the 4n +2 th, 4n +3 th and 4n +4 th frames are tracking picture frames, and n is an integer greater than or equal to 0. Of course, different detection frames may correspond to different numbers of tracking frames, such as the first frame corresponding to the second to fourth frames, the fifth frame corresponding to the sixth and seventh frames, etc.

Thus, the feature points of the user face image of the second frame picture can be acquired based on the feature points of the user face image of the first frame picture, and the feature points of the user face image of the third frame picture can be acquired based on the feature points of the user face image of the second frame picture.

Based on the picture frame distribution principle of the panoramic picture, the panoramic picture display method of the embodiment comprises the following steps:

step S101, detecting the type of a current picture frame;

step S102, if the current picture frame is a detection picture frame, obtaining a user face image to be displayed according to the panoramic picture information and the panoramic sound information of the current picture frame; carrying out display operation on the face image of the user to be displayed;

step S103, if the current picture frame is a tracking picture frame, acquiring a reference image feature point of a user face image displayed by the previous picture frame; acquiring a corresponding user face image in the tracking picture frame based on the reference face image feature point; and updating the displayed user face image by using the user face image in the current picture frame.

The following describes in detail a specific flow of each step of the panoramic image display method according to the present embodiment.

In step S101, an electronic device (e.g., a user fixed terminal) acquires a panoramic picture to be played, and then detects a picture frame type of a current picture frame to be displayed, where the picture frame type may be a detected picture frame or a tracking picture frame. If the current frame is the detection frame, turning to step S102; if the current picture frame is the tracking picture frame, it goes to step S103.

In step S102, if the current picture frame is the detection picture frame, the electronic device acquires the panoramic picture information and the panoramic sound information of the current picture frame. The panoramic picture information and the panoramic sound information are collected by the panoramic camera, the panoramic picture information comprises picture information which takes the panoramic camera as the center and is within 360 degrees, the panoramic sound information comprises sound information which takes the panoramic camera as the center and is within 360 degrees, the panoramic picture information comprises picture position information used for representing position information corresponding to the picture information, and the panoramic sound information comprises sound position information used for representing the position information corresponding to the sound information.

And then the electronic equipment performs face recognition operation on the acquired panoramic picture information, so that user face images of all users in the current scene and corresponding image position information of the user face images in the panoramic picture can be acquired.

The face recognition can be performed through facial feature points (such as five officers) of the user, or can be performed through facial contours of the user, and the face recognition is performed through the relative position relationship of the face on the human body.

Then, the electronic device compares the acquired image position information with the acquired sound position information (it is assumed that only one user is speaking currently, that is, only one sound position information is present in the panoramic sound information), that is, the image position information of the user face image is compared with the sound position information of the panoramic sound information in position, and the user face image corresponding to the matched image position information is used as the user face image to be displayed.

And finally, the electronic equipment performs display operation on the acquired facial image of the user to be displayed. Specifically, the electronic device can adjust the user face image to be displayed according to the size of the set image, so that the adjusted user face image can be displayed better through the user display position with the fixed shape.

In step S103, if the current frame is the tracking frame, the electronic device acquires the reference facial image feature points of the user facial image shown in the previous frame. The reference facial image feature points here are feature points of the previous picture frame for characterizing the user's facial image.

In the step, the electronic device searches the corresponding reference facial image feature points in the current picture frame to obtain the current facial image feature points in the current picture frame, and due to the mismatching of partial features, the feature points which can be searched in the current picture frame are less than or equal to the feature points in the previous picture frame, so that the number of the current facial image feature points is less than or equal to the number of the reference facial image feature points.

And then the electronic equipment restores the user face image in the current picture frame based on the acquired feature points of the current face image, and performs display operation on the user face image.

And finally, the electronic equipment sets the current facial image feature point in the current picture frame as the reference facial image feature point of the current picture frame so as to be convenient for calling the next picture frame.

This completes the presentation process of the user's face image in the panoramic picture by the panoramic picture presentation method of the present embodiment.

The panoramic picture display method based on frame processing of the embodiment carries out user face recognition on the detected picture frames and carries out facial image tracking on the tracked picture frames, thereby effectively ensuring the face picture display effect in each frame of picture, avoiding the occurrence of frame skipping and further improving the picture display efficiency.

Referring to fig. 2, fig. 2 is a flowchart illustrating a panorama image displaying method based on framing according to a second embodiment of the present invention. The panoramic picture display method of the present embodiment may be implemented by using the electronic device, and the panoramic picture display method of the present embodiment includes:

step S201, detecting the type of the current picture frame; if the current frame is the detection frame, go to step S202; if the current frame is the tracking frame, go to step S206;

step S202, acquiring panoramic picture information and panoramic sound information of a current picture frame; wherein the panoramic sound information includes sound location information;

step S203, carrying out face recognition operation on the panoramic picture information to acquire a user face image of the current scene and corresponding image position information;

step S204, comparing the image position information with the sound position information to obtain a user face image to be displayed;

step S205, displaying at least part of user face images of users in the current picture frame according to the user face images to be displayed and the displayed user face images;

step S206, determining the type of the previous picture frame, and if the previous picture frame is a detection picture frame, turning to step S207; if the previous frame is the tracking frame, go to step S211;

step S207, collecting reference face image characteristic points of a user face image displayed in a previous picture frame;

step S208, acquiring corresponding reference human body image feature points of the user human body image according to the reference facial image feature points of the user facial image displayed in the previous picture frame;

step S209, based on the reference human body image characteristic points, searching corresponding current human body image characteristic points in the current picture frame; the number of the current human body image characteristic points is less than or equal to the number of the reference human body image characteristic points;

step S210, acquiring a user face image in the current picture frame based on the current human body image feature point in the current picture frame, and setting the current human body image feature point in the current picture frame as a reference human body image feature point of the current picture frame; go to step S214;

step S211, obtaining the reference human body image characteristic points of the user human body image displayed by the previous picture frame;

step S212, searching a corresponding current human body image characteristic point in a current picture frame based on a reference human body image characteristic point of a user human body image displayed in a previous picture frame; the number of the current human body image characteristic points is less than or equal to the number of the reference human body image characteristic points;

step S213, acquiring a user face image in the current picture frame based on the current human body image feature point in the current picture frame, and setting the current human body image feature point in the current picture frame as a reference human body image feature point of the current picture frame; go to step S214;

step S214, matching the corresponding user face image in the current picture frame with the displayed user face image;

step S215, using the corresponding user face image in the current frame, performs an update operation on the matched displayed user face image.

In step S201, an electronic device (e.g., a user fixed terminal) acquires a panoramic picture to be played, and then detects a picture frame type of a current picture frame to be displayed, where the picture frame type may be a detected picture frame or a tracking picture frame. If the current frame is the detection frame, go to step S202; if the current picture frame is the tracking picture frame, it goes to step S206.

In step S202, if the current picture frame is the detection picture frame, the electronic device acquires the panoramic picture information and the panoramic sound information of the current picture frame. The panoramic picture information and the panoramic sound information are collected by the panoramic camera, the panoramic picture information comprises picture information which takes the panoramic camera as the center and is within 360 degrees, and the panoramic sound information comprises sound information which takes the panoramic camera as the center and is within 360 degrees.

In step S203, the electronic device performs a face recognition operation on the acquired panoramic picture information, so that user face images of all users in the current scene can be acquired. The picture position information of the user face image in the panoramic picture is then taken as the image position information of the user face image.

In step S204, the electronic device acquires all panoramic sound information; and then confirming the image position information matched with the sound position information of the panoramic sound information, namely when the position difference value between the sound position information and the image position information is smaller than a set value, the electronic equipment determines that the sound position information is matched with the image position information. And then the electronic equipment takes the user face image corresponding to the matched image position information as the user face image to be displayed.

Because the sound position information acquired by the panoramic camera has certain precision, if the distance difference between the image position information of a plurality of users is small, the corresponding image position information can be distinguished only through the sound position information or can not be distinguished, the electronic equipment can determine the image position information corresponding to the sound position information by identifying the user action information in the panoramic picture, and further determine the face image of the user to be displayed. The user action information may be a hand-lifting action or a mouth action of the user, or the like. If the electronic device recognizes that certain sound position information matches two user face images, the user face image corresponding to the user who is doing a hand-lifting action or the user with a mouth action can be set as the user face image to be displayed.

If a plurality of users speak at the same time, the electronic device may acquire a plurality of sound position information at the same time, and further acquire a plurality of image position information and a user face image. In order to avoid the interference of the conversation noise of other users to the display of the face image of the user, the corresponding sound position information is collected only when the sound volume information corresponding to the user is larger than a set value, so that the corresponding image position information and the face image of the user can be searched.

In step S205, the electronic device displays the user face image of at least part of the users in the current frame according to the acquired user face image to be displayed and the displayed user face image.

Specifically, the electronic device may match the user face image to be presented with the presented user face image, so that the presented user face image determined to need updating may be updated.

And then the electronic equipment uses the user face image to be displayed to perform an updating operation on the matched displayed user face image, if the user face image to be displayed does not match any displayed user face image, the user face image to be displayed is used for replacing one displayed user face image to be displayed.

This completes the presentation process of the user face image of the detection screen frame of the present embodiment.

In step S206, if the current frame is the tracking frame, the electronic device determines the type of the previous frame, and if the previous frame is the detection frame, the electronic device goes to step S207; if the previous picture frame is the tracking picture frame, the process goes to step S211.

In step S207, if the previous frame is a detection frame, since the previous frame acquires the user face image through picture pixel detection, the accuracy of the acquired user face image is high, and the electronic device can directly acquire the reference face image feature points of the user face image shown in the previous frame.

In step S208, since the person in the panoramic image may turn around or be occluded, the tracking of the user face image is performed based on only the reference face image feature points of the user face image, and in some cases, the reference face image feature points are all lost.

Therefore, in this step, the electronic device may obtain the reference human body image feature points of the corresponding user human body image according to the reference facial image feature points of the user facial image shown in the previous frame, so as to increase the number of image feature points corresponding to the user facial image.

Referring to fig. 3, fig. 3 is a flowchart of step S208 of the panorama picture displaying method based on framing according to the second embodiment of the present invention. The step S208 includes:

in step S301, the electronic device constructs a user facial feature frame based on the reference image feature points of the user facial image.

In step S302, the electronic device translates the user facial feature frame to below the frame by at least one frame position (i.e., the position of the neck of the user to the body), and sets both the translated region and the region of the user facial feature frame as the user human body image feature acquisition region.

Step S303, in the user human body image feature acquisition region acquired in step S302, the electronic device acquires the reference human body image feature points of the corresponding user human body image.

Therefore, the number of the reference human body image characteristic points is far greater than that of the reference facial image characteristic points, and partial characteristic points of the reference human body image characteristic points are less susceptible to human actions and obstructions.

In step S209, the electronic device searches for a corresponding current human body image feature point in the current frame based on the reference human body image feature point acquired in step S208. Namely, the image feature point of the previous picture frame is used for searching the corresponding image feature point in the current picture frame. Because part of the feature points are lost along with the movement of the object in the panoramic picture, the number of the feature points of the current human body image is less than or equal to that of the feature points of the reference human body image.

In step S210, the electronic device obtains the user face image in the current picture frame based on the current human body image feature point in the current picture frame, specifically, the position of the user face feature frame can be found based on the position of the current human body image feature point, and then the image in the user face feature frame is taken as the user face image.

And then the electronic equipment sets the current human body image characteristic point in the current picture frame as the reference human body image characteristic point of the current picture frame so as to be convenient for calling the next picture frame.

In step S211, if the previous frame is the tracking frame, the electronic device can directly capture the reference human body image feature points of the user human body image displayed in the previous frame.

In step S212, the electronic device searches for a corresponding current human body image feature point in the current frame based on the reference human body image feature point of the user human body image shown in the previous frame. Namely, the image feature point of the previous picture frame is used for searching the corresponding image feature point in the current picture frame. Because part of the feature points are lost along with the movement of the object in the panoramic picture, the number of the feature points of the current human body image is less than or equal to that of the feature points of the reference human body image.

In step S213, the electronic device obtains the user face image in the current picture frame based on the current human body image feature point in the current picture frame, and may specifically find the position of the user face feature frame based on the position of the current human body image feature point, and then take the image in the user face feature frame as the user face image.

In step S214, the electronic device matches the corresponding user face image in the current screen frame with the presented user face image to determine the presented user face image that needs to be updated.

In step S215, the electronic device performs an update operation on the matched presented user face image using the user face image to be presented.

This completes the presentation process of the user face image of the tracking picture frame of the present embodiment.

The user face images of all the detection picture frames and the tracking picture frames in the panoramic picture are displayed, and then the displaying process of the user face images in the panoramic picture of the panoramic picture displaying method of the embodiment is completed.

On the basis of the first embodiment, the panoramic picture display method based on framing processing of the embodiment improves the recognition accuracy of the user facial image of the detected picture frame through the user action information and the sound volume information; the image characteristic points are expanded through the human body picture of the user, and the identification accuracy of the facial image of the user tracking the picture frame is improved.

In this embodiment, the type of each frame of the panoramic picture is manually set, for example, the 4n +1 th frame is a detection frame, and the 4n +2, 4n +3, and 4n +4 th frames are tracking frames, and each tracking frame may lose part of image feature points, so that the tracking frames corresponding to the same detection frame may be limited by a set number, for example, 1 detection frame corresponds to 3 tracking frames. Therefore, the overall display quality of the user face image display in the panoramic picture can be effectively improved, and the situation that the corresponding user face image area cannot be found due to the fact that the image feature points corresponding to the tracking picture certificate are few is avoided. Meanwhile, the display operation of the user face image of the tracking picture frame consumes less system resources, so that the overall panoramic picture display method consumes less system resources, and the picture display efficiency is improved.

The type of the current picture frame can be determined based on the number of matching feature points between the current human body image feature point of the current picture frame and the reference human body image feature point of the previous picture frame.

If the number of the matched feature points is larger than the set proportion (such as 90% or 95%) of the number of the reference human body image feature points of the previous picture frame, that is, the difference between the current human body image feature points of the current picture frame and the reference human body image feature points of the previous picture frame is smaller, the reference human body image feature points of the previous picture frame can be continuously used for assisting in displaying the user face image of the current picture frame, that is, the current picture frame can be continuously set as a tracking picture frame for displaying the user face image.

If the number of the matched characteristic points is less than or equal to the set proportion of the number of the reference human body image characteristic points of the previous picture frame, namely the difference between the current human body image characteristic points of the current picture frame and the reference human body image characteristic points of the previous picture frame is larger, the current picture frame is set as a detection picture frame, and the display of the facial image of the user is realized by carrying out image identification on the current picture frame of the panoramic picture.

Referring to fig. 4, fig. 4 is a schematic structural diagram of a panoramic image display apparatus according to a first embodiment of the present invention, the panoramic image display apparatus according to the present embodiment can be implemented by using the first embodiment of the panoramic image display method according to the framing processing, and the panoramic image display apparatus 40 according to the present embodiment includes a detection frame display module 41 and a tracking frame display module 42.

The detection picture frame display module 41 is configured to, if the current picture frame is the detection picture frame, obtain a user face image to be displayed according to the panoramic picture information and the panoramic sound information of the current picture frame; carrying out display operation on the face image of the user to be displayed; the tracking picture frame displaying module 42 is configured to, if the current picture frame is the tracking picture frame, obtain a reference facial image feature point of a user facial image displayed in a previous picture frame; acquiring a corresponding user face image in the tracking picture frame based on the reference face image feature point; and updating the displayed user face image by using the user face image in the current picture frame.

Referring to fig. 5, fig. 5 is a schematic structural diagram of a frame display module of a detection frame of a panoramic image display apparatus according to a first embodiment of the present invention. The detected frame display module 41 includes a frame and sound information obtaining unit 51, a face recognition unit 52, a comparison unit 53 and a first display unit 54.

The picture and sound information acquiring unit 51 is configured to acquire panoramic picture information and panoramic sound information of the current picture frame; the face recognition unit 52 is configured to perform face recognition operation on the panoramic picture information to obtain a user face image of the current scene and corresponding image position information; the comparison unit 53 is configured to perform comparison operation on the image position information and the sound position information to obtain a user face image to be displayed; the first presentation unit 54 is configured to present the user face images of at least some users in the current picture frame according to the user face image to be presented and the presented user face image.

Referring to fig. 6, fig. 6 is a schematic structural diagram of a tracking frame display module of a first embodiment of a panorama display apparatus according to the present invention. The tracking-picture frame presentation module 42 includes a reference face image feature-point acquisition unit 61, a current face image feature-point search unit 62, and a user face image acquisition unit 63.

The reference facial image feature point acquisition unit 61 is configured to acquire reference facial image feature points of a user facial image shown in a previous frame; the current facial image feature point searching unit 62 is configured to search a corresponding current facial image feature point in the current picture frame based on a reference facial image feature point of the user facial image shown in the previous picture frame; the user face image acquisition unit 63 is configured to acquire a user face image in the current picture frame based on the current face image feature points, and set the current face image feature points in the current picture frame as reference face image feature points of the current picture frame.

When the panorama picture displaying apparatus 40 of the present embodiment is used, the panorama picture displaying apparatus 40 first obtains a panorama picture to be played, and then detects a picture frame type of a current picture frame to be displayed, where the picture frame type may be a detected picture frame or a tracking picture frame.

If the current frame is the detection frame, the picture and sound information acquisition unit 51 of the detection frame presentation module 41 acquires the panoramic picture information and the panoramic sound information of the current frame. The panoramic picture information and the panoramic sound information are collected by the panoramic camera, the panoramic picture information comprises picture information which takes the panoramic camera as the center and is within 360 degrees, the panoramic sound information comprises sound information which takes the panoramic camera as the center and is within 360 degrees, the panoramic picture information comprises picture position information used for representing position information corresponding to the picture information, and the panoramic sound information comprises sound position information used for representing the position information corresponding to the sound information.

Subsequently, the face recognition unit 52 of the detection frame presentation module 41 performs a face recognition operation on the acquired panoramic picture information, so that the user face images of all users in the current scene and the corresponding image position information of the user face images in the panoramic picture can be acquired.

Then, the comparing unit 53 of the detection frame displaying module 41 performs a comparison operation between the acquired image position information and the acquired sound position information (it is assumed that only one user is currently speaking, that is, only one sound position information is currently provided in the panoramic sound information), that is, performs a position comparison between the image position information of the user face image and the sound position information of the panoramic sound information, and takes the user face image corresponding to the matched image position information as the user face image to be displayed.

Finally, the first displaying unit 54 of the detection frame displaying module 41 performs a displaying operation on the acquired user face image to be displayed, and may display the user face images of at least some users in the current frame according to the user face image to be displayed and the displayed user face image.

Specifically, the first display unit 54 of the detection frame display module 41 may adjust the facial image of the user to be displayed according to the size of the set image, so that the adjusted facial image of the user can be displayed better through the user display position with a fixed shape.

If the current picture frame is the tracking picture frame, the reference facial image feature point acquisition unit 61 of the tracking picture frame presentation module 42 acquires the reference facial image feature points of the user facial image presented in the previous picture frame. The reference facial image feature points here are feature points of the previous picture frame for characterizing the user's facial image.

The current facial image feature point searching unit 62 of the tracking picture frame displaying module 42 searches the corresponding reference facial image feature point in the current picture frame to obtain the current facial image feature point in the current picture frame, and because of the mismatch of partial features, the feature points that can be searched in the current picture frame are less than or equal to the feature points in the previous picture frame, so the number of the current facial image feature points is less than or equal to the number of the reference facial image feature points.

The user face image acquisition unit 63 of the follow-up picture frame presentation module 42 then restores the user face image in the current picture frame based on the acquired current face image feature points, and performs a presentation operation on the user face image.

Finally, the user face image acquisition unit 63 of the tracking picture frame presentation module 42 sets the current face image feature point in the current picture frame as the reference face image feature point of the current picture frame, so that the next picture frame is called.

This completes the presentation process of the user's face image in the panoramic picture of the panoramic picture presentation apparatus 40 of the present embodiment.

The panorama picture display device based on framing processing of this embodiment carries out user face identification to detecting the picture frame, carries out facial image tracking to the picture frame of tracking, can effectually guarantee the face picture display effect in every frame picture, avoids the emergence of frame skipping phenomenon, and then has improved picture display efficiency.

Referring to fig. 7, fig. 7 is a schematic structural diagram of a panoramic picture display apparatus according to a second embodiment of the present invention. The panorama picture displaying apparatus of the present embodiment can be implemented using the second embodiment of the framing processing-based panorama picture displaying method described above, and the panorama picture displaying apparatus 70 of the present embodiment includes a detection picture frame displaying module 71 and a tracking picture frame displaying module 72.

On the basis of the first embodiment, the tracking picture frame displaying module 72 of this embodiment includes a reference facial image feature point obtaining unit 81, a first reference human body image feature point obtaining unit 82, a first current human body image feature point searching unit 83, a first user facial image obtaining unit 84, a second reference human body image feature point obtaining unit 85, a second current human body image feature point searching unit 86, a second user facial image obtaining unit 87, and a second displaying unit 88. Referring to fig. 8, fig. 8 is a schematic structural diagram of a tracking frame display module of a second embodiment of a panorama display apparatus according to the present invention.

The reference facial image feature point acquiring unit 81 is configured to acquire a reference facial image feature point of a user facial image displayed in a previous picture frame if the previous picture frame is a detection picture frame; the first reference human body image feature point acquiring unit 82 is configured to acquire a reference human body image feature point of a corresponding user human body image according to a reference facial image feature point of a user facial image displayed in a previous picture frame; the first current human body image feature point searching unit 83 is configured to search a corresponding current human body image feature point in a current picture frame based on the reference human body image feature point; the first user face image obtaining unit 84 is configured to obtain a user face image in a current picture frame based on a current human body image feature point in the current picture frame, and set the current human body image feature point in the current picture frame as a reference human body image feature point of the current picture frame; the second reference human body image feature point acquiring unit 85 is configured to acquire a reference human body image feature point of the user human body image displayed in the previous picture frame, if the previous picture frame is the tracking picture frame; the second current human body image feature point searching unit 86 is configured to search a corresponding current human body image feature point in the current picture frame based on a reference human body image feature point of the user human body image displayed in the previous picture frame; the second user face image obtaining unit 87 is configured to obtain a user face image in the current picture frame based on the current human body image feature point in the current picture frame, and set the current human body image feature point in the current picture frame as a reference human body image feature point of the current picture frame; the second display unit 88 is used for matching the corresponding user face image in the current picture frame with the displayed user face image; and updating the matched displayed user face image by using the corresponding user face image in the current picture frame.

Referring to fig. 9, fig. 9 is a schematic structural diagram of a first reference human body image feature point obtaining unit of a tracking picture frame display module according to a second embodiment of the panorama picture display apparatus of the present invention. The first reference human body image feature point obtaining unit 82 includes a user face feature frame constituting subunit 91, a human body image feature collection area setting subunit 92, and a reference human body image feature point collecting subunit 93.

The user facial feature frame construction sub-unit 91 is configured to construct a user facial feature frame based on the reference facial image feature points of the user facial image; the human body image feature acquisition area setting subunit 92 is configured to translate the user facial feature frame to below the picture frame by at least one frame position, and set an area through which translation passes and an area of the user facial feature frame as a user human body image feature acquisition area; the reference human body image feature point collecting subunit 93 is configured to collect, in the user human body image feature collecting region, a reference human body image feature point of the corresponding user human body image.

When the panorama picture displaying apparatus 70 of this embodiment is used, the panorama picture displaying apparatus 70 first obtains a panorama picture to be played, and then detects a picture frame type of a current picture frame to be displayed, where the picture frame type may be a detected picture frame or a tracking picture frame.

If the current frame is a detection frame, the picture and sound information acquisition unit of the detection frame presentation module 71 acquires the panoramic picture information and the panoramic sound information of the current frame.

The face recognition unit of the detection frame presentation module 71 performs face recognition operation on the acquired panoramic picture information, so that user face images of all users in the current scene can be acquired. The picture position information of the user face image in the panoramic picture is then taken as the image position information of the user face image.

The comparison unit of the detection picture frame display module 71 acquires all panoramic sound information; then, image position information matching the sound position information of the panoramic sound information is confirmed, that is, when the position difference value between the sound position information and the image position information is smaller than a set value, the comparison unit determines that the sound position information matches the image position information. And then the comparison unit takes the user face image corresponding to the matched image position information as the user face image to be displayed.

The first display unit of the detection frame display module 71 displays the user face images of at least some users in the current frame according to the acquired user face images to be displayed and the displayed user face images.

Specifically, the first display unit may match the user face image to be displayed with the displayed user face image, so that the displayed user face image determined to need updating may be updated.

And then the first display unit uses the user face image to be displayed to perform an updating operation on the matched displayed user face image, and if the user face image to be displayed does not match any displayed user face image, the user face image to be displayed is used for replacing one displayed user face image to be displayed to perform display.

This completes the presentation process of the user face image by the detection picture frame presentation module 71.

If the current frame is a tracking frame, the tracking frame presentation module 72 determines the type of the previous frame.

If the previous frame is a detected frame, since the previous frame acquires the user face image through image pixel detection, the accuracy of the user face image acquired by the previous frame is high, and the reference face image feature point acquiring unit 81 of the tracking frame displaying module 72 can directly acquire the reference face image feature point of the user face image displayed in the previous frame.

Since a person in the panoramic image may turn around or be blocked, the user face image is tracked based on only the reference face image feature points of the user face image, and in some cases, all the reference face image feature points are lost.

The first reference human body image feature point obtaining unit 82 of the tracking picture frame displaying module 72 obtains the reference human body image feature points of the corresponding user human body image according to the reference facial image feature points of the user facial image displayed in the previous picture frame, so as to increase the number of the image feature points corresponding to the user facial image.

The step of obtaining the reference human body image characteristic points comprises the following steps:

the user facial feature frame construction sub-unit 91 of the first reference human image feature point acquisition unit 82 constructs a user facial feature frame based on the reference image feature points of the user facial image.

The human body image feature collection area setting subunit 92 of the first reference human body image feature point obtaining unit 82 translates the user face feature frame to below the frame by at least one frame position (i.e., the position from the neck to the body of the user), and sets both the translated area and the area of the user face feature frame as the user human body image feature collection area.

In the acquired user human body image feature acquisition region, the reference human body image feature point acquisition subunit 93 of the first reference human body image feature point acquisition unit 82 acquires the reference human body image feature points of the corresponding user human body image.

The first current human body image feature point search unit 83 of the tracking picture frame presentation module 72 searches for a corresponding current human body image feature point in the current picture frame based on the acquired reference human body image feature point. Namely, the image feature point of the previous picture frame is used for searching the corresponding image feature point in the current picture frame. Because part of the feature points are lost along with the movement of the object in the panoramic picture, the number of the feature points of the current human body image is less than or equal to that of the feature points of the reference human body image.

The first user face image obtaining unit 84 of the tracking picture frame presentation module 72 obtains the user face image in the current picture frame based on the current human body image feature point in the current picture frame, and specifically finds the position of the user face feature frame based on the position of the current human body image feature point, and then takes the image in the user face feature frame as the user face image.

The first user face image acquisition unit 84 then sets the current human body image feature point in the current picture frame as the reference human body image feature point of the current picture frame for calling of the next picture frame.

If the previous frame is the tracking frame, the second reference human body image feature point obtaining unit 85 of the tracking frame displaying module 72 may directly collect the reference human body image feature points of the user human body image displayed in the previous frame.

The second current human body image feature point searching unit 86 of the tracking picture frame displaying module 72 searches the corresponding current human body image feature point in the current picture frame based on the reference human body image feature point of the user human body image displayed in the previous picture frame. Namely, the image feature point of the previous picture frame is used for searching the corresponding image feature point in the current picture frame. Because part of the feature points are lost along with the movement of the object in the panoramic picture, the number of the feature points of the current human body image is less than or equal to that of the feature points of the reference human body image.

The second user face image obtaining unit 87 of the tracking picture frame presentation module 72 obtains the user face image in the current picture frame based on the current human body image feature points in the current picture frame, specifically, finds the position of the user face feature frame based on the position of the current human body image feature points, and then takes the image in the user face feature frame as the user face image.

The second user face image obtaining unit 87 then sets the current human body image feature point in the current picture frame as the reference human body image feature point of the current picture frame for calling of the next picture frame.

The second presentation unit 88 of the follow-up picture frame presentation module 72 matches the corresponding user face image in the current picture frame with the presented user face image to determine the presented user face image that needs to be updated.

The second presentation unit 88 performs an update operation on the matched presented user face image using the user face image to be presented.

This completes the presentation of the user's face image by the follow-up picture frame presentation module 72.

The user face images of all the detection frame and the tracking frame in the panoramic image are displayed, and the displaying process of the user face image in the panoramic image of the panoramic image displaying apparatus 70 of the present embodiment is completed.

On the basis of the first embodiment, the panoramic picture presentation apparatus based on framing processing of the present embodiment improves the recognition accuracy of the user face image of the detected picture frame by the user action information and the sound volume information; the image characteristic points are expanded through the human body picture of the user, and the identification accuracy of the facial image of the user tracking the picture frame is improved.

The following describes a specific working principle of the panoramic picture displaying method and the panoramic picture displaying apparatus based on framing processing according to an embodiment of the present invention. Referring to fig. 10, fig. 10 is a flowchart illustrating a panoramic image displaying method and a panoramic image displaying apparatus according to an embodiment of the present invention. The panorama picture displaying apparatus of the present embodiment is provided on a fixed terminal 110 for receiving and displaying a panorama picture.

The step of the fixed terminal 110 displaying the face image of the user in the panoramic picture includes:

in step S1001, the fixed terminal 110 receives a panoramic picture frame by frame and identifies a type of a picture frame of the received panoramic picture, and a schematic diagram of the picture frame of the panoramic picture may be as shown in fig. 11 a. The display screen of the fixed terminal 110 performs user face image presentation using one panorama picture presentation position 1101 and four fixed user presentation positions 1102.

In step S1002, if the current frame is the detection frame 111, the fixed terminal 110 acquires the panoramic picture information and the panoramic sound information of the current frame.

Step S1003, the fixed terminal 110 performs face recognition operation on the obtained panoramic picture information and obtains user face images of all users in the current picture frame; the picture position information of the user face image in the panoramic picture is then taken as the image position information of the user face image.

In step S1004, the fixed terminal 110 acquires all the panoramic sound information, then confirms the image position information matched with the sound position information of the panoramic sound information, and takes the user face image corresponding to the matched image position information as the user face image to be displayed.

In step S1005, the fixed terminal 100 displays the user face image to be displayed using the fixed user display position. The fixed user display positions for displaying the user face images can be highlighted, the user face images displayed by other fixed user display positions can be directly updated based on the user face images in the corresponding areas in the corresponding panoramic picture information, and the panoramic picture display positions can display the whole panoramic picture. As shown in particular in fig. 11 b.

In step S1006, the fixed terminal then obtains the next frame, which is the tracking frame 112, and since the previous frame is the detection frame, the fixed terminal can directly obtain the reference facial image feature points of the user facial images displayed by the four fixed user display positions of the previous frame.

In step S1007, the fixed terminal generates a user face feature frame of the user face image shown in the previous frame. As shown in detail in region a of fig. 11 c.

And step S1008, the fixed terminal translates the user facial feature frame to a frame position below the picture frame to obtain a user human body image feature acquisition area. As shown in particular in the area b of fig. 11 c.

Step S1009, the fixed terminal generates the reference human body image feature point of the previous frame picture according to the reference human body image feature point in the user human body image feature acquisition area.

Step S1010, the fixed terminal searches current human body image feature points corresponding to the reference human body image feature points in the current picture frame, and obtains a user face image in the current picture frame according to the current human body image feature points; and then setting the current human body image characteristic point in the current picture frame as the reference human body image characteristic point of the current picture frame so as to call the next picture frame.

In step S1011, the fixed terminal displays the user face image in the current frame acquired in step S1010, and since the fixed terminal acquires the user face image in the current frame according to the feature points of the user face image displayed in the previous frame, the fixed terminal only updates the displayed user face image. As shown in particular in fig. 11 d.

In step S1012, the fixed terminal then obtains the next frame, which is the tracking frame 113, and since the previous frame is the tracking frame, the fixed terminal can directly call the reference human body image feature points of the user face image displayed by the four fixed user display positions of the previous frame.

Step S1013, the fixed terminal searches the current human body image characteristic points corresponding to the reference human body image characteristic points in the current picture frame, and obtains the user face image in the current picture frame according to the current human body image characteristic points. And then setting the current human body image characteristic point in the current picture frame as the reference human body image characteristic point of the current picture frame so as to call the next picture frame.

In step S1014, the fixed terminal displays the user face image in the current frame acquired in step S1013, and since the fixed terminal acquires the user face image in the current frame according to the feature points of the user face image displayed in the previous frame, the fixed terminal only updates the displayed user face image.

In step S1015, the fixed terminal then acquires the next frame, which is the detection frame 114, and at this time, the fixed terminal may discard the user face image of the previous frame, and acquire the panoramic image information and the panoramic sound information of the current frame again to acquire the user face image to be displayed.

In step S1016, the fixed terminal displays the facial image of the user to be displayed using the fixed user display position. And if the current fixed user display position is completely used, replacing the displayed user face image with the longest update time with the user face image to be displayed, and highlighting the updated fixed user display position. The user face images displayed by other fixed user display positions can be directly updated based on the user face images in the corresponding areas in the corresponding panoramic picture information, and the panoramic picture display positions can display the whole panoramic picture. As shown in particular in fig. 11 e.

In this way, the display operation of the user face image in the detection picture frame and the tracking picture frame is sequentially performed, thereby completing the display operation of the user face image of the entire panoramic picture.

The panoramic picture display method and device based on framing processing of the invention carries out user face recognition on the detected picture frame and carries out face image tracking on the tracked picture frame, thereby effectively ensuring the face picture display effect in each frame of picture, avoiding the occurrence of frame skipping phenomenon and further improving the picture display efficiency; the technical problem that the frame skipping phenomenon easily occurs in the conventional panoramic picture display method and device, and the picture display efficiency is further influenced is effectively solved.

As used herein, the terms "component," "module," "system," "interface," "process," and the like are generally intended to refer to a computer-related entity: hardware, a combination of hardware and software, or software in execution. For example, a component may be, but is not limited to being, a process running on a processor, an object, an executable, a thread of execution, a program, and/or a computer. By way of illustration, both an application running on a controller and the controller can be a component. One or more components can reside within a process and/or thread of execution and a component may be localized on one computer and/or distributed between two or more computers.

Fig. 12 and the following discussion provide a brief, general description of an operating environment of an electronic device in which the panoramic image display apparatus of the present invention may be implemented. The operating environment of FIG. 12 is only one example of a suitable operating environment and is not intended to suggest any limitation as to the scope of use or functionality of the operating environment. Example electronic devices 1212 include, but are not limited to, wearable devices, head-mounted devices, medical health platforms, personal computers, server computers, hand-held or laptop devices, mobile devices (such as mobile phones, Personal Digital Assistants (PDAs), media players, and the like), multiprocessor systems, consumer electronics, minicomputers, mainframe computers, distributed computing environments that include any of the above systems or devices, and the like.

Although not required, embodiments are described in the general context of "computer readable instructions" being executed by one or more electronic devices. Computer readable instructions may be distributed via computer readable media (discussed below). Computer readable instructions may be implemented as program modules, such as functions, objects, Application Programming Interfaces (APIs), data structures, etc. that perform particular tasks or implement particular abstract data types. Typically, the functionality of the computer readable instructions may be combined or distributed as desired in various environments.

Fig. 12 illustrates an example of an electronic device 1212 that includes one or more embodiments of the panoramic picture presentation apparatus of the present invention. In one configuration, the electronic device 1212 includes at least one processing unit 1216 and memory 1218. Depending on the exact configuration and type of electronic device, memory 1218 may be volatile (such as RAM), non-volatile (such as ROM, flash memory, etc.) or some combination of the two. This configuration is illustrated in fig. 12 by dashed line 1214.

In other embodiments, electronic device 1212 may include additional features and/or functionality. For example, device 1212 may also include additional storage (e.g., removable and/or non-removable) including, but not limited to, magnetic storage, optical storage, and the like. Such additional storage is illustrated in fig. 12 by storage 1220. In one embodiment, computer readable instructions to implement one or more embodiments provided herein may be in storage 1220. Storage 1220 may also store other computer readable instructions to implement an operating system, an application program, and the like. Computer readable instructions may be loaded in memory 1218 for execution by processing unit 1216, for example.

The term "computer readable media" as used herein includes computer storage media. Computer storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions or other data. Memory 1218 and storage 1220 are examples of computer storage media. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, Digital Versatile Disks (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can accessed by electronic device 1212. Any such computer storage media may be part of electronic device 1212.

Electronic device 1212 may also include communication connection(s) 1226 that allow electronic device 1212 to communicate with other devices. Communication connection(s) 1226 may include, but are not limited to, a modem, a Network Interface Card (NIC), an integrated network interface, a radio frequency transmitter/receiver, an infrared port, a USB connection, or other interfaces for connecting electronic device 1212 to other electronic devices. Communication connection 1226 may include a wired connection or a wireless connection. Communication connection(s) 1226 may transmit and/or receive communication media.

The term "computer readable media" may include communication media. Communication media typically embodies computer readable instructions or other data in a "modulated data signal" such as a carrier wave or other transport mechanism and includes any information delivery media. The term "modulated data signal" may include signals that: one or more of the signal characteristics may be set or changed in such a manner as to encode information in the signal.

Electronic device 1212 may include input device(s) 1224 such as keyboard, mouse, pen, voice input device, touch input device, infrared camera, video input device, and/or any other input device. Output device(s) 1222 such as one or more displays, speakers, printers, and/or any other output device may also be included in device 1212. The input device 1224 and the output device 1222 may be connected to the electronic device 1212 via wired connections, wireless connections, or any combination thereof. In one embodiment, an input device or an output device from another electronic device may be used as input device 1224 or output device 1222 for electronic device 1212.

Components of electronic device 1212 may be connected by various interconnects, such as a bus. Such interconnects may include Peripheral Component Interconnect (PCI), such as PCI express, Universal Serial Bus (USB), firewire (IEEE1394), optical bus structures, and the like. In another embodiment, components of electronic device 1212 may be interconnected by a network. For example, memory 1218 may be comprised of multiple physical memory units located in different physical locations interconnected by a network.

Those skilled in the art will realize that storage devices utilized to store computer readable instructions may be distributed across a network. For example, an electronic device 1230 accessible via a network 1228 may store computer readable instructions to implement one or more embodiments provided by the present invention. Electronic device 1212 may access electronic device 1230 and download a part or all of the computer readable instructions for execution. Alternatively, electronic device 1212 may download pieces of the computer readable instructions, as needed, or some instructions may be executed at electronic device 1212 and some at electronic device 1230.

Various operations of embodiments are provided herein. In one embodiment, the one or more operations may constitute computer readable instructions stored on one or more computer readable media, which when executed by an electronic device, will cause the computing device to perform the operations. The order in which some or all of the operations are described should not be construed as to imply that these operations are necessarily order dependent. Those skilled in the art will appreciate alternative orderings having the benefit of this description. Moreover, it should be understood that not all operations are necessarily present in each embodiment provided herein.

Also, although the disclosure has been shown and described with respect to one or more implementations, equivalent alterations and modifications will occur to others skilled in the art based upon a reading and understanding of this specification and the annexed drawings. The present disclosure includes all such modifications and alterations, and is limited only by the scope of the appended claims. In particular regard to the various functions performed by the above described components (e.g., elements, resources, etc.), the terms used to describe such components are intended to correspond, unless otherwise indicated, to any component which performs the specified function of the described component (e.g., that is functionally equivalent), even though not structurally equivalent to the disclosed structure which performs the function in the herein illustrated exemplary implementations of the disclosure. In addition, while a particular feature of the disclosure may have been disclosed with respect to only one of several implementations, such feature may be combined with one or more other features of the other implementations as may be desired and advantageous for a given or particular application. Furthermore, to the extent that the terms "includes," has, "" contains, "or variants thereof are used in either the detailed description or the claims, such terms are intended to be inclusive in a manner similar to the term" comprising.

Each functional unit in the embodiments of the present invention may be integrated into one processing module, or each unit may exist alone physically, or two or more units are integrated into one module. The integrated module can be realized in a hardware mode, and can also be realized in a software functional module mode. The integrated module, if implemented in the form of a software functional module and sold or used as a stand-alone product, may also be stored in a computer readable storage medium. The storage medium mentioned above may be a read-only memory, a magnetic or optical disk, etc. Each apparatus or system described above may perform the method in the corresponding method embodiment.

In summary, although the present invention has been disclosed in the foregoing embodiments, the serial numbers before the embodiments are used for convenience of description only, and the sequence of the embodiments of the present invention is not limited. Furthermore, the above embodiments are not intended to limit the present invention, and those skilled in the art can make various changes and modifications without departing from the spirit and scope of the present invention, therefore, the scope of the present invention shall be limited by the appended claims.

Claims

1. A panoramic picture display method based on framing processing is used for displaying a panoramic picture, and is characterized in that the panoramic picture comprises detection picture frames and corresponding tracking picture frames, wherein one detection picture frame corresponds to at least one tracking picture frame; and the tracking picture frame is arranged behind the corresponding detection picture frame; the panoramic picture display method comprises the following steps:

if the current picture frame is the detection picture frame, acquiring a user face image to be displayed according to the panoramic picture information and the panoramic sound information of the detection picture frame; displaying the user face image to be displayed;

if the current picture frame is the tracking picture frame, acquiring a reference facial image feature point of a user facial image displayed by the previous picture frame; acquiring a corresponding user face image in the tracking picture frame based on the reference face image feature point; updating the displayed user face image by using the user face image in the tracking picture frame;

the step of obtaining the user face image to be displayed according to the panoramic picture information and the panoramic sound information of the detection picture frame comprises the following steps:

acquiring panoramic picture information and panoramic sound information of a current picture frame; wherein the panoramic sound information includes sound location information;

performing face recognition operation on the panoramic picture information to acquire a user face image of the current scene and corresponding image position information;

comparing the image position information with the sound position information to obtain a user face image to be displayed;

obtaining a reference facial image feature point of a user facial image shown in a previous picture frame; the step of acquiring a corresponding user face image in the tracking picture frame based on the reference face image feature point includes:

acquiring reference face image feature points of a user face image displayed by the previous picture frame;

searching a corresponding current facial image feature point in the current picture frame based on a reference facial image feature point of a user facial image shown in the previous picture frame; the number of the current facial image feature points is less than or equal to the number of the reference facial image feature points;

and acquiring a user face image in the current picture frame based on the current face image feature point, and setting the current face image feature point in the current picture frame as a reference face image feature point of the current picture frame.

2. The framing processing-based panorama picture presentation method according to claim 1, wherein said step of performing a presentation operation on said user's face image to be presented includes:

and displaying at least part of the user face images of the users in the current picture frame according to the user face images to be displayed and the displayed user face images.

3. The framing processing-based panoramic picture presentation method according to claim 2, wherein the step of presenting the user face images of at least some users in the current picture frame according to the user face image to be presented and the presented user face image comprises:

matching the user face image to be displayed with the displayed user face image;

and using the user face image to be displayed to update the matched displayed user face image.

4. The framing processing-based panoramic picture presentation method according to claim 1, wherein the reference facial image feature points of the user facial image presented in the previous picture frame are acquired; the step of acquiring a corresponding user face image in the tracking picture frame based on the reference face image feature point includes:

if the previous picture frame is a detection picture frame, acquiring a reference facial image feature point of a user facial image displayed by the previous picture frame;

acquiring a reference human body image characteristic point of a corresponding user human body image according to the reference facial image characteristic point of the user facial image displayed in the previous picture frame;

searching a corresponding current human body image characteristic point in the current picture frame based on the reference human body image characteristic point; the number of the current human body image feature points is less than or equal to that of the reference human body image feature points;

acquiring a user face image in the current picture frame based on the current human body image feature point in the current picture frame, and setting the current human body image feature point in the current picture frame as a reference human body image feature point of the current picture frame;

the step of obtaining the corresponding reference human body image feature points of the user human body image according to the reference facial image feature points of the user facial image displayed in the previous picture frame comprises the following steps:

constructing a user facial feature frame based on the reference facial image feature points of the user facial image;

translating the user facial feature frame to at least one frame position below the picture frame, and setting a region through which translation passes and a region of the user facial feature frame as a user human body image feature acquisition region;

and acquiring reference human body image characteristic points of the corresponding user human body image in the user human body image characteristic acquisition region.

5. The framing processing-based panoramic picture presentation method according to claim 4, wherein the reference facial image feature points of the user facial image presented in the previous picture frame are acquired; the step of acquiring a corresponding user face image in the tracking picture frame based on the reference face image feature point includes:

if the previous picture frame is a tracking picture frame, acquiring reference human body image characteristic points of the user human body image displayed by the previous picture frame;

searching a corresponding current human body image characteristic point in the current picture frame based on the reference human body image characteristic point of the user human body image displayed in the previous picture frame; the number of the current human body image feature points is less than or equal to that of the reference human body image feature points;

and acquiring a user face image in the current picture frame based on the current human body image feature point in the current picture frame, and setting the current human body image feature point in the current picture frame as a reference human body image feature point of the current picture frame.

6. The framing process-based panorama picture presentation method according to claim 1, wherein said step of updating the presented user face image using the user face image in the current picture frame comprises:

matching the corresponding user face image in the current picture frame with the displayed user face image;

and updating the matched and displayed user face image by using the corresponding user face image in the current picture frame.

7. A panoramic picture display device based on framing processing is used for displaying a panoramic picture, and is characterized in that the panoramic picture comprises detection picture frames and corresponding tracking picture frames, wherein one detection picture frame corresponds to at least one tracking picture frame; and the tracking picture frame is arranged behind the corresponding detection picture frame; the panorama picture display apparatus includes:

the detection picture frame display module is used for acquiring a user face image to be displayed according to the panoramic picture information and the panoramic sound information of the detection picture frame if the current picture frame is the detection picture frame; displaying the user face image to be displayed; and

the tracking picture frame display module is used for acquiring a reference facial image feature point of a user facial image displayed by a previous picture frame if the current picture frame is the tracking picture frame; acquiring a corresponding user face image in the tracking picture frame based on the reference face image feature point; updating the displayed user face image by using the user face image in the tracking picture frame;

obtaining a reference facial image feature point of a user facial image shown in a previous picture frame; the step of acquiring a corresponding user face image in the current picture frame based on the reference face image feature point includes: