CN118034620A

CN118034620A - Screen-throwing media resource display method and virtual display device based on target identification

Info

Publication number: CN118034620A
Application number: CN202211394098.4A
Authority: CN
Inventors: 王大勇; 李斌; 何灿
Original assignee: Hisense Visual Technology Co Ltd
Current assignee: Hisense Visual Technology Co Ltd
Priority date: 2022-11-08
Filing date: 2022-11-08
Publication date: 2024-05-14

Abstract

The application relates to the technical field of virtual display, and provides a screen-throwing media resource display method and virtual display equipment based on target identification.

Description

Screen-throwing media resource display method and virtual display device based on target identification

Technical Field

Background

In the development of the current virtual display devices such as AR/VR, the virtual display device is enabled to support the screen projection initiated by other devices so as to increase the media resources played by the virtual display device, and the development of the virtual display devices such as AR/VR is one of the requirements for improving the diversity of the played contents of the virtual display devices such as AR/VR.

At present, a common scheme for increasing media resources played by virtual display devices such as AR/VR is as follows: and displaying media resources in intelligent terminals such as intelligent mobile phones, intelligent televisions, tablets and the like in a virtual mirror window of virtual display equipment such as AR/VR and the like in a screen throwing mode. However, in the actual screen projection process, the screen of the intelligent terminal is likely to keep a bright screen state, and under the condition of VR perspective mode or AR superposition display, the content displayed in the bright screen state of the intelligent terminal can form visual interference on the media resource displayed by the virtual mirror image window of the virtual display device, so that the watching effect of the user is affected.

Therefore, the problem that visual interference of a screen of an intelligent terminal to a virtual window of a virtual display device in the screen throwing process is to be solved is urgent.

Disclosure of Invention

The embodiment of the application provides a screen-throwing media resource display method and virtual display equipment based on target identification, which are used for reducing visual interference of a screen of an intelligent terminal initiating screen throwing on a virtual mirror window, so that user experience is improved.

In one aspect, an embodiment of the present application provides a method for displaying a screen-throwing media asset based on target recognition, including:

responding to a screen throwing request sent by an intelligent terminal, and receiving media resources sent by the intelligent terminal;

acquiring a real scene image acquired by a binocular camera, and identifying the real scene image to acquire position information and size information of a screen of the intelligent terminal;

Determining the position and the size of a virtual mirror window according to the position information and the size information of the screen so that the virtual mirror window completely covers the screen of the intelligent terminal;

And rendering and displaying the media resources of the intelligent terminal screen in the virtual mirror window.

In another aspect, an embodiment of the present application provides a virtual display device, including a multi-camera, a processor, a memory, and a communication interface, where the multi-camera, the communication interface, the memory, and the process are connected by a bus;

the memory stores a computer program, and the processor performs the following operations according to the computer program:

receiving a screen throwing request sent by an intelligent terminal and media resources sent by the intelligent terminal through the communication interface;

In another aspect, an embodiment of the present application provides a computer readable storage medium, where computer executable instructions are stored, where the computer executable instructions are configured to cause a computer device to execute steps of a method for displaying a screen-casting medium resource based on object recognition provided by the embodiment of the present application.

In the screen-throwing media information display method and the virtual display device based on target identification, after receiving the screen-throwing request sent by the intelligent terminal, the acquired real scene image acquired by the binocular camera is identified, the position information and the size information of the screen of the intelligent terminal are determined, and the dynamic self-adaptive adjustment of the virtual mirror image window in the virtual reality scene is realized by utilizing the position information and the size information of the screen, so that the virtual mirror image window can completely cover the screen of the intelligent terminal, the media resources displayed by the screen of the intelligent terminal in the screen-throwing process are reduced, the visual interference on the media resources displayed by the virtual mirror image window of the virtual display device is improved, and the viewing experience of a user is improved.

Drawings

In order to more clearly illustrate the embodiments of the application or the technical solutions of the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described, and it is obvious that the drawings in the description below are some embodiments of the application, and other drawings can be obtained according to these drawings without inventive effort for a person skilled in the art.

Fig. 1 is a schematic view of a screen-projection scene provided by an embodiment of the present application;

Fig. 2A is a schematic diagram of a VR headset according to an embodiment of the present application;

fig. 2B is a schematic view of AR glasses according to an embodiment of the present application;

FIG. 3 is a diagram showing the effect of visual disturbance according to an embodiment of the present application;

FIG. 4 is a flowchart of a method for displaying a screen-throwing media asset based on target identification according to an embodiment of the present application;

Fig. 5 is a schematic diagram of a process for establishing a screen connection according to an embodiment of the present application;

FIG. 6 is a frame diagram of a multi-view stereoscopic recognition algorithm provided by an embodiment of the application;

Fig. 7A is a diagram of a single-object stereoscopic recognition effect according to an embodiment of the present application;

fig. 7B is a diagram of multi-objective stereoscopic recognition effect according to an embodiment of the present application;

FIG. 7C is a diagram of another multi-object stereo recognition effect provided by an embodiment of the present application;

Fig. 8 is a flowchart of a method for determining position information and size information of a screen of an intelligent terminal according to an embodiment of the present application;

Fig. 9 is a view of a screen effect after resolving visual interference according to an embodiment of the present application;

FIG. 10 is a schematic diagram of displaying a plurality of windows on a virtual desktop according to an embodiment of the present application;

FIG. 11 is a diagram showing the effect of a method for displaying a virtual mirror window according to an embodiment of the present application;

Fig. 12 is a screen display effect diagram of a plurality of intelligent terminals according to an embodiment of the present application;

fig. 13 is a view of a screen display effect of another intelligent terminal according to an embodiment of the present application;

FIG. 14 is a flowchart of a complete target recognition-based screen media asset display method according to an embodiment of the present application;

Fig. 15 is a block diagram of a virtual display device according to an embodiment of the present application.

Detailed Description

For the purpose of making the objects, technical solutions and advantages of the embodiments of the present application more apparent, the technical solutions of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application, and it is apparent that the described embodiments are some embodiments of the technical solutions of the present application, but not all embodiments. All other embodiments, based on the embodiments described in the present document, which can be obtained by a person skilled in the art without any creative effort, are within the scope of protection of the technical solutions of the present application.

Virtual display devices (e.g., AR glasses, VR head mounted displays) are devices that employ AR/VR technology to present virtual pictures to a user to enhance the user's viewing experience. The virtual pictures can come from media resources (such as a 2D film source, a panoramic film source and the like) in intelligent terminals such as intelligent mobile phones, intelligent televisions, tablets and the like, so that the diversity of the played contents of the virtual display equipment is increased.

However, in the actual screen projection process, the screen of the intelligent terminal is likely to keep a bright screen state, and under the condition of VR perspective mode and AR superposition display, the content displayed in the bright screen state of the intelligent terminal can form visual interference on the media resources displayed by the virtual window of the virtual display device, so that the watching effect of a user is affected.

In view of this, the embodiment of the application provides a screen-throwing media resource display method based on target recognition, which performs 3D target recognition according to images acquired by a camera on a virtual display device to obtain the size and position of an intelligent terminal initiating media resource screen throwing, and based on the size and position of the intelligent terminal, dynamically adjusts the position and size of a virtual mirror window in a self-adaptive manner, thereby covering the screen of the intelligent terminal, solving visual interference and improving visual experience of a user.

Referring to fig. 1, for a schematic view of a screen projection scenario provided in an embodiment of the present application, a screen projection connection is established between a virtual display device 100 and an intelligent terminal 200 in a wired or wireless manner, and media resources projected by the intelligent terminal 200 are displayed through the virtual display device 100 in a virtual mirror window. The media resource put in by the intelligent terminal 200 may be a local 2D patch source or a panoramic patch source, or may be a 2D patch source or a panoramic patch source loaded from a server through a network.

The 2D film source at least comprises a 2D picture or a 2D video, and the panoramic film source at least comprises a panoramic picture or a panoramic video.

In an embodiment of the present application, virtual display device 100 includes, but is not limited to, VR head mounted display 101 and AR glasses 102; the smart terminal 200 includes, but is not limited to, a smart television 201, a smart phone 202, a laser television 203 (also understood as a curtain of a projector), a tablet 204, and a computer; the server 300 may be an independent physical server, a server cluster or a distributed system formed by a plurality of physical servers, or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, a content delivery network (Content Delivery Network, CDN), basic cloud computing services such as big data and an artificial intelligent platform. The intelligent terminal 200 and the server 300 may be directly or indirectly connected through wired or wireless communication, and the present application is not limited herein.

Typically, the virtual display device itself is mounted with a plurality of RGB cameras or infrared cameras.

For example, taking the VR head mounted display of fig. 2A as an example, cameras with different orientations are respectively installed at four positions of the upper left, lower left, upper right and lower right of the VR head mounted display housing, and images with different viewing angles can be acquired. In VR perspective mode, the user can see objects in the real world through the four multi-view cameras.

For another example, taking the AR glasses in fig. 2B as an example, one camera is provided at each of both sides and the middle of the frame of the AR glasses, and images of the real world are acquired from different positions and virtual pictures are superimposed and displayed in the images of the real world.

By utilizing images acquired by the multi-camera on the virtual display device, 3D objects in the visual field range of the camera can be identified through a stereoscopic vision algorithm, and the information such as the size and the position of the 3D objects can be estimated while the type of the 3D objects is identified.

When the screen is projected under the VR perspective mode or the AR superimposed display, the user may see the screen of the intelligent terminal initiating the screen projection when watching the media resource projected onto the virtual mirror window.

Taking an intelligent terminal as an intelligent electricity as an example, the intelligent television is taken as a screen throwing starting end, a currently displayed media resource is thrown into a virtual mirror window of the AR glasses, as shown in fig. 3, 301 is a real picture currently played by a screen of the intelligent television, and 302 is a virtual picture synchronously played by the virtual mirror window of the AR glasses. Because the real picture 301 currently played by the screen of the smart television is the same as the virtual picture 302 played by the virtual mirror window of the AR glasses during screen casting, and the screen of the smart television is relatively larger, although the brightness of the virtual picture 302 played by the virtual mirror window of the AR glasses is higher than that of the real picture 301, in the field of view, the real picture 301 currently played by the screen of the smart television has a certain interference to the viewing of the virtual picture 302 played by the virtual mirror window of the AR glasses.

Therefore, the embodiment of the application utilizes the identification result of the stereoscopic vision algorithm to cover the screen of the intelligent terminal in the screen throwing process, thereby solving the visual interference of the content displayed by the screen of the intelligent terminal on the media resource thrown to the virtual mirror window and improving the watching experience of the user.

Referring to fig. 4, a flowchart of a target recognition-based screen media resource display method is provided in an embodiment of the present application, where the flowchart is executed by a virtual display device with AR or VR technology, and mainly includes the following steps:

S401: and responding to the screen throwing request sent by the intelligent terminal, and receiving the media resource sent by the intelligent terminal.

In order to increase the diversity of the content played by the virtual display device, the virtual display device provides a virtual desktop function, so that a 2D film source and a panorama film source played by the intelligent terminal are supported. The virtual desktop has extensibility and can draw at least one window to display media resources which need to be played by the virtual display equipment.

By means of the virtual desktop function, after the intelligent terminal and the virtual display device are connected through communication, the intelligent terminal can cast the media resource currently displayed by the screen into the virtual mirror window of the virtual display device.

For example, taking an intelligent terminal as a TV and a virtual display device as AR glasses as an example, as shown in fig. 5, a user interacts with the TV through a remote controller, and for a media resource displayed on a current TV screen, the TV pops up a screen-throwing page widget on the screen, after the user clicks a "confirm" button through the remote controller, the TV pops up a screen-throwing device widget on the upper right corner of the screen, after the user selects an "AR glasses" device in the screen-throwing device widget, the TV sends a screen-throwing request to the AR glasses, after the AR glasses receive the screen-throwing request, a screen-throwing connection is established with the TV according to a device identifier carried in the screen-throwing request, and after the screen-throwing connection is established between the two, the TV sends the media resource displayed on the current screen to the AR glasses.

Optionally, the device identifier may be information such as a model number of the intelligent terminal, an IP of the intelligent terminal, a name of the intelligent terminal, and the like.

In some embodiments, after the intelligent terminal establishes a screen-casting connection with the virtual display device, screen-casting data may also be sent, where the screen-casting data includes at least one of a name, an ID, and a uniform resource locator (Uniform Resource Locator, URL) of a media resource currently displayed by a screen of the intelligent terminal. The virtual display device loads corresponding media resources from the server according to the screen throwing data.

S402: and acquiring a real scene image acquired by the binocular camera, and identifying the real scene image to acquire the position information and the size information of the screen of the intelligent terminal.

Along with the development of the deep learning technology, for images collected by the multi-eye cameras, a stereoscopic vision algorithm can be adopted to perform target recognition, a recognition framework is shown in fig. 6, binocular images collected by the left and right binocular cameras are extracted, for each monocular image, a 2D detection frame of a 3D object is calculated, the size, angle and key points of the 3D object are estimated, 3D detection estimation is performed by combining the 2D detection frames, and a 3D detection result can be updated by reducing the secondary projection errors of the 2D frames and the key points.

As shown in fig. 7A, an identification effect diagram of a single 3D object provided by the embodiment of the present application is shown, where the length, width and height of a 3D detection frame may represent the size of a cup, the position of the 3D detection frame in an XYZ coordinate system may represent the position of the cup, and the Z axis is parallel to illumination of a camera, so the size of the Z axis coordinate may reflect the distance from the cup to the camera, as in fig. 7A, the distance from the cup to the camera is 98.35 centimeters (cm).

In some embodiments, multiple 3D objects may be included in the same image, and each 3D object may be identified using a stereoscopic algorithm for object recognition.

For example, as shown in fig. 7B, an identification effect diagram of a plurality of 3D objects is provided in an embodiment of the present application, where each cup corresponds to one 3D detection frame, and each 3D detection frame may reflect a position and a size of a corresponding cup.

In some embodiments, multiple 3D objects may be included in the same image, and a stereoscopic algorithm may be used to perform object recognition, so that each 3D object may be identified, and the detected multiple 3D objects may be taken as a whole.

For example, as shown in fig. 7C, another recognition effect diagram of multiple 3D objects provided by the embodiment of the present application is shown, where a stereoscopic algorithm is used to recognize that two cups correspond to one 3D detection frame (not shown in fig. 7C) for each cup, and according to the 3D detection frames of the two cups, a large detection frame surrounding the two 3D detection frames is determined, and the positions and sizes of the two cups can be reflected by the large 3D detection frame.

Alternatively, in the embodiment of the present application, a stereo vision algorithm based on deep learning may use a stereo-RPN network.

In the embodiment of the application, the screen of the intelligent terminal can be a screen (such as a screen of a mobile phone or a screen of a television) integrated on the intelligent terminal equipment, or can be a screen (such as a curtain of a projector) independent of the intelligent equipment.

The determination method of the position information and the size information of the screen of the intelligent terminal in the three-dimensional coordinate system based on the depth learning stereoscopic vision algorithm is shown in fig. 8, and mainly comprises the following steps:

S4021: and identifying the screen of the intelligent terminal according to the real scene images acquired by the binocular camera, and respectively obtaining 2D detection frames of the screen.

S4022: the size, angle and key points of the screen in each real scene image are estimated, and a 3D detection frame surrounding the screen is determined by combining 2D detection frames.

S4023: and determining the position information and the size information of the screen of the intelligent terminal according to the 3D detection frame.

Position information and size information of a screen of the intelligent terminal are determined, and then shielding processing can be carried out on the screen to solve the problem of visual interference.

S403: and determining the position and the size of the virtual mirror window according to the position information and the size information of the screen of the intelligent terminal so that the virtual mirror window completely covers the screen of the intelligent terminal.

Specifically, the position of the screen is set to the position of the virtual mirror window according to the position information of the screen of the intelligent terminal, and the size of the screen is set to the size of the virtual mirror window according to the size information of the screen of the intelligent terminal, so that the virtual mirror window can completely cover the screen of the intelligent terminal.

Optionally, in some embodiments, the virtual mirror window is a window of a media resource for playing a screen throw, which is drawn in real time on the virtual desktop according to position information and size information of a screen of the intelligent terminal.

Optionally, in other embodiments, the virtual mirror window is a window that is drawn in advance on the virtual desktop and used for playing the media resource of the screen, and then is adjusted according to the position information and the size information of the screen of the intelligent terminal.

In some embodiments, the 3D screen-throwing scene may be built in a 3D homepage of the virtual display device, so as to accelerate the starting of the virtual desktop, and further improve the screen-throwing efficiency.

S404: and rendering and displaying the media resources of the intelligent terminal screen in the virtual mirror window.

Still taking the application scenario shown in fig. 3 as an example, the display effect after screen projection by using the method provided by the embodiment of the application is shown in fig. 9, and as can be seen from fig. 9, the virtual mirror window of the AR glasses completely covers the screen of the TV, so that the user can only see the media resources projected onto the virtual mirror window by the TV and cannot see the media resources displayed on the screen of the TV, thereby solving the problem of visual interference caused by the screen of the TV.

According to the screen-throwing media resource display method based on target identification, after a screen throwing request sent by an intelligent terminal is received, the acquired real scene image acquired by the binocular camera is identified, the position information and the size information of the screen of the intelligent terminal are determined, and the position information and the size information of the screen are utilized to realize dynamic self-adaptive adjustment of the virtual mirror window in the virtual reality scene, so that the virtual mirror window can completely cover the screen of the intelligent terminal, media resources displayed by the screen of the intelligent terminal in the screen throwing process are reduced, visual interference on the media resources displayed by the virtual mirror window of the virtual display device is avoided, and viewing experience of a user is improved.

The virtual desktop of the virtual display device is malleable and may display one or more windows. Thus, in some embodiments, a virtual configuration window containing virtual display device configuration information is displayed while rendering media resources that display a screen cast of the intelligent terminal in the virtual mirror window, wherein the virtual mirror window and the virtual configuration window are independent of each other and do not overlap.

As shown in fig. 10, a schematic diagram of displaying a plurality of windows on a virtual desktop of a virtual display device is shown, where (a) is a display effect of displaying a media resource displayed on a screen of a TV onto a virtual mirror window of an AR eye in the related art, a user of the screen-casting method can see a media resource 301 displayed on a screen of the TV, a media resource 302 displayed on a virtual mirror window and configuration information 303 displayed on a virtual configuration window, a screen of the TV forms visual interference to the virtual mirror window, and (b) is a display effect of displaying a media resource displayed on a screen of the TV onto a virtual mirror window of the AR eye by adopting the method provided by the embodiment of the present application, and the user of the screen-casting method can see a media resource 302 displayed on a screen of the virtual mirror window and the configuration information 303 displayed on the virtual configuration window, so as to solve the visual interference problem of a screen of the TV.

As can be seen from comparing the display effects of (a) and (b) in fig. 10, since the virtual mirror window and the virtual configuration window are independent and do not overlap each other, the position and the size of the virtual configuration window are not changed when the position and the size of the virtual mirror window are determined.

Considering that when the distance between the screen of the intelligent terminal and the virtual display device is far, the position of the virtual mirror window is far, according to the principle that the distance is small and the distance is large, the virtual mirror window seen by human eyes is small, and the information of the media resource put in can not be seen clearly. Therefore, in some embodiments, after determining the position information of the screen of the intelligent terminal, comparing the distance between the screen included in the position information and the virtual display device with a preset distance threshold, and when the distance is greater than the preset distance threshold, setting the position of the screen of the intelligent terminal as the position of the virtual mirror window, increasing the size information of the screen according to a preset proportion, and taking the increased size as the size of the virtual mirror window.

For example, as shown in fig. 11, when the distance between the virtual display device and the TV is greater than the preset distance threshold, (a) is an effect diagram of directly setting the size of the screen of the TV to the size of the virtual mirror window of the AR glasses, and (b) is an effect diagram of setting the size of the screen of the TV to the size of the virtual mirror window of the AR glasses after increasing by a preset ratio.

Considering that a plurality of intelligent terminals and a virtual display device are possibly connected in the same network in a home network, the intelligent terminals can initiate screen projection to the virtual display device, and media resources displayed by the intelligent terminals can be different. In order to see media resources put in by each intelligent terminal through the virtual display equipment, the embodiment of the application fully utilizes the ductility of the virtual desktop, and a corresponding virtual mirror window is drawn for the screen of each intelligent terminal in the screen throwing process.

Optionally, in some embodiments, taking two intelligent terminals as examples, when both the two intelligent terminals initiate a screen-throwing request to the virtual display device and the screens of the two intelligent terminals are identified, the virtual display device determines the position and the size of the first virtual mirror window according to the identified position information and the size information of the screen of the first intelligent terminal, so that the first virtual mirror window completely covers the screen of the first intelligent terminal, renders and displays the media resource of the screen-throwing of the first intelligent terminal in the first virtual mirror window, and determines the position and the size of the second virtual mirror window according to the position information and the size information of the screen of the second intelligent terminal, so that the second virtual mirror window completely covers the screen of the second intelligent terminal, and renders and displays the media resource of the screen-throwing of the second intelligent terminal in the second virtual mirror window; the first intelligent terminal and the second intelligent terminal are different in screen-throwing media resources.

For example, as shown in fig. 12, in the scenario shown in fig. 12 (a), when a user views a TV play played by a TV 1201, the user uses a mobile phone 1202 to remotely communicate with a small terminal, through man-machine interaction, the TV and the mobile phone both send a screen-throwing request to an AR glasses, the AR glasses establish a screen-throwing connection with the TV and the mobile phone after receiving the screen-throwing request of the TV and the mobile phone, respectively receive a video frame of the TV throwing and a chat interface of the mobile phone throwing, enable a binocular camera on the AR glasses to collect a real scene image, identify position information and size information of a screen of the TV 1201, and identify position information and size information of a screen of the mobile phone 1202; according to the position information and the size information of the screen of the TV 1201, the position and the size of the first virtual mirror window (e.g. 1203 in (b)) are determined so that the first virtual mirror window 1203 completely covers the screen of the TV 1201, video frames of TV delivery are rendered and displayed in the first virtual mirror window 1203, meanwhile, according to the position information and the size information of the screen of the mobile phone 1202, the position and the size of the second virtual mirror window (e.g. 1204 in (b)) are determined so that the second virtual mirror window 1204 completely covers the screen of the mobile phone 1202, and a chat interface of mobile phone delivery is rendered and displayed in the second virtual mirror window 1204.

Alternatively, in other embodiments, when a plurality of intelligent terminals send a screen projection request to the virtual display device, the screens of the plurality of intelligent terminals may be drawn on one virtual mirror window.

Taking two intelligent terminals as an example, when the two intelligent terminals initiate screen throwing requests to the virtual display device and the screens of the two intelligent terminals are identified, the virtual display device determines the position information of a virtual mirror window according to the position information of the screen of the first intelligent terminal and the position information of the screen of the second intelligent terminal, and determines the size information of the virtual mirror window according to the size information of the screen of the first intelligent terminal and the size information of the screen of the second intelligent terminal, so that the virtual mirror window completely covers the screen of the first intelligent terminal and the screen of the second intelligent terminal; rendering and displaying the media resource of the first intelligent terminal screen in a first area of the virtual mirror window, and rendering and displaying the media resource of the second intelligent terminal screen in a second area of the virtual mirror window; the first intelligent terminal and the second intelligent terminal are different in screen-throwing media resources.

For example, as shown in fig. 13, in the scenario shown in fig. 13 (a), when a user views a TV play played by a TV 1301, the user uses a mobile phone 1302 to remotely communicate with a small figure, through man-machine interaction, the TV and the mobile phone both send a screen-throwing request to AR glasses, the AR glasses establish screen-throwing connection with the TV and the mobile phone after receiving the screen-throwing request of the TV and the mobile phone, respectively receive a video frame thrown by the TV and a chat interface thrown by the mobile phone, enable a binocular camera on the AR glasses to collect a real scene image, identify position information and size information of a screen of the TV 1301, and identify position information and size information of a screen of the mobile phone 1302; according to the position information and the size information of the screen of the TV 1301 and the position information and the size information of the screen of the mobile phone 1302, the position and the size of a virtual mirror window (e.g. 1303 in (b)) are determined, so that the virtual mirror window 1303 completely covers the screen of the TV 1301 and the screen of the mobile phone 1202, and video frames put in the TV 1301 are rendered and displayed in a first area 1303_1 of the virtual mirror window 1303, and at the same time, a chat interface put in the mobile phone 1302 is rendered and displayed in a second area 1303_2 of the virtual mirror window 1303.

In the embodiment of the application, the position of the intelligent terminal in the real world is considered to be fixed, the virtual display device rotates along with the rotation of the head in the screen throwing process, and in order to prevent the visual interference of the screen of the intelligent terminal on the virtual mirror window in the rotation process, the position of the virtual mirror window in the 3D space is kept fixed, so that the screen of the intelligent terminal is always covered.

Referring to fig. 14, a complete flowchart of a target recognition-based screen media asset display method is provided for implementation of the present application, where the flowchart mainly includes the following steps:

s1401, determining whether a screen throwing request sent by the intelligent terminal is received according to the screen throwing service state, if not, executing S1402, and if yes, executing S1403.

The screen-throwing service state can be determined according to a screen-throwing protocol, and when the screen-throwing service is started, the screen-throwing service state indicates that a screen-throwing request sent by the intelligent terminal can be received.

S1402: and the virtual desktop is not started, and the 3D film source is normally displayed.

S1403: and starting the virtual desktop.

To expedite the start of the virtual desktop, a 3D drop screen scene may be built into the 3D homepage of the virtual display device.

S1404: whether the adjustment switch of the virtual mirror window is opened is determined, if not, S1405 is executed, and if yes, S1406 is executed.

After the adjusting switch of the virtual mirror window is started, the position and the size of the virtual mirror window can be adaptively adjusted according to the position and the size of the intelligent terminal, so that the problem of visual interference is solved.

Wherein the position and size of the virtual mirror window before adjustment is default.

S1405: and giving prompt information to turn on an adjusting switch of the virtual mirror window.

S1406: and acquiring a real scene image acquired by the binocular camera.

S1407: it is determined whether the screen of the intelligent terminal is recognized from the binocular reality scene image, if not, S1408 is executed, and if yes, S1409 is executed.

S1408: and adjusting the visual angle of the virtual display device, and re-acquiring the image of the real scene.

S1409: and determining position information and size information of a screen of the intelligent terminal.

S1410: and adjusting the position and the size of the virtual mirror window according to the position information and the size information of the screen of the intelligent terminal so as to cover the screen of the intelligent terminal.

S1411: and displaying the media resources put in by the intelligent terminal in the virtual mirror window.

Based on the same technical conception, the embodiment of the application provides virtual display equipment which can be an AR product or a VR product, and can realize the steps of the screen media resource display method based on target identification and achieve the same technical effect.

Referring to fig. 15, the virtual display device includes a multi-camera 1501, a processor 1502, a memory 1503, and a communication interface 1504, the multi-camera 1501, the communication interface 1504, the memory 1503, and the processor 1502 being connected by a bus 1505;

The memory 1503 stores a computer program, and the processor 1502 performs the following operations according to the computer program:

receiving a screen throwing request sent by an intelligent terminal and media resources sent by the intelligent terminal through the communication interface 1504;

Optionally, the virtual mirror window is displayed while the media resource of the intelligent terminal screen is rendered and displayed in the virtual mirror window, and the virtual mirror window and the virtual configuration window are independent and do not overlap.

Optionally, the location information includes a distance between the screen and the virtual display device;

When the distance is greater than a preset distance threshold, the processor 1502 determines, according to the position information and the size information of the screen, the position and the size of the virtual mirror window so that the virtual mirror window completely covers the screen of the intelligent terminal, and specifically includes:

Setting the position of the screen as the position of the virtual mirror window, increasing the size information of the screen according to a preset proportion, and taking the increased size as the size of the virtual mirror window.

Optionally, when the number of the intelligent terminals is multiple, each screen of the intelligent terminals corresponds to one virtual mirror window;

The processor 1502 determines, according to the position information and the size information of the screen, the position and the size of a virtual mirror window so that the virtual mirror window completely covers the screen of the intelligent terminal, and renders and displays a media resource of the screen projection of the intelligent terminal in the virtual mirror window, which specifically includes:

Determining the position and the size of a first virtual mirror window according to the position information and the size information of the screen of the first intelligent terminal so that the first virtual mirror window completely covers the screen of the first intelligent terminal, and rendering and displaying media resources of the screen of the first intelligent terminal in the first virtual mirror window; and

Determining the position and the size of a second virtual mirror window according to the position information and the size information of the screen of the second intelligent terminal so that the second virtual mirror window completely covers the screen of the second intelligent terminal, and rendering and displaying media resources of the screen of the second intelligent terminal in the second virtual mirror window;

The first intelligent terminal is used for displaying the first screen, and the second intelligent terminal is used for displaying the second screen.

Optionally, when the number of the intelligent terminals is multiple, the screens of the multiple intelligent terminals correspond to one virtual mirror window;

The processor 1502 determines, according to the position information and the size information of the screen, the position and the size of the virtual mirror window so that the virtual mirror window completely covers the screen of the intelligent terminal, and renders and displays a media resource of the screen projection of the intelligent terminal in the virtual mirror window, which specifically includes:

Determining the position information of the virtual mirror window according to the position information of the screen of the first intelligent terminal and the position information of the screen of the second intelligent terminal, and determining the size information of the virtual mirror window according to the size information of the screen of the first intelligent terminal and the size information of the screen of the second intelligent terminal so that the virtual mirror window completely covers the screen of the first intelligent terminal and the screen of the second intelligent terminal;

Rendering and displaying the media resource of the first intelligent terminal screen in a first area of the virtual mirror window, and rendering and displaying the media resource of the second intelligent terminal screen in a second area of the virtual mirror window;

Optionally, the position of the virtual mirror window in the 3D space is fixed after the determination.

Optionally, the processor 1502 identifies a real scene image acquired by the binocular camera, and obtains position information and size information of a screen of the intelligent terminal, which specifically includes:

identifying a screen of the intelligent terminal according to the real scene image acquired by the binocular camera, and respectively obtaining 2D detection frames of the screen;

Estimating the size, angle and key points of the screen, and determining a 3D detection frame surrounding the screen by combining 2D detection frames;

And determining the position information and the size information of the screen of the intelligent terminal according to the 3D detection frame.

It should be noted that fig. 15 is only an example, and shows hardware necessary for the virtual display device to execute the steps of the screen media asset display method based on object recognition provided by the embodiment of the present application. Not shown, the virtual display device also includes conventional devices for AR/VR products such as speakers, radios, power supplies, display screens, left and right eye lenses, and the like.

The Processor referred to in fig. 15 of the embodiments of the present application may be a central processing unit (Central Processing Unit, CPU), a general purpose Processor, a graphics Processor (Graphics Processing Unit, GPU) a digital signal Processor (DIGITAL SIGNAL Processor, DSP), an Application-specific integrated Circuit (ASIC), a field programmable gate array (Field Programmable GATE ARRAY, FPGA) or other programmable logic device, transistor logic device, hardware component, or any combination thereof.

The embodiment of the application also provides a computer readable storage medium for storing instructions which, when executed, can complete the screen-throwing media resource display method based on target identification in the previous embodiment.

The embodiment of the application also provides a computer program product for storing a computer program for executing the screen-throwing media resource display method based on target identification in the previous embodiment.

It will be appreciated by those skilled in the art that embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.

The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to the application. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

It will be apparent to those skilled in the art that various modifications and variations can be made to the present application without departing from the spirit or scope of the application. Thus, it is intended that the present application also include such modifications and alterations insofar as they come within the scope of the appended claims or the equivalents thereof.

Claims

1. The screen-throwing media asset display method based on target identification is characterized by being applied to virtual display equipment and comprising the following steps of:

2. The method of claim 1, wherein a virtual configuration window containing the virtual display device configuration information is displayed while rendering media resources displaying the smart terminal screen in the virtual mirror window, the virtual mirror window and the virtual configuration window being independent of each other and non-overlapping.

3. The method of claim 1, wherein the location information comprises a distance of the screen from the virtual display device;

When the distance is greater than a preset distance threshold, determining the position and the size of a virtual mirror window according to the position information and the size information of the screen so that the virtual mirror window completely covers the screen of the intelligent terminal, including:

4. The method of claim 1, wherein when the number of the intelligent terminals is plural, a screen of each intelligent terminal corresponds to one virtual mirror window;

Determining the position and the size of a virtual mirror window according to the position information and the size information of the screen so that the virtual mirror window completely covers the screen of the intelligent terminal, and rendering and displaying the media resource of the screen projection of the intelligent terminal in the virtual mirror window, wherein the method comprises the following steps:

5. The method of claim 1, wherein when the number of the intelligent terminals is plural, the screens of the plural intelligent terminals correspond to one virtual mirror window;

Determining the position and the size of the virtual mirror window according to the position information and the size information of the screen so that the virtual mirror window completely covers the screen of the intelligent terminal, and rendering and displaying the media resource of the screen projection of the intelligent terminal in the virtual mirror window, wherein the method comprises the following steps:

6. The method of any of claims 1-5, wherein a position of the virtual mirror window in 3D space is fixed after determining.

7. The method of any of claims 1-5, wherein identifying the real scene image captured by the binocular camera, obtaining location information and size information of a screen of the intelligent terminal, comprises:

8. A virtual display device comprising a multi-view camera, a processor, a memory, and a communication interface, wherein the multi-view camera, the communication interface, the memory, and the process are connected by a bus;

9. The virtual display device of claim 8, wherein a virtual configuration window containing configuration information of the virtual display device is displayed while rendering media resources displaying the smart terminal screen in the virtual mirror window, the virtual mirror window and the virtual configuration window being independent of each other and non-overlapping.

10. The virtual display device of claim 8, wherein the location information comprises a distance of the screen from the virtual display device;