CN111640191A - Projection screen picture acquisition processing method based on VR (virtual reality) all-in-one machine - Google Patents

Abstract

The invention discloses a method for capturing and processing a screen projection screen based on a VR all-in-one machine. The updated screen of a 3D application is put into a screen projection module, and while the screen is updated, different threads in the same process can also share textures. resource. Compared with the prior art, the technical solution of the present invention can flexibly handle multiple screen projection requests, and can perfectly adapt to the size of the target screen projection terminal. At the same time, the captured and processed images are clear and complete, without large black borders, and have little impact on the system rendering load. Whether it is used by users or made publicity materials, it is a very good solution.

Description

A method of capturing and processing screen recording images based on VR all-in-one machine

technical field

The present invention relates to the technical field of VR, and in particular, to a method for capturing and processing a recording screen based on a VR all-in-one machine.

Background technique

VR (Virtual Reality, virtual reality technology) is a computer simulation system that can create and experience virtual worlds. It uses computers to generate a simulation environment, which is a multi-source information fusion, interactive three-dimensional dynamic vision and A system of entity behavior that immerses the user in the simulation environment.

When the VR device performs screen projection processing, the single-eye image in the all-in-one machine can be projected to an external device or recorded as a video file. There are two common screen projection processing methods at present: the first method in the prior art is: one is to directly record the entire screen as a picture source with the help of the virtual screen of the android system. The second method in the prior art: using the Presentation solution of the android system. The Apk renders the screen to be cast to a separate SurfaceView, and then associates the SurfaceView with the virtual screen with Presentation. Finally pass the picture to the virtual screen.

However, obviously, the above two technical solutions in the prior art still have some objective technical problems: the technical defect of the above-mentioned first method is that the entire screen is directly recorded by using the virtual screen of the android system as a picture source. The biggest disadvantage of this method is that the projected images are two circular images, the images seen by the receiving end are not beautiful, and the user experience is poor. The technical defect of the above-mentioned second method is: the disadvantage of the presentation scheme using the android system is that the rendering efficiency is low, the surfaceflinger needs to be passed, and it is not easy to add post-processing links (for example, adding watermarks). Specifically, when the image data transmission is processed by surfaceflinger, there will be an additional operation between copying the image of the apk to the virtual screen, the data transmission delay and system load will increase, and the synchronization between the copy operation and the rendering of the apk will not be perfect. Causes the projected screen to have screen tearing.

Therefore, how to overcome the above technical problems is a problem to be solved by those skilled in the art.

SUMMARY OF THE INVENTION

In view of this, an embodiment of the present invention proposes a method for capturing and processing a screen recording image based on an all-in-one VR machine, so as to solve the above-mentioned technical problem.

An embodiment of the present invention proposes a method for capturing and processing a screen-casting screen based on a VR all-in-one machine. The updated screen of a 3D application is put into the screen-casting module, and while the screen is updated, it is also possible to realize different images under the same process. Threads share texture resources, including the following steps:

Step S100: after the background screen projection service unit is started, a background main rendering thread is created, and a texture of OpenGLES is created simultaneously; the texture of the OpenGLES is used as a texture area in the middle of the data for providing shared texture resources;

Step S200: The screencasting module that has a screencasting requirement sends a screencasting request to the background screencasting service unit; the background screencasting service unit identifies the received screencasting request, and identifies the Surface-Projection request included in the screencasting request. The ID number of x and the size information of the screen required by Surface-x; the size information includes the height size and width size of the screen required by Surface-x;

Step S300: after receiving the screen projection request in step S200, the background screen projection service unit creates a rendering thread A based on the Surface-x in the screen projection request and shares the context with the background main rendering thread; the rendering thread A Share the context with the background main rendering thread to share the texture resources recorded by the texture of OpenGLES in the background main rendering thread; the rendering thread A corresponds to the Surface-x one-to-one; in the background screen projection service unit Other rendering threads in the OpenGL ES texture resources are called arbitrarily; at the same time, the background screen projection service unit will create a Surface-APP based on the OpenGLES texture of the background main rendering thread;

Step S400: the background screen projection service unit transfers the Surface-APP to the 3D application through the AIDL interface, and the Surface-APP is used to establish an information link connection between the 3D application and the background screen projection service unit;

Step S500: the 3D application itself includes a 3D main rendering thread, and after the 3D application receives the Surface-APP, the 3D application creates a rendering thread B that shares a context with the 3D main rendering thread based on the Surface-APP; The rendering thread B is used to render and draw the monocular image in the 3D main rendering thread, and place the rendered monocular image on the Surface-APP image to ensure that the Surface-APP image is updated, so that the The update screen is added to the OpenGLES texture.

Preferably, as an implementable solution, the screen projection module includes an Rtmp screen projection module and a Miracast screen projection module.

Preferably, as an implementable embodiment; in step S500, it also includes the operation steps of cropping the updated picture to fit the size of the screen projection module:

Step S510: After adding the update screen to the texture of OpenGLES, call the size information recorded in the Surface-x of the target screen projection module; crop according to the size information to ensure that the size of the update screen is suitable for the screen projection module. match.

Preferably, as an implementable embodiment; in step S500, the operation steps of adding a logo watermark to the update screen are also included:

Step S520: After adding the update picture to the texture of OpenGLES, obtain the logo watermark to be added and add it to the update picture, and finally put the update picture into the target screen projection module.

Preferably, as an implementable embodiment; in the above step S520: the obtaining of the logo watermark that needs to be added and adding it to the update screen specifically includes the following operation steps:

Step S5210: After adding the update picture to the texture of OpenGLES, the rendering thread A obtains the logo watermark to be added, adds it to the update picture, and organizes the picture with the logo watermark added into the texture resources of the texture of OpenGLES ;

Step S5220: The target screen projection module accesses the texture of OpenGLES, and according to the ID number of its Surface-x, retrieves the update screen corresponding to the same link as the ID number of the Surface-x, and puts it on the target screen projection in the module.

The embodiments of the present invention have the following technical advantages:

The embodiment of the present invention proposes a technical solution for a method for capturing and processing a projected screen image based on an all-in-one VR machine, which at least includes the following technical advantages in the following two aspects;

1. The method for capturing and processing a projected screen based on an all-in-one VR machine provided by the embodiment of the present invention reduces the system rendering load. The specific reason for reducing the system rendering load is mainly to pass the image of the 3D application to the background projection through the AIDL interface. Screen service, not processed by surfaceflinger. 2. The method for capturing and processing a video recording screen based on the VR all-in-one machine provided by the embodiment of the present invention realizes the shared texture area between different rendering threads in the same process, thereby simplifying the processing operation and reducing the rendering storage space. The system rendering load is further reduced; the research found that in the prior art, different threads in the same process are isolated from each other, and texture sharing cannot be achieved; therefore, after the technical solution of the present application is implemented, other subsequent creations in the same process The rendering thread can also freely access to share the texture resources recorded by the OpenGLES texture in the background main rendering thread.

Description of drawings

In order to illustrate the technical solutions of the present invention more clearly, the accompanying drawings required in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention, and therefore should not be It is regarded as the limitation of the protection scope of the present invention. In the various figures, similar components are numbered similarly.

FIG. 1 shows a schematic flowchart of the main flow of a method for capturing and processing a screen shot based on a VR all-in-one machine according to an embodiment of the present invention;

FIG. 2 shows a schematic structural diagram of a control principle of a method for capturing and processing a screen shot based on an all-in-one VR machine according to an embodiment of the present invention.

Labels: 10-screencasting module; 20-background screencasting service unit; 21-background main rendering thread; 22-OpenGLES texture; 30-3D application; 31-3D main rendering thread.

Detailed ways

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, but not all of the embodiments.

The components of the embodiments of the invention generally described and illustrated in the drawings herein may be arranged and designed in a variety of different configurations. Thus, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the invention as claimed, but is merely representative of selected embodiments of the invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work fall within the protection scope of the present invention.

Hereinafter, the terms "comprising", "having" and their cognates, which may be used in various embodiments of the present invention, are only intended to denote particular features, numbers, steps, operations, elements, components, or combinations of the foregoing, and should not be construed as first excluding the presence or addition of one or more other features, numbers, steps, operations, elements, components or combinations of the foregoing or the possibility of a combination of the foregoing.

Furthermore, the terms "first", "second", "third", etc. are only used to differentiate the description and should not be construed as indicating or implying relative importance.

Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which various embodiments of this invention belong. The terms (such as those defined in commonly used dictionaries) will be interpreted as having the same meaning as the contextual meaning in the relevant technical field and will not be interpreted as having an idealized or overly formal meaning, unless explicitly defined in the various embodiments of the present invention.

Example 1

As shown in FIG. 1 , the following describes in detail the method for capturing and processing a screen recording image based on the VR all-in-one machine in Embodiment 1 of the present invention.

The first implementation of the present invention proposes a method for capturing and processing a screen projection screen based on a VR all-in-one machine. The updated screen of a 3D application is put into the screen projection module. While updating the screen, it can also realize the sharing of different threads under the same process. Texture resources, including the following steps:

Step S100: After the background screen projection service unit is started, a background main rendering thread is created, and a texture of OpenGLES is created at the same time; the texture of the OpenGLES is used as a texture area at the middle end of the data to provide shared texture resources; that is, it should be noted that, The background main rendering thread involved in step S100 is different from the 3D main rendering thread in step S500; renderthread refers to the rendering thread RenderThread, which is a rendering thread in the Android system.

Step S200: The screencasting module that has a screencasting requirement sends a screencasting request to the background screencasting service unit; the background screencasting service unit identifies the received screencasting request, and identifies the Surface-x included in the screencasting request. ID number and size information of the screen required by Surface-x; the size information includes the height size and width size of the screen required by Surface-x; for example: the screen projection request includes Surface-1 and Surface-1 The size information of the desired screen. Another screen projection module may send Surface-2 and the size information of the desired screen of Surface-2; (that is, the x in Surface-x is the ID number);

Step S300: after receiving the screen projection request in step S200, the background screen projection service unit creates a rendering thread A based on the Surface-x in the screen projection request and shares the context with the background main rendering thread; the rendering thread A Share the context with the background main rendering thread to share the texture resources recorded by the texture of OpenGLES in the background main rendering thread; the rendering thread A is in one-to-one correspondence with the Surface-x; and the rendering thread A is also used to The additional logo is added to the monocular screen, and the drawing results are sorted into the texture resources of the OpenGLES texture; other rendering threads in the background screen projection service unit arbitrarily call the texture resources recorded by the OpenGLES texture; at the same time, the background screen projection The service unit will create a Surface-APP based on the OpenGLES texture of the background main rendering thread; the above screencasting modules include Rtmp screencasting modules, Miracast screencasting modules or other forms of screencasting modules.

Step S400: the background screen projection service unit transfers the Surface-APP to the 3D application (or 3D application APK) through the AIDL interface, and the Surface-APP is used to establish an information link between the 3D application and the background screen projection service unit road connection;

Step S500: the 3D application itself includes a 3D main rendering thread, and after the 3D application receives the Surface-APP, the 3D application creates a rendering thread B that shares a context with the 3D main rendering thread based on the Surface-APP; The rendering thread B is used to render and draw the monocular picture (or monocular texture, that is, every time the 3D application changes the picture, it needs to be processed by the 3D main rendering thread) in the 3D main rendering thread, and the rendered monocular picture is dropped. On the screen of the Surface-APP, to ensure that the screen of the Surface-APP is updated, so that the updated screen is finally added to the texture of OpenGLES; that is, it should be noted that the rendering thread B renders and draws the 3D main rendering thread. In the monocular picture, the rendered monocular picture is placed on the picture of the Surface-APP to ensure that the picture of the Surface-APP is updated, and the Surface-APP is created by the OpenGLES texture in the background service unit. , so the background service unit can also know the change of each frame of the Surface-APP image data. At this time, the content of the OpenGLES texture in the background service unit is already the latest image.

It should be noted that the background main rendering thread and rendering thread A created in the pre-procedure step S100 are in the process of the background screen projection service unit, but the 3D main rendering thread and rendering thread B created later are in the 3D application. On the one hand, the 3D application creates a rendering thread B based on the Surface-APP, then the single-eye texture is drawn in the rendering thread B, and the final picture result will fall on the Surface-APP. The following is an example: For example, rendering thread B renders at a frame rate of 30FPS, and the Surface-APP screen will be updated at a frame rate of 30FPS. On the other hand, Surface-APP is created by the background service unit implementation. Therefore, the background service unit can also know the changes of each frame of the Surface-APP, that is to say, every time the screen is updated in the rendering thread B of the 3D application, the background service unit can receive a system notification to inform that the Surface-APP has been updated. . At this time, the content of texture 1 in the background service unit is already the latest picture.

In the process of video rendering development using the Open Graphics Library for Embedded Systems (OpenGLES), textures need to be created according to the decoded video frame data, and images are rendered according to the created textures. In the technical solution adopted in the embodiment of the present invention, the texture of OpenGLES is used as the texture area at the middle end of the data to provide shared texture resources;

In the above-mentioned step S300, it should be noted that the above-mentioned rendering thread A and the above-mentioned background main rendering thread realize a shared context, and the ultimate essential purpose is to share the texture resources recorded by the OpenGLES texture in the background main rendering thread, so that other rendering threads (For example: rendering thread B1, rendering thread B2, rendering thread B3, etc.) can also access the texture resources recorded by the texture of OpenGLES; therefore, the purpose of sharing texture resources is when other rendering threads in the same process are drawing , you can arbitrarily call the texture resources of the OpenGLES texture using the background main rendering thread to draw directly. In addition, the shared context exists only between different threads of the same process. It should be noted that the background main rendering thread and rendering thread A created in the pre-procedure step S100 are in the process of the background screen projection service unit; however, the 3D main rendering thread and rendering thread B created later are in the 3D application. It should be noted that each time the above-mentioned background screen projection service unit receives the screen projection request in step S100, it will create a rendering thread A based on the Surface-x in the screen projection request, that is, for each Surface-x, it will create a rendering thread A. Create a rendering thread A; a rendering thread A will be specifically produced for each Surface-x ID number.

As shown in FIG. 2 , the control principle of the application of the method for capturing and processing a screen-casting screen based on the VR all-in-one machine provided by the embodiment of the present invention is shown in FIG. 2 . Rendering thread 21, OpenGLES texture 22 in the background main rendering thread, rendering thread A, Surface-x, Surface-APP, 3D application 30, 3D main rendering thread 31, rendering thread B, etc.

In step S500, it also includes the operation steps of cropping the updated picture to fit the size of the screen projection module:

Step S510: After adding the update screen to the texture of OpenGLES, call the size information recorded in the Surface-x of the target screen projection module; crop according to the size information to ensure that the size of the update screen is suitable for the screen projection module. match.

It should be noted that the existing technology does not crop the monocular image. The monocular image is generally square. However, the display screen of the external screen projection module is all rectangular. It is placed on the display screen of the external screen projection module, so there will be large black borders on both sides, which seriously affects the appearance. However, in the embodiment of the present invention, the cropping and adaptation method is adopted to avoid the appearance of large black borders.

In step S500, it also includes the operation steps of adding a logo watermark to the update screen:

Step S520: After adding the update picture to the texture of OpenGLES, obtain the logo watermark to be added and add it to the update picture, and finally put the update picture into the target screen projection module.

It should be noted that the existing technology cannot add a watermark to the update picture, and the solution of the embodiment of the present invention can add the update picture to the texture of OpenGLES, and then obtain the logo watermark and add it to the update picture.

In a specific technical solution, in the above step S520: the obtaining of the logo watermark that needs to be added and adding it to the update screen specifically includes the following operation steps:

Step S5210: After adding the update picture to the texture of OpenGLES, the rendering thread A obtains the logo watermark to be added, adds it to the update picture, and organizes the picture with the logo watermark added into the texture resources of the texture of OpenGLES ;

Step S5220: The target screen projection module accesses the texture of OpenGLES, and according to the ID number of its Surface-x, retrieves the update screen corresponding to the same link as the ID number of the Surface-x, and puts it on the target screen projection in the module.

To sum up, the method for capturing and processing a screen-casting screen based on the VR all-in-one machine provided by the embodiments of the present invention can flexibly handle multiple screen-casting requests, and can perfectly adapt to the size of the target screen-casting terminal. And the picture is clear and complete, without big black borders. Small impact on system rendering load. Whether it is used by users or made publicity materials, it is a very good solution.

The above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention. should be included within the protection scope of the present invention.

Claims (5)

1. A projection screen picture collecting and processing method based on a VR (virtual reality) all-in-one machine is characterized in that an updated picture of a 3D application is projected into a projection screen module, and texture resources can be shared by different threads in the same process while the picture is updated, and the method specifically comprises the following operation steps:

step S100: after the background screen projection service unit is started, a background main rendering thread is created, and meanwhile, an OpenGLES texture is created; the texture of OpenGLES is used as a texture area of a data middle end for providing shared texture resources;

step S200: the screen projection module with the screen projection requirement sends a screen projection request to a background screen projection service unit; the background screen projection service unit identifies the received screen projection request and identifies the ID number of the Surface-x and the size information of the required picture of the Surface-x contained in the screen projection request; the size information comprises the height size and the width size of a screen required to be projected by Surface-x;

step S300: after receiving the screen-casting request in step S200, the background screen-casting service unit creates a rendering thread a based on Surface-x in the screen-casting request and shares a context with the background main rendering thread; the rendering thread A and the background main rendering thread share a context and are used for sharing texture resources recorded by the OpenGLES textures in the background main rendering thread; the rendering thread A corresponds to the Surface-x one by one; randomly calling texture resources recorded by the textures of OpenGLES by other rendering threads in the background screen projection service unit; meanwhile, the background screen-casting service unit can create a Surface-APP based on the texture of the OpenGLES of the background main rendering thread;

step S400: the background screen-casting service unit transmits Surface-APP to the 3D application through an AIDL interface, and the Surface-APP is used for establishing information link connection between the 3D application and the background screen-casting service unit;

step S500: the 3D application comprises a 3D main rendering thread, and after the 3D application receives the Surface-APP, the 3D application creates a rendering thread B which shares the context with the 3D main rendering thread based on the Surface-APP; and the rendering thread B is used for rendering and drawing a monocular picture in the 3D main rendering thread, and the rendered monocular picture is dropped on the picture of the Surface-APP so as to ensure that the picture of the Surface-APP is updated, and finally the updated picture is added into the texture of the OpenGLES.

2. The VR integrated machine-based screen projection and recording picture collection processing method of claim 1, wherein the screen projection module comprises an Rtmp screen projection module and a Miracast screen projection module.

3. The VR all-in-one machine-based screen projection and recording picture collection processing method of claim 2, further comprising, in step S500, an operation step of clipping the updated picture to fit the size of the screen projection module:

step S510: after the updated picture is added into the texture of OpenGLES, the size information recorded in the Surface-x of the screen projection module of the target is called; and cutting according to the size information to ensure that the updated picture size is matched with the screen projection module.

4. The VR all-in-one machine-based screen projection collection processing method of claim 3, further comprising, in step S500, an operation step of adding a logo watermark to the updated screen:

step S520: and after the updated picture is added into the texture of the OpenGLES, obtaining a logo watermark to be added and adding the logo watermark into the updated picture, and finally putting the updated picture into a target screen projection module.

5. The VR integrated machine based screen shot collection processing method of claim 4, wherein in step S520: the method for acquiring the logo watermark to be added and adding the logo watermark to the updated picture specifically comprises the following operation steps:

step S5210: after the updated picture is added into the texture of the OpenGLES, the rendering thread A acquires a logo watermark needing to be added, adds the logo watermark to the updated picture, and arranges the picture added with the logo watermark into the texture resource of the texture of the OpenGLES;

step S5220: and the screen projection module of the target accesses the texture of OpenGLES, calls an updating picture corresponding to the ID number of the Surface-x on the same link according to the ID number of the Surface-x, and projects the updating picture into the screen projection module of the target.

Images