CN110505471B - Head-mounted display equipment and screen acquisition method and device thereof - Google Patents
Head-mounted display equipment and screen acquisition method and device thereof Download PDFInfo
- Publication number
- CN110505471B CN110505471B CN201910688257.3A CN201910688257A CN110505471B CN 110505471 B CN110505471 B CN 110505471B CN 201910688257 A CN201910688257 A CN 201910688257A CN 110505471 B CN110505471 B CN 110505471B
- Authority
- CN
- China
- Prior art keywords
- screen
- virtual
- trigger operation
- head
- mounted display
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 52
- 238000005457 optimization Methods 0.000 claims abstract description 21
- 230000004044 response Effects 0.000 claims description 13
- 238000012216 screening Methods 0.000 claims description 3
- 238000005520 cutting process Methods 0.000 claims description 2
- 230000000694 effects Effects 0.000 abstract description 19
- 230000006870 function Effects 0.000 description 26
- 230000008569 process Effects 0.000 description 14
- 238000010586 diagram Methods 0.000 description 12
- 238000012545 processing Methods 0.000 description 12
- 230000003287 optical effect Effects 0.000 description 7
- 238000003860 storage Methods 0.000 description 7
- 238000004590 computer program Methods 0.000 description 5
- 230000003190 augmentative effect Effects 0.000 description 3
- 238000012958 reprocessing Methods 0.000 description 3
- 230000003044 adaptive effect Effects 0.000 description 2
- 230000006978 adaptation Effects 0.000 description 1
- 230000000903 blocking effect Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 210000003128 head Anatomy 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000003754 machining Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012805 post-processing Methods 0.000 description 1
- 230000001960 triggered effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/14—Digital output to display device ; Cooperation and interconnection of the display device with other functional units
- G06F3/1454—Digital output to display device ; Cooperation and interconnection of the display device with other functional units involving copying of the display data of a local workstation or window to a remote workstation or window so that an actual copy of the data is displayed simultaneously on two or more displays, e.g. teledisplay
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/30—Image reproducers
- H04N13/332—Displays for viewing with the aid of special glasses or head-mounted displays [HMD]
- H04N13/344—Displays for viewing with the aid of special glasses or head-mounted displays [HMD] with head-mounted left-right displays
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/41—Structure of client; Structure of client peripherals
- H04N21/4104—Peripherals receiving signals from specially adapted client devices
- H04N21/4122—Peripherals receiving signals from specially adapted client devices additional display device, e.g. video projector
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/433—Content storage operation, e.g. storage operation in response to a pause request, caching operations
- H04N21/4334—Recording operations
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/435—Processing of additional data, e.g. decrypting of additional data, reconstructing software from modules extracted from the transport stream
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/436—Interfacing a local distribution network, e.g. communicating with another STB or one or more peripheral devices inside the home
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs
- H04N21/4402—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving reformatting operations of video signals for household redistribution, storage or real-time display
- H04N21/440263—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving reformatting operations of video signals for household redistribution, storage or real-time display by altering the spatial resolution, e.g. for displaying on a connected PDA
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/47—End-user applications
- H04N21/472—End-user interface for requesting content, additional data or services; End-user interface for interacting with content, e.g. for content reservation or setting reminders, for requesting event notification, for manipulating displayed content
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/47—End-user applications
- H04N21/478—Supplemental services, e.g. displaying phone caller identification, shopping application
- H04N21/4781—Games
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
- H04N21/81—Monomedia components thereof
- H04N21/816—Monomedia components thereof involving special video data, e.g 3D video
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Human Computer Interaction (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Databases & Information Systems (AREA)
- Processing Or Creating Images (AREA)
- Testing, Inspecting, Measuring Of Stereoscopic Televisions And Televisions (AREA)
Abstract
The application discloses a head-mounted display device and a screen acquisition method and device thereof. The method comprises the following steps: acquiring an original image corresponding to a 3D image displayed on a screen of the head-mounted display device in a screen acquisition optimization mode; responding to a specified trigger operation, and generating a corresponding screen acquisition file according to the original image; the appointed trigger operation comprises screen capture trigger operation, screen recording trigger operation and screen projection trigger operation. The embodiment of the application avoids the problems of poor subsequent playing and displaying effects and poor user experience caused by directly collecting the images which are subjected to left-right split screen and have distortion.
Description
Technical Field
The application relates to the technical field of head-mounted display equipment, in particular to head-mounted display equipment and a screen acquisition method and device thereof.
Background
When many existing mobile terminal devices (e.g., mobile phones) perform screen capture operations such as screen projection, screen recording, screen capture and the like, the obtained image content is consistent with that displayed on the mobile phone screen, and is a 2D image, which is a default state of a mobile terminal system such as an Android system itself and is also a scene that users and developers are accustomed to.
However, if the screen capture operation of the mobile phone is applied to the head-mounted display device (such as a virtual reality VR device), since the 3D image is displayed in the screen of the head-mounted display device, the screen capture is directly performed, and then the image in the obtained screen projection file, screen recording file, and screen capture file is a left-right split image with distortion, which affects the subsequent viewing effect.
Disclosure of Invention
The invention provides a head-mounted display device and a screen acquisition method and device thereof, which are used for generating a screen projection file/screen recording file/screen capture file based on an original image, and solving the problems of poor subsequent playing and display effects and poor user experience caused by directly acquiring distorted images which are split left and right.
According to an aspect of the present application, there is provided a screen capture method of a head-mounted display device, including:
acquiring an original image corresponding to a 3D image displayed on a screen of the head-mounted display device in a screen acquisition optimization mode;
responding to a specified trigger operation, and generating a corresponding screen acquisition file according to the original image;
the appointed trigger operation comprises screen capture trigger operation, screen recording trigger operation and screen projection trigger operation.
According to another aspect of the present application, there is provided a screen capture apparatus of a head-mounted display device, including:
the image acquisition module is used for acquiring an original image corresponding to the 3D image displayed on the screen of the head-mounted display device in a screen acquisition optimization mode;
the screen acquisition module is used for responding to specified trigger operation and generating a corresponding screen acquisition file according to the original image; the appointed trigger operation comprises screen capture trigger operation, screen recording trigger operation and screen projection trigger operation.
According to still another aspect of the present application, there is provided a head mounted display device including: a processor, and a memory configured to store computer-executable instructions,
the executable instructions, when executed by the processor, implement the steps of a method according to one aspect of the present application.
By applying the screen acquisition scheme of the head-mounted display equipment, the original image corresponding to the 3D image displayed on the screen of the head-mounted display equipment is acquired in the screen acquisition optimization mode, the corresponding screen acquisition file is generated according to the original image in response to the specified trigger operation, so that on one hand, when the head-mounted display equipment performs the functions of screen capture/screen recording/screen projection and the like related to display content acquisition, the screen capture/screen recording/screen projection file is optimized on the premise of not influencing the display of the screen of the head-mounted display equipment, the subsequent display effect of the screen capture/screen recording/screen projection file is improved, various application scenes such as screen capture/screen recording/screen projection are supported, and the universality is strong. On the other hand, the original image corresponding to the 3D image in the head-mounted display device is used for generating the screen acquisition file, so that the lossless processing of the content is guaranteed, meanwhile, the process of converting the 3D image into the 2D image is avoided, and the performance consumption is lower.
Drawings
FIG. 1 is a schematic flow chart diagram illustrating a screen capture method of a head-mounted display device according to an embodiment of the present application;
FIG. 2 is a flow chart of a screen capture method for a head mounted display device according to another embodiment of the present application;
FIG. 3 is a block diagram of a screen capture device of a head mounted display device according to an embodiment of the present application;
FIG. 4 is a block diagram of a head mounted display device according to one embodiment of the present application.
Detailed Description
In order to make the aforementioned objects, features and advantages of the present invention comprehensible, embodiments accompanied with figures are described in further detail below. It is to be understood that the embodiments described are only a few embodiments of the present invention, and not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
For ease of understanding, screen capture is described herein. The screen acquisition is an operation for multimedia acquisition on an equipment screen, and mainly comprises screen capture, screen recording and screen projection. The screen capture is an operation of capturing the display content on the screen and saving the display content into an image. The screen recording mainly comprises the process of acquiring multimedia data of a screen of the terminal and generating a video file. The screen projection is to share the content of a terminal (such as a head-mounted display device and a mobile phone) to be projected to other terminal devices (usually a television) with a display screen for playing.
Head Mounted Display (HMD), i.e., Head Display, transmits optical signals to a user through various Head mounted Display devices, and can realize different effects such as Virtual Reality (VR), Augmented Reality (AR), Mixed Reality (MR), and the like. The differences among the VR (Virtual Reality) device, the AR (Augmented Reality) device, and the MR device (Mixed Reality) are that the objects seen by the user in the Virtual Reality device are all Virtual; the augmented reality device is used for overlaying virtual things onto real things; the mixed reality device virtualizes real things and then overlays the real things into a virtual world.
Use VR equipment as an example, the eye part shows about mostly adopting in the present VR application, two images of simultaneous output when the screen display promptly: a left view image corresponding to a left eye and a right view image corresponding to a right eye, thereby realizing a 3D effect by binocular disparity. That is, on the VR device, although the user sees a normal image, the user sees the images through the VR device, respectively through the left and right eyes and the optical lens, and if the user directly sees the content on the display screen of the VR device, the user may find two images that are split into left and right screens and have distortion.
When the head-mounted display device performs operations such as screen capture/screen recording/screen projection, the obtained image is a left-right split screen image with distortion as described above, which may affect the subsequent display effect of the image. In this regard, the embodiments of the present application propose: the content seen by the head-mounted display device for the user is separated from the content when the functions of screen capture/screen recording/screen projection and the like are realized, so that the display effect of screen capture/screen recording/screen projection files and the user experience are improved. Here, the symbol "/" indicates or.
FIG. 1 is a schematic flow chart diagram illustrating a screen capture method of a head-mounted display device according to an embodiment of the present application; referring to fig. 1, the screen capture method of the head-mounted display device of the embodiment includes:
step S101, in a screen acquisition optimization mode, acquiring an original image corresponding to a 3D image displayed on a screen of the head-mounted display device.
In practical application, a screen acquisition optimization mode can be configured in the head-mounted display device, and the user interaction through the head-mounted display device receives the switch selection of the screen acquisition optimization mode from the user. For example, a control instruction of a screen acquisition optimization mode input by a user is received, if the control instruction is an opening instruction, the screen acquisition optimization mode is entered, and an original image corresponding to a 3D image displayed on a screen of the head-mounted display device is acquired. That is, when a start instruction of a user is received, the head-mounted display device enters a screen capture optimization mode, and the screen capture file is optimized in the screen capture optimization mode.
Of course, in some embodiments, the user may also select to turn off the screen capture optimization mode, so as to directly perform operations such as screen capture, screen recording, screen projection and the like on the 3D image displayed on the screen of the head-mounted display device, thereby adapting to the screen capture needs in a specific scene.
And step S102, responding to the appointed trigger operation, and generating a corresponding screen acquisition file according to the original image. The appointed trigger operation comprises screen capture trigger operation, screen recording trigger operation and screen projection trigger operation.
And after judging that the screen acquisition optimization mode is entered, the system detects the appointed trigger operation of the user, and when the appointed trigger operation is detected, responds to the occurrence of the appointed trigger operation and generates a corresponding screen acquisition file according to the original image acquired in the last step. The original image here is an image corresponding to a 3D image displayed on the screen of the head-mounted display device, note: the 3D image is obtained by processing an original image according to the requirement of the head-mounted display device on the 3D display effect, the 3D image comprises two images of left and right split screens, the 3D image has certain distortion, and the original image is an original 2D image generated by an application (such as a game application and a video application) in the head-mounted display device.
The appointed triggering operation comprises one or more of screen capturing, screen recording and screen projecting, namely, the method and the device support various screen acquisition application scenes and are high in universality. The difference between screen capture and screen capture is that screen capture captures a screen image at a time point, and screen capture records and stores each video frame displayed on a screen in a period of time (for example, 5 seconds, and the screen capture has a start time and an end time) into a video stream. And the screen projection is to transmit each video frame displayed on the screen to another terminal in real time for playing. Different from screen capture, screen recording and screen projection in the prior art, the screen capture file is generated according to the original image corresponding to the screen image instead of directly capturing the screen image of the head-mounted display device.
As shown in fig. 1, the screen acquisition method for the head-mounted display device according to the embodiment of the application solves the problems of poor subsequent display effect and poor user experience of screen capture/recording/screen projection files of the head-mounted display device, supports various application scenarios such as screen capture/recording/screen projection, and has strong universality. And the original image corresponding to the 3D image in the head-mounted display equipment is used for generating the screen acquisition file, so that theoretically, the lossless processing of the content can be realized, the display effect is better, meanwhile, the process of converting the 3D image into the 2D image is avoided, and the performance consumption is lower.
In view of different requirements of screen capture files in different application scenes, for example, the resolution of a required image is high when a screen is captured, and the resolution can be lower when the screen is recorded, and a corresponding screen capture file is directly generated according to an original image, so that the problem that the subsequent display effect is poor due to the fact that a 3D image is captured can be solved, but image editing and image processing cannot be performed.
The implementation steps of the method of this embodiment are described below with reference to fig. 2, where fig. 2 is a flowchart of a screen capture method of a head-mounted display device according to another embodiment of the present application, see fig. 2,
an original image of an HMD application is acquired.
Here, the HMD is a head-mounted display device, and it is understood that applications are running in the head-mounted display device, and the applications are 3D applications, such as 3D games.
And (5) SDK treatment.
The application of the head-mounted display device needs to be developed by using a specific SDK (Software Development Kit), and the application generates an optical image which does not meet the requirements of the HMD device, and then the SDK processes the original image to generate an optical image which needs to be displayed by the HMD, namely, the 3D image which is adaptive to binocular and adaptive to the optical structure of the HMD device in fig. 2.
And judging whether machining is needed, if so, reprocessing is carried out according to the function, and if not, the reprocessing is output to a physical display screen.
For a 3D image, the image may be generally directly displayed on a physical screen of a head-mounted display device, a game picture is rendered by considering some special scenes, for example, an application, and a normal process is to display a game picture in the head-mounted display device, but if a screen is currently recorded, it may be necessary to superimpose contents such as an interface element (e.g., a common red dot) indicating that the screen is currently recorded and a recording time on the game picture, which is a case of reprocessing the 3D image according to a function.
Referring to fig. 2, it is determined whether there is a screen capture/recording/projection trigger operation for the original image, and no processing is performed for performance reasons in the case where the screen capture/recording/projection function is not activated.
That is, if the specified trigger operation is not detected, the original image is not processed.
The detection of the trigger operation is a function of the head-mounted display device, and may be implemented by using the prior art, for example, a user triggers the start of the screen capture function through a shortcut key of the head-mounted display device, or triggers the start of the screen capture function through a voice command, and the like, which is not described herein too much.
With reference to fig. 2, when a specific trigger operation is detected, that is, when a screen capture/screen recording/screen projection trigger operation exists, the embodiment of the present application processes the original image, so as to prevent the screen capture file/screen recording file/screen projection file from including left and right split screens and having a distorted image.
In one embodiment, in response to a specified trigger operation, a virtual expansion screen is created, each of the acquired original images is output to the virtual expansion screen, and a corresponding screen capture file is generated based on the content of the virtual expansion screen.
Here, the creating of the virtual extension screen in response to the specified trigger operation includes: creating at least one virtual extended screen in response to one or more of the screen capture trigger operation, the screen recording trigger operation, and the screen projection trigger operation.
Referring to fig. 2, a virtual extension screen is created according to the functional requirements (i.e., the functional requirements of screen capture/screen recording/screen projection), and the original image is processed as required and output to the virtual extension screen. For example, when a user wants to obtain a clear image during screen capture, the parameters of the screen and the image can be set to be more precise, and when the user wants to have smooth content during screen recording and screen projection, the parameters (resolution) of the screen recording and screen projection can be set to be smaller, so that the smoothness is better.
Next, based on the content of the virtual extended screen, generating a corresponding screen capture file, including: and coding the content of the virtual extended screen to generate an image file corresponding to screen capture triggering operation, or generate a video file corresponding to screen recording triggering operation/screen projection triggering operation. Namely, the virtual expansion screen content is encoded into a video stream corresponding to the screen recording function, and is output and stored as a video file. And coding and saving the content of the virtual expansion screen into an image file corresponding to the screen capturing function. And coding the content of the virtual expansion screen into a video corresponding to the screen projection function, and storing the video into a video file.
It should be noted that, in this embodiment, the content of the virtual extended screen may be correspondingly processed according to specific needs, so that the user is not influenced to view the 3D content through the physical display screen, and a file with better screen capture/recording/screen projection display effects may be obtained.
The foregoing description has been given by taking an example in which the screen capture/recording/projection functions are simultaneously compatible, and only one virtual extended screen is created. In this case, the size, format, and the like of the image obtained by screen capture, the video frame obtained by screen recording, and the video frame obtained by screen projection are the same. The three functions of screen capture/screen recording/screen projection share a plurality of processing procedures, thereby reducing the performance consumption of the system.
Of course, if the requirements for the image for screen capture/screen recording/screen projection are different greatly, a plurality of virtual extended screens can be created for respective processing. That is, in the embodiment of the present application, creating the virtual extension screen in response to the specified trigger operation includes: responding to the screen capture triggering operation, and creating a virtual expansion screen matched with the screen capture parameters according to the screen capture parameters from the screen capture application; or responding to the screen recording triggering operation, and creating a virtual expansion screen matched with the recording parameters according to the recording parameters from the screen recording application; or responding to the screen projection triggering operation, and creating a virtual expansion screen matched with the screen projection parameters according to the screen projection parameters from the screen projection application; the screen projection parameters are determined based on at least one of the current network state, the data volume of the content to be projected, the resolution of the projected screen end and the screen proportion of the projected screen end.
In one embodiment, if the screen capture trigger operation is currently detected, a first virtual extended screen adapted to the screen capture parameters (such as resolution) is created according to the screen capture trigger operation (the virtual extended screen of the present application has a resolution attribute, and a user can customize an attribute value of the resolution to adapt to different requirements), the acquired original image is output to the first virtual extended screen, and then the screen capture image is generated based on the first virtual extended screen. If the screen capture triggering operation and the screen recording triggering operation are detected currently, a first virtual expansion screen matched with screen capture parameters (such as resolution) is created according to the screen capture triggering operation, the acquired original image is output to a second virtual expansion screen, the screen capture image is generated based on the first virtual expansion screen subsequently, a second virtual expansion screen matched with the screen recording parameters (such as resolution) is created according to the screen recording triggering operation, the acquired original image is output to the second virtual expansion screen, and screen recording video data is generated based on the second virtual expansion screen subsequently.
Like this, handle screen capture and record screen respectively, to the screen capture function, can export more clear image, to the record screen function, can export the definition low but the data bulk occupies less record screen video of storage space to satisfy the respective demand of different functions. Therefore, the extensibility of the embodiment is good.
The head-mounted display device is provided with a screen capture application, a screen recording application and a screen projection application, wherein the screen capture application comprises preset screen capture parameters, such as the definition and the size of a screen capture image, the same screen recording application also comprises preset screen recording parameters, and the screen projection application also comprises screen projection parameters. The difference of the screen projection applied to the screen recording application and the screen capturing application is that the screen projection is an operation process which has high recording and playing real-time performance and has compatibility requirements on receiving end equipment, so that the screen projection parameters are determined based on at least one of information of the current network state, the data volume of the content to be projected, the resolution of the screen projected end and the screen ratio of the screen projected end. In the embodiment, the screen projection effect is optimized by improving parameters, for example, the best parameter is dynamically selected according to the network state, the content data volume, the resolution of the screen projected end, the screen ratio of the screen projected end and the like to meet the screen projection requirement. For example, due to different screen scales of the televisions, the 3D picture of the head-mounted display device cannot be stretched, otherwise, the scales are easy to be wrong, and the user experience is poor. In the embodiment, the ratio adaptation of the screen projection content and the television screen is ensured by modifying the parameters of the virtual extended screen and the parameters of the output video stream, the screen projection content is directly output according to the parameters required by the television, and the screen projection effect is improved. For another example, when a mobile phone or a head-mounted display device is used for projecting a screen, the situation of unsmooth and unsmooth blocking often occurs, the scheme can optimize the problem, improve the definition of the video if the network communication state is good, and reduce the definition of the video if the network state is poor, which can be realized by modifying the parameters of the virtual extended screen and the parameters of the output video stream.
In this embodiment, outputting each of the acquired original images to the virtual expansion screen includes: cutting the original image and outputting the cut original image to the virtual expansion screen; or screening the acquired multiple frames of original images, and outputting each screened original image to the virtual expansion screen.
That is to say, after the original image is acquired, the original image is cut according to the parameter requirement in the screen capture application, and the cut original image is output to the virtual expansion screen, so that the image processing requirement is met. In addition, when screen recording and other operations are carried out, the acquired images can be screened, and only the screened original images are output to the virtual extended screen, so that the obtained screen recording file is prevented from containing video contents irrelevant to target application, and the screen recording effect is improved.
In this embodiment, after outputting each of the acquired original images to the virtual expansion screen, the method further includes: and embedding watermark images into the original images to obtain the content of the virtual extended screen. The watermarking is to perform uniqueness marking on the screen capture file, the screen recording file or the screen projection file generated by the head-mounted display device, so that the source of the screen acquisition files such as the screen capture file can be identified, and disputes can be avoided.
The following describes a screen acquisition process of the head-mounted display device by taking the implementation of the screen recording function as an example:
the 3D application is run in the head-mounted display device, and the image of the 3D application is displayed on the screen of the head-mounted display device.
The user triggers the screen recording function, and the system responds to the screen recording function triggering event and creates a virtual extended screen.
The image of the 3D application is displayed on the screen of the head-mounted display device after being overlaid with the red point indicating the screen recording state and the screen recording time (namely, the content seen by the user is modified).
The original image (non-adapted dual-purpose image) is output to the virtual expansion screen.
It is emphasized that, here, the original image is output to the virtual extended screen, so there are no interface elements indicating the screen recording state red dot and the screen recording time that the user sees on the head-mounted display device, and since such information has no meaning in the screen recording file, the user experience of the embodiment of the present application is better.
And encoding the content of the virtual extended screen into a video file.
The video parameters can be customized while coding, and the virtual expansion screen can also be used for screenshot, so that the flexibility is better, and the performance consumption is low. And the post-processing process during screen recording can be reduced. For example, in the screen recording process, it is not desirable to record the picture at the beginning or the end of the video, then the current picture can be directly determined in the screen recording function, and if the current picture is not the picture of the target 3D application (such as a game application), the images are controlled not to be output to the virtual extended screen, so that the problem that the screen recording file is modified after recording, and the operation is complex is avoided.
To this end, an image with split screen distortion and screen recording information for 3D applications is displayed on a physical screen of the head-mounted display device of this embodiment; the files obtained by screen recording are single-eye distortion-free images which can be directly watched by the user (without the help of head-mounted display equipment), screen recording information is not available, the screen recording information is more reasonable, and experience is better.
Therefore, by the screen acquisition method of the head-mounted display device, the obtained files such as screen recording/screen capturing/screen projecting can be directly watched by the user, the display effect is better, and the problem of poor user experience caused by screen recording of the 3D image in the prior art is solved. Moreover, when the screen capture/screen recording/screen projection functions are simultaneously triggered, the screen capture/screen recording/screen projection functions can be simultaneously compatible, namely, the screen capture/screen recording/screen projection functions and other application scenes are compatible, and the universality is good; the three screen acquisition functions can be completed by only creating one virtual expansion screen, the performance consumption of the system is low, a plurality of virtual screens can be created for respective processing, and the expansibility is good. The parameters of the virtual expansion screen and the parameters of the original image support self-definition, can be balanced in performance and effect, and is high in flexibility.
Compared with the scheme that the 3D image is obtained and then processed into the 2D image, the method and the device have higher efficiency; and moreover, when 3D is converted into 2D, image loss and performance consumption are ensured, the original image is directly used, the content can be processed in a lossless mode, the effect is better, meanwhile, the process of converting the image is omitted, and the performance consumption is lower.
It should be noted that the terms "first," "second," and the like in the description and claims of this application and in the drawings described above are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the objects so used are interchangeable under appropriate circumstances such that the embodiments of the invention described herein are capable of operation in sequences other than those illustrated or described herein.
Fig. 3 is a block diagram of a screen capture apparatus of a head-mounted display device according to an embodiment of the present application, and referring to fig. 3, a screen capture apparatus 300 of a head-mounted display device according to the embodiment includes:
an image obtaining module 301, configured to obtain, in a screen acquisition optimization mode, an original image corresponding to a 3D image displayed on a screen of the head-mounted display device;
the screen acquisition module 302 is used for responding to a specified trigger operation and generating a corresponding screen acquisition file according to the original image; the appointed trigger operation comprises screen capture trigger operation, screen recording trigger operation and screen projection trigger operation.
In an embodiment of the present application, the screen capture module 302 is specifically configured to, in response to a specified trigger operation, create a virtual extended screen, output each of the acquired original images onto the virtual extended screen, and generate a corresponding screen capture file based on the content of the virtual extended screen.
In an embodiment of the present application, the screen capture module 302 is specifically configured to create at least one virtual extension screen in response to one or more of the screen capture trigger operation, the screen recording trigger operation, and the screen projection trigger operation.
In an embodiment of the present application, the screen capture module 302 is specifically configured to, in response to the screen capture trigger operation, create a virtual extension screen adapted to a screen capture parameter from a screen capture application according to the screen capture parameter; or responding to the screen recording triggering operation, and creating a virtual expansion screen matched with the recording parameters according to the recording parameters from the screen recording application; or responding to the screen projection triggering operation, and creating a virtual expansion screen matched with the screen projection parameters according to the screen projection parameters from the screen projection application; the screen projection parameters are determined based on at least one of the current network state, the data volume of the content to be projected, the resolution of the projected screen end and the screen proportion of the projected screen end.
In an embodiment of the present application, the screen capture module 302 is specifically configured to perform clipping processing on the original image and output the clipped image to the virtual expansion screen; or screening the acquired multiple frames of original images, and outputting each screened original image to the virtual expansion screen.
In an embodiment of the present application, the screen capture module 302 is specifically configured to, after the obtained original images are output to the virtual expansion screen, embed watermark images in the original images to obtain content of the virtual expansion screen; and coding the content of the virtual extended screen to generate an image file corresponding to screen capture triggering operation, or coding the content of the virtual extended screen to generate a video file corresponding to screen recording triggering operation.
In an embodiment of the present application, the screen capture module 302 is specifically configured to receive a control instruction of a screen capture optimization mode input by a user, enter the screen capture optimization mode if the control instruction is a start instruction, and obtain an original image corresponding to a 3D image displayed on a screen of the head-mounted display device.
The exemplary explanation about the functions performed by the modules in the screen capture device of the head-mounted display device shown in fig. 3 is consistent with the exemplary explanation in the foregoing method embodiment, and is not repeated here.
FIG. 4 is a block diagram of a head mounted display device according to one embodiment of the present application. As shown in fig. 4, the head mounted display device 400 comprises a processor 401, and a memory 402 configured to store computer executable instructions, which when executed by the processor 401, implement the steps of the screen capture method of the head mounted display device as previously described.
Furthermore, the logic instructions in the memory 402 may be implemented in software functional units and stored in a computer readable storage medium when sold or used as a stand-alone product. Based on such understanding, the technical solution of the present invention or a part thereof, which essentially contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method described in the embodiments of the present application. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.
As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
It is to be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.
In the description of the present invention, numerous specific details are set forth. It is understood, however, that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures and techniques have not been shown in detail in order not to obscure an understanding of this description. Similarly, it should be appreciated that in the foregoing description of exemplary embodiments of the invention, various features of the invention are sometimes grouped together in a single embodiment, figure, or description thereof for the purpose of streamlining the disclosure and aiding in the understanding of one or more of the various inventive aspects. However, the disclosed method should not be interpreted as reflecting an intention that: that the invention as claimed requires more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive aspects lie in less than all features of a single foregoing disclosed embodiment. Thus, the claims following the detailed description are hereby expressly incorporated into this detailed description, with each claim standing on its own as a separate embodiment of this invention.
While the foregoing is directed to embodiments of the present invention, other modifications and variations of the present invention may be devised by those skilled in the art in light of the above teachings. It should be understood by those skilled in the art that the foregoing detailed description is for the purpose of illustrating the invention rather than the foregoing detailed description, and that the scope of the invention is defined by the claims.
Claims (7)
1. A screen capture method for a head-mounted display device, comprising:
acquiring an original image corresponding to a 3D image displayed on a screen of the head-mounted display device in a screen acquisition optimization mode;
responding to a specified trigger operation, and generating a corresponding screen acquisition file according to the original image;
the appointed trigger operation comprises screen capture trigger operation, screen recording trigger operation and screen projection trigger operation;
wherein the generating of the corresponding screen capture file from the original image in response to the specified trigger operation comprises:
in response to a specified trigger operation, creating a virtual extension screen,
outputting each acquired original image to the virtual expansion screen, and generating a corresponding screen acquisition file based on the content of the virtual expansion screen;
wherein the original image is an original 2D image generated by an application in the head-mounted display device corresponding to the 3D image displayed on the screen of the head-mounted display device;
wherein after the outputting of each of the acquired original images onto the virtual expansion screen, the method further comprises:
embedding watermark images into the original images to obtain the content of the virtual extended screen;
the generating of the corresponding screen capture file based on the content of the virtual extended screen includes:
and coding the content of the virtual extended screen to generate an image file corresponding to screen capture triggering operation, or coding the content of the virtual extended screen to generate a video file corresponding to screen recording triggering operation, or coding the content of the virtual extended screen to generate a video file corresponding to screen projection triggering operation.
2. The method of claim 1, wherein the creating a virtual expansion screen in response to a specified trigger operation comprises:
creating at least one virtual extended screen in response to one or more of the screen capture trigger operation, the screen recording trigger operation, and the screen projection trigger operation.
3. The method of claim 1, wherein the creating a virtual expansion screen in response to a specified trigger operation comprises:
responding to the screen capture triggering operation, and creating a virtual expansion screen matched with the screen capture parameters according to the screen capture parameters from the screen capture application; or the like, or, alternatively,
responding to the screen recording triggering operation, and creating a virtual expansion screen matched with the recording parameters according to the recording parameters from the screen recording application; or the like, or, alternatively,
responding to the screen projection triggering operation, and creating a virtual expansion screen matched with the screen projection parameters according to the screen projection parameters from the screen projection application; the screen projection parameters are determined based on at least one of the current network state, the data volume of the content to be projected, the resolution of the projected screen end and the screen proportion of the projected screen end.
4. The method of claim 1, wherein said outputting each of said captured original images onto said virtual expanded screen comprises:
cutting the original image and outputting the cut original image to the virtual expansion screen;
or screening the acquired multiple frames of original images, and outputting each screened original image to the virtual expansion screen.
5. The method of any one of claims 1-4, wherein in the screen acquisition optimization mode, acquiring an original image corresponding to a 3D image displayed on the head mounted display device screen comprises:
receiving a control instruction of a screen acquisition optimization mode input by a user,
and if the control instruction is a starting instruction, entering a screen acquisition optimization mode, and acquiring an original image corresponding to the 3D image displayed on the screen of the head-mounted display device.
6. A screen capture device for a head-mounted display device, comprising:
the image acquisition module is used for acquiring an original image corresponding to the 3D image displayed on the screen of the head-mounted display device in a screen acquisition optimization mode;
the screen acquisition module is used for responding to specified trigger operation and generating a corresponding screen acquisition file according to the original image; the appointed trigger operation comprises screen capture trigger operation, screen recording trigger operation and screen projection trigger operation;
the screen acquisition module is specifically used for responding to specified trigger operation, creating a virtual extended screen, outputting each acquired original image to the virtual extended screen, and generating a corresponding screen acquisition file based on the content of the virtual extended screen;
wherein the original image is an original 2D image generated by an application in the head-mounted display device corresponding to the 3D image displayed on the screen of the head-mounted display device;
the screen acquisition module is further configured to, after the obtained original images are output to the virtual expansion screen, embed watermark images in the original images to obtain content of the virtual expansion screen; and coding the content of the virtual extended screen to generate an image file corresponding to screen capture triggering operation, or coding the content of the virtual extended screen to generate a video file corresponding to screen recording triggering operation, or coding the content of the virtual extended screen to generate a video file corresponding to screen projection triggering operation.
7. A head-mounted display device, comprising: a processor, and a memory configured to store computer-executable instructions,
the executable instructions, when executed by the processor, implement the steps of the method of any one of claims 1-5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910688257.3A CN110505471B (en) | 2019-07-29 | 2019-07-29 | Head-mounted display equipment and screen acquisition method and device thereof |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910688257.3A CN110505471B (en) | 2019-07-29 | 2019-07-29 | Head-mounted display equipment and screen acquisition method and device thereof |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110505471A CN110505471A (en) | 2019-11-26 |
| CN110505471B true CN110505471B (en) | 2021-09-14 |
Family
ID=68587565
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910688257.3A Active CN110505471B (en) | 2019-07-29 | 2019-07-29 | Head-mounted display equipment and screen acquisition method and device thereof |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110505471B (en) |
Families Citing this family (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2022151864A1 (en) * | 2021-01-18 | 2022-07-21 | 海信视像科技股份有限公司 | Virtual reality device |
| CN112732089A (en) * | 2021-01-18 | 2021-04-30 | 海信视像科技股份有限公司 | Virtual reality equipment and quick interaction method |
| CN112732088B (en) * | 2021-01-18 | 2023-01-20 | 海信视像科技股份有限公司 | Virtual reality equipment and monocular screen capturing method |
| CN114286142B (en) * | 2021-01-18 | 2023-03-28 | 海信视像科技股份有限公司 | Virtual reality equipment and VR scene screen capturing method |
| CN112965651A (en) * | 2021-03-16 | 2021-06-15 | 深圳市前海手绘科技文化有限公司 | Optimization method for demonstrating video source output |
| CN115248655A (en) * | 2021-04-28 | 2022-10-28 | 闪耀现实(无锡)科技有限公司 | Method and apparatus for displaying information |
| CN114630101B (en) * | 2021-09-30 | 2024-05-31 | 海信视像科技股份有限公司 | Display device, VR device and display control method of virtual reality application content |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105242827A (en) * | 2015-09-28 | 2016-01-13 | 小米科技有限责任公司 | Screen capture method, apparatus and system |
| CN105677142A (en) * | 2015-12-29 | 2016-06-15 | 上海孩子国科教设备有限公司 | Method and client terminal for acquiring screenshot |
| CN105760090A (en) * | 2016-02-25 | 2016-07-13 | 惠州Tcl移动通信有限公司 | Screenshot obtaining method and system based on mobile terminal and mobile terminal |
| CN106201275A (en) * | 2016-07-29 | 2016-12-07 | 北京小米移动软件有限公司 | Snapshot is as processing method and processing device |
Family Cites Families (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2012133586A (en) * | 2010-12-21 | 2012-07-12 | Ntt Docomo Inc | Display device, screen image transfer method and program |
| CN103338405A (en) * | 2013-06-03 | 2013-10-02 | 四川长虹电器股份有限公司 | Screen capture application method, equipment and system |
| CN104407829B (en) * | 2014-11-06 | 2018-01-23 | 北京凌阳益辉科技有限公司 | A kind of image mirrors display methods and its device |
| CN105678684B (en) * | 2014-11-18 | 2020-11-03 | 中兴通讯股份有限公司 | Method and device for intercepting image |
| CN204440491U (en) * | 2015-03-26 | 2015-07-01 | 天机数码创新技术有限公司 | A kind of augmented reality system with a key screenshotss sharing function |
| CN105404455B (en) * | 2015-12-16 | 2019-02-22 | 北京奇虎科技有限公司 | The method and device of screenshotss is realized to mobile terminal using terminal |
| CN105959666A (en) * | 2016-06-30 | 2016-09-21 | 乐视控股（北京）有限公司 | Method and device for sharing 3d image in virtual reality system |
| CN106911850B (en) * | 2017-02-27 | 2020-05-19 | 陈惠金 | Mobile terminal and screen capturing method thereof |
| CN108108117B (en) * | 2018-01-23 | 2021-01-19 | 中兴通讯股份有限公司 | Screen capturing method and device and terminal |
| CN109255990A (en) * | 2018-09-30 | 2019-01-22 | 杭州乔智科技有限公司 | A kind of tutoring system based on AR augmented reality |
| CN109275019A (en) * | 2018-10-30 | 2019-01-25 | 青岛海信电器股份有限公司 | When playing 3D video to the processing method and processing device of current display screen shot |
| CN109523462A (en) * | 2018-11-14 | 2019-03-26 | 北京奇艺世纪科技有限公司 | A kind of acquisition methods and device of VR video screenshotss image |
| CN109934764A (en) * | 2019-01-31 | 2019-06-25 | 北京奇艺世纪科技有限公司 | Processing method, device, terminal, server and the storage medium of panoramic video file |
-
2019
- 2019-07-29 CN CN201910688257.3A patent/CN110505471B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105242827A (en) * | 2015-09-28 | 2016-01-13 | 小米科技有限责任公司 | Screen capture method, apparatus and system |
| CN105677142A (en) * | 2015-12-29 | 2016-06-15 | 上海孩子国科教设备有限公司 | Method and client terminal for acquiring screenshot |
| CN105760090A (en) * | 2016-02-25 | 2016-07-13 | 惠州Tcl移动通信有限公司 | Screenshot obtaining method and system based on mobile terminal and mobile terminal |
| CN106201275A (en) * | 2016-07-29 | 2016-12-07 | 北京小米移动软件有限公司 | Snapshot is as processing method and processing device |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110505471A (en) | 2019-11-26 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110505471B (en) | Head-mounted display equipment and screen acquisition method and device thereof | |
| CN112585978B (en) | Generating a composite video stream for display in VR | |
| US9055258B2 (en) | Video display apparatus and video display method | |
| JP6283108B2 (en) | Image processing method and apparatus | |
| CA2559131C (en) | Stereoscopic parameter embedding apparatus and stereoscopic image reproducer | |
| EP2235685B1 (en) | Image processor for overlaying a graphics object | |
| US8654131B2 (en) | Video image processing apparatus and video image processing method | |
| KR101528312B1 (en) | Method for editing video and apparatus therefor | |
| JP5383798B2 (en) | System and method for marking three-dimensional film | |
| CN113064684B (en) | Virtual reality equipment and VR scene screen capturing method | |
| CN113206971B (en) | Image processing method and display device | |
| WO2014155670A1 (en) | Stereoscopic video processing device, stereoscopic video processing method, and stereoscopic video processing program | |
| US10616502B2 (en) | Camera preview | |
| CN113747240A (en) | Video processing method, apparatus, storage medium, and program product | |
| JP2014209707A (en) | Device and method for video reproduction | |
| JP6934052B2 (en) | Display control device, display control method and program | |
| CN110708591B (en) | Image processing method and device and electronic equipment | |
| CN106412718A (en) | Rendering method and device for subtitles in 3D space | |
| CN115002335B (en) | Video processing method, apparatus, electronic device, and computer-readable storage medium | |
| CN115661418A (en) | Mixed reality display device, method, equipment and storage medium | |
| CN114915798A (en) | Real-time video generation method, multi-camera live broadcast method and device | |
| US20140056524A1 (en) | Image processing device, image processing method, and program | |
| CN115665461B (en) | Video recording method and virtual reality device | |
| CN109104626B (en) | Method and equipment for filtering images in live webcast | |
| EP4335108A1 (en) | Systems and methods for processing volumetric images |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |