CN112532962A - Panoramic video subtitle display method and display equipment - Google Patents

Panoramic video subtitle display method and display equipment Download PDF

Info

Publication number
CN112532962A
CN112532962A CN202011330993.0A CN202011330993A CN112532962A CN 112532962 A CN112532962 A CN 112532962A CN 202011330993 A CN202011330993 A CN 202011330993A CN 112532962 A CN112532962 A CN 112532962A
Authority
CN
China
Prior art keywords
perspective
subtitle
displaying
plane
panoramic video
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202011330993.0A
Other languages
Chinese (zh)
Inventor
任子健
刘帅
史东平
吴连朋
王宝云
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Qingdao Hisense Media Network Technology Co Ltd
Juhaokan Technology Co Ltd
Original Assignee
Qingdao Hisense Media Network Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Qingdao Hisense Media Network Technology Co Ltd filed Critical Qingdao Hisense Media Network Technology Co Ltd
Priority to CN202011330993.0A priority Critical patent/CN112532962A/en
Publication of CN112532962A publication Critical patent/CN112532962A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/10Processing, recording or transmission of stereoscopic or multi-view image signals
    • H04N13/106Processing image signals
    • H04N13/172Processing image signals image signals comprising non-image signal components, e.g. headers or format information
    • H04N13/183On-screen display [OSD] information, e.g. subtitles or menus
    • GPHYSICS
    • G02OPTICS
    • G02BOPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/01Head-up displays
    • G02B27/017Head mounted
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/30Image reproducers
    • H04N13/332Displays for viewing with the aid of special glasses or head-mounted displays [HMD]
    • H04N13/344Displays for viewing with the aid of special glasses or head-mounted displays [HMD] with head-mounted left-right displays
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/30Image reproducers
    • H04N13/366Image reproducers using viewer tracking
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/30Image reproducers
    • H04N13/366Image reproducers using viewer tracking
    • H04N13/383Image reproducers using viewer tracking for tracking with gaze detection, i.e. detecting the lines of sight of the viewer's eyes

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Physics & Mathematics (AREA)
  • Human Computer Interaction (AREA)
  • General Physics & Mathematics (AREA)
  • Optics & Photonics (AREA)
  • Controls And Circuits For Display Device (AREA)

Abstract

The application discloses a panoramic video subtitle display method and display equipment. In the present application, in response to a user perspective switching from a first perspective to a second perspective, a first angle of rotation of the first perspective to the second perspective along an X-axis and a second angle of rotation of the first perspective to the second perspective along a Y-axis are determined; rotating a spherical rectangular mesh for displaying a subtitle image along an X-axis according to the first rotation angle, and rotating the spherical rectangular mesh along a Y-axis according to the second rotation angle; and displaying the subtitle image on the spherical rectangular grid. Wherein the spherical rectangular mesh faces the center of a sphere used to display the panoramic video. By adopting the method and the device, the panoramic video subtitle can be adapted to the panoramic video image, so that the watching experience of a user is improved.

Description

Panoramic video subtitle display method and display equipment
Technical Field
The application relates to the technical field of panoramic videos, in particular to a panoramic video subtitle display method and display equipment.
Background
Panoramic video is a new multimedia form developed based on 360-degree panoramic images, and is converted into dynamic panoramic video by continuously playing a series of static panoramic images. The panoramic video is generally formed by splicing video images of all directions collected by a panoramic camera through software, is played by using a special player, projects a planar video into a 360-degree panoramic mode, and presents the planar video to a viewer with a full-surrounding space view of 360 degrees in the horizontal direction and 180 degrees in the vertical direction. The viewer can control the playing of the panoramic video in modes of head motion, eyeball motion, remote controller control and the like, so that the viewer can experience the experience of being personally on the scene. As a new heterogeneous multimedia Service, a panoramic video Service stream contains multiple data types such as audio, video, text, interaction, control command, etc., and has diversified Quality of Service (QoS) requirements.
Because the panoramic video takes a spherical surface as a playing carrier, how to display the panoramic video caption to improve the watching experience of a user is a technical problem to be solved at present.
Disclosure of Invention
The application provides a panoramic video subtitle display method and display equipment, which are used for adapting display of panoramic video subtitles to panoramic video images so as to improve user experience.
In a first aspect, there is provided a display device comprising:
a display connected with the controller and configured to display the panoramic video image;
a memory coupled to the controller and configured to store computer instructions;
the controller configured to perform the following operations in accordance with the computer instructions:
in response to a user perspective switching from a first perspective to a second perspective, determining a first rotational angle along an X-axis from the first perspective to the second perspective and a second rotational angle along a Y-axis from the first perspective to the second perspective;
rotating a spherical rectangular mesh for displaying a subtitle image along an X-axis according to the first rotation angle, and rotating the spherical rectangular mesh along a Y-axis according to the second rotation angle; wherein the spherical rectangular mesh is oriented toward the center of a sphere used to display panoramic video;
and displaying the subtitle image on the spherical rectangular grid.
In a second aspect, there is provided a display device comprising:
a display connected with the controller and configured to display the panoramic video image;
a memory coupled to the controller and configured to store computer instructions;
the controller configured to perform the following operations in accordance with the computer instructions:
in response to a user perspective switching from a first perspective to a second perspective, determining a first rotational angle along an X-axis from the first perspective to the second perspective and a second rotational angle along a Y-axis from the first perspective to the second perspective;
rotating at least one plane for displaying caption texts along an X axis according to the first rotation angle, and rotating the at least one plane along a Y axis according to the second rotation angle; wherein each plane of the at least one plane faces a center of a sphere for displaying the panoramic video, and each plane is for displaying one caption character;
and respectively displaying corresponding subtitle characters on the at least one plane.
In a third aspect, a method for displaying subtitles of a panoramic video is provided, including:
in response to a user perspective switching from a first perspective to a second perspective, determining a first rotational angle along an X-axis from the first perspective to the second perspective and a second rotational angle along a Y-axis from the first perspective to the second perspective;
rotating a spherical rectangular mesh for displaying a subtitle image along an X-axis according to the first rotation angle, and rotating the spherical rectangular mesh along a Y-axis according to the second rotation angle; wherein the spherical rectangular mesh is oriented toward the center of a sphere used to display panoramic video;
and displaying the subtitle image on the spherical rectangular grid.
In a fourth aspect, a method for displaying subtitles of a panoramic video is provided, including:
in response to a user perspective switching from a first perspective to a second perspective, determining a first rotational angle along an X-axis from the first perspective to the second perspective and a second rotational angle along a Y-axis from the first perspective to the second perspective;
rotating at least one plane for displaying caption texts along an X axis according to the first rotation angle, and rotating the at least one plane along a Y axis according to the second rotation angle; wherein each plane of the at least one plane faces a center of a sphere for displaying the panoramic video, and each plane is for displaying one caption character;
and respectively displaying corresponding subtitle characters on the at least one plane.
In a fifth aspect, there is provided a computer-readable storage medium having stored thereon computer-executable instructions for causing a computer to perform the method of the third or fourth aspect as described above.
In the above embodiment of the present application, when the user view angle is switched, according to the rotation angle of the user view angle along the X axis and along the Y axis, the spherical rectangular grid or at least one plane for displaying the subtitles is correspondingly rotated, and the subtitles are displayed on the spherical rectangular grid or at least one plane after the rotation, because the spherical rectangular grid or at least one plane faces the center of the panorama video sphere, and according to the rotation angle of the user view angle in the horizontal direction and the vertical direction, the spherical rectangular grid or at least one first plane is rotated, so that the subtitle direction is consistent with the video content direction, and the viewing experience of the user can be improved.
Drawings
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings needed to be used in the description of the embodiments or the prior art will be briefly introduced below, and it is obvious that the drawings in the following description are some embodiments of the present application, and for those skilled in the art, other drawings can be obtained according to these drawings without inventive exercise.
Fig. 1 illustrates a block diagram of a VR head mounted display device provided by an embodiment of the present application;
fig. 2 is a block diagram schematically showing a hardware configuration of a display device 200 according to an exemplary embodiment;
fig. 3 is a schematic diagram illustrating a spherical rectangular grid for displaying an image-based subtitle according to an embodiment of the present application;
fig. 4 is a schematic diagram illustrating rotation of a spherical rectangular grid provided by an embodiment of the present application;
fig. 5 is a flowchart illustrating a panoramic video subtitle display method for image-based subtitles according to an embodiment of the present application;
fig. 6 is a schematic diagram illustrating a plurality of planes for displaying text subtitles according to an embodiment of the present application;
fig. 7 is a flowchart illustrating a method for displaying a subtitle of panoramic video in text type according to an embodiment of the present application.
Detailed Description
To make the objects, embodiments and advantages of the present application clearer, the following description of exemplary embodiments of the present application will clearly and completely describe the exemplary embodiments of the present application with reference to the accompanying drawings in the exemplary embodiments of the present application, and it is to be understood that the described exemplary embodiments are only a part of the embodiments of the present application, and not all of the embodiments.
All other embodiments, which can be derived by a person skilled in the art from the exemplary embodiments described herein without inventive step, are intended to be within the scope of the claims appended hereto. In addition, while the disclosure herein has been presented in terms of one or more exemplary examples, it should be appreciated that aspects of the disclosure may be implemented solely as a complete embodiment.
It should be noted that the brief descriptions of the terms in the present application are only for the convenience of understanding the embodiments described below, and are not intended to limit the embodiments of the present application. These terms should be understood in their ordinary and customary meaning unless otherwise indicated.
The terms "first," "second," "third," and the like in the description and claims of this application and in the above-described drawings are used for distinguishing between similar or analogous objects or entities and are not necessarily intended to limit the order or sequence of any particular one, Unless otherwise indicated. It is to be understood that the terms so used are interchangeable under appropriate circumstances such that the embodiments described herein are, for example, capable of operation in sequences other than those illustrated or otherwise described herein.
Furthermore, the terms "comprises" and "comprising," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a product or device that comprises a list of elements is not necessarily limited to those elements explicitly listed, but may include other elements not expressly listed or inherent to such product or device.
The term "module," as used herein, refers to any known or later developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware and/or software code that is capable of performing the functionality associated with that element.
The embodiment of the application provides a panoramic video subtitle display method and display equipment. The display device can be a head-mounted display device, a smart phone, a tablet computer, a notebook computer and other devices with a panoramic video playing function and an interaction function.
Taking a Virtual Reality (VR) head-mounted display device as an example, fig. 1 exemplarily shows a structure diagram of a VR head-mounted display device provided in an embodiment of the present application. As shown in fig. 1, the VR head mounted display device 10 includes a lens group 101 and a display terminal 102 disposed directly in front of the lens group 101, where the lens group 101 is composed of a left display lens 1011 and a right display lens 1012. When a user wears the VR head-mounted display device 10, the human eye can watch the panoramic video image displayed by the display terminal 102 through the lens group 101, and experience VR effects.
A hardware configuration block diagram of a display device 200 according to an exemplary embodiment is exemplarily shown in fig. 2.
The display apparatus 200 includes a controller 250, a tuner demodulator 210, a communicator 220, a detector 230, an external device interface 240, a memory 260, a user interface 265, a video processor 270, a display 275, an audio processor 280, an audio output interface 285, and a power supply 290.
A display 275 for receiving image signals from the processor output, and performing components for displaying video content and images and menu manipulation interfaces, such as a user manipulation UI interface generated in the display device 200 and used for controlling the display device 200, may be presented. The display 275 includes a display screen assembly for presenting a picture and a driving assembly for driving the display according to the type of the display 275. Display 275 may be a projection display and may also include a projection device and a projection screen.
The communicator 220 is a component for communicating with an external device or an external server according to various communication protocol types. For example: the communicator may include at least one of a Wifi chip, a bluetooth communication protocol chip, a wired ethernet communication protocol chip, and other network communication protocol chips or near field communication protocol chips, and an infrared receiver.
The display apparatus 200 may establish control signal and data signal transmission and reception with the external control apparatus 100 or the content providing apparatus through the communicator 220.
The user interface 265 may be used to receive infrared control signals for controlling the apparatus 100 (e.g., an infrared remote control, etc.).
The detector 230 is a signal used by the display apparatus 200 to collect an external environment or interact with the outside. The detector 230 includes a light receiver, an image collector, a temperature sensor, and the like. The detector 230 may also be a sound collector or the like, such as a microphone, which may be used to receive the user's voice. Illustratively, a voice signal including a control instruction of the user to control the display apparatus 200, or to collect an ambient sound for recognizing an ambient scene type, so that the display apparatus 200 can adapt to an ambient noise.
As shown in fig. 2, the first interface (input/output interface) 255 is configured to allow data transmission between the controller 250 and an external other device or other controller 250. Such as receiving video signal data and audio signal data of an external device, or command instruction data, etc.
The external device interface 240 may include, but is not limited to, the following: the interface can be any one or more of a high-definition multimedia interface (HDMI), an analog or data high-definition component input interface, a composite video input interface, a USB input interface, an RGB port and the like. The plurality of interfaces may form a composite input/output interface.
As shown in fig. 2, the tuner demodulator 210 is configured to receive broadcast television signals in a wired or wireless receiving manner, may perform modulation and demodulation processing such as amplification, mixing, resonance, and the like, and demodulate audio and video signals from a plurality of wireless or wired broadcast television signals, where the audio and video signals may include television audio and video signals carried in a television channel frequency selected by a user and EPG data signals.
The frequency point demodulated by the tuning demodulator 210 is controlled by the controller 250, and the controller 250 can send out a control signal according to the user selection, so that the modem responds to the television signal frequency selected by the user and modulates and demodulates the television signal carried by the frequency.
The controller 250 and the tuner-demodulator 210 may be located in different separate devices, that is, the tuner-demodulator 210 may also be located in an external device of the main device where the controller 250 is located, such as an external set-top box. Therefore, the set top box outputs the television audio and video signals modulated and demodulated by the received broadcast television signals to the main body equipment, and the main body equipment receives the audio and video signals through the first input/output interface.
The controller 250 controls the operation of the display device and responds to the user's operation through various software control programs stored in the memory. The controller 250 may control the overall operation of the display apparatus 200. For example: in response to receiving a user command for selecting a UI object to be displayed on the display 275, the controller 250 may perform an operation related to the object selected by the user command.
As shown in fig. 2, the controller 250 includes at least one of a Random Access Memory 251 (RAM), a Read-Only Memory 252 (ROM), a video processor 270, an audio processor 280, other processors 253 (e.g., a Graphics Processing Unit (GPU), a Central Processing Unit 254 (CPU), a Communication Interface (Communication Interface), and a Communication Bus 256(Bus), which connects the respective components.
The RAM 251 is used to store temporary data for the operating system or other programs that are running.
The ROM 252 is used to store instructions for various system boots.
The ROM 252 stores a Basic Input Output System (BIOS). The system is used for completing power-on self-test of the system, initialization of each functional module in the system, a driver of basic input/output of the system and booting an operating system.
When the power of the display apparatus 200 starts to be started upon receipt of the power-on signal, the CPU executes a system start instruction in the ROM 252, and copies temporary data of the operating system stored in the memory to the RAM 251 so as to start or run the operating system. After the start of the operating system is completed, the CPU copies the temporary data of the various application programs in the memory to the RAM 251, and then, the various application programs are started or run.
The CPU processor 254 is used to execute operating system and application program instructions stored in memory. And executing various application programs, data and contents according to various interactive instructions received from the outside so as to finally display and play various audio and video contents. The CPU processor 254 may comprise a plurality of processors. The plurality of processors may include a main processor and one or more sub-processors. A main processor for performing some operations of the display apparatus 200 in a pre-power-up mode and/or operations of displaying a screen in a normal mode. One or more sub-processors for one operation in a standby mode or the like.
The graphics processor 253 is used to generate various graphics objects, such as: icons, operation menus, user input instruction display graphics, and the like. The display device comprises an arithmetic unit which carries out operation by receiving various interactive instructions input by a user and displays various objects according to display attributes. And the system comprises a renderer for rendering various objects obtained based on the arithmetic unit, wherein the rendered objects are used for being displayed on a display.
The video processor 270 is configured to receive an external video signal, and perform video processing such as decompression, decoding, scaling, noise reduction, frame rate conversion, resolution conversion, image synthesis, and the like according to a standard codec protocol of the input signal, so as to obtain a signal to be displayed or played on the direct display device 200. The video processor 270 includes a demultiplexing module, a video decoding module, an image synthesizing module, a frame rate conversion module, a display formatting module, and the like.
The demultiplexing module is used for demultiplexing the input audio and video data stream, and if the input MPEG-2 is input, the demultiplexing module demultiplexes the input audio and video data stream into a video signal and an audio signal. The video decoding module is used for processing the video signal after demultiplexing, including decoding and scaling processing. And the image synthesis module is used for carrying out superposition mixing processing on the GUI signal input by the user or generated by the user and the video image after the zooming processing by the graphic generator so as to generate an image signal for display. The frame rate conversion module is configured to convert an input video frame rate, such as a 60Hz frame rate into a 120Hz frame rate or a 240Hz frame rate, and the normal format is implemented in, for example, an interpolation frame mode. The display format module is used for converting the received video output signal after the frame rate conversion, and changing the signal to conform to the signal of the display format, such as outputting an RGB data signal.
The graphic processor 253 and the video processor may be integrated with each other or may be separately configured, and may perform processing of a graphic signal output to the display during the integrated configuration, and may perform different functions during the separate configuration, for example, a GPU + frc (frame Rate conversion) architecture.
The audio processor 280 is configured to receive an external audio signal, decompress and decode the received audio signal according to a standard codec protocol of the input signal, and perform noise reduction, digital-to-analog conversion, and amplification processing to obtain an audio signal that can be played in a speaker.
Video processor 270 may comprise one or more chips. The audio processor may also comprise one or more chips.
The video processor 270 and the audio processor 280 may be separate chips or may be integrated together with the controller in one or more chips.
An audio output for receiving sound signals output from audio processor 280 under the control of controller 250, such as: the speaker 286, and an external sound output terminal of a generating device that can output to an external device, in addition to the speaker carried by the display device 200 itself, such as: external sound interface or earphone interface, etc., and may also include a near field communication module in the communication interface, for example: and the Bluetooth module is used for outputting sound of the Bluetooth loudspeaker.
The power supply 290 supplies power to the display device 200 from the power input from the external power source under the control of the controller 250. The power supply 290 may include a built-in power supply circuit installed inside the display apparatus 200, or may be a power supply interface installed outside the display apparatus 200 to provide an external power supply in the display apparatus 200.
A user interface 265 for receiving an input signal of a user and then transmitting the received user input signal to the controller 250. The user input signal may be a remote controller signal received through an infrared receiver, and various user control signals may be received through the network communication module.
The memory 260 includes a memory storing various software modules for driving the display device 200. Such as: various software modules stored in the first memory, including: at least one of a basic module, a detection module, a communication module, a display control module, a browser module, and various service modules.
The base module is a bottom layer software module for signal communication between various hardware in the display device 200 and for sending processing and control signals to the upper layer module. The detection module is used for collecting various information from various sensors or user input interfaces, and the management module is used for performing digital-to-analog conversion and analysis management.
Currently, there are two representative methods for displaying a subtitle in a panoramic video.
In the first method, a plane is created on which subtitles are displayed directly. However, as the panoramic video is played by taking the spherical surface as a playing carrier, the two-dimensional plane and the spherical surface cannot be well fused, the subtitles can disperse the attention of the user, reduce the playing effect of the panoramic video, and influence the watching experience of the user.
In the second method, the image of the corresponding viewpoint area is taken out through the viewpoint position of the current user, then the image and the subtitle are overlapped to generate an overlapped image, and finally the overlapped image replaces the original image in the view field range to realize the display of the VR video subtitle. The method can well fuse the caption and the panoramic video together to form a whole, but the film source of the panoramic video is two-dimensional, and due to the relation of spherical surface and plane projection, the original film source has great deformation near the two poles of the spherical surface, and the deformation disappears only after the original film source is projected to the spherical surface, but the caption itself has no deformation, so when the video content at the two poles of the spherical surface is watched, the undeformed caption and the deformed video area are superposed and displayed on the spherical surface after superposition, the video content is normal, but the caption can generate deformation due to the projection relation, thereby affecting the watching experience of a user.
Panoramic video captions are generally divided into two types:
image type subtitles: the image-type subtitle refers to a subtitle file in which subtitle content is stored in an image format. The subtitle content which needs to be displayed at different display time can be stored as different subtitle files according to different display time, and when the subtitle is displayed, the corresponding subtitle file is obtained according to the current time to be displayed. Or all subtitle contents can be stored as a subtitle file, and when the subtitle is displayed, the subtitle image slice at the corresponding position in the subtitle file can be acquired according to the current time to be displayed.
Character type subtitles: a text subtitle refers to a subtitle file in which subtitle content is stored in a text format, such as a file that may be in txt format or html format or other format, from which each character in the subtitle may be extracted. When displaying the caption, the corresponding character sequence (such as one or more characters) in the caption file can be obtained according to the current time to display.
For Chinese subtitles, one character is a Chinese character or a punctuation mark; for an english caption, a character is an english letter or a punctuation mark. For subtitles in other languages, the analogy can be repeated.
The subtitles of the panoramic video are different from the subtitles of the traditional two-dimensional video, and because a user can randomly move the viewing angle to watch the contents of different areas of the panoramic video when watching the panoramic video, the subtitle position of the panoramic video cannot be fixed and needs to move along with the movement of the viewing angle of the user, so that the user can see the subtitle contents at any time.
According to the type of the image panoramic video caption, a panoramic video caption display method and a display device for the image subtitle, and a panoramic video caption display method and a display device for the text subtitle are respectively provided. The panoramic video subtitle display method provided by the embodiment of the application can be executed by the display equipment, can be suitable for displaying local panoramic videos and can also be suitable for displaying online panoramic videos (including two modes of on-demand and live broadcasting).
The following describes in detail a panorama video subtitle display method for an image-type subtitle and a text-type subtitle, respectively.
In the panoramic video subtitle display method for image-based subtitles provided by the embodiment of the application, when a user view angle is switched, a spherical rectangular grid for displaying subtitles is correspondingly rotated according to the rotation angle of the user view angle along an X axis and a Y axis of a world coordinate system, and the subtitles are displayed on the rotated spherical rectangular grid. The orientation of the spherical rectangular grid is used for displaying the spherical center of a sphere of a panoramic video, and the spherical rectangular grid can be rotated according to the rotation angles of the visual angle of a user in the horizontal direction and the vertical direction, so that the direction of a subtitle is consistent with the direction of video content, and the watching experience of the user can be improved.
The panoramic video takes a spherical surface as a playing carrier, the radius of the spherical surface for displaying the panoramic video is assumed to be R, and the size of the field angle of the display device is W × H (wherein W represents the width of the field angle, and H represents the height of the field angle).
In this embodiment, a spherical rectangular grid is created, and the spherical rectangular grid is used as a carrier for displaying the panoramic video subtitle. The spherical rectangular grid is matched with the spherical surface of the sphere for playing the panoramic video, so that the subtitles and the panoramic video can be well fused, and the watching experience of a user is improved.
Specifically, the normal of the spherical rectangular grid points to the center of the sphere of the panoramic video, and the spherical radius of the spherical rectangular grid is smaller than or equal to the radius R of the sphere for displaying the panoramic video, so that the spherical rectangular grid can be matched with the spherical surface of the sphere.
Further, the horizontal span of the spherical rectangular grid is not greater than the viewing angle width W of the display device, and the vertical span of the spherical rectangular grid is not greater than the height H of the viewing angle of the display device, so that the subtitles are displayed within the user viewing area (i.e., the user viewing area).
Further, the horizontal span of the spherical rectangular grid may be smaller than the viewing angle width W of the display device, and the vertical span of the spherical rectangular grid may be smaller than the viewing angle height H of the display device, so that the subtitles are all displayed in the user viewing angle area. Further, in order to avoid the subtitle not obstructing other display contents in the user's view, the vertical span of the spherical rectangular grid may be set to be smaller than H/3 in the embodiment of the present application. Of course, the vertical span of the spherical rectangular grid may be set to other values as long as the height H is smaller than the field angle of the display device.
A spherical rectangular grid for displaying image captions is exemplarily shown in fig. 3. As shown, the spherical rectangular grid 301 is located in the field angle area (i.e. the user viewing area) 300 of the display device, the horizontal span W of the spherical rectangular grid 301 is smaller than W (W is the width of the user viewing area), and the vertical span H of the spherical rectangular grid 301 is smaller than H/3(H is the height of the user viewing area).
In the embodiment of the present application, the initial position of the spherical rectangular grid may be set, and the spherical rectangular grid is subsequently rotated correspondingly according to the change of the user view angle.
The initial position of the spherical rectangular grid needs to ensure that the spherical rectangular grid is completely located in the user's view field. Specifically, the spherical rectangular grid faces the center of sphere (i.e. the normal of the rectangular grid points to the center of sphere), so that the spherical rectangular grid and the spherical surface are ensured to be matched. The spherical rectangular grid is positioned in the horizontal direction, namely the upper side and the lower side of the spherical rectangular grid are kept parallel to the X-Z plane of the world coordinate system, so that the subtitle is consistent with the direction of the video content at the beginning.
The subtitles of the panoramic video are different from the subtitles of the traditional two-dimensional video, and because a user can randomly move the viewing angle to watch the contents of different areas of the panoramic video when watching the panoramic video, the subtitle position of the panoramic video cannot be fixed and needs to move along with the movement of the viewing angle of the user, so that the user can see the subtitle contents at any time. In the embodiment of the application, the panoramic video subtitle is kept horizontal, namely, only rotates around the X axis and the Y axis of a world coordinate system along with the view angle of a user. Because the content of the panoramic video is consistent with the normal visual angle of the user, namely, the panoramic video is displayed in the horizontal direction, if the subtitle rotates around the Z axis, the subtitle direction forms an included angle with the horizontal direction, namely, the subtitle direction is inconsistent with the video content direction, and the visual experience of the user is greatly reduced.
Fig. 4 exemplarily shows a rotation diagram of a spherical rectangular grid. As shown, the spherical rectangular grid rotates only around the X-axis and Y-axis of the world coordinate system to ensure that the caption direction coincides with the video content direction.
A panorama video subtitle display method for an image-type subtitle will be described in detail with reference to the accompanying drawings.
Fig. 5 is a flowchart illustrating a method for displaying subtitles of panoramic video in image type according to an embodiment of the present application. The process can be executed by the display device, and the process can be realized by a software mode or a mode of combining software and hardware. As shown, the process includes the following steps:
s501: in response to a user perspective switching from a first perspective to a second perspective, a first rotational angle along the X-axis from the first perspective to the second perspective and a second rotational angle along the Y-axis from the first perspective to the second perspective are determined.
In this step, during the panoramic video playing process, if the user's view angle is switched (i.e., the user's view angle is moved), an angle (referred to as a first rotation angle rotation _ X) by which the user's view angle is rotated around the world coordinate system X-axis and an angle (referred to as a second rotation angle rotation _ Y) by which the user's view angle is rotated around the world coordinate system Y-axis are obtained.
S502: the method includes rotating a spherical rectangular mesh for displaying a subtitle image along an X-axis according to a first rotation angle, and rotating the spherical rectangular mesh along a Y-axis according to a second rotation angle.
In this step, the spherical rectangular mesh for displaying the subtitle image is rotated along the X-axis by a first rotation angle rotation _ X, and the spherical rectangular mesh is rotated along the X-axis by a second rotation angle rotation _ Y.
S503: and displaying the caption image on the rotated spherical rectangular grid.
In this step, the subtitle image may be used as the texture of the spherical rectangular grid, and the texture rendering may be performed on the spherical rectangular grid to realize the display of the subtitle. If the subtitle content does not need to be updated (i.e., the time for updating the screen is not yet reached) when the user' S view angle is switched, the original subtitle image is redisplayed in the rotated spherical rectangular grid in S503.
Further, the method also comprises the following steps: and acquiring a subtitle image to be displayed according to the subtitle image display time, and displaying the subtitle image to be displayed in the spherical rectangular grid. Namely, when the next section of caption needs to be displayed is judged according to the caption display time, the caption file is loaded and analyzed to obtain the next section of caption image to be displayed, the caption image is bound with the spherical rectangular grid and used as the texture to perform texture rendering on the spherical rectangular grid, and therefore the next section of caption is displayed on the spherical rectangular grid.
In the above-mentioned embodiment of the present application, when the user visual angle is switched, according to the rotation angle of the user visual angle along the X axis and along the Y axis, the spherical rectangular grid for displaying the caption is correspondingly rotated, and the caption is displayed on the spherical rectangular grid after the rotation, because the spherical rectangular grid faces the spherical center of the panoramic video sphere, the caption deformation can be avoided, and according to the rotation angle of the user visual angle in the horizontal direction and the vertical direction, the spherical rectangular grid is rotated, so that the caption direction is consistent with the video content direction, and the watching experience of the user can be improved.
In the panoramic video subtitle display method for text type subtitles provided by the embodiment of the application, when a user visual angle is switched, at least one plane for displaying subtitles (wherein each plane is used for displaying one character) is correspondingly rotated according to the rotation angles of the user visual angle along the X axis and the Y axis of a world coordinate system, and the subtitles are displayed on the rotated plane. The plane faces the center of a sphere used for displaying the panoramic video, and can be rotated according to the rotation angles of the user visual angle in the horizontal direction and the vertical direction, so that the direction of the subtitles is consistent with the direction of the video content, and the watching experience of the user can be improved.
The panoramic video takes a spherical surface as a playing carrier, the radius of the spherical surface for displaying the panoramic video is assumed to be R, and the size of the field angle of the display device is W × H (wherein W represents the width of the field angle, and H represents the height of the field angle).
In the embodiment, at least one plane is created in the caption area, the at least one plane is used as a carrier for displaying the panoramic video caption, and the number of the planes is adapted to the maximum number of characters in the caption to be displayed at one time. One of the planes displays a character, for example, for Chinese subtitles, one of the planes displays a Chinese character or a punctuation mark; for example, for an english caption, a plane displays an english caption or a punctuation mark. Each plane faces the center of a sphere used for playing the panoramic video, so that the subtitles and the panoramic video can be well fused, and the watching experience of a user is improved.
In some embodiments, at least one plane in the subtitle region may be arranged in N rows (N is an integer greater than or equal to 1), and each row may include M planes (M is an integer greater than or equal to 1), so that a maximum of M × N characters may be displayed at a time.
In other embodiments, in the subtitle area, each horizontal line may include M planes, but the value of the line number N may not be fixed, and the value of the line number N may be determined according to the total number of characters in the subtitle to be displayed, for example, the line number is equal to the total number of characters/M, and the calculation result is rounded down to obtain the line number of the subtitle at the current time, so as to avoid the loss of the characters.
In some embodiments, the distance from the center of each of the at least one plane to the center of the sphere used to display the panoramic video is equal to the radius R of the sphere, thereby enabling all planes to approximately coincide with the sphere of the sphere. In other embodiments, each of the at least one plane has four corner points of a rectangle that are equidistant from the center of the sphere by a radius equal to the radius of the sphere, such that the rectangle is inscribed on the sphere.
The area where the at least one plane is located is called a subtitle area. The horizontal span of the subtitle region is not greater than the field angle width W of the display device and the vertical span of the spherical rectangular grid is not greater than the height H of the field angle of the display device, so that the subtitle is displayed within the user field of view region (i.e., the user field of view).
Further, the horizontal span of the subtitle region may be smaller than the width W of the field angle of the display device, and the vertical span of the subtitle region may be smaller than the height H of the field angle of the display device, so that the subtitles are all displayed in the user field angle region. Further, in order to avoid that the subtitle does not obstruct other display contents in the user's view, the vertical span of the subtitle region may be set to be less than H/3 in the embodiment of the present application. Of course, the vertical span of the subtitle region may be set to other values as long as it is smaller than the height H of the field angle of the display device.
If the horizontal span of the caption area is W1, the vertical span is H1, and the screen in the caption area comprises at most M lines in the horizontal direction and at most N lines in the vertical direction, the horizontal included angle between two adjacent planes in the horizontal direction is W1/M, and the vertical included angle between two adjacent planes in the vertical direction is H1/N, so that each plane can be arranged according to a certain radian.
As illustrated in fig. 6, a plurality of planes for displaying text type subtitles. As shown, the subtitle area 601 includes a plurality of planes (each dotted square in the figure represents a plane) each for displaying a character, the planes being arranged in two rows. The subtitle area 601 is located in a field angle area (i.e., user viewing area) 600 of the display device, a horizontal span W of the subtitle area 601 is smaller than W (W is a width of the user viewing area), and a vertical span H of the subtitle area 601 is smaller than H/3(H is a height of the user viewing area).
In the embodiment of the application, the initial position of the plane can be set, and the plane is correspondingly rotated according to the change of the visual angle of the user.
The initial position of the plane needs to ensure that the subtitle area is completely located in the user's view. Specifically, each plane is oriented toward the center of the sphere (i.e., the normal of the rectangular grid is directed toward the center of the sphere), which ensures that the planes approximately coincide with the sphere. The planes are made to lie in a horizontal direction, i.e., rotated about the X and Y axes of the world coordinate system only with the user's perspective. Because the content of the panoramic video is consistent with the normal visual angle of the user, namely, the panoramic video is displayed in the horizontal direction, if the subtitle rotates around the Z axis, the subtitle direction forms an included angle with the horizontal direction, namely, the subtitle direction is inconsistent with the video content direction, and the visual experience of the user is greatly reduced.
Since each plane is used for displaying a character, each character of the caption can be individually output as a character entity, each character has a respective initial position and angle, each plane faces to the center of sphere (i.e. the normal of the plane points to the center of sphere), and then each plane rotates along with the rotation of the user's view angle.
A method for displaying a subtitle of a panoramic video for a text subtitle will be described in detail with reference to the accompanying drawings.
Fig. 7 is a flowchart illustrating a method for displaying a subtitle of panoramic video in text type according to an embodiment of the present application. The process can be executed by the display device, and the process can be realized by a software mode or a mode of combining software and hardware. As shown, the process includes the following steps:
s701: in response to a user perspective switching from a first perspective to a second perspective, a first rotational angle along the X-axis from the first perspective to the second perspective and a second rotational angle along the Y-axis from the first perspective to the second perspective are determined.
In this step, during the panoramic video playing process, if the user's view angle is switched (i.e., the user's view angle is moved), an angle (referred to as a first rotation angle rotation _ X) by which the user's view angle is rotated around the world coordinate system X-axis and an angle (referred to as a second rotation angle rotation _ Y) by which the user's view angle is rotated around the world coordinate system Y-axis are obtained.
S702: rotating at least one plane for displaying caption text along an X-axis according to the first rotation angle, and rotating the at least one plane along a Y-axis according to the second rotation angle.
In this step, each plane for displaying caption characters is rotated along the X-axis by a first rotation angle rotation _ X, respectively, and each plane is rotated along the X-axis by a second rotation angle rotation _ Y, respectively.
S703: and respectively displaying corresponding subtitle characters on the at least one plane.
In this step, if the subtitle content does not need to be updated (i.e., the time for updating the screen is not yet reached) when the user' S view angle is switched, the original subtitle characters are redisplayed on the rotated plane in S703.
Further, the method also comprises the following steps: and acquiring the subtitle characters to be displayed according to the subtitle display time, and displaying the subtitle characters to be displayed in the plane. Namely, when the next section of caption is judged to be displayed according to the caption display time, the caption file is loaded, the caption file is analyzed, the next section of caption characters to be displayed is obtained, each character is independently output, namely, only one character is displayed on each plane.
Alternatively, if the number of characters in the subtitle that needs to be displayed currently is less than the number of the planes, the subtitle characters may be displayed in the partial planes, for example, the subtitle characters may be displayed using a screen in a middle position, so that the subtitle text is displayed in a centered manner, thereby improving the viewing experience of the user.
Alternatively, if the number of characters in the subtitle to be displayed currently is large, the subtitle text can be displayed in a line feed manner by using a multi-line plane.
In the above embodiment of the present application, when the user visual angle is switched, according to the rotation angle of the user visual angle along the X axis and along the Y axis, the plane for displaying the subtitles is correspondingly rotated, and the subtitle characters are displayed on the rotated plane, because the plane faces the spherical center of the panoramic video sphere, the subtitle can be prevented from being deformed, and according to the rotation angle of the user visual angle in the horizontal direction and the vertical direction, the plane is rotated, so that the subtitle direction is consistent with the video content direction, and the viewing experience of the user can be improved.
Embodiments of the present application also provide a computer-readable storage medium for storing instructions that, when executed, may implement the methods of the foregoing embodiments.
The embodiments of the present application also provide a computer program product for storing a computer program, where the computer program is used to execute the method of the foregoing embodiments.
Finally, it should be noted that: the above embodiments are only used for illustrating the technical solutions of the present application, and not for limiting the same; although the present application has been described in detail with reference to the foregoing embodiments, it should be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some or all of the technical features may be equivalently replaced; and the modifications or the substitutions do not make the essence of the corresponding technical solutions depart from the scope of the technical solutions of the embodiments of the present application.
The foregoing description, for purposes of explanation, has been presented in conjunction with specific embodiments. However, the illustrative discussions above are not intended to be exhaustive or to limit the embodiments to the precise forms disclosed above. Many modifications and variations are possible in light of the above teaching. The embodiments were chosen and described in order to best explain the principles and the practical application, to thereby enable others skilled in the art to best utilize the embodiments and various embodiments with various modifications as are suited to the particular use contemplated.

Claims (10)

1. A display device, comprising:
a display connected with the controller and configured to display the panoramic video image;
a memory coupled to the controller and configured to store computer instructions;
the controller configured to perform the following operations in accordance with the computer instructions:
in response to a user perspective switching from a first perspective to a second perspective, determining a first rotational angle along an X-axis from the first perspective to the second perspective and a second rotational angle along a Y-axis from the first perspective to the second perspective;
rotating a spherical rectangular mesh for displaying a subtitle image along an X-axis according to the first rotation angle, and rotating the spherical rectangular mesh along a Y-axis according to the second rotation angle; wherein the spherical rectangular mesh is oriented toward the center of a sphere used to display panoramic video;
and displaying the subtitle image on the spherical rectangular grid.
2. The display device of claim 1, wherein the controller is further configured to:
acquiring a subtitle image to be displayed according to the subtitle image display time;
and displaying the subtitle image to be displayed in the spherical rectangular grid.
3. The display device of claim 1, wherein a horizontal span of the spherical rectangular grid is less than a field angle width of the display device and a vertical span of the spherical rectangular grid is less than a field angle height of the display device.
4. The display device of any of claims 1-3, wherein a normal of the spherical rectangular grid points to a center of a sphere of the panoramic video sphere, a spherical radius of the spherical rectangular grid being less than or equal to a radius of the sphere for displaying the panoramic video.
5. A display device, comprising:
a display connected with the controller and configured to display the panoramic video image;
a memory coupled to the controller and configured to store computer instructions;
the controller configured to perform the following operations in accordance with the computer instructions:
in response to a user perspective switching from a first perspective to a second perspective, determining a first rotational angle along an X-axis from the first perspective to the second perspective and a second rotational angle along a Y-axis from the first perspective to the second perspective;
rotating at least one plane for displaying caption texts along an X axis according to the first rotation angle, and rotating the at least one plane along a Y axis according to the second rotation angle; wherein each plane of the at least one plane faces a center of a sphere for displaying the panoramic video, and each plane is for displaying one caption character;
and respectively displaying corresponding subtitle characters on the at least one plane.
6. The display device of claim 5, wherein the controller is further configured to:
acquiring at least one subtitle character to be displayed according to the subtitle display time;
and displaying the at least one subtitle character to be displayed on the at least one plane.
7. The display device of claim 1, wherein a horizontal span of a subtitle region in which the at least one plane is located is less than a field angle width of the display device, and a vertical span of the subtitle region is less than a field angle height of the display device.
8. The display device of any of claims 1-3, wherein the distance from four corner points of each of the at least one plane to the center of the sphere used to display the panoramic video is equal to the radius of the sphere.
9. A method for displaying subtitles of a panoramic video, comprising:
in response to a user perspective switching from a first perspective to a second perspective, determining a first rotational angle along an X-axis from the first perspective to the second perspective and a second rotational angle along a Y-axis from the first perspective to the second perspective;
rotating a spherical rectangular mesh for displaying a subtitle image along an X-axis according to the first rotation angle, and rotating the spherical rectangular mesh along a Y-axis according to the second rotation angle; wherein the spherical rectangular mesh is oriented toward the center of a sphere used to display panoramic video;
and displaying the subtitle image on the spherical rectangular grid.
10. A method for displaying subtitles of a panoramic video, comprising:
in response to a user perspective switching from a first perspective to a second perspective, determining a first rotational angle along an X-axis from the first perspective to the second perspective and a second rotational angle along a Y-axis from the first perspective to the second perspective;
rotating at least one plane for displaying caption texts along an X axis according to the first rotation angle, and rotating the at least one plane along a Y axis according to the second rotation angle; wherein each plane of the at least one plane faces a center of a sphere for displaying the panoramic video, and each plane is for displaying one caption character;
and respectively displaying corresponding subtitle characters on the at least one plane.
CN202011330993.0A 2020-11-24 2020-11-24 Panoramic video subtitle display method and display equipment Pending CN112532962A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202011330993.0A CN112532962A (en) 2020-11-24 2020-11-24 Panoramic video subtitle display method and display equipment

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202011330993.0A CN112532962A (en) 2020-11-24 2020-11-24 Panoramic video subtitle display method and display equipment

Publications (1)

Publication Number Publication Date
CN112532962A true CN112532962A (en) 2021-03-19

Family

ID=74993061

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202011330993.0A Pending CN112532962A (en) 2020-11-24 2020-11-24 Panoramic video subtitle display method and display equipment

Country Status (1)

Country Link
CN (1) CN112532962A (en)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113286138A (en) * 2021-05-17 2021-08-20 聚好看科技股份有限公司 Panoramic video display method and display equipment
WO2023221409A1 (en) * 2022-05-20 2023-11-23 北京字跳网络技术有限公司 Subtitle rendering method and apparatus for virtual reality space, device, and medium

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105916001A (en) * 2016-05-12 2016-08-31 乐视控股(北京)有限公司 Video barrage display method and device
CN106210861A (en) * 2016-08-23 2016-12-07 上海幻电信息科技有限公司 The method and system of display barrage
US20170142389A1 (en) * 2015-11-18 2017-05-18 Le Holdings (Beijing) Co., Ltd. Method and device for displaying panoramic videos
CN108737907A (en) * 2017-04-18 2018-11-02 杭州海康威视数字技术股份有限公司 A kind of method and device generating subtitle
CN109561263A (en) * 2018-11-23 2019-04-02 重庆爱奇艺智能科技有限公司 3D subtitle effect is realized in the 3D video of VR equipment
CN111669664A (en) * 2020-06-17 2020-09-15 Oppo广东移动通信有限公司 Video playing method, video playing device, electronic equipment and storage medium

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20170142389A1 (en) * 2015-11-18 2017-05-18 Le Holdings (Beijing) Co., Ltd. Method and device for displaying panoramic videos
CN105916001A (en) * 2016-05-12 2016-08-31 乐视控股(北京)有限公司 Video barrage display method and device
CN106210861A (en) * 2016-08-23 2016-12-07 上海幻电信息科技有限公司 The method and system of display barrage
CN108737907A (en) * 2017-04-18 2018-11-02 杭州海康威视数字技术股份有限公司 A kind of method and device generating subtitle
CN109561263A (en) * 2018-11-23 2019-04-02 重庆爱奇艺智能科技有限公司 3D subtitle effect is realized in the 3D video of VR equipment
CN111669664A (en) * 2020-06-17 2020-09-15 Oppo广东移动通信有限公司 Video playing method, video playing device, electronic equipment and storage medium

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113286138A (en) * 2021-05-17 2021-08-20 聚好看科技股份有限公司 Panoramic video display method and display equipment
WO2023221409A1 (en) * 2022-05-20 2023-11-23 北京字跳网络技术有限公司 Subtitle rendering method and apparatus for virtual reality space, device, and medium

Similar Documents

Publication Publication Date Title
CN113330736B (en) Display and image processing method
CN111246266A (en) Display equipment and UI (user interface) display method during rotation
CN112073788B (en) Video data processing method and device and display equipment
CN112533021A (en) Display method and display equipment
CN113206992A (en) Method for converting projection format of panoramic video and display equipment
CN112565839A (en) Display method and display device of screen projection image
CN114793294A (en) Display device and boot animation display method
CN111176603A (en) Image display method for display equipment and display equipment
CN112532962A (en) Panoramic video subtitle display method and display equipment
CN111866593A (en) Display device and startup interface display method
CN112672131A (en) Panoramic video image display method and display equipment
CN112055256A (en) Image processing method and display device for panoramic image
CN112073795B (en) Video data processing method and device and display equipment
CN111930233A (en) Panoramic video image display method and display equipment
CN112399233A (en) Display device and position self-adaptive adjusting method of video chat window
CN112783380A (en) Display apparatus and method
CN111212293A (en) Image processing method and display device
CN111954043B (en) Information bar display method and display equipment
CN113473024A (en) Display device, holder camera and camera control method
CN111064982B (en) Display control method, storage medium and display device
CN113518257A (en) Multi-system screen projection processing method and equipment
CN112565736A (en) Panoramic video display method and display equipment
CN112055245A (en) Color subtitle realization method and display device
CN113395554A (en) Display device
CN113573118B (en) Video picture rotating method and display equipment

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
RJ01 Rejection of invention patent application after publication

Application publication date: 20210319

RJ01 Rejection of invention patent application after publication