CN108924534B - Panoramic image display method, client, server and storage medium - Google Patents

Panoramic image display method, client, server and storage medium Download PDF

Info

Publication number
CN108924534B
CN108924534B CN201710474090.1A CN201710474090A CN108924534B CN 108924534 B CN108924534 B CN 108924534B CN 201710474090 A CN201710474090 A CN 201710474090A CN 108924534 B CN108924534 B CN 108924534B
Authority
CN
China
Prior art keywords
additional content
user
data
content data
orientation
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN201710474090.1A
Other languages
Chinese (zh)
Other versions
CN108924534A (en
Inventor
周扬
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Tencent Technology Shenzhen Co Ltd
Original Assignee
Tencent Technology Shenzhen Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Tencent Technology Shenzhen Co Ltd filed Critical Tencent Technology Shenzhen Co Ltd
Priority to CN201710474090.1A priority Critical patent/CN108924534B/en
Priority to PCT/CN2018/088923 priority patent/WO2018233459A1/en
Publication of CN108924534A publication Critical patent/CN108924534A/en
Application granted granted Critical
Publication of CN108924534B publication Critical patent/CN108924534B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/30—Image reproducers
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/30—Image reproducers
    • H04N13/332—Displays for viewing with the aid of special glasses or head-mounted displays [HMD]
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/30—Image reproducers
    • H04N13/366—Image reproducers using viewer tracking
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/30—Image reproducers
    • H04N13/366—Image reproducers using viewer tracking
    • H04N13/383—Image reproducers using viewer tracking for tracking with gaze detection, i.e. detecting the lines of sight of the viewer's eyes

Abstract

The application discloses a display method of a panoramic image, which comprises the following steps: receiving data of a panoramic image from an application server and displaying the panoramic image; receiving input additional content data in response to an operation of a first user to publish additional content; sending the additional content data and the view direction data of the first user to the application server, so that the application server determines a view direction change range of the first user in a period of time according to the view direction data, wherein the period of time comprises a time point of the first user publishing the additional content data, and the application server or a second application client determines whether the second application client displays the additional content data when displaying the panoramic image according to the view direction change range and the current view direction data of the second user. The application also discloses a corresponding client, a server and a storage medium.

Description

Panoramic image display method, client, server and storage medium
Technical Field
The present application relates to the field of computer technologies, and in particular, to a method for displaying a panoramic image, a method for controlling display of a panoramic image, an application client, an application server, and a storage medium.
Background
With the development of intelligent user equipment, Virtual Reality (VR) technology is increasingly applied to common user equipment, such as smart phones, tablet computers, PCs and the like, so that various VR software (such as VR video APPs) appears, a user wears VR wearable equipment such as helmets and glasses connected with the user equipment, a VR panoramic image displayed by the VR software can be viewed, and immersive experience in a three-dimensional space view can be obtained.
At present, there are some VR platforms, VR application clients may be installed in user equipment, and each VR application client can access VR panoramic image data and present the data to a user by accessing a VR application server in the VR platform. Such as: VR application client can obtain VR video data and play from VR application server, and the user has worn the first equipment of showing of VR (such as helmet, glasses) of connecting VR application client and just can experience immersive panoramic video.
Disclosure of Invention
The application example provides a panoramic image display method, which is applied to a first application client and comprises the following steps: receiving data of a panoramic image from an application server and displaying the panoramic image; receiving input additional content data in response to an operation of a first user to publish additional content; sending the additional content data and the view direction data of the first user to the application server, so that the application server determines a view direction change range of the first user within a period of time according to the view direction data, wherein the period of time includes a time point when the first user releases the additional content data, so that the application server or a second application client determines whether the second application client displays the additional content data when displaying the panoramic image according to the view direction change range and the current view direction data of the second user, and the second user is a user of the second application client.
The application example provides a panoramic image display method, which is applied to an application client and comprises the following steps: receiving data of a panoramic image from an application server and displaying the panoramic image; receiving at least one piece of additional content data from the application server and a range of view orientation changes over a period of time for a first user publishing each piece of additional content data; wherein, for a piece of additional content data, the period of time includes a point in time at which the piece of additional content data is published; for each piece of additional content data, determining whether the additional content data can be presented according to the visual field orientation change range of the piece of additional content data and visual field orientation data of a second user, wherein the second user is a user of the application client; and displaying the determined displayable pieces of additional content data.
The application example provides a display control method of a panoramic image, which is applied to an application server and comprises the following steps: sending data of a panoramic image to a plurality of application clients so that the application clients respectively display the panoramic image; when additional content data uploaded by a first user and uploaded by any application client in the plurality of application clients are received, receiving visual field orientation data of the first user, and determining a visual field orientation change range of the first user within a period of time according to the visual field orientation data, wherein the period of time comprises a time point when the first user releases the additional content data; and respectively sending the currently received additional content data and the respective visual field orientation change range of the first user to the plurality of application clients, so that for each additional content data, determining whether the additional content data can be displayed according to the visual field orientation change range of the additional content data and the visual field orientation data of a second user, and displaying the determined displayable additional content data, wherein the second user is a user of the application client who receives the additional content data.
The application example provides a display control method of a panoramic image, which is applied to an application server and comprises the following steps: sending data of a panoramic image to a plurality of application clients so that the application clients respectively display the panoramic image; when additional content data uploaded by a first user and uploaded by any application client in the plurality of application clients are received, receiving visual field orientation data of the first user, and determining a visual field orientation change range of the first user within a period of time according to the visual field orientation data, wherein the period of time comprises a time point when the first user releases the additional content data; the following operations are performed for each application client: for each piece of additional content data which is currently received, determining whether the application client can display the additional content data according to the visual field orientation change range of the piece of additional content data and visual field orientation data of a second user, wherein the second user is a user of the application client; and sending the determined additional content data which can be displayed by the application client to the application client so as to enable the application client to display the additional content data which can be displayed.
The application example provides an application client, including: the panoramic display module receives data of the panoramic image from the application server and displays the panoramic image; the receiving module is used for responding to the operation of publishing the additional content by the first user and receiving the input additional content data; the sending module is used for sending the additional content data and the view direction data of the first user to the application server so that the application server determines a view direction change range of the first user within a period of time according to the view direction data, wherein the period of time comprises a time point when the first user issues the additional content data, so that the application server or a second application client determines whether the second application client displays the additional content data or not when displaying the panoramic image according to the view direction change range and the current view direction data of the second user, and the second user is a user of the application client.
In some examples, the client further comprises: a determination module, responsive to the operation of posting the additional content, determining a range of change in orientation of the first user's field of view over a period of time before and after a point in time at which the additional content data was posted; wherein the view orientation data sent by the sending module to the application server includes the view orientation variation range.
In some examples, the determination module further determines a dwell time of the first user within the range of view orientation changes for a period of time before and after a point in time at which the additional content data is published; wherein the view orientation data sent by the sending module to the application server further includes the dwell time; wherein the application server or the second application client determines whether the second application client presents the additional content data when presenting the panoramic image according to the field of view orientation change range, the dwell time, and the second user's current field of view orientation data.
In some examples, the sending module sends the view orientation data including the first user's current view orientation parameter to the application server at intervals.
The application example provides an application client, including: the panoramic display module receives data of the panoramic image from the application server and displays the panoramic image; the receiving module receives at least one piece of additional content data from the application server and the visual field orientation change range of a first user publishing each piece of additional content data in a period of time; wherein, for a piece of additional content data, the period of time includes a point in time at which the piece of additional content data is published; a determination module that determines, for each piece of additional content data, whether the additional content data can be presented based on the range of change in the orientation of the field of view of the piece of additional content data and the orientation of the field of view of the second user; the panoramic display module displays the determined displayable additional content data, and the second user is a user of the application client.
In some examples, the receiving module, when receiving the additional content data and the range of orientation of the field of view thereof from the application server, further receives a presentation of the additional content data; and the panoramic display module displays the additional content data according to the display mode.
In some examples, the determining module further determines a presentation manner of the additional content data according to the field of view orientation variation range of the additional content data; and the panoramic display module displays the additional content data according to the display mode.
In some examples, the determining module determines that the piece of additional content data can be presented when a visual dwell region corresponding to the field of view orientation variation range of the piece of additional content data covers a visual dwell region corresponding to the field of view orientation data of the second user.
In some examples, the receiving module, when receiving the additional content data and its range of view orientation changes from the application server, further receives a dwell time for the range of view orientation changes; the determining module determines that the piece of additional content data can be displayed when the visual dwell area corresponding to the visual field orientation change range of the piece of additional content data covers the visual dwell area corresponding to the visual field orientation data of the second user and the dwell time corresponding to the visual field orientation change range reaches a predetermined threshold value.
The application example provides an application server, including: the sending module is used for sending data of the panoramic image to a plurality of application clients so as to respectively display the panoramic image; the determining module is used for receiving the visual field orientation data of a first user when receiving additional content data uploaded by any one of the application clients and published by the first user, and determining a visual field orientation change range of the first user within a period of time according to the visual field orientation data, wherein the period of time comprises a time point when the first user publishes the additional content data; the sending module is configured to send each piece of currently received additional content data and the respective view direction change range of the first user to the plurality of application clients, so that for each piece of additional content data, it is determined whether the additional content data can be displayed according to the view direction change range of the piece of additional content data and view direction data of a second user, and the determined displayable pieces of additional content data are displayed, where the second user is a user of the application client that receives the additional content data.
In some examples, for each piece of additional content data, the determining module further determines a presentation manner of the piece of additional content data according to the field-of-view orientation variation range of the additional content data;
the sending module further sends the display mode to the application clients so that the application clients display the additional content data according to the display mode.
The application example provides an application server, including: the sending module is used for sending data of the panoramic image to a plurality of application clients so as to respectively display the panoramic image; the determining module is used for receiving the visual field orientation data of a first user when receiving additional content data uploaded by any one of the application clients and published by the first user, and determining a visual field orientation change range of the first user within a period of time according to the visual field orientation data, wherein the period of time comprises a time point when the first user publishes the additional content data; the sending module executes the following operations aiming at each application client: for each piece of additional content data which is currently received, determining whether the application client can display the additional content data according to the visual field orientation change range of the piece of additional content data and visual field orientation data of a second user, wherein the second user is a user of the application client; and sending the determined additional content data which can be displayed by the application client to the application client so as to enable the application client to display the additional content data which can be displayed.
The present example provides a storage medium storing a data processing program for causing a computer to execute the method as described above.
By adopting the technical scheme provided by the application, the display effect of the panoramic image and the additional content in the VR system can be improved.
Drawings
In order to more clearly illustrate the technical solutions in the examples of the present application, the drawings needed to be used in the description of the examples are briefly introduced below, and it is obvious that the drawings in the following description are only some examples of the present application, and it is obvious for a person skilled in the art to obtain other drawings based on these drawings without inventive exercise.
FIG. 1 is a schematic diagram of a system architecture to which the present application relates;
FIG. 2 is a flow chart of a method according to an example of the present application;
FIG. 3 is a schematic three-dimensional virtual space according to an example of the present application;
FIG. 4 is a flow chart of a method of an example of the present application;
FIG. 5 is a flow chart of a method of an example of the present application;
FIG. 6 is a flow chart of a method of an example of the present application;
FIG. 7 is a message interaction diagram of an example of the present application;
FIG. 8 is a block diagram of an application client according to an example of the present application;
FIG. 9 is a block diagram of an application client according to an example of the present application;
FIG. 10 is a block diagram of an application server according to an example of the present application;
FIG. 11 is a block diagram of an application server according to an example of the present application; and
fig. 12 is a block diagram of an apparatus according to some examples of the present application.
Detailed Description
The technical solutions in the examples of the present application will be clearly and completely described below with reference to the drawings in the examples of the present application, and it is obvious that the described examples are only a part of the examples of the present application, and not all examples. All other examples, which can be obtained by a person skilled in the art without making any inventive step based on the examples in this application, are within the scope of protection of this application.
The panoramic image display method can be applied to VR systems. Fig. 1 shows a VR system 100, comprising: terminal device 101, VR device 102 and server 103. The terminal device 101 may run a VR client (i.e., VRAPP), and the VR device 102 may include a Controller (Controller) operable by a user and wearable devices (e.g., various VR head display devices, VR body sensing devices, etc.). The VR client in the terminal device 101 may perform information interaction with the VR device 102 to provide an immersive VR image for the user and complete a corresponding operation function. Specifically, the VR client can display corresponding VR image data for the user according to the position information and motion information of the user in the virtual space, which are provided by the wearable device, so as to bring immersive experience to the user; the VR client may also perform corresponding operations in response to instructions from the user operating the controller, such as: open/pause/close video files, make comments (e.g., barrage information), and so forth. Corresponding VR application server software (VR server or VR platform for short) can be run in the server 103, and the VR server can provide various VR panoramic image data for each VR client, such as: panoramic pictures, panoramic videos, VR games, etc. may perform corresponding processing in response to a service request from a VR client, such as: providing a page for selecting the panoramic video for the VR client, providing data of the VR video selected by the VR client for playing, publishing additional content (such as comment information) provided by the VR client, and the like.
Here, the terminal device 101 refers to a terminal device having a data calculation processing function, and includes but is not limited to a smart phone (with a communication module installed), a palm pc, a tablet pc, a smart television (SmartTV), and the like. These communication terminals are all installed with an operating system, including but not limited to: the Android operating system, the Symbian operating system, the Windows mobile operating system, the apple iPhone OS operating system, and the like. The VR Head-mounted display (HMD) includes a screen capable of displaying real-time images, and may further include an eye tracking module capable of obtaining a real-time motion trajectory of an eye of a person. The VR device may further include a tracking system and a calculation processing unit, where the tracking system is capable of tracking real-time position information and motion information of the user in the real three-dimensional space, and the calculation processing unit is capable of obtaining the real-time position information and motion information of the user from the tracking system, and calculating three-dimensional coordinates and motion information of the head of the user in the virtual three-dimensional space, view direction data of the user in the virtual three-dimensional space, and three-dimensional coordinates of the real-time gaze point of the user in the virtual three-dimensional space.
When a user wears the VR device to view a VR image, additional content of the VR image (such as making a comment, adding an annotation or a label, and the like) can be created by the controller, and in a VR system, such additional content as a comment is usually presented in the VR image in the form of bullet-screen information. When different users issue different barrage information for the same VR media content (e.g., a VR video), because the directions and viewing directions of the users are different, the VR images viewed by the users are also different (images corresponding to different viewing angles/viewpoints in the same VR video), that is, the VR images for which the barrage information issued by the users is different. This is because, a VR media content (e.g., a panoramic picture or a panoramic video) is formed by splicing a plurality of pictures or a plurality of groups of frame images taken at different orientations and angles in an actual scene, each picture or each group of frame images can be mapped to a specific position in a virtual space, and when providing a VR image to a user, a VR client determines position information and motion information of the user in the virtual space according to currently acquired position information and motion information of the user in the actual scene, and further provides the user with a picture or a frame image corresponding to the specific position in the virtual space. Therefore, VR image screen content seen by individual users may be different for the same VR media content. For example, when a user wears a VR head-mounted display device, a VR client plays a VR video at a default initial view, which is randomly assigned by a VR server, and may be different for different users. Further, when the head of the user moves, the playing angle of view of the VR video changes accordingly, so that the video pictures seen by the user are inconsistent with the video pictures seen by other users. Therefore, when a VR client plays a VR media content, if the bullet screen information is published, the bullet screen information is displayed in the VR image, and if different bullet screen information published for the VR media content by different users is displayed in the currently played VR image at the same time, a trouble is brought to the currently watching user, because there are many possible bullet screen information unrelated to the currently played VR image, so that the display effect of the VR image and the bullet screen information is seriously affected, the waste of processing resources is also caused, and the system performance is affected.
In order to solve the above problem, some examples of the present application provide a method for displaying a panoramic image, where the method may be applied to a first application client 101, that is, a VR client that can post additional content (such as comments) on a VR panoramic image. As shown in fig. 2, the method includes the following process 200:
step 201: the first application client receives data of the panoramic image from the application server and displays the panoramic image.
Step 202: the first application client receives input additional content data in response to the first user publishing the additional content.
Here, when the first user wants to publish the additional content, the handheld controller or a peripheral device such as a keyboard and a mouse of the terminal device running the first application client may be operated, and then an instruction for publishing the additional content may be issued to the first application client, so that the first application client starts to receive the additional content data (e.g., comment information) input by the first user. In a VR scenario, a user may issue the above instruction to publish additional content by pressing a specific button on a controller or performing a specific gesture operation on a handheld controller, may input comment information in a text form through a virtual keyboard presented in a VR image or may input comment information in a voice form through a microphone on a VR wearable device, and a VR client (i.e., a first application client) may receive the comment information in a text or voice form and may convert the comment information in a voice form into comment information in a text form. The VR client can send comment information in the form of text to be published by the user to the VR server through the network, and thus the VR server can provide the comment information to each VR client.
Step 203: the first application client sends the additional content data and the view direction data of the first user to the application server, so that the application server determines a view direction change range of the first user within a period of time according to the view direction data, wherein the period of time includes a time point when the first user issues the additional content data, and the application server or a second application client determines whether the second application client displays the additional content data or not according to the view direction change range and the current view direction data of the second user. Wherein the second application client may be determined to present the additional content data while presenting the panoramic image when the range of change in the orientation of the field of view of the first user covers (or otherwise encompasses) the orientation of the field of view represented by the second user's current orientation of view data. Here, the second user is a user of the second application client, and the first user is a user of the first application client.
In this example, the VR client (i.e., the first application client) publishing the additional content may obtain, according to real-time user data transferred by a wearable device (VR headset, glasses, or body sensing device) connected to the VR client, view orientation data of a current user (i.e., the first user) in the virtual space, and may send the view orientation data to the VR server. This field of view orientation data may include: three-dimensional position coordinates (x, y, z) of the user's head in virtual space (also called viewpoint coordinates), user's head, and the likeIn this case, based on a series of empirical data, such as an optimal viewing distance (e.g., 2.5 meters), a field of view width FOV (e.g., 90 degrees), parameters of a visual dwell plane of the user, including a length and a width of the plane, on which three-dimensional position coordinates of each point in the virtual space can be determined, based on three-dimensional position coordinates of the user's head in the virtual space, such as, as shown in FIG. 3, in a three-dimensional virtual space 300 based on a three-dimensional coordinate system XOY, position coordinates of the user's head P are (x, y, z), the field of view parameters of the user's head are (α, γ), based on which the user's visual dwell plane 301 is determined, that is, based on visual field orientation data of the first user at a point in time, a visual dwell plane (or so-called visual dwell region) can be determined, and then based on the first user's visual field orientation data, a change in the visual dwell plane can be determined, which the change in the direction of the first user's visual field of view orientation can be characterized, where the change in the first visual dwell range of the visual dwell plane is determined, the change in a time, which the change in the first visual dwell range of the visual field of view orientation of view0,y0,z0) And a viewing angle parameter (α)0,β0,γ0) Position coordinate of end point (x)1,y1,z1) And a viewing angle parameter (α)1β 1, γ 1), and may also include the position coordinates (x) of the intermediate pointi,yi,zi) And a viewing angle parameter (α)iβ i, γ i), the range of change of the determined visual dwell plane may be characterized by a set of visual dwell planes, the second user's current visual dwell plane may also be determined based on the current view orientation data of the other user (i.e., the second user). depending on whether some or all of the second user's current visual dwell plane is covered by the range of change of the first user's aforementioned visual dwell plane, it may be determined whether the first user's published additional content is presented when the second user's second application client is currently presenting a panoramic imageRather, the visual resting plane of the second user and the plane in the variation range of the visual resting plane of the first user achieve a certain degree of coincidence such that they are presented to the user in the virtual space in the panoramic image frame substantially the same. In this way, when part or all of the current visual retention plane of the second user is covered by the variation range of the above visual retention plane of the first user, it is described that the panoramic image picture (e.g. the picture content of the panoramic video or the picture content of the panoramic picture) seen by the first user when the first user delivers the additional content is substantially the same as the panoramic image picture currently seen by the second user, the additional content (e.g. the pop-up comment) delivered by the first user is associated with the panoramic image picture currently seen by the second user, and the additional content delivered by the first user can be presented on the panoramic image picture currently presented by the second application client of the second user.
In the above example, the position coordinates (i.e., viewpoint coordinates) of the user in the virtual space are taken into account, and in some VR systems (such as a cell phone VR box) the viewpoint coordinates of the user are not taken into account, then, in some examples, the view orientation data of step 203 may include view angle parameters (α, γ) of the head of the user0,β0,γ0) Viewing angle parameter of endpoint (α)1,β1,γ1) And may also include a view angle parameter for the intermediate point (α)i,βi,γi)。
In some examples, simplified viewing angle parameters may also be used in step 203, such as an angle α ' between a projection of the user's gaze on a horizontal plane and a coordinate axis (e.g., the X-axis of FIG. 3) on the horizontal plane, the viewing orientation data of step 203 may include a viewing angle parameter α 'The parameters of the surround may include a viewing angle variation range of the origin point viewing angle parameter α'0Viewing angle parameter α 'of terminal point'1Based on such viewing angle parameter, the above step 203 can also be implemented, when the viewing angle variation range of some additional content data covers the viewing angle of the current second user, that is, the visual staying area corresponding to the viewing angle orientation variation range in the additional content data is considered to cover the visual staying area corresponding to the viewing angle orientation data of the current second user.2Whether it belongs to the viewing angle parameter variation range α 'of the first user'0～α’1It may be determined whether the second application client is currently showing the panoramic image showing the additional content posted by the first user. Here, the second application client is an application client used by the second user.
In some examples, the period of time may be a period of time from a point in time before the point in time the first user posted the additional content to a point in time after the point in time, such as: the time for issuing the bullet screen is 9:00, and the time period can be 8: 59-9: 01. In summary, the length of this time period and the starting time point may be preset according to implementation requirements, such as: the set period length is n seconds or minutes, and the starting time point is m seconds or minutes before the time point of issuing the bullet screen.
In the above embodiment, based on the view direction data of each user, the client may be applied to select the currently published additional content for display when displaying the panoramic image, for a single user, the additional content published by other users is related to the picture content of the current panoramic image, the display of the additional content does not cause interference to the user, and a certain prompt and guidance can be provided. Such as: therefore, by adopting the scheme, the additional content of the bullet screen can be effectively utilized to guide the user to better watch the panoramic image, the watching trouble of the user caused by the change of the azimuth and the visual angle of the panoramic image is reduced, the blind operation (such as the operation of a controller, the turning head movement and the like) caused by the trouble is reduced, the display quantity of the additional content (the additional content published by a part of users in a targeted display part at the application client side) is also reduced, the processing resources of the system are saved, the processing efficiency is improved, and the overall performance of the system is improved.
In some examples, the method described above may further comprise: in response to the first user posting additional content, the first application client determines a range of change in the orientation of the first user's field of view for a period of time before and after a point in time at which the additional content data was posted. Wherein the view orientation data sent by the first application client to the application server includes the view orientation variation range.
In some examples, the method described above may further comprise: the first application client determines a dwell time of the first user within the range of view orientation changes for a period of time before and after a point in time at which the additional content data is published by the first user. Wherein the view orientation data sent by the first application client to the application server further comprises the dwell time; wherein the application server or the second application client determines whether the second application client presents the additional content data when presenting the panoramic image according to the field of view orientation change range, the dwell time, and the second user's current field of view orientation data. Here, two conditions are considered in determining whether additional content is to be presented: the visual field orientation change range of the first user covers the current visual field orientation of the second user, and the accumulated staying time of the visual field orientation change range of the second user and the first user reaches a preset threshold value. Determining to present the additional content posted by the first user when both of the conditions are satisfied. Here, the field of view orientation variation range of the first user includes one or more field of view orientations (each field of view orientation may correspond to one visual dwell plane), each field of view orientation has its own dwell time, and the cumulative dwell time of the field of view orientation variation range can be obtained by summing the dwell time of the respective field of view orientations. When the accumulated staying time reaches a predetermined threshold, it indicates that the first user has a relatively stable view direction within the view direction variation range, and is an effective viewing process, and the published additional content is also related to the panoramic image frame content corresponding to the view direction variation range. By further considering the condition of accumulated residence time, the additional content can be displayed more specifically, and the appropriate additional content can be selected for the second user to be displayed more accurately.
In the above example, the manner of sending the view orientation data by the first application client includes at least the following two ways:
1. when the first user posts additional content (such as a bullet screen), the first application client determines the dwell time of the first user in each view orientation during a period before and after the time point when the first user posts the additional content data, and sends the relevant data (including parameters characterizing the view orientation and the dwell time) of each view orientation of the first user during the period to the application server. In this way, the application server can determine the change range of the visual field orientation of the first user and the accumulated staying time in the time period according to the visual field orientation data which is sent by the first application client and is related to the additional content. Optionally, the first application client may also calculate the accumulated residence time of the first user in each view direction in this period of time, and then send the accumulated residence time to the application server, so that the application server may directly obtain the accumulated residence time of the first user in the view direction change range in this period of time.
2. At preset time intervals, the first application client sends the relevant data (such as parameters representing the current visual field orientation) of the current visual field orientation of the first user to the application server, and when the first user issues additional content, the additional content and the relevant data of the current visual field orientation of the first user are also sent to the application server. In this way, the application server can obtain the first user's full amount of view orientation data as well as behavioral data for posting additional content. In this case, the application server can calculate, for an additional content posted by the first user, a visual field orientation change range and a cumulative stay time of the first user for a period of time before and after a time point at which the additional content was posted.
In some examples, when a user is watching a VR video on an on-tape HMD head-up, the VR client performs playback initialization, and the VR server may randomly set the initial perspective parameters for the user (α)0,β0,γ0) And transmitting the video to the VR client, and starting to play the VR video by the VR client according to the visual angle parameter. Thus, when different users take the HMD for head display, their respective initial viewing angle parameters may be different, that is, their initial frames of VR video may be different. After the VR video starts playing, the VR client records the time information of the current user for publishing the bullet screen and the view orientation data when the bullet screen is published, including the position coordinate (x)i,yi,zi) And a viewing angle parameter (α)i,βi,γi) And sending the data of the bullet screen, the recorded view direction data and the time information to the VR server. In some examples, the VR client may record the view orientation data of the user every unit time (the specific value is not limited) and report the view orientation data to the VR server. In this way, the VR server may obtain, from each VR client, a total amount of user behavior data (view orientation change data and behavior data for delivering a bullet screen) based on the time series, and the VR server may determine, for each bullet screen, a view orientation change range and an accumulated stay time of the user delivering the bullet screen in a period before and after a time point at which the bullet screen is delivered, based on the total amount of user behavior data.
In some examples, when the VR video starts to be played, the VR server has not collected the above view orientation data related to the barrage, and at this time, when the VR server sets the initial view angle parameters for each user, the view angles of the users can be uniformly distributed. After the VR video is played for a period of time, the VR server has collected some view orientation data related to the bullet screen, and thus can determine a view orientation range (mainly, a view angle range) corresponding to the maximum bullet screen, so that when a user takes the HMD for head display, the VR client can obtain view parameters from the VR server when performing playback initialization, and the view parameters can be determined according to the view angle range corresponding to the maximum bullet screen, so that each VR client can first position a picture to a VR video picture associated with the maximum bullet screen when performing playback initialization, that is, obtain corresponding VR video image data according to the view angle parameters for playing, and thus, the VR server can guide VR video watching of each user by using the collected bullet screen data, so that the user can first see the video picture corresponding to the maximum bullet screen. That is to say, the VR server can analyze the view angle range in which the user is most interested according to the barrage data collected over a period of time, and then can guide the subsequent user to watch the VR video content in the view angle range.
In some instances, the user may not wear the HMD head display, the VR client may determine that the user is to view a panoramic image, such as a VR video, on the 2D plane, and may enter an automatic play mode, and the VR client may automatically adjust the perspective of the currently presented panoramic image according to predetermined perspective change rules without adjusting the perspective and/or viewpoint of the currently presented panoramic image in response to the view orientation data (e.g., user position coordinate changes, perspective change parameters, etc.) conveyed by the HMD head display worn by the user. For the user, the image frame that he sees is moving in a certain direction, for example, from right to left. Because the VR client can obtain the view orientation data (for example, the user view angle parameter when a certain bullet screen is published) corresponding to each bullet screen from the VR server, the VR client can make the playing view angle of the current image change along with the movement of the currently displayable bullet screen, or make the currently displayable bullet screen move along with the change of the playing view angle of the current image, so that the bullet screen seen by the user moves along with the corresponding image frame synchronously. In VR video broadcast scene, the user need not manual adjustment visual angle, can realize that the video broadcast visual angle of VR carries out synchronous motion along with the barrage.
The present application example further provides a method for presenting a panoramic image, where the method may be applied to an application client (such as the second application client in the foregoing example), that is, a VR client that can present additional content (such as comments) for a VR panoramic image. As shown in fig. 4, the method includes the following process 400:
step 401: the application client receives data of the panoramic image from the application server and displays the panoramic image.
Step 402: the application client receives at least one piece of additional content data from the application server and the visual field orientation change range of a first user publishing each piece of additional content data in a period of time; wherein, for a piece of additional content data, the period of time includes a point in time at which the piece of additional content data is published.
Step 403: for each piece of additional content data, the application client determines whether the additional content data can be presented according to the field of view orientation variation range of the piece of additional content data and the field of view orientation data of the second user. The second user is a user of the application client.
Step 404: and the application client displays the determined displayable pieces of additional content data.
In this example, the application client may receive the additional content posted by each first user and the viewing direction variation range when the panoramic image is displayed, and determine which one or more of the additional content posted by the first users is displayable according to the viewing direction data of the second user and the viewing direction variation range of each first user. By adopting the scheme, the additional content aiming at the current panoramic image picture content can be selected for displaying.
In some examples, the method may further comprise: when the additional content data and the visual field orientation change range thereof are received from the application server, the application client further receives a display mode of the additional content data; wherein the additional content data is presented in the presentation manner.
In some examples, the application server may classify the behavior of each first user for delivering the bullet screen according to the collected view orientation data before and after each first user delivers the bullet screen:
first, stationary barrage: before and after the user publishes the bullet screen, the visual field orientation change (including the horizontal direction or the vertical direction) of the user is smaller than a certain threshold value, namely the user basically does not turn the head. I.e., to indicate that the quality area of the video presented to the user has not changed, this bullet screen is simply a bullet screen with a fixed field of view facing inward.
Second, directional bullet screen: before and after the user publishes the bullet screen, the visual field orientation change (including the horizontal direction or the vertical direction) of the user is larger than a certain threshold value, namely, the user generates obvious turning behavior. I.e. to indicate that the quality area of the video presented to the user has changed. For example, the following three cases:
in the first situation, the user does not rotate the head basically before the bullet screen is published, and rotates the head after the bullet screen is published.
And in the second situation, the user rotates the head before the bullet screen is published, and the user basically does not rotate the head after the bullet screen is published.
And in the third case, the user always turns around before the bullet screen is published and after the bullet screen is published.
Through the analysis, the application server can determine the display mode of each bullet screen, such as: for the static barrage of the first type, the display in a static mode can be determined, and for the directional barrage of the second type, the display in a dynamic mode can be determined. The application client displays the bullet screen according to a display mode indicated by the application server, wherein when the display mode indicates that the bullet screen is static, the bullet screen is displayed in a static mode, such as: the bullet screen is displayed at a fixed position, and further the bullet screen can disappear after being displayed for a period of time, or the bullet screen disappears when a second user turns around; when the display mode indicates that the bullet screen is directional, determining a pointing direction, and displaying the bullet screen in a dynamic mode according to the pointing direction, for example: the bullet screen can be displayed in a fixed position or in a moving mode along the pointing direction when the arrow is displayed. By adopting the method to analyze and display the bullet screen, more accurate watching prompts can be given to users, such as: a bullet screen 'front high-energy early warning' is displayed in a horror panoramic video picture in a moving mode from right to left, and a user can know that a horror picture is about to appear on the left; for another example: a bullet screen is statically displayed in a panoramic video picture, a user can know that the bullet screen is related to the current picture, when the user turns the head and changes the visual direction, the video picture can be correspondingly changed, and correspondingly, the bullet screen can disappear, so that the interference to the user is reduced.
In some examples, the determining the pointing direction includes: determining the pointing direction according to the field of view orientation variation range of the received plural pieces of additional content data. Such as: when the application client selects some barrages to be displayed, a pointing direction is uniformly determined for the dynamically displayed barrages, and the pointing direction is determined according to the view direction change data corresponding to the plurality of barrages. Optionally, the respective pointing directions may be determined according to the respective viewing direction variation ranges corresponding to the respective pieces of additional content data, that is, the pointing directions of the displayed dynamic barrages may be different.
In some examples, the analyzing of the view orientation data before and after the first user posts the additional content (e.g., the barrage) to classify the behavior of each first user in posting the additional content may be implemented on the application client side, and the method may further include: the application client determines a display mode of the additional content data according to the view direction change range corresponding to the additional content data; wherein the additional content data is presented in the presentation manner.
In some examples, the determining the pointing direction from the field of view orientation variation range of the received pieces of additional content data includes: determining a moving direction of the visual field of a majority of the first users as the pointing direction according to the visual field orientation variation range of a plurality of pieces of additional content data. Here, the direction of movement of the visual field of the majority of the first users may be selected as the pointing direction, or the direction of movement of the visual field of the majority of the first users corresponding to the determined additional contents that can be presented may be selected. Therefore, the visual field moving direction of most of the users in the user group can be adopted to guide the users to watch, and the guiding accuracy and effectiveness in the bullet screen display can be further improved.
In some examples, the determining the pointing direction from the field of view orientation variation range of the received pieces of additional content data includes: determining a moving direction of a field of view of a specified type of the first user as the pointing direction according to the field of view orientation variation range of a plurality of pieces of additional content data. Here, the direction of the movement of the field of view of the user-level higher may be selected from the first user, and the user-level higher may be determined according to user information registered by the user in an application platform (e.g., a panoramic video platform or a virtual reality platform or a social application platform for logging in a social application account used by the VR client), accumulated user-level points, and the like. Such as: a direction of view movement of a opinion leader (KOL) user or a VIP user may be selected, and so forth. Therefore, the visual field moving direction of the user with influence in the user group can be utilized to guide the user to watch, more interesting applications in the panoramic image watching process can be realized, the display effect of the panoramic image is improved, and the watching experience of the user is improved.
In some examples, the determining whether the additional content data can be presented according to the range of viewing orientation change of the piece of additional content data and the viewing orientation data of the second user includes: when the visual dwell area corresponding to the field of view orientation variation range of the piece of additional content data covers the visual dwell area corresponding to the field of view orientation data of the second user, the application client determines that the piece of additional content data can be presented.
In some examples, further comprising: when receiving the additional content data and the viewing orientation variation range thereof from the application server, the application client further receives a stay time of the viewing orientation variation range; when the visual dwell area corresponding to the view direction change range of the piece of additional content data covers the visual dwell area corresponding to the view direction data of the second user, and the dwell time corresponding to the view direction change range reaches a predetermined threshold, the application client determines that the piece of additional content data can be presented.
Here, the specific method for determining whether the additional content data can be displayed has been described in detail above, and is not described in detail here.
Some examples of the present application also provide a method for controlling presentation of a panoramic image, which may be applied to an application server 103 (e.g., a VR server), as shown in fig. 5, and the method includes the following processes 500:
step 501: and the application server sends the data of the panoramic image to a plurality of application clients so as to respectively display the panoramic image.
Here, the application server may provide a panoramic image service, such as a VR video online play service, for each application client, which may provide data of a corresponding panoramic image for the application client in response to a request of the application client. For a media content (a VR video or a panoramic picture), which may be accessed by multiple application clients simultaneously, the application server sends the panoramic image data of the media content to each application client for presentation.
Step 502: when additional content data uploaded by a first user and uploaded by any application client in a plurality of application clients are received, an application server receives visual field orientation data of the first user, and determines a visual field orientation change range of the first user within a period of time according to the visual field orientation data, wherein the period of time comprises a time point when the first user releases the additional content data.
Step 503: and respectively sending the currently received additional content data and the respective view direction change range of the first user to the plurality of application clients, so that for each additional content data, determining whether the additional content data can be displayed according to the view direction change range of the additional content data and the view direction data of the second user, and displaying the determined displayable additional content data. Here, the second user refers to a user of the application client that receives the additional content data.
In this example, each application client may obtain, from the application server, each piece of currently published additional content data and a range of change in the viewing direction of the publishing user of each piece of additional content data, and each application client may screen, according to the received pieces of information and the viewing direction data of its own current user, displayable additional content data from the received pieces of additional content data. That is to say, the application client can screen out the additional content (such as the barrage) suitable for being displayed within the viewing direction range of the user, so that the display effect of the additional content can be improved, the displayed additional content is more related to the current panoramic image picture content, the displayed additional content is more readable for the user, the viewing trouble brought to the user by the change of the orientation and the viewing angle of the panoramic image is reduced, the blind operation (such as the operation of a controller, the turning movement and the like) performed by the user due to the trouble is reduced, and the display amount of the additional content (the additional content released by a part of users is displayed in a targeted manner at the application client side) is also reduced, so that the processing resources of the system are saved, the processing efficiency is improved, and the overall performance of the system is improved.
In some examples, the method further comprises: for each piece of additional content data, the application server determines the display mode of the piece of additional content data according to the view direction change range of the additional content data, and sends the display mode to the plurality of application clients so that the application clients display the piece of additional content data according to the display mode.
Some examples of the present application further propose a method for controlling presentation of a panoramic image, which can be applied to the application server 103, as shown in fig. 6, the method includes the following processing 600:
step 601: and the application server sends the data of the panoramic image to a plurality of application clients so as to respectively display the panoramic image.
Step 602: when additional content data uploaded by a first user and uploaded by any application client in the plurality of application clients are received, an application server receives visual field orientation data of the first user, and determines a visual field orientation change range of the first user within a period of time according to the visual field orientation data, wherein the period of time comprises a time point when the first user releases the additional content data.
Step 603: the application server executes the steps 604-605 for each application client:
step 604: and for each piece of additional content data which is currently received, the application server determines whether the application client can show the additional content data according to the visual field orientation change range of the piece of additional content data and the visual field orientation data of the second user. The second user is a user of the application client.
Step 605: and the application server sends the determined additional content data which can be displayed by the application client to the application client so as to enable the application client to display the additional content data which can be displayed.
In the above example, the application server may, for each application client, screen out the additional content that can be presented by the application client from the currently published additional content data, and send the additional content to the application client, so that similar technical effects to those in the foregoing example can also be achieved.
Fig. 7 illustrates a message interaction diagram of an example of the present application. In this example, the application client is a VR client, the application server is a VR server, the additional content is a barrage, and the panoramic image is a VR video. Fig. 7 shows only one VR client and VR server, and in an actual scenario, for the presentation of one VR video, multiple VR clients perform message interaction with the VR server, and each VR client may perform the processing shown in fig. 7. As shown in fig. 7, for a currently playing VR video, the interaction process 700 of the VR client and the VR server is as follows:
step 701-702: and recording the view direction change data in a period of time before and after the current time point when the user issues the bullet screen when the VR client issues the bullet screen, and sending the view direction change data and the data of the bullet screen to the VR server. Further, the VR server collects the visual field orientation change data and the barrage data of each user from each VR client. Here, one VR client corresponds to one user who logs in to the VR client using a user account of the VR application to view VR video.
Step 703: the VR server obtains data for a bullet screen-view orientation-time series in which there is a series of data records for each published bullet screen, including: the bullet screen content, the view direction change range in a period of time before and after the time point of delivering the bullet screen, and the accumulated residence time corresponding to the view direction change range.
Step 704: the VR server sends the bullet screen-view orientation-time series of data to each VR client.
Step 705: after receiving the data of the bullet screen-view direction-time sequence, the VR client may determine one or more bullet screens and display modes (static or directional) to be displayed by the client according to the data records of the bullet screens in the sequence and the view direction data of the current user.
The specific implementation of the above steps has been described in detail, and is not described herein again.
Some examples of the present application propose an application client, as shown in fig. 8, the application client 800 includes the following modules:
and a panorama display module 801 for receiving data of the panoramic image from the application server and displaying the panoramic image.
The receiving module 802 receives input additional content data in response to a first user publishing additional content.
The sending module 803 sends the additional content data and the view direction data of the first user to the application server, so that the application server determines a view direction change range of the first user within a period of time according to the view direction data, where the period of time includes a time point when the first user issues the additional content data, so that the application server or a second application client determines whether the second application client displays the additional content data when displaying the panoramic image according to the view direction change range and current view direction data of the second user. The second user is a user of the second application client.
In some examples, the application client 800 may further include:
a determining module 804, responsive to the posting of the additional content, determines a range of change in the orientation of the first user's field of view over a period of time before and after a point in time at which the additional content data was posted. Wherein the view orientation data sent by the sending module 803 to the application server includes the view orientation variation range.
In some examples, the determining module 804 may further determine a dwell time of the first user within the range of view orientation changes for a period of time before and after a point in time at which the additional content data is published. Wherein the view orientation data sent by the sending module 803 to the application server further comprises the dwell time; and the application server or the second application client determines whether the second application client displays the additional content data when displaying the panoramic image according to the field of view orientation change range, the stay time and the current field of view orientation data of the second user.
In some examples, the sending module 803 sends the view orientation data including the first user's current view orientation parameter to the application server at intervals.
The application client in the above example can provide the additional content published by the user and the related view orientation data to the application server, so that the application server or another application client can determine the additional content that can be displayed by the other application client, and the display effect of the additional content can be improved.
Some examples of the present application also propose an application client, as shown in fig. 9, the application client 900 may include the following modules:
a panorama displaying module 901, which receives data of the panoramic image from the application server and displays the panoramic image.
A receiving module 902, configured to receive at least one piece of additional content data from the application server and a view orientation change range of a first user publishing each piece of additional content data in a period of time; wherein, for a piece of additional content data, the period of time includes a point in time at which the piece of additional content data is published.
The determining module 903 determines, for each piece of additional content data, whether the additional content data can be presented according to the field of view orientation variation range of the piece of additional content data and the field of view orientation data of the second user. The panorama displaying module 901 displays the determined displayable pieces of additional content data. The second user is a user of the application client.
In some examples, the receiving module 902, when receiving the additional content data and the range of orientation of the field of view thereof from the application server, may further receive a presentation of the additional content data. And a panorama display module 901 configured to display the additional content data according to the display mode.
In some examples, the determining module 903 further determines a presentation manner of the additional content data according to the field of view orientation variation range of the additional content data; and a panorama display module 901 configured to display the additional content data according to the display mode.
In some examples, the determining module 903 determines that the piece of additional content data can be presented when the visual dwell area corresponding to the field of view orientation variation range of the piece of additional content data covers the visual dwell area corresponding to the field of view orientation data of the second user.
In some examples, the receiving module 902, when receiving the additional content data and its range of view orientation changes from the application server, further receives a dwell time for the range of view orientation changes; the determining module 903 determines that the piece of additional content data can be displayed when the visual staying area corresponding to the field of view orientation variation range of the piece of additional content data covers the visual staying area corresponding to the field of view orientation data of the second user and the staying time corresponding to the field of view orientation variation range reaches a predetermined threshold.
The application client in the above example may determine the additional content data that can be currently displayed according to the additional content data acquired from the application server and the variation range of the orientation of the field of view thereof, and may further select the additional content in a targeted manner for display based on the orientation of the field of view of the user.
Some examples of the present application propose an application server, as shown in fig. 10, the application server 1000 may include the following modules:
the sending module 1001 sends data of the panoramic image to a plurality of application clients, so that the application clients respectively display the panoramic image.
The determining module 1002, when receiving additional content data published by a first user and uploaded by any one of the plurality of application clients, receives view orientation data of the first user, and determines a change range of the view orientation of the first user within a period of time according to the view orientation data, where the period of time includes a time point when the first user publishes the additional content data.
The sending module 1001 sends each piece of currently received additional content data and the respective view direction change range of the first user to the plurality of application clients, so that for each piece of additional content data, it determines whether the additional content data can be displayed according to the view direction change range of the piece of additional content data and the view direction data of the second user, and displays each piece of determined displayable additional content data. The second user is a user of the application client that receives the attachment content data.
In some examples, for each piece of additional content data, the determining module 1002 may further determine a presentation manner of the piece of additional content data according to the field of view orientation variation range of the additional content data; the sending module 1001 further sends the display mode to the plurality of application clients, so that the application clients display the additional content data in the display mode.
The application server in the above example may collect each additional content data and view direction data from each application client, determine a view direction change range corresponding to each additional content data, and distribute the view direction change range to each application client, so that each application client can select displayable additional content data based on the view direction change range, thereby improving the additional content display effect.
The present example provides an application server, and as shown in fig. 11, the application server 1100 may include the following modules:
the sending module 1101 sends data of the panoramic image to a plurality of application clients so that the application clients respectively display the panoramic image.
The determining module 1102 is configured to receive, when additional content data published by a first user and uploaded by any one of the plurality of application clients is received, view orientation data of the first user, and determine, according to the view orientation data, a change range of the view orientation of the first user within a period of time, where the period of time includes a time point at which the first user publishes the additional content data.
A sending module 1101, configured to perform the following operations for each application client: for each piece of additional content data which is currently received, determining whether the application client can display the additional content data according to the visual field orientation change range of the piece of additional content data and visual field orientation data of a second user, wherein the second user is a user of the application client; and sending the determined additional content data which can be displayed by the application client to the application client so as to enable the application client to display the additional content data which can be displayed.
The application server in the above example may collect each additional content data and view direction data from each application client, determine a view direction change range corresponding to each additional content data, and determine additional content data that each application client can display, thereby improving an additional content display effect.
The specific implementation principle of the functions of the above modules has been described in the foregoing, and is not described herein again.
In addition, the panorama image display method, the application client, the application server and the modules thereof in the examples of the present application may be integrated into one processing unit, or each module may exist alone physically, or two or more devices or modules may be integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.
In some examples, the application client and/or application server described above may be run on various computing devices that may perform panoramic image data processing and loaded in memory of the computing device.
Fig. 12 is a block diagram showing the components of a computing device on which the application client and/or the application server are located. As shown in fig. 12, the computing device 1200 includes one or more processors (CPUs) 1202, a communications module 1204, a memory 1206, a user interface 1210, and a communication bus 1208 interconnecting these components.
The processor 1202 can receive and transmit data via the communication module 1204 to enable network communication and/or local communication.
The user interface 1210 includes one or more output devices 1212 including one or more speakers and/or one or more visual displays. The user interface 1210 also includes one or more input devices 1214, including, for example, a keyboard, a mouse, a voice command input unit or microphone, a touch screen display, a touch sensitive tablet, a gesture capture camera or other input buttons or controls, and the like.
The memory 1206 may be a high-speed random access memory such as DRAM, SRAM, DDR RAM, or other random access solid state memory devices; or non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices.
The memory 1206 stores a set of instructions executable by the processor 1202, including:
an operating system 1216 including programs for handling various basic system services and for performing hardware related tasks;
the application 1218 includes various programs for implementing panoramic image data processing, and such programs can implement the processing flows in the above examples, such as the application client 800 shown in fig. 8, the application client 900 shown in fig. 9, the application server 1000 shown in fig. 10, and/or the application server 1100 shown in fig. 11.
In some examples, the application client 800 may include at least one of the various modules 801-804 shown in FIG. 8, and each of the modules 801-804 may store machine executable instructions. The processor 1202 may implement the functions of at least one of the modules 801-804 by executing machine-executable instructions of at least one of the modules 801-804 in the memory 1206.
In some examples, the application client 900 can include at least one of the modules 901-903 shown in FIG. 9, and each of the modules 901-903 can store machine executable instructions. The processor 1202 may further implement the functions of at least one of the modules 901 to 903 by executing machine executable instructions of at least one of the modules 901 to 903 in the memory 1206.
In some examples, the application server 1000 may include at least one of the modules 1001-1002 shown in FIG. 10, at least one of the modules 1001-1002 may store machine executable instructions. The processor 1202 may further implement the functions of at least one of the modules 1001-1002 by executing the machine-executable instructions of at least one of the modules 1001-1002 in the memory 1206.
In some examples, the application server 1100 may include at least one of the modules 1101-1102 shown in FIG. 11, at least one of the modules 1101-1102 may store machine-executable instructions. The processor 1202 may further implement the functions of at least one of the modules 1101-1102 by executing machine-executable instructions of at least one of the modules 1101-1102 in the memory 1206.
In addition, each of the examples of the present application may be realized by a data processing program executed by a data processing apparatus such as a computer. It is clear that the data processing program constitutes the invention. Further, the data processing program, which is generally stored in one storage medium, is executed by directly reading the program out of the storage medium or by installing or copying the program into a storage device (such as a hard disk and/or a memory) of the data processing device. Such a storage medium therefore also constitutes the present invention. The storage medium may use any type of recording means, such as a paper storage medium (e.g., paper tape, etc.), a magnetic storage medium (e.g., a flexible disk, a hard disk, a flash memory, etc.), an optical storage medium (e.g., a CD-ROM, etc.), a magneto-optical storage medium (e.g., an MO, etc.), and the like.
The present application therefore also discloses a non-volatile storage medium having stored therein a data processing program for executing any one of the examples of the method of the present application.
In addition, the method steps described in this application may be implemented by hardware, for example, logic gates, switches, Application Specific Integrated Circuits (ASICs), programmable logic controllers, embedded microcontrollers, and the like, in addition to data processing programs. Such hardware capable of implementing the methods described herein may also constitute the present application.
The above description is only a preferred example of the present application and should not be taken as limiting the present application, and any modifications, equivalents, improvements and the like made within the spirit and principle of the present application should be included in the scope of the present application.

Claims (37)

1. A method for displaying a panoramic image is applied to a first application client, and comprises the following steps:
receiving data of a panoramic image from an application server and displaying the panoramic image;
receiving input additional content data in response to an operation of a first user to publish additional content;
determining a dwell time of the first user within a range of view orientation changes corresponding to a period of time before and after a point in time at which the additional content is published;
sending the additional content data and view orientation data of the first user to the application server, the view orientation data including the dwell time; so that the application server determines a variation range of the field of view orientation of the first user in the period of time according to the field of view orientation data, wherein if the application server or a second application client determines that the variation range of the field of view orientation covers the current field of view orientation of a second user, and the dwell time of the first user in the variation range of the field of view orientation reaches a predetermined threshold, it is determined that the second application client presents the additional content data while presenting the panoramic image, and the second user is a user of the second application client.
2. The method of claim 1, further comprising: in response to the posting of the additional content, determining a range of change in orientation of the first user's field of view within a period of time before and after a point in time at which the additional content data was posted;
wherein the view orientation data sent to the application server includes the view orientation variation range.
3. The method of claim 1, further comprising: the application server determines a display mode of the additional content according to the visual field orientation data of the first user;
wherein if the change of the orientation of the first user's field of view is less than a threshold value within a period of time before and after the point in time at which the first user posted the additional content, the presentation style is determined to be a static presentation style;
and if the change of the orientation of the visual field of the first user is larger than a threshold value in a period of time before and after the time point of the first user publishing the additional content, determining that the showing mode is a dynamic showing mode.
4. The method of claim 3, wherein the second application client presents the additional content at a fixed location when the presentation style is a static presentation style;
and when the display mode is a dynamic display mode, the second application client moves to display the additional content according to the pointing direction.
5. The method of claim 1, wherein the first user's field of view orientation data is sent to the application server at intervals, the field of view orientation data being current field of view orientation data, the current field of view orientation data including data related to the first user's current field of view orientation.
6. The method of claim 1, wherein the view orientation data is transmitted to the application server when the first user posts additional content, the view orientation data comprising: respective view orientation data for the first user over the period of time, the respective view orientation data including data relating to the first user in the respective view orientation.
7. A method for displaying panoramic images is applied to an application client, and comprises the following steps:
receiving data of a panoramic image from an application server and displaying the panoramic image;
receiving at least one piece of additional content data from the application server and a range of change in a direction of a field of view of a first user publishing each piece of additional content data for a period of time before and after a point in time at which the additional content data is published;
receiving, from the application server, a dwell time of the first user over the period of time within the range of view orientation changes;
for each piece of additional content data, if the variation range of the visual field orientation of the piece of additional content data covers the visual field orientation of a second user, and the dwell time of the first user in the variation range of the visual field orientation reaches a preset threshold value, determining that the additional content data can be presented, wherein the second user is a user of the application client;
and displaying the determined displayable pieces of additional content data.
8. The method of claim 7, further comprising:
when the additional content data and the visual field orientation change range thereof are received from the application server, further receiving a display mode of the additional content data;
wherein the additional content data is presented in the presentation manner.
9. The method of claim 7, further comprising:
determining a display mode of the additional content data according to the view direction change range of the additional content data;
wherein the additional content data is presented in the presentation manner.
10. The method of claim 8 or 9,
when the presentation mode indicates that the additional content is static, presenting the additional content data in a static mode;
and when the display mode indicates that the additional content is directional, determining a pointing direction, and displaying the additional content data in a dynamic mode according to the pointing direction.
11. The method of claim 10, wherein the determining a pointing direction comprises: determining the pointing direction according to the field of view orientation variation range of the received plural pieces of additional content data.
12. The method of claim 11, wherein the determining the pointing direction from the field of view orientation variation range of the received plurality of pieces of additional content data comprises:
determining a moving direction of the visual field of a majority of the first users as the pointing direction according to the visual field orientation variation range of a plurality of pieces of additional content data.
13. The method of claim 11, wherein the determining the pointing direction from the field of view orientation variation range of the received plurality of pieces of additional content data comprises:
determining a moving direction of a field of view of a specified type of the first user as the pointing direction according to the field of view orientation variation range of a plurality of pieces of additional content data.
14. A display control method of a panoramic image is applied to an application server, and comprises the following steps:
sending data of a panoramic image to a plurality of application clients so that the application clients respectively display the panoramic image;
when additional content data uploaded by a first user and published by any application client in the plurality of application clients are received, visual field orientation data of the first user are received, and a visual field orientation change range of the first user in a period of time before and after a time point of publishing the additional content data is determined according to the visual field orientation data; the field of view orientation data further includes a dwell time of the first user over the range of field of view orientation changes over the period of time;
and respectively sending the currently received additional content data and the respective view direction change range and the dwell time of the first user to the plurality of application clients, so that for each additional content data, if the view direction change range of the additional content data is determined to cover the view direction of a second user, and the dwell time of the first user in the view direction change range reaches a predetermined threshold, determining that the additional content data can be displayed, and displaying the determined displayable additional content data, wherein the second user is the user of the application client who receives the additional content data.
15. The method of claim 14, further comprising:
and for each piece of additional content data, determining a display mode of the piece of additional content data according to the view direction change range of the additional content data, and sending the display mode to the plurality of application clients so as to display the piece of additional content data according to the display mode.
16. The method of claim 15, wherein the presentation is determined to be a static presentation if the orientation of the first user's field of view changes by less than a threshold amount over a period of time before and after the point in time at which the first user posted the additional content;
and if the change of the orientation of the visual field of the first user is larger than a threshold value in a period of time before and after the time point of the first user publishing the additional content, determining that the showing mode is a dynamic showing mode.
17. The method of claim 16, wherein the plurality of application clients present the additional content at a fixed location when the presentation is a static presentation;
and when the display mode is a dynamic display mode, the plurality of application clients move to display the additional content according to the pointing direction.
18. A display control method of a panoramic image is applied to an application server, and comprises the following steps:
sending data of a panoramic image to a plurality of application clients so that the application clients respectively display the panoramic image;
when additional content data uploaded by a first user and published by any application client in the plurality of application clients are received, visual field orientation data of the first user are received, and a visual field orientation change range of the first user in a period of time before and after a time point of publishing the additional content data is determined according to the visual field orientation data; the field of view orientation data further includes a dwell time of the first user over the range of field of view orientation changes over the period of time;
the following operations are performed for each application client:
for each piece of additional content data which is currently received, if the visual field orientation change range of the piece of additional content data covers the visual field orientation of a second user and the dwell time of the first user in the visual field orientation change range reaches a preset threshold value, determining that the application client can display the additional content data, wherein the second user is the user of the application client;
and sending the determined additional content data which can be displayed by the application client to the application client so as to enable the application client to display the additional content data which can be displayed.
19. The method of claim 18, further comprising: the application server determines a display mode of the additional content according to the visual field orientation data of the first user; wherein if the change of the orientation of the first user's field of view is less than a threshold value within a period of time before and after the point in time at which the first user posted the additional content, the presentation style is determined to be a static presentation style;
and if the change of the orientation of the visual field of the first user is larger than a threshold value in a period of time before and after the time point of the first user publishing the additional content, determining that the showing mode is a dynamic showing mode.
20. The method of claim 19, wherein the plurality of application clients present the additional content at a fixed location when the presentation is a static presentation;
and when the display mode is a dynamic display mode, the plurality of application clients move to display the additional content according to the pointing direction.
21. An application client, comprising:
the panoramic display module receives data of the panoramic image from the application server and displays the panoramic image;
the receiving module is used for responding to the operation of publishing the additional content by the first user and receiving the input additional content data;
a determining module, for determining the stay time of the first user in the view orientation change range corresponding to a period of time before and after the time point of publishing the additional content;
a sending module that sends the additional content data and the view orientation data of the first user to the application server, the view orientation data including the dwell time; so that the application server determines a variation range of the field of view orientation of the first user in the period of time according to the field of view orientation data, wherein if the application server or a second application client determines that the variation range of the field of view orientation covers the current field of view orientation of a second user, and the dwell time of the first user in the variation range of the field of view orientation reaches a predetermined threshold, it is determined that the second application client presents the additional content data while presenting the panoramic image, and the second user is a user of the application client.
22. The application client of claim 21, wherein:
a determination module further responsive to the operation of posting additional content for determining a range of change in orientation of the first user's field of view over a period of time before and after a point in time at which the additional content data was posted;
wherein the view orientation data sent by the sending module to the application server includes the view orientation variation range.
23. The application client according to claim 21, wherein the application server further determines a presentation of the additional content based on the first user's view orientation data;
wherein if the change of the orientation of the field of view of the first user is less than a threshold value within a period of time before and after the point in time when the first user delivers the additional content, the application server determines that the presentation mode is a static presentation mode;
and if the change of the orientation of the visual field of the first user is larger than a threshold value in a period of time before and after the time point of the first user publishing the additional content, the application server determines that the display mode is a dynamic display mode.
24. The application client according to claim 23, wherein the second application client presents the additional content at a fixed location when the presentation is a static presentation;
and when the display mode is a dynamic display mode, the second application client moves to display the additional content according to the pointing direction.
25. The application client according to claim 21, wherein the sending module sends the first user's view orientation data to the application server at intervals, the view orientation data being current view orientation data, the current view orientation data including data related to the first user's current view orientation.
26. The application client of claim 21, wherein the sending module sends the view orientation data to the application server when the first user posts additional content, the view orientation data comprising: respective view orientation data for the first user over the period of time, the respective view orientation data including data relating to the first user in the respective view orientation.
27. An application client, comprising:
the panoramic display module receives data of the panoramic image from the application server and displays the panoramic image;
a receiving module that receives at least one piece of additional content data from the application server and a range of change in a direction of a field of view of a first user publishing each piece of additional content data for a period of time before and after a point in time when the additional content data is published; receiving, from the application server, a dwell time of the first user over the period of time within the range of view orientation changes;
a determining module, configured to determine, for each piece of additional content data, that the additional content data can be presented if the variation range of the orientation of the field of view of the piece of additional content data covers the orientation of the field of view of a second user, and the retention time of the first user within the variation range of the orientation of the field of view reaches a predetermined threshold, the second user being a user of the application client;
and the panoramic display module displays the determined displayable additional content data.
28. The application client of claim 27,
the receiving module is used for further receiving the display mode of the additional content data when the additional content data and the visual field orientation change range thereof are received from the application server;
and the panoramic display module displays the additional content data according to the display mode.
29. The application client of claim 27,
the determining module is used for further determining a display mode of the additional content data according to the view direction change range of the additional content data;
and the panoramic display module displays the additional content data according to the display mode.
30. An application server, comprising:
the sending module is used for sending data of the panoramic image to a plurality of application clients so as to respectively display the panoramic image;
the determining module is used for receiving the visual field orientation data of a first user when receiving additional content data uploaded by any one of the application clients and published by the first user, and determining a visual field orientation change range of the first user in a period of time before and after a time point of publishing the additional content data according to the visual field orientation data; the field of view orientation data further includes a dwell time of the first user over the range of field of view orientation changes over the period of time;
the sending module is used for sending the currently received additional content data and the respective view direction change range and the dwell time of the first user to the plurality of application clients respectively so that for each additional content data, if the view direction change range of the additional content data is determined to cover the view direction of a second user and the dwell time of the first user in the view direction change range reaches a preset threshold value, the additional content data can be displayed, and the determined displayable additional content data are displayed, wherein the second user is the user of the application client receiving the additional content data.
31. The application server of claim 30, wherein, for each additional content data,
the determining module is further used for determining the display mode of the additional content data according to the view direction change range of the additional content data;
the sending module further sends the display mode to the application clients so that the application clients display the additional content data according to the display mode.
32. The application server of claim 31, wherein the determination module further determines that the presentation is a static presentation if the orientation of the first user's field of view changes by less than a threshold amount over a period of time before and after a point in time when the first user posted the additional content;
the determination module further determines that the presentation mode is a dynamic presentation mode if a change in orientation of the field of view of the first user is greater than a threshold for a period of time before and after a point in time when the first user posted the additional content.
33. The application server of claim 32, wherein the plurality of application clients present the additional content at a fixed location when the presentation is a static presentation;
and when the display mode is a dynamic display mode, the plurality of application clients move to display the additional content according to the pointing direction.
34. An application server, comprising:
the sending module is used for sending data of the panoramic image to a plurality of application clients so as to respectively display the panoramic image;
the determining module is used for receiving the visual field orientation data of a first user when receiving additional content data uploaded by any one of the application clients and published by the first user, and determining a visual field orientation change range of the first user in a period of time before and after a time point of publishing the additional content data according to the visual field orientation data; the field of view orientation data further includes a dwell time of the first user over the range of field of view orientation changes over the period of time;
the sending module executes the following operations aiming at each application client: for each piece of additional content data which is currently received, if the visual field orientation change range of the piece of additional content data and the visual field orientation of a second user are reached, and the dwell time of the first user in the visual field orientation change range reaches a preset threshold value, determining that the application client can display the additional content data, wherein the second user is a user of the application client; and sending the determined additional content data which can be displayed by the application client to the application client so as to enable the application client to display the additional content data which can be displayed.
35. The application server of claim 34, wherein for each piece of additional content data, the determining module further determines a presentation manner of the piece of additional content data according to the field-of-view orientation variation range of the additional content data; the determining module further determines that the presentation mode is a static presentation mode if the change in the orientation of the field of view of the first user is less than a threshold value for a period of time before and after a point in time when the first user posted the additional content;
the determination module further determines that the presentation mode is a dynamic presentation mode if a change in orientation of the field of view of the first user is greater than a threshold for a period of time before and after a point in time when the first user posted the additional content.
36. The application server of claim 35, wherein the plurality of application clients present the additional content at a fixed location when the presentation is a static presentation;
and when the display mode is a dynamic display mode, the plurality of application clients move to display the additional content according to the pointing direction.
37. A computer-readable storage medium, characterized in that a data processing program for causing a computer to execute the method according to any one of claims 1 to 20 is stored.
CN201710474090.1A 2017-06-21 2017-06-21 Panoramic image display method, client, server and storage medium Active CN108924534B (en)

Priority Applications (2)

Application Number Priority Date Filing Date Title
CN201710474090.1A CN108924534B (en) 2017-06-21 2017-06-21 Panoramic image display method, client, server and storage medium
PCT/CN2018/088923 WO2018233459A1 (en) 2017-06-21 2018-05-30 Panoramic image display method and display control method, computing device, and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201710474090.1A CN108924534B (en) 2017-06-21 2017-06-21 Panoramic image display method, client, server and storage medium

Publications (2)

Publication Number Publication Date
CN108924534A CN108924534A (en) 2018-11-30
CN108924534B true CN108924534B (en) 2020-03-17

Family

ID=64402793

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201710474090.1A Active CN108924534B (en) 2017-06-21 2017-06-21 Panoramic image display method, client, server and storage medium

Country Status (2)

Country Link
CN (1) CN108924534B (en)
WO (1) WO2018233459A1 (en)

Families Citing this family (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110231867A (en) * 2019-05-31 2019-09-13 重庆爱奇艺智能科技有限公司 The method and apparatus of the display properties of barrage are adjusted in virtual reality device
CN111369320A (en) * 2020-03-05 2020-07-03 中国建设银行股份有限公司 Vehicle display method and system
CN113271411B (en) * 2021-04-17 2024-02-27 圆周率科技(常州)有限公司 Panoramic image viewing method and playing device
CN116260986A (en) * 2021-12-10 2023-06-13 华为技术有限公司 Bullet screen display method, device and system of free view video

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106331879A (en) * 2016-08-26 2017-01-11 传线网络科技（上海）有限公司 Screen comment display method and device based on panoramic video
CN106878786A (en) * 2017-03-07 2017-06-20 北京小米移动软件有限公司 The display methods and device of barrage in VR videos

Family Cites Families (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9778740B2 (en) * 2015-04-10 2017-10-03 Finwe Oy Method and system for tracking an interest of a user within a panoramic visual content
CN105898375B (en) * 2016-07-01 2019-01-22 上海幻电信息科技有限公司 A kind of barrage supplying system and method for pushing
CN106385587B (en) * 2016-09-14 2019-08-02 三星电子（中国）研发中心 Share the method, apparatus and system at virtual reality visual angle

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106331879A (en) * 2016-08-26 2017-01-11 传线网络科技（上海）有限公司 Screen comment display method and device based on panoramic video
CN106878786A (en) * 2017-03-07 2017-06-20 北京小米移动软件有限公司 The display methods and device of barrage in VR videos

Also Published As

Publication number Publication date
CN108924534A (en) 2018-11-30
WO2018233459A1 (en) 2018-12-27

Similar Documents

Publication Publication Date Title
TWI571130B (en) Volumetric video presentation
US9855504B2 (en) Sharing three-dimensional gameplay
CN107911737B (en) Media content display method and device, computing equipment and storage medium
CN108924534B (en) Panoramic image display method, client, server and storage medium
US9024842B1 (en) Hand gestures to signify what is important
TWI669635B (en) Method and device for displaying barrage and non-volatile computer readable storage medium
US11770599B2 (en) Techniques to set focus in camera in a mixed-reality environment with hand gesture interaction
US10770113B2 (en) Methods and system for customizing immersive media content
WO2016194441A1 (en) Three-dimensional advertising space determination system, user terminal, and three-dimensional advertising space determination computer
CN115150555B (en) Video recording method, device, equipment and medium
WO2021088973A1 (en) Live stream display method and apparatus, electronic device, and readable storage medium
CN113114985A (en) Information processing method and information processing device
CN113542846B (en) AR barrage display method and device
US20230405475A1 (en) Shooting method, apparatus, device and medium based on virtual reality space
CN117478931A (en) Information display method, information display device, electronic equipment and storage medium
CN115454247A (en) Virtual content interaction method and device, electronic equipment and storage medium
WO2020070756A1 (en) A system and a method for providing a virtual tour
TW202118291A (en) Electronic device and a subtitle-embedding method for virtual reality video
CN116896648A (en) Virtual reality live broadcast wheat connecting method, system, device, equipment and storage medium
CN106125940A (en) virtual reality interactive interface management method and device

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant