CN117278733B - Display method and system of panoramic camera in VR head display - Google Patents
Display method and system of panoramic camera in VR head display Download PDFInfo
- Publication number
- CN117278733B CN117278733B CN202311557636.1A CN202311557636A CN117278733B CN 117278733 B CN117278733 B CN 117278733B CN 202311557636 A CN202311557636 A CN 202311557636A CN 117278733 B CN117278733 B CN 117278733B
- Authority
- CN
- China
- Prior art keywords
- image
- panoramic
- panoramic image
- picture
- head display
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 37
- 238000005520 cutting process Methods 0.000 claims abstract description 25
- 238000012544 monitoring process Methods 0.000 claims abstract description 15
- 238000004364 calculation method Methods 0.000 claims description 11
- 238000012545 processing Methods 0.000 claims description 11
- 230000005540 biological transmission Effects 0.000 claims description 6
- 238000000605 extraction Methods 0.000 claims description 4
- 230000004044 response Effects 0.000 abstract description 3
- 230000003993 interaction Effects 0.000 abstract description 2
- 210000003128 head Anatomy 0.000 description 94
- 230000000694 effects Effects 0.000 description 5
- 230000001186 cumulative effect Effects 0.000 description 4
- 238000010586 diagram Methods 0.000 description 4
- 230000001360 synchronised effect Effects 0.000 description 3
- 238000006243 chemical reaction Methods 0.000 description 2
- 238000009826 distribution Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000007935 neutral effect Effects 0.000 description 2
- 238000003860 storage Methods 0.000 description 2
- 239000003855 balanced salt solution Substances 0.000 description 1
- 238000004590 computer program Methods 0.000 description 1
- 238000005094 computer simulation Methods 0.000 description 1
- 229940053083 eye stream Drugs 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 238000004806 packaging method and process Methods 0.000 description 1
- 239000000243 solution Substances 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 239000000126 substance Substances 0.000 description 1
- 238000013519 translation Methods 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/30—Image reproducers
- H04N13/332—Displays for viewing with the aid of special glasses or head-mounted displays [HMD]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
- H04N13/106—Processing image signals
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/20—Image signal generators
- H04N13/296—Synchronisation thereof; Control thereof
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/30—Image reproducers
- H04N13/366—Image reproducers using viewer tracking
- H04N13/371—Image reproducers using viewer tracking for tracking viewers with different interocular distances; for tracking rotational head movements around the vertical axis
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Closed-Circuit Television Systems (AREA)
Abstract
The invention is applicable to the technical field of VR interaction, and discloses a display method and a system for panoramic photography in a VR head display, wherein the display method for panoramic photography in the VR head display comprises the following steps: acquiring pictures through two groups of panoramic cameras simultaneously to obtain a first panoramic image and a second panoramic image; cutting, and cutting out pictures to obtain a first image stream and a second image stream; transmitting the first image stream and the second image stream to VR head display equipment, monitoring the VR head display equipment in real time, identifying the deflection direction of the VR head display equipment, and calculating a real-time deflection angle; rotating the two groups of panoramic cameras based on real-time deflection angles, and updating the acquired first panoramic image and second panoramic image to generate a panoramic image to be output; the invention can provide 3D images for VR head display equipment, can provide the image rotation requirement of a certain angle by carrying out image acquisition once, can process the currently acquired images in advance and output the images in real time, and improves the response speed.
Description
Technical Field
The invention belongs to the technical field of VR interaction, and particularly relates to a display method and a system for panoramic photography in a VR head display.
Background
The panoramic multi-camera vision system is one of panoramic cameras, a plurality of sensors with different orientations are packaged in the panoramic multi-camera vision system, and panoramic effects are obtained by performing image stitching operation on split pictures; the mainstream product is structured by packaging several two million image sensors and field-of-view independent short focus lenses in a unified housing.
Virtual reality technology (VR) is a computer simulation system that can create and experience a virtual world, using a computer to create a simulated environment into which a user is immersed; the virtual reality technology is to use data in real life, combine electronic signals generated by computer technology with various output devices to convert the electronic signals into phenomena which can be perceived by people, wherein the phenomena can be real and cut objects in reality or substances which can not be seen by naked eyes, and the phenomena are shown by a three-dimensional model.
In current VR equipment, if want to realize real-time VR picture update, need equipment have high arithmetic processing ability, when the user wears VR equipment and moves or rotate fast, the picture variation is big, and data processing volume is big, has higher requirement to the data processing ability of equipment, otherwise can appear the picture unsmooth, delay high problem.
Disclosure of Invention
The technical problem to be solved by the invention is to provide a display method and a system for panoramic photography in a VR head display, which can provide 3D images for VR head display equipment, can provide the image rotation requirement of a certain angle by single image acquisition, can process the currently acquired images in advance and output the images in real time, improves the response speed and reduces the image display delay.
In order to solve the technical problems, the invention provides the following technical scheme:
a method of displaying panoramic photography in a VR head display, the method comprising:
the method comprises the steps that picture collection is carried out through two groups of panoramic cameras simultaneously, a first panoramic image and a second panoramic image are obtained, and the first panoramic image and the second panoramic image are obtained through collection of the two panoramic cameras respectively;
cutting the first panoramic image and the second panoramic image, and cutting out pictures of the first panoramic image and the second panoramic image based on corner information to obtain a first image stream and a second image stream, wherein the pictures of the first image stream and the second image stream are larger than the pictures displayed by VR head display equipment;
transmitting the first image stream and the second image stream to VR head display equipment, monitoring the VR head display equipment in real time, identifying the deflection direction of the VR head display equipment, and calculating a real-time deflection angle;
and rotating the two groups of panoramic cameras based on the real-time deflection angle, and updating the acquired first panoramic image and second panoramic image to generate a panoramic image to be output.
The following is a further optimization of the above technical solution according to the present invention:
the step of clipping the first panoramic image and the second panoramic image, and clipping the first panoramic image and the second panoramic image based on corner information to obtain a first image stream and a second image stream specifically comprises the following steps:
identifying the first panoramic image and the second panoramic image, and determining display zero points of the first panoramic image and the second panoramic image based on the identification result;
cutting the first panoramic image and the second panoramic image, and calculating the area of the zero display picture based on corner information to obtain a first source picture and a second source picture;
and performing image processing on the first source picture and the second source picture, and converting the first source picture and the second source picture into a format for display of the VR head display device to obtain a first image stream and a second image stream.
Further optimizing: the step of transmitting the first image stream and the second image stream to the VR head display device, monitoring the VR head display device in real time, identifying the deflection direction of the VR head display device, and calculating the real-time deflection angle comprises the following steps:
transmitting the first image stream and the second image stream to VR head display equipment, and displaying a 3D picture through the VR head display equipment;
monitoring VR head display equipment, determining the deflection direction of the VR head display equipment, recording information of the deflection angle of the VR head display equipment, and calculating the accumulated picture deflection angle;
and comparing the accumulated picture deflection angle with a preset value, and judging whether the panoramic camera needs to be rotated or not according to the comparison result.
Further optimizing: the step of generating the panoramic image to be output specifically includes:
when the real-time deflection angle is larger than a preset value, rotating the two groups of panoramic cameras;
image acquisition is carried out in real time in the process of rotating the panoramic camera, and an updated first panoramic image and a second panoramic image are obtained;
and when the real-time deflection angle is not larger than a preset value, the panoramic camera is not rotated, and the first panoramic image and the second panoramic image are subjected to picture cutting based on the real-time deflection angle, so that the panoramic image to be output is obtained.
Further optimizing: the two groups of panoramic cameras are arranged in parallel at preset intervals.
Further optimizing: the transmission frame rate of the first image stream and the second image stream is greater than the display frame rate of the VR head display device.
The invention also provides a display system of the panoramic camera in the VR head display, which comprises:
the image acquisition module is used for simultaneously acquiring pictures through two groups of panoramic cameras to obtain a first panoramic image and a second panoramic image, wherein the first panoramic image and the second panoramic image are acquired through the two panoramic cameras respectively;
the picture cutting module is used for cutting the first panoramic image and the second panoramic image, and cutting pictures of the first panoramic image and the second panoramic image based on corner information to obtain a first image stream and a second image stream, wherein the pictures of the first image stream and the second image stream are larger than the pictures displayed by the VR head display device;
the deflection angle calculation module is used for transmitting the first image stream and the second image stream to the VR head display device, monitoring the VR head display device in real time, identifying the deflection direction of the VR head display device and calculating the real-time deflection angle;
the picture adjusting module is used for rotating the two groups of panoramic cameras based on the real-time deflection angle, updating the acquired first panoramic image and second panoramic image and generating a panoramic image to be output.
Further optimizing: the picture clipping module comprises:
the zero calibration module is used for identifying the first panoramic image and the second panoramic image, and determining the display zero points of the first panoramic image and the second panoramic image based on the identification result;
the source picture extraction unit is used for cutting the first panoramic image and the second panoramic image, calculating the area where the zero point display picture is located based on corner information, and obtaining a first source picture and a second source picture;
and the image stream generating unit is used for carrying out image processing on the first source picture and the second source picture, converting the first source picture and the second source picture into a format for the VR head display device to display, and obtaining a first image stream and a second image stream.
Further optimizing: the deflection angle calculation module includes:
the image transmission unit is used for transmitting the first image stream and the second image stream to the VR head display device, and 3D picture display is carried out through the VR head display device;
the picture deflection calculation unit is used for monitoring the VR head display equipment, determining the deflection direction of the VR head display equipment, recording information on the deflection angle of the VR head display equipment and calculating the accumulated picture deflection angle;
and the equipment adjusting and judging unit is used for comparing the accumulated picture deflection angle with a preset value and judging whether the panoramic camera needs to be rotated or not according to the comparison result.
Further optimizing: the picture adjusting module includes:
the device rotating unit is used for rotating the two groups of panoramic cameras when the real-time deflection angle is larger than a preset value;
the image updating unit is used for acquiring images in real time in the process of rotating the panoramic camera to obtain an updated first panoramic image and a second panoramic image;
and the panoramic picture updating unit is used for not rotating the panoramic camera when the real-time deflection angle is not larger than a preset value, and cutting pictures of the first panoramic image and the second panoramic image based on the real-time deflection angle to obtain the panoramic image to be output.
The invention adopts the technical scheme, has ingenious conception and reasonable structure, adopts two groups of panoramic cameras to collect images, can simultaneously obtain two groups of panoramic images with different angles, processes the two groups of panoramic images with different angles, transmits the two groups of panoramic images to the VR head display equipment, can provide 3D images for the VR head display equipment, can collect images once, can provide the image rotation requirement with a certain angle, can process the currently collected images in advance, output the images in real time, improves the response speed and reduces the image display delay.
Drawings
FIG. 1 is a flow chart of a method according to an embodiment of the invention;
FIG. 2 is a flowchart of steps for cropping a first panoramic image and a second panoramic image, and performing picture cropping on the first panoramic image and the second panoramic image based on corner information to obtain a first image stream and a second image stream in an embodiment of the present invention;
FIG. 3 is a flowchart showing steps of transmitting a first image stream and a second image stream to a VR head display device, monitoring the VR head display device in real time, recognizing a deflection direction, and calculating a real-time deflection angle in the embodiment of the present invention;
FIG. 4 is a flowchart of the steps of rotating two groups of panoramic cameras based on real-time deflection angles, and updating the acquired first panoramic image and second panoramic image to generate a panoramic image to be output in an embodiment of the present invention;
FIG. 5 is a block diagram of a display system for panoramic photography in a VR head display in an embodiment of the present invention;
FIG. 6 is a schematic diagram of a frame cropping module according to an embodiment of the present invention;
FIG. 7 is a schematic diagram of a deflection angle calculation module according to an embodiment of the present invention;
fig. 8 is a block diagram of a frame adjustment module according to an embodiment of the invention.
Detailed Description
As shown in fig. 1-4, a display method of panoramic photography in VR head display includes:
s100, simultaneously acquiring pictures through two groups of panoramic cameras to obtain a first panoramic image and a second panoramic image, wherein the first panoramic image and the second panoramic image are acquired through the two panoramic cameras respectively.
In this step, the existing VR head display device generally directly uses the existing VR data, so that the existing VR head display device can send pictures in real time, the pictures to be acquired in real time are difficult to display in real time, and the 3D effect cannot be realized, in this embodiment, two panoramic cameras are arranged in parallel, the distance between the two panoramic cameras is within the normal range of the distance between eyes, so as to realize the effect of simulating eyes, two groups of panoramic cameras can simulate the left eyes and the right eyes of a person, and respectively obtain a first panoramic image and a second panoramic image.
S200, cutting the first panoramic image and the second panoramic image, and cutting out pictures of the first panoramic image and the second panoramic image based on corner information to obtain a first image stream and a second image stream, wherein the pictures of the first image stream and the second image stream are larger than the pictures displayed by the VR head display device.
In this step, the first panoramic image and the second panoramic image are identified, when preliminary display is performed, the central positions of the first panoramic image and the second panoramic image are determined, the positions of the two groups of panoramic cameras in the first panoramic image and the second panoramic image are determined based on the central positions, two straight lines are drawn on the first panoramic image and the second panoramic image, the straight lines respectively pass through the two central positions and are parallel, the two straight lines respectively swing by taking the respective corresponding central positions as axes, the two straight lines swing synchronously, firstly, the first swing to the left until any group of straight lines is contacted with the images of the panoramic cameras in the first panoramic image and the second panoramic image, then the second swing to the right until any group of straight lines is contacted with the images of the panoramic cameras in the first panoramic image and the second panoramic image, then the picture corresponding to the left swing range is the picture which can be used for VR display, the picture is intercepted, the angle of the swing range is [ -a DEG+a DEG ], when the position of the straight lines is 0 DEG is taken as a zero point, the picture is intercepted, so that a first image stream and a second stream is generated, and a left image and a right image are respectively corresponding to the picture of the first stream and the second stream are generated.
S300, transmitting the first image stream and the second image stream to the VR head display device, monitoring the VR head display device in real time, identifying the deflection direction of the VR head display device, and calculating a real-time deflection angle.
In this step, the first image stream and the second image stream are transmitted to the VR head display device, at this time, the left eye of the user can see the picture in the first image stream, the right eye of the user can see the picture in the second image stream, and the two simulated eyes can see the picture, at this time, the picture seen by the user is the 3D picture, when the user moves and rotates the VR head display device, the deflection angle of the VR head display device is detected, and the deflection direction is determined, and because the range available for displaying the picture is [ -a°, +a° ], the first panoramic image and the second panoramic image which are currently acquired cannot meet the requirements provided by the picture, in order to ensure the stability of picture display, when the real-time deflection angle of the user is greater than a/2 °, the position of the panoramic camera is actively adjusted, and when the user moves, the panoramic camera is driven to synchronously rotate by a/2 ° through the rotating base, and if the user moves, the panoramic camera is driven to synchronously move.
S400, rotating the two groups of panoramic cameras based on real-time deflection angles, and updating the acquired first panoramic image and second panoramic image to generate a panoramic image to be output.
In the step, two groups of panoramic cameras are rotated based on real-time deflection angles, pictures which are acquired at present and output to VR head display equipment are still positioned at zero positions of a first panoramic image and a second panoramic image at present through rotation, when the deflection angle of the head of a user is smaller, if the deflection angle of the head of the user is smaller than a/2 DEG, the first panoramic image and the second panoramic image can still meet the use requirement of the user in a short time, the panoramic cameras do not need to be moved or rotated, at the moment, the head of the user deflects b DEG, the zero is taken as a starting point, an image stream is obtained from a newly acquired picture according to the head swinging direction of the user and is output to the VR head display equipment, if the panoramic cameras move or rotate, the first panoramic image and the second panoramic image are updated, the range of the zero and the display picture is redetermined [ -a DEG, +a DEG ] is cut into a panoramic image based on the zero point and [ -a DEG ], and the panoramic image to be output is obtained, and the panoramic image to be output is directly cut and transmitted according to the rotation angle of the head of the user when the picture to be output is required.
As shown in fig. 2, the step of clipping the first panoramic image and the second panoramic image, and clipping the first panoramic image and the second panoramic image based on the corner information to obtain a first image stream and a second image stream specifically includes:
s201, the first panoramic image and the second panoramic image are identified, and the display zero points of the first panoramic image and the second panoramic image are determined based on the identification results.
In this step, the first panoramic image and the second panoramic image are identified, and the image positions of the panoramic cameras contained in the first panoramic image and the second panoramic image are identified, and because the two groups of panoramic cameras can be mutually shielded, if the first panoramic image and the second panoramic image are not updated, the picture of the panoramic camera can appear to enter the field of view of a user, so that the user experience is affected, after the positions of the panoramic cameras are identified, the available range in the picture is determined by drawing a straight line, and after the range is determined, the neutral line is taken as a zero point, so that the display zero point is obtained.
S202, cutting the first panoramic image and the second panoramic image, and calculating the area of the zero point display picture based on the corner information to obtain a first source picture and a second source picture.
In this step, the first panoramic image and the second panoramic image are cropped, where the cropping process is determined based on the available range, and the available range is [ -a°, +a° ], where 0 ° is zero, and since the frame collected by the panoramic camera is a circular area centered on the panoramic camera, there are actually two available ranges, and if the available range is [ -30 °, +30° ], and corresponds to a sector where the circle is located, the corresponding sector area corresponding to 180-240 ° is also the available range, and one of the two available ranges is selected randomly, and in the above example, the images where the two panoramic cameras are located in the range of 60-180 ° and the range of 240-360 ° respectively, so as to obtain the first source frame and the second source frame.
S203, performing image processing on the first source picture and the second source picture, and converting the first source picture and the second source picture into a format for display of the VR head display device to obtain a first image stream and a second image stream.
In this step, image processing is performed on the first source frame and the second source frame, so that the first source frame and the second source frame can be applied to the VR head-display device, the images need to be processed, so that after the images are displayed by the VR head-display device, the human eye frame can be simulated, and format conversion is performed on the images, so as to generate a first image stream and a second image stream.
As shown in fig. 3, the steps of transmitting the first image stream and the second image stream to the VR head display device, monitoring the VR head display device in real time, identifying the deflection direction thereof, and calculating the real-time deflection angle specifically include:
s301, the first image stream and the second image stream are transmitted to the VR head display device, and 3D image display is carried out through the VR head display device.
In this step, the first image stream and the second image stream are transmitted to the VR head display device, and specifically, display parameters of the VR head display device, such as pixel distribution of a display screen and update frequency of the display screen, are first identified, so that the first image stream and the second image stream are processed and displayed on the VR head display device.
S302, monitoring the VR head display equipment, determining the deflection direction of the VR head display equipment, recording information on the deflection angle of the VR head display equipment, and calculating the accumulated picture deflection angle.
In this step, the VR head display device is monitored, in order to ensure that the user can update the picture in real time during the moving process, the swing angle of the VR head display device needs to be monitored, when the swing angle is smaller than the preset value, it is indicated that the first panoramic image and the second panoramic image which are currently collected still can meet the display requirement, specifically, the cumulative picture deflection angle is the total angle of the user swinging during the process of using the first panoramic image and the second panoramic image, if the user moves to the left by 10 ° in the a-B time period, the B-C time moves to the right by 2 ° and the C-D time moves to the right by 3 ° and the cumulative picture deflection angle is (-10+2+3) °.
S303, comparing the accumulated picture deflection angle with a preset value, and judging whether the panoramic camera needs to be rotated or not according to the comparison result.
In this step, the accumulated frame deflection angle is compared with the preset value, specifically, the preset value may be set to a/2, when the accumulated frame deflection angle is greater than the preset value, it is indicated that the available frame margins of the first panoramic image and the second panoramic image are not too large, and the acquisition needs to be performed again, otherwise, it is indicated that the margins are sufficient, and the acquisition does not need to be performed again.
As shown in fig. 4, the step of generating the panoramic image to be output by rotating two groups of panoramic cameras based on real-time deflection angles and updating the acquired first panoramic image and second panoramic image specifically includes:
s401, rotating the two groups of panoramic cameras when the real-time deflection angle is larger than a preset value.
In this step, when real-time deflection angle is greater than the default, rotate two sets of panoramic camera, two sets of panoramic camera are fixed simultaneously when a set of rotating base, and the line terminal point of two sets of panoramic camera is located rotating shaft of rotating base, and rotating base also can drive panoramic camera translation.
S402, image acquisition is carried out in real time in the process of rotating the panoramic camera, and an updated first panoramic image and an updated second panoramic image are obtained.
S403, when the real-time deflection angle is not larger than a preset value, the panoramic camera is not rotated, and the first panoramic image and the second panoramic image are subjected to picture cutting based on the real-time deflection angle, so that the panoramic image to be output is obtained.
In this step, image acquisition is performed in real time during the process of rotating the panoramic camera, so as to achieve the purpose of updating the first panoramic image and the second panoramic image, when the real-time deflection angle is not greater than a preset value, the panoramic camera is not required to be rotated, the current first panoramic image and the current second panoramic image are continuously used, under the condition of updating, the zero point and the available range are required to be redetermined according to the steps disclosed in this embodiment, and a new source picture is obtained through clipping.
As shown in fig. 5, the present invention further provides a display system for panoramic photography in a VR head display, where the system includes:
the image acquisition module 100 is configured to acquire images through two groups of panoramic cameras simultaneously, and obtain a first panoramic image and a second panoramic image, where the first panoramic image and the second panoramic image are acquired through two panoramic cameras respectively.
In this system, current VR head display equipment generally directly uses current VR data, consequently, can carry out the picture delivery in real time, be difficult to real-time demonstration to the picture that needs real-time collection, and also can't realize 3D effect, in this embodiment, be provided with two panoramic cameras, two panoramic cameras set up side by side, the interval between them is in normal human eye interval scope, thereby realize the effect of simulating the human eye, two sets of panoramic cameras can simulate human left eye and right eye, obtain first panoramic image and second panoramic image respectively, because two sets of panoramic cameras are parallel arrangement, can influence each other, be input in the picture of other side by side, therefore the range that can intercept in first panoramic image and the second panoramic image is limited, in order to satisfy picture collection on the all-round, panoramic camera installs on rotating base, rotating base can drive panoramic camera synchronous rotation, rotating base also can drive the panoramic camera and wholly remove.
The picture clipping module 200 is configured to clip the first panoramic image and the second panoramic image, and clip the first panoramic image and the second panoramic image based on the corner information, so as to obtain a first image stream and a second image stream, where the pictures of the first image stream and the second image stream are larger than the pictures displayed by the VR head display device.
In the system, the picture cropping module 200 identifies the first panoramic image and the second panoramic image, when the first panoramic image and the second panoramic image are primarily displayed, the central positions of the first panoramic image and the second panoramic image are determined based on the central positions, two straight lines are drawn on the first panoramic image and the second panoramic image and respectively pass through the two central positions, the two straight lines are parallel, the two straight lines swing respectively by taking the corresponding central positions as axes, the two straight lines swing synchronously, firstly, the first straight line swings leftwards until any group of straight lines is contacted with the images of the panoramic camera in the first panoramic image and the second panoramic image, then the second straight line swings rightwards until any group of straight lines is contacted with the images of the panoramic camera in the first panoramic image and the second panoramic image, then the picture corresponding to the leftward and rightward swinging range is the picture which can be used for VR display, the picture is intercepted, the angle of the swinging range is [ -a DEG, a DEG ], when the angle of the swinging range is 0 DEG, the position of the straight lines is used as a zero point, the picture stream is intercepted, and the picture stream is generated, the first picture stream and the second picture stream is corresponding to the left eye stream, and the picture stream are generated.
The deflection angle calculation module 300 is configured to transmit the first image stream and the second image stream to the VR head display device, monitor the VR head display device in real time, identify a deflection direction of the VR head display device, and calculate a real-time deflection angle.
In the system, the deflection angle calculation module 300 transmits the first image stream and the second image stream to the VR head display device, at this time, the left eye of the user can see the picture in the first image stream, the right eye of the user can see the picture in the second image stream, so as to view the pictures with two simulated eyes, at this time, the picture seen by the user is a 3D picture, when the user moves and rotates the VR head display device, the deflection angle of the VR head display device is detected, and the deflection direction of the VR head display device is determined, and because the range available for displaying the picture is [ -a°, +a° ], the first panoramic image and the second panoramic image which are acquired currently cannot meet the requirements provided by the picture, in order to ensure the stability of picture display, when the real-time deflection angle of the user is greater than a/2 °, the position of the panoramic camera is actively adjusted, and the panoramic camera is driven to synchronously rotate by the rotation base, if the user moves, the panoramic camera is driven to synchronously move.
The picture adjusting module 400 is configured to rotate two groups of panoramic cameras based on real-time deflection angles, update the acquired first panoramic image and second panoramic image, and generate a panoramic image to be output.
In the system, the picture adjusting module 400 rotates two groups of panoramic cameras based on real-time deflection angles, pictures which are currently acquired and output to the VR head display device are still at zero positions of the current first panoramic image and the current second panoramic image through rotation, when the deflection angle of the head of a user is smaller, if the deflection angle of the head of the user is smaller than a/2 DEG, the first panoramic image and the second panoramic image still can meet the use requirement of the user in a short time, the panoramic cameras do not need to be moved or rotated, at the moment, the head of the user deflects b DEG, the zero point is taken as a starting point, an image stream is obtained from a newly acquired picture according to the head swinging direction of the user and is output to the VR head display device, if the panoramic cameras move or rotate, the first panoramic image and the second panoramic image are updated, the range of zero point and the display picture is redetermined, the first panoramic image and the second panoramic image are cut based on the zero point and the range of [ -a DEG, +a DEG ], +a DEG, the panoramic image to be output is obtained, and when the picture to be output is required, the picture is directly acquired and transmitted according to the rotation angle of the head of the user.
As shown in fig. 6, the screen cropping module 200 includes:
the zero calibration module 201 is configured to identify the first panoramic image and the second panoramic image, and determine display zero points of the first panoramic image and the second panoramic image based on the identification result.
In this module, the zero calibration module 201 identifies the first panoramic image and the second panoramic image, identifies the image positions of the panoramic cameras contained therein, and because the two groups of panoramic cameras can be mutually blocked, if the first panoramic image and the second panoramic image are not updated, the picture of the panoramic camera can appear to enter the field of view of the user, which affects the user experience, after identifying the positions of the panoramic cameras, the available range in the picture is determined by drawing a straight line, and after determining the range, the neutral line is taken as the zero point, so as to obtain the display zero point.
The source frame extraction unit 202 is configured to crop the first panoramic image and the second panoramic image, calculate an area where the zero point display frame is located based on the corner information, and obtain a first source frame and a second source frame.
In this module, the source frame extraction unit 202 performs clipping on the first panoramic image and the second panoramic image, where the clipping process is determined based on the available range, and the available range is [ -a°, +a° ], where 0 ° is zero, and since the frame collected by the panoramic camera is a circular area centered on the panoramic camera, there are actually two available ranges, and if the available range is [ -30 °, +30° ], and corresponds to a sector where the circle is located, the corresponding sector where the circle is located, and then the corresponding sector where the circle is located, and one of the two available ranges is selected randomly, and in the above example, the images where the two panoramic cameras are located in the range of 60 ° to 180 ° and the range of 240 ° to 360 °, respectively, so as to obtain the first source frame and the second source frame.
The image stream generating unit 203 is configured to perform image processing on the first source picture and the second source picture, and convert the first source picture and the second source picture into a format for display by the VR head display device, so as to obtain a first image stream and a second image stream.
In this module, the image stream generating unit 203 performs image processing on the first source frame and the second source frame, and in order to be applicable to the VR head display device, the images need to be processed, so that after the images are displayed by the VR head display device, the images can simulate the human eye frame, and format conversion is performed on the images, so as to generate a first image stream and a second image stream.
As shown in fig. 7, the deflection angle calculation module 300 includes:
an image transmission unit 301, configured to transmit the first image stream and the second image stream to a VR head display device, and perform 3D image display through the VR head display device.
In this module, the image transmission unit 301 transmits the first image stream and the second image stream to the VR head display device, specifically, first identifies the display parameters of the VR head display device, such as the pixel distribution of the display screen and the update frequency of the display screen, so as to process the first image stream and the second image stream, and display them on the VR head display device.
The frame deflection calculating unit 302 is configured to monitor the VR head display device, determine a deflection direction thereof, record information on a deflection angle of the VR head display device, and calculate an accumulated frame deflection angle.
In this module, the frame deflection calculation unit 302 monitors the VR head display device, in order to ensure that the frame can be updated in real time during the moving process of the user, it needs to monitor the swing angle of the VR head display device, and when the swing angle is smaller than the preset value, it indicates that the first panoramic image and the second panoramic image that are currently collected still can meet the display requirement, specifically, the cumulative frame deflection angle is the total swing angle of the user during the process of using the first panoramic image and the second panoramic image, for example, the user moves to the left by 10 ° in the a-B time period, the B-C time moves to the right by 2 ° and the C-D time moves to the right by 3 °, and the cumulative frame deflection angle is (-10+2+3) °.
The device adjustment determining unit 303 is configured to compare the accumulated frame deflection angle with a preset value, and determine whether the panoramic camera needs to be rotated according to the comparison result.
In this module, the device adjustment determining unit 303 compares the accumulated frame deflection angle with a preset value, specifically, the preset value may be set to a/2, and when the accumulated frame deflection angle is greater than the preset value, it indicates that the available frame margins of the first panoramic image and the second panoramic image are not large, and the acquisition needs to be performed again, otherwise, it indicates that the margins are sufficient, and the acquisition does not need to be performed again.
As shown in fig. 8, the picture adjustment module 400 includes:
and the device rotating unit 401 is used for rotating the two groups of panoramic cameras when the real-time deflection angle is larger than a preset value.
In this module, when the real-time deflection angle of the device rotation unit 401 is greater than a preset value, two groups of panoramic cameras are rotated, and when two groups of panoramic cameras are simultaneously fixed on a group of rotating bases, and the connecting line terminal point of the two groups of panoramic cameras is located on the rotating shaft of the rotating base, the rotating base can also drive the panoramic cameras to translate.
The image updating unit 402 is configured to perform image acquisition in real time during rotation of the panoramic camera, so as to obtain an updated first panoramic image and second panoramic image.
The panorama image updating unit 403 is configured to, when the real-time deflection angle is not greater than a preset value, not rotate the panorama camera, and perform image cropping on the first panorama image and the second panorama image based on the real-time deflection angle, to obtain a panorama image to be output.
In the module, image acquisition is performed in real time in the process of rotating the panoramic camera so as to achieve the purpose of updating the first panoramic image and the second panoramic image, when the real-time deflection angle is not larger than a preset value, the panoramic camera is not required to be rotated, the current first panoramic image and the current second panoramic image are continuously used, under the condition of updating, zero points and available ranges are required to be determined again according to the steps disclosed in the embodiment, and a new source picture is obtained through cutting.
It should be understood that, although the steps in the flowcharts of the embodiments of the present invention are shown in order as indicated by the arrows, these steps are not necessarily performed in order as indicated by the arrows; the steps are not strictly limited to the order of execution unless explicitly recited herein, and the steps may be executed in other orders; moreover, at least some of the steps in various embodiments may include multiple sub-steps or stages that are not necessarily performed at the same time, but may be performed at different times, nor do the order in which the sub-steps or stages are performed necessarily performed in sequence, but may be performed alternately or alternately with at least a portion of the sub-steps or stages of other steps or other steps.
Those skilled in the art will appreciate that all or part of the processes in the methods of the above embodiments may be implemented by a computer program for instructing relevant hardware, where the program may be stored in a non-volatile computer readable storage medium, and where the program, when executed, may include processes in the embodiments of the methods described above.
Wherein any reference to memory, storage, database, or other medium used in the various embodiments provided herein may include non-volatile and/or volatile memory; the nonvolatile memory can include Read Only Memory (ROM), programmable ROM (PROM), electrically Programmable ROM (EPROM), electrically Erasable Programmable ROM (EEPROM), or flash memory; volatile memory can include Random Access Memory (RAM) or external cache memory; by way of illustration and not limitation, RAM is available in a variety of forms such as Static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double Data Rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous Link DRAM (SLDRAM), memory bus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), among others.
The technical features of the above-described embodiments may be arbitrarily combined, and all possible combinations of the technical features in the above-described embodiments are not described for brevity of description, however, as long as there is no contradiction between the combinations of the technical features, they should be considered as the scope of the description.
The foregoing examples illustrate only a few embodiments of the invention, which are described in detail and are not to be construed as limiting the scope of the invention; it should be noted that it is possible for a person skilled in the art to make several variants and modifications without departing from the concept of the invention, all of which fall within the scope of protection of the invention; accordingly, the scope of protection of the present invention is to be determined by the appended claims.
The foregoing description of the preferred embodiments of the invention is not intended to be limiting, but rather is intended to cover all modifications, equivalents, and alternatives falling within the spirit and principles of the invention.
Claims (4)
1. The display method of the panoramic camera in the VR head display is characterized by comprising the following steps:
the method comprises the steps that picture collection is carried out through two groups of panoramic cameras simultaneously, a first panoramic image and a second panoramic image are obtained, and the first panoramic image and the second panoramic image are obtained through collection of the two panoramic cameras respectively;
cutting the first panoramic image and the second panoramic image, and cutting out pictures of the first panoramic image and the second panoramic image based on corner information to obtain a first image stream and a second image stream, wherein the pictures of the first image stream and the second image stream are larger than the pictures displayed by VR head display equipment;
transmitting the first image stream and the second image stream to VR head display equipment, monitoring the VR head display equipment in real time, identifying the deflection direction of the VR head display equipment, and calculating a real-time deflection angle;
rotating the two groups of panoramic cameras based on real-time deflection angles, and updating the acquired first panoramic image and second panoramic image to generate a panoramic image to be output;
the step of clipping the first panoramic image and the second panoramic image, and clipping the first panoramic image and the second panoramic image based on corner information to obtain a first image stream and a second image stream specifically comprises the following steps:
identifying the first panoramic image and the second panoramic image, and determining display zero points of the first panoramic image and the second panoramic image based on the identification result;
cutting the first panoramic image and the second panoramic image, and calculating the area of the zero display picture based on corner information to obtain a first source picture and a second source picture;
performing image processing on the first source picture and the second source picture, and converting the first source picture and the second source picture into a format for display of VR head display equipment to obtain a first image stream and a second image stream;
the step of transmitting the first image stream and the second image stream to the VR head display device, monitoring the VR head display device in real time, identifying the deflection direction of the VR head display device, and calculating the real-time deflection angle comprises the following steps:
transmitting the first image stream and the second image stream to VR head display equipment, and displaying a 3D picture through the VR head display equipment;
monitoring VR head display equipment, determining the deflection direction of the VR head display equipment, recording information of the deflection angle of the VR head display equipment, and calculating the accumulated picture deflection angle;
comparing the accumulated picture deflection angle with a preset value, and judging whether the panoramic camera needs to be rotated or not according to the comparison result;
the step of generating the panoramic image to be output specifically includes:
when the real-time deflection angle is larger than a preset value, rotating the two groups of panoramic cameras;
image acquisition is carried out in real time in the process of rotating the panoramic camera, and an updated first panoramic image and a second panoramic image are obtained;
and when the real-time deflection angle is not larger than a preset value, the panoramic camera is not rotated, and the first panoramic image and the second panoramic image are subjected to picture cutting based on the real-time deflection angle, so that the panoramic image to be output is obtained.
2. The method for displaying panoramic cameras in a VR head display of claim 1, wherein the two sets of panoramic cameras are arranged in parallel at a preset interval.
3. The method of claim 2, wherein the transmission frame rate of the first image stream and the second image stream is greater than the display frame rate of the VR head display device.
4. A display system for panoramic photography in a VR head display, the system comprising:
the image acquisition module is used for simultaneously acquiring pictures through two groups of panoramic cameras to obtain a first panoramic image and a second panoramic image, wherein the first panoramic image and the second panoramic image are acquired through the two panoramic cameras respectively;
the picture cutting module is used for cutting the first panoramic image and the second panoramic image, and cutting pictures of the first panoramic image and the second panoramic image based on corner information to obtain a first image stream and a second image stream, wherein the pictures of the first image stream and the second image stream are larger than the pictures displayed by the VR head display device;
the deflection angle calculation module is used for transmitting the first image stream and the second image stream to the VR head display device, monitoring the VR head display device in real time, identifying the deflection direction of the VR head display device and calculating the real-time deflection angle;
the picture adjusting module is used for rotating the two groups of panoramic cameras based on the real-time deflection angle, updating the acquired first panoramic image and second panoramic image and generating a panoramic image to be output;
the picture clipping module comprises:
the zero calibration module is used for identifying the first panoramic image and the second panoramic image, and determining the display zero points of the first panoramic image and the second panoramic image based on the identification result;
the source picture extraction unit is used for cutting the first panoramic image and the second panoramic image, calculating the area where the zero point display picture is located based on corner information, and obtaining a first source picture and a second source picture;
the image stream generating unit is used for carrying out image processing on the first source picture and the second source picture, converting the first source picture and the second source picture into a format for the VR head display equipment to display, and obtaining a first image stream and a second image stream;
the deflection angle calculation module includes:
the image transmission unit is used for transmitting the first image stream and the second image stream to the VR head display device, and 3D picture display is carried out through the VR head display device;
the picture deflection calculation unit is used for monitoring the VR head display equipment, determining the deflection direction of the VR head display equipment, recording information on the deflection angle of the VR head display equipment and calculating the accumulated picture deflection angle;
the device adjusting judging unit is used for comparing the accumulated picture deflection angle with a preset value and judging whether the panoramic camera needs to be rotated or not according to the comparison result;
the picture adjusting module includes:
the device rotating unit is used for rotating the two groups of panoramic cameras when the real-time deflection angle is larger than a preset value;
the image updating unit is used for acquiring images in real time in the process of rotating the panoramic camera to obtain an updated first panoramic image and a second panoramic image;
and the panoramic picture updating unit is used for not rotating the panoramic camera when the real-time deflection angle is not larger than a preset value, and cutting pictures of the first panoramic image and the second panoramic image based on the real-time deflection angle to obtain the panoramic image to be output.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311557636.1A CN117278733B (en) | 2023-11-22 | 2023-11-22 | Display method and system of panoramic camera in VR head display |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311557636.1A CN117278733B (en) | 2023-11-22 | 2023-11-22 | Display method and system of panoramic camera in VR head display |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN117278733A CN117278733A (en) | 2023-12-22 |
| CN117278733B true CN117278733B (en) | 2024-03-19 |
Family
ID=89204897
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311557636.1A Active CN117278733B (en) | 2023-11-22 | 2023-11-22 | Display method and system of panoramic camera in VR head display |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117278733B (en) |
Citations (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2011082919A (en) * | 2009-10-09 | 2011-04-21 | Sony Corp | Image processing device and method, and program |
| KR20110103692A (en) * | 2010-03-15 | 2011-09-21 | 한국과학기술연구원 | Panorama video display device and display control method using user motion |
| TW201223271A (en) * | 2010-09-22 | 2012-06-01 | Sony Corp | Image processing device, imaging device, and image processing method and program |
| CN106909221A (en) * | 2017-02-21 | 2017-06-30 | 北京小米移动软件有限公司 | Image processing method and device based on VR systems |
| US9807384B1 (en) * | 2017-01-13 | 2017-10-31 | Optofidelity Oy | Method, apparatus and computer program product for testing a display |
| WO2018035811A1 (en) * | 2016-08-25 | 2018-03-01 | 北京小米移动软件有限公司 | Panoramic photographing method, terminal, rotating assembly and panoramic photographing device |
| CN108174240A (en) * | 2017-12-29 | 2018-06-15 | 哈尔滨市舍科技有限公司 | Panoramic video playback method and system based on user location |
| CN109672867A (en) * | 2018-12-28 | 2019-04-23 | 上海上实龙创智慧能源科技股份有限公司 | A kind of display methods during panoramic shooting is aobvious at VR |
| CN113936119A (en) * | 2020-06-28 | 2022-01-14 | 华为技术有限公司 | Data rendering method, system and device |
| CN114637392A (en) * | 2020-12-15 | 2022-06-17 | 华为技术有限公司 | Display method and electronic equipment |
| CN115390253A (en) * | 2022-09-20 | 2022-11-25 | 立讯精密科技(南京)有限公司 | Scene display method and device, electronic equipment and storage medium |
| KR20230121951A (en) * | 2022-02-13 | 2023-08-22 | 주식회사아트릭트 | Image tracking system using object recognition information based on Virtual Reality, and image tracking method thereof |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107316273B (en) * | 2016-04-27 | 2021-05-18 | 深圳看到科技有限公司 | Panoramic image acquisition device and acquisition method |
| WO2018134946A1 (en) * | 2017-01-19 | 2018-07-26 | 株式会社ソニー・インタラクティブエンタテインメント | Image generation device, and image display control device |
| US20180342043A1 (en) * | 2017-05-23 | 2018-11-29 | Nokia Technologies Oy | Auto Scene Adjustments For Multi Camera Virtual Reality Streaming |
-
2023
- 2023-11-22 CN CN202311557636.1A patent/CN117278733B/en active Active
Patent Citations (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2011082919A (en) * | 2009-10-09 | 2011-04-21 | Sony Corp | Image processing device and method, and program |
| KR20110103692A (en) * | 2010-03-15 | 2011-09-21 | 한국과학기술연구원 | Panorama video display device and display control method using user motion |
| TW201223271A (en) * | 2010-09-22 | 2012-06-01 | Sony Corp | Image processing device, imaging device, and image processing method and program |
| WO2018035811A1 (en) * | 2016-08-25 | 2018-03-01 | 北京小米移动软件有限公司 | Panoramic photographing method, terminal, rotating assembly and panoramic photographing device |
| US9807384B1 (en) * | 2017-01-13 | 2017-10-31 | Optofidelity Oy | Method, apparatus and computer program product for testing a display |
| CN106909221A (en) * | 2017-02-21 | 2017-06-30 | 北京小米移动软件有限公司 | Image processing method and device based on VR systems |
| CN108174240A (en) * | 2017-12-29 | 2018-06-15 | 哈尔滨市舍科技有限公司 | Panoramic video playback method and system based on user location |
| CN109672867A (en) * | 2018-12-28 | 2019-04-23 | 上海上实龙创智慧能源科技股份有限公司 | A kind of display methods during panoramic shooting is aobvious at VR |
| CN113936119A (en) * | 2020-06-28 | 2022-01-14 | 华为技术有限公司 | Data rendering method, system and device |
| CN114637392A (en) * | 2020-12-15 | 2022-06-17 | 华为技术有限公司 | Display method and electronic equipment |
| KR20230121951A (en) * | 2022-02-13 | 2023-08-22 | 주식회사아트릭트 | Image tracking system using object recognition information based on Virtual Reality, and image tracking method thereof |
| CN115390253A (en) * | 2022-09-20 | 2022-11-25 | 立讯精密科技(南京)有限公司 | Scene display method and device, electronic equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN117278733A (en) | 2023-12-22 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US6570566B1 (en) | Image processing apparatus, image processing method, and program providing medium | |
| CN111552076B (en) | Image display method, AR glasses and storage medium | |
| EP3065049A2 (en) | Interactive video display method, device, and system | |
| CN109743626B (en) | Image display method, image processing method and related equipment | |
| TWI547901B (en) | Simulating stereoscopic image display method and display device | |
| CN107479712B (en) | Information processing method and device based on head-mounted display equipment | |
| KR20180027361A (en) | Methods, devices and systems for automatic zoom when playing an augmented reality scene | |
| JP7167134B2 (en) | Free-viewpoint image generation method, free-viewpoint image display method, free-viewpoint image generation device, and display device | |
| CN106303230B (en) | Video processing method and device | |
| CN110765828A (en) | Visual recognition method and system | |
| KR20170044451A (en) | System and Method for Controlling Remote Camera using Head mount display | |
| US20230239457A1 (en) | System and method for corrected video-see-through for head mounted displays | |
| CN110751728A (en) | Virtual reality equipment and method with BIM building model mixed reality function | |
| CN110969706B (en) | Augmented reality device, image processing method, system and storage medium thereof | |
| CN109725730B (en) | Head-mounted display device and driving method thereof, display system and driving method thereof | |
| CN117278733B (en) | Display method and system of panoramic camera in VR head display | |
| CN109858319A (en) | Image processing equipment and control method and non-transitory computer-readable storage media | |
| CN113225480A (en) | Image acquisition method, image acquisition device, electronic equipment and medium | |
| EP3805899A1 (en) | Head mounted display system and scene scanning method thereof | |
| CN111898622B (en) | Information processing method, information display method, model training method, information display system, model training system and equipment | |
| TWI542194B (en) | Three-dimensional image processing system, apparatus and method for the same | |
| CN115268650A (en) | Picture screen capturing method and device, head-mounted virtual reality equipment and storage medium | |
| CN110581977A (en) | video image output method and device and three-eye camera | |
| CN111142825B (en) | Multi-screen visual field display method and system and electronic equipment | |
| CN106331475A (en) | Image acquisition method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |