WO2023221923A1

WO2023221923A1 - Video processing method and apparatus, electronic device and storage medium

Info

Publication number: WO2023221923A1
Application number: PCT/CN2023/094199
Authority: WO
Inventors: 林达浩; 辛洋
Original assignee: 影石创新科技股份有限公司
Priority date: 2022-05-19
Filing date: 2023-05-15
Publication date: 2023-11-23
Also published as: CN117131225A

Abstract

Provided are a video processing method and apparatus, an electronic device, and a storage medium. The video processing method comprises: a control device acquiring preview posture data corresponding to a user previewing a panoramic picture, and sending the preview posture data to a recording device (110); then acquiring panoramic video data sent by the recording device, the panoramic video data being generated by the recording device according to the recorded panoramic image data and the preview posture data (120); then extracting the preview posture data and the panoramic image data from the panoramic video data (130); finally, playing, according to the preview posture data, a video corresponding to the panoramic image data (140). Therefore, the video can be played at a preview angle of the user when the video corresponding to the panoramic image data is played.

Description

Video processing method, device, electronic equipment and storage medium

Technical field

This application relates to the field of image processing technology, and specifically to a video processing method, device, electronic equipment and storage medium.

Background technique

In the existing technology, after the drone collects the panoramic video through the camera, the panoramic video is directly transmitted to other electronic devices. After other electronic devices obtain the panoramic video, since the panoramic video does not include the user during the recording process, The viewing angle requires the user to adjust the viewing angle to watch the video at the corresponding angle.

Contents of the invention

Embodiments of the present application provide a video processing method, device, electronic device, and storage medium. The video processing method can play the video corresponding to the panoramic image data at the preview angle when the user previews the panoramic image.

In the first aspect, embodiments of the present application provide a video processing method, including:

Obtain the corresponding preview posture data when the user previews the panoramic picture, and send the preview posture data to the recording device;

Obtain the panoramic image data sent by the recording device. The panoramic image data is generated by the recording device based on the recorded panoramic image data and preview posture data;

Extract preview pose data and panoramic image data from panoramic image data;

Play the video corresponding to the panoramic image data according to the preview posture data.

In a second aspect, embodiments of the present application provide a video processing device, including:

The sending module is used to obtain the corresponding preview posture data when the user previews the panoramic picture, and sends the preview posture data to the recording device;

The acquisition module is used to obtain the panoramic image data sent by the recording device. The panoramic image data is generated by the recording device based on the recorded panoramic image data and preview posture data;

Extraction module, used to extract preview pose data and panoramic image data from panoramic image data;

The playback module is used to play the video corresponding to the panoramic image data based on the preview posture data.

In a third aspect, embodiments of the present application provide an electronic device, including a memory storing executable program code, and a processor coupled to the memory; the processor calls the executable program code stored in the memory to execute the method provided by the embodiment of the present application. steps in the video processing method or the data processing method.

In the fourth aspect, embodiments of the present application provide a storage medium. The storage medium stores a plurality of instructions, and the instructions are suitable for loading by the processor to execute steps in the video processing method or data processing method provided by the embodiments of the present application.

In the embodiment of this application, the control device can obtain the corresponding preview posture data when the user previews the panoramic picture, and send the preview posture data to the recording device; then obtain the panoramic image data sent by the recording device, and the panoramic image data is recorded by the recording device according to The panoramic image data and preview posture data are generated; then the preview posture data and panoramic image data are extracted from the panoramic image data; and finally the video corresponding to the panoramic image data is played based on the preview posture data. Thus, when playing the video corresponding to the panoramic image data, the video can be played from the user's preview angle.

Description of the drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the description of the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application. For those skilled in the art, other drawings can also be obtained based on these drawings without exerting creative efforts.

Figure 1 is a schematic diagram of a data transmission scenario between a recording device and a control device provided by an embodiment of the present application.

Figure 2 is a schematic diagram of a three-dimensional space provided by an embodiment of the present application.

Figure 3 is a first flowchart of the video processing method provided by the embodiment of the present application.

Figure 4 is a second schematic flowchart of the video processing method provided by the embodiment of the present application.

FIG. 5 is a third flowchart of the video processing method provided by the embodiment of the present application.

Figure 6 is a first flow diagram of the data processing method provided by the embodiment of the present application.

Figure 7 is a schematic structural diagram of a video processing device provided by an embodiment of the present application.

Figure 8 is a schematic structural diagram of a data processing device provided by an embodiment of the present application.

Figure 9 is a schematic structural diagram of an electronic device provided by an embodiment of the present application.

Detailed ways

The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without making creative efforts fall within the scope of protection of this application.

In related technologies, after the drone collects the panoramic video through the camera, the panoramic video is directly transmitted to other electronic devices. After other electronic devices obtain the panoramic video, since the panoramic video does not include the user's data during the recording process, Viewing angle, users need to adjust the viewing angle themselves to watch the video at the corresponding angle.

In order to solve the above technical problems, embodiments of the present application provide a video processing method, device, electronic device, and storage medium. This video processing method can be applied to VR (Virtual Reality, virtual reality) glasses, and can also be applied to various electronic devices such as smartphones and computers.

Please refer to Figure 1. Figure 1 is a schematic diagram of a data transmission scenario between a recording device and a control device provided by an embodiment of the present application.

In this embodiment of the present application, the recording device may record panoramic image data, the control device may preview the panoramic image recorded by the recording device, and the control device may also be used to play the panoramic image data recorded by the recording device.

As shown in Figure 1, S1 is the control device, and S2 is the recording device. The control device can be VR glasses, and the recording device can be a drone. The drone is equipped with a 360-degree panoramic camera, and the recording device can record real-life scenes. panoramic picture in three-dimensional space.

It should be noted that the control device can also be a mobile phone, computer or other electronic device with computing capabilities. The recording device can also be an electronic device such as a panoramic camera. The drones and VR glasses shown in Figure 1 are only examples and should not be regarded as limitations of this application.

Please continue to refer to Figure 1. In actual application scenarios, after the control device and the recording device establish a communication connection, the recording device can transmit the recorded panoramic image to the control device, and the user can preview and control the panoramic image through the control device. The device will obtain the preview posture data corresponding to the user when previewing the panoramic image, and then send the preview posture data to the recording device.

For example, the panoramic picture previewed by the user on the control device may be a low-definition picture. For example, when the panoramic image data recorded by the recording device is the data corresponding to the original panoramic image, but if the recording device transmits the data corresponding to the original panoramic image to the control device, due to the large data size, a high delay will occur during the transmission process, which is not conducive to Users preview in real time.

Therefore, the recording device can convert the acquired panoramic image data into preview panoramic image data with a lower resolution, and then send the preview panoramic image data to the control device. After receiving the preview panoramic image data, the control device can preview the panoramic picture according to the preview panoramic image data, and the control device sends the preview data when the user previews the panoramic picture to the recording device. For example, the control device can obtain the preview posture data of the user when previewing the panoramic screen through sensors such as gyroscopes and acceleration sensors.

After the recording device receives the preview posture data, the recording device generates panoramic image data based on the recorded panoramic image data and preview posture data, and then the recording device sends the panoramic image data to the control device.

After the control device receives the panoramic image data, the control device extracts the panoramic image data to extract preview posture data and panoramic image data, and finally plays the video corresponding to the panoramic image data based on the preview posture data.

The control device can determine the user's preview angle based on the preview gesture data, and then play the video corresponding to the panoramic image data based on the preview angle.

For the control device to preview the content of the panoramic image according to the preview panoramic image data, please refer to Figure 2 as well. Figure 2 is a schematic diagram of a three-dimensional space provided by an embodiment of the present application.

Among them, the control device can first determine the space center point A2, and then establish a three-dimensional space A1. This three-dimensional space corresponds to the three-dimensional space where the recording device is located. The position of the space center point in the three-dimensional space corresponds to the real three-dimensional space of the recording device. location in. In the process of controlling the device to establish a three-dimensional space, the north direction obtained by the recording device through the fusion magnetometer can be determined as the standard direction corresponding to the center point of the space, and then the three-dimensional space is established based on this standard direction.

The three-dimensional space is a virtual space. When the recording device transmits the preview panoramic image data to the control device, the control device can render the preview panoramic image data in the three-dimensional space to obtain a preview panoramic picture.

When the preview panoramic image data transmitted by the recording device is a data stream, it is rendered in three-dimensional space. What is dyed is panoramic video.

When the user uses the control device to preview the panoramic image, the control device can use sensors to record the user's corresponding preview posture data. For example, the user's corresponding preview posture data can be determined through data recorded by sensors such as gyroscopes and acceleration sensors.

For example, when the user wears the control device, the control device can obtain the user's preview posture data. For example, the user rotates his head to look in a certain direction, and the control device can record the preview posture data corresponding to that direction. The preview posture data can be European. Pull angle data. Among them, the Euler angle is a set of three independent angular parameters used to uniquely determine the position of a fixed-point rotating rigid body, consisting of the nutation angle θ, the precession angle ψ and the rotation angle φ.

As shown in Figure 2, when the control device is rendering a panoramic image or panoramic video, the control device can determine the corresponding viewing angle B1 and viewing distance C1. Under the viewing angle B1 and viewing distance C1, the panoramic view is The picture previewed by the user is determined from the image or panoramic video. This screen is then displayed on the display area of the control device. Among them, the visible distance C1 is the distance between the space center point A2 and the space point A3.

In order to understand the video processing method provided by the embodiment of the present application in more detail, please refer to Figure 3 , which is a first flow diagram of the video processing method provided by the embodiment of the present application. The video processing method may include the following steps:

110. Obtain the corresponding preview posture data when the user previews the panoramic image, and send the preview posture data to the recording device.

In some embodiments, when the user uses the control device to preview the panoramic image sent by the recording device, the control device can record the user's preview gesture data and send the preview gesture data to the recording device.

For example, the control device can use a gyroscope, acceleration sensor, etc. to obtain the user's corresponding preview gesture data. For example, when the control device is worn by the user or held by the user, the corresponding sensor data can be obtained through the sensor, and then the sensor data can be used to determine the user. Preview attitude data, for example, preview attitude data includes Euler angle data, angular velocity data, acceleration data, etc.

For another example, when the user uses the mouse or touch to slide the screen to preview the panoramic picture, the control device can convert the user's mouse movement information or finger sliding information into Preview posture data corresponding to when the user previews the panoramic picture, such as mouse movement information or finger sliding information, is used to determine the direction selected by the user when previewing the panoramic picture, thereby determining the direction as the preview posture data.

In some implementations, after the control device obtains the user's preview gesture data, it can associate the preview gesture data with a specific time, and then send the preview gesture data after the associated time to the recording device.

120. Obtain the panoramic image data sent by the recording device. The panoramic image data is generated by the recording device based on the recorded panoramic image data and preview posture data.

In some implementations, after the control device sends the preview posture data to the recording device, the recording device generates panoramic image data based on the recorded panoramic image data and preview posture data. Panoramic image data can be understood as data including image data and posture data.

The control device can receive the panoramic image data sent by the recording device through wireless communication.

130. Extract preview pose data and panoramic image data from panoramic image data.

After acquiring the panoramic image data, the control device can analyze the panoramic image data to obtain panoramic image data and preview attitude data.

For example, if the panoramic image data and preview posture data are panoramic image data generated through a preset encoding format, the control device can use a corresponding decoding method to decode the panoramic image data, thereby obtaining the panoramic image data and preview posture data.

It should be noted that the control device can also determine the time information corresponding to the preview posture data and the time information corresponding to the panoramic image data, and then correspond the preview posture data and the panoramic image data at the same time.

In some implementations, when the control device parses the panoramic image data, the control device can smooth the panoramic image data to filter out useless data or noise data therein.

140. Play the video corresponding to the panoramic image data according to the preview posture data.

In some implementations, when the user needs to play the video corresponding to the panoramic image data, the control device can first determine the starting point of time that needs to be played, and then determine the preview gesture data and panoramic image data corresponding to the starting point of playing based on the starting point of time. Then the screen corresponding to the panoramic image data is rendered through the preview posture data in chronological order, thereby rendering the user's preview angle picture.

When the electronic device continuously displays images from multiple frames of preview angles, the control device plays the video corresponding to the panoramic image data.

That is to say, in this embodiment of the present application, the control device can determine the user's preview angle based on the preview gesture data, and then play the video corresponding to the panoramic image data based on the preview angle.

It should be noted that the device that plays the video of the panoramic image data is not limited to the control device, but can also be other electronic devices with playback capabilities.

Please refer to Figure 2 as well. As shown in Figure 2, when the preview posture data is Euler angle data, the control device can determine the user's preview angle in the three-dimensional space based on the Euler angle data, and determine the preview angle. corresponding screen.

When the display area of the control device displays images corresponding to multiple frames of preview angles, the video corresponding to the panoramic image data is played at the user's preview angle.

Please continue to refer to FIG. 4 , which is a second schematic flowchart of the video processing method provided by an embodiment of the present application. The video processing method may include the following steps:

201. Obtain preview posture data.

For another example, when the user uses the mouse or touch to slide the screen to preview the panoramic picture, the control device can convert the user's mouse movement information or finger sliding information into corresponding preview gesture data when the user previews the panoramic picture. For example, mouse movement information or finger sliding information is used to determine the direction selected by the user when previewing the panoramic screen, thereby determining the direction. is the preview posture data.

202. Determine the difference between the first preview gesture data corresponding to the current video frame and the second preview gesture data corresponding to the previous video frame.

In some implementations, the control device may determine the first preview gesture data corresponding to the current video frame and the second preview gesture data corresponding to the previous video frame, and then determine the first preview gesture data corresponding to the current video frame and the previous video frame The difference between the corresponding second preview pose data.

When the preview posture data is Euler angle data, the first preview posture data is the first Euler angle data, and the second posture data is the second Euler angle data.

Specifically, the control device converts the first Euler angle data of the current video frame into a first space vector, converts the second Euler angle data of the previous video frame into a second space vector, and finally determines the first space vector and the first space vector. The difference between two space vectors. The difference between the first space vector and the second space vector is the difference between the first preview posture data corresponding to the current video frame and the second preview posture data corresponding to the previous video frame.

203. If the difference is within the preset difference range, stop sending the first preview gesture data to the recording device.

In some embodiments, the control device can set a preset difference range, and the difference between the first preview posture data corresponding to the current video frame and the second preview posture data corresponding to the previous video frame is within the preset difference range. Within the range, it is considered that the user's preview posture changes very little, for example, the user's head changes very little.

For example, the preset difference range is 0 to 5 degrees, and the difference between the first preview posture data corresponding to the current video frame and the second preview posture data corresponding to the previous video frame is 4 degrees, then the current video frame is determined The difference between the corresponding first preview gesture data and the second preview gesture data corresponding to the previous video frame is within a preset difference range.

At this time, the control device can stop sending the preview data of the current video frame to the recording device, reducing data transmission to the recording device, so that the amount of panoramic image data finally generated by the recording device based on the preview posture data and panoramic image data is smaller. . At the same time, it can reduce the transmission of useless data and improve the data transmission efficiency between the data control device and the recording device.

204. If the difference is outside the preset difference range, send the first preview posture data to the recording device.

In some implementations, if the difference is outside the preset difference range, it means that the user's preview posture has changed significantly compared with the current video frame and the previous video frame. At this time, the first preview posture data can be sent. Give recording equipment.

205. Obtain the panoramic image data sent by the recording device based on the preview posture data.

In some embodiments, after the recording device receives the first preview gesture data, the recording device can determine the timestamp corresponding to the first preview gesture data, and then associate it with the panoramic image data of the same time based on the timestamp.

The recording device can associate the multi-frame preview posture data with its corresponding panoramic image data, thereby generating panoramic image data.

For example, the preview posture data corresponding to time 10:01 is associated with its corresponding panoramic image data, and the preview posture data corresponding to time 10:02 is associated with its corresponding panoramic image data. In this way, after the preview posture data and the panoramic image data are associated, the panoramic image data is obtained.

206. Obtain the edited posture data generated after editing the preview posture data.

In some implementations, after the control device receives the panoramic image data, the user can edit the preview posture data in the panoramic image data, thereby generating edited posture data. For example, after selecting a key frame, the user can edit the preview posture data corresponding to the key frame to obtain the edited posture data.

For example, the first key frame corresponds to the first target preview posture data, and after the user edits the first target preview posture data, the first edited posture data corresponding to the first key frame is obtained. The second key frame corresponds to the second target preview posture data, and after the user edits the second target preview posture data, the second edited posture data corresponding to the second key frame is obtained.

In some implementations, the user can also change other parameters of the panoramic image data, such as changing the viewing angle corresponding to the key frame after the user selects the key frame.

207. Play the video corresponding to the panoramic image data based on the edited posture data and preview posture data.

In some implementations, the control device can determine the preview angle of the picture corresponding to the key frame through the edited posture data corresponding to the key frame, determine the preview angle of the picture corresponding to other frames in the panoramic image data through the preview posture data, and finally determine the preview angle of the picture corresponding to other frames in the panoramic image data through Preview of the screen corresponding to the key frame Angle plays the picture corresponding to the key frame, and plays the picture corresponding to other frames through the preview angle of the picture corresponding to other frames.

In some embodiments, the control device can also smooth the posture data corresponding to the panoramic image data of each frame in the panoramic image data through the edited posture data and the preview posture data, and then play the panoramic view according to the smoothed posture data. The picture corresponding to the panoramic image data of each frame in the image data.

This enables the video corresponding to the panoramic image data to be played based on the edited posture data and the preview posture data.

In order to understand in more detail the video corresponding to the panoramic image data played according to the edited posture data and preview posture data in the embodiment of the present application, please continue to refer to Figure 5. Figure 5 is a third flow diagram of the video processing method provided by the embodiment of the present application. . The video processing method may also include the following steps:

301. Determine the first key frame corresponding to the panoramic image data and the first target preview posture data corresponding to the first key frame.

In some embodiments, taking the first key frame edited by the user as an example, the control device determines the first key frame of the panoramic image data edited by the user and the first target preview posture data corresponding to the first key frame, where the first target preview posture The data is extracted from the panoramic image data of the control device. The first key frame is a key frame corresponding to the panoramic image data selected by the user in the panoramic image data.

302. Obtain the first edited posture data generated after editing the first target preview posture data.

For example, the preview posture data is Euler angle data. After the user determines the first key frame, edit the first target Euler angle data corresponding to the first key frame, thereby obtaining the edited Euler angle corresponding to the first key frame. data.

303. Determine whether there is a second key frame within the preset time range corresponding to the first key frame.

For example, the preset time range corresponding to the first key frame is the first five seconds before the first key frame and the last five seconds after the first key frame. The control device can determine whether there is a user selected to edit within this preset time range. Second keyframe.

If there is a second key frame within the preset time range corresponding to the first key frame, step 304 is entered. If there is no second key frame within the preset time range corresponding to the first key frame, step 306 is entered.

304. If there is a second key frame within the preset time range corresponding to the first key frame, obtain the second edited posture data corresponding to the second key frame.

For example, if there is a second key frame within the preset time range corresponding to the first key frame, the control device can determine the second target preview posture data corresponding to the second key frame, and then obtain the second target preview posture data edited by the user. Generated second edited pose data.

305. Play the video corresponding to the panoramic image data between the first key frame and the second key frame based on the first edited posture data and the second edited posture data, and play the panoramic image data outside the preset time range based on the preview posture data. corresponding video.

Specifically, the control device can perform interpolation calculation on the posture data corresponding to each frame of panoramic image data between the first key frame and the second key frame based on the first edited posture data and the second edited posture data to obtain the first key frame. Interpolation posture data corresponding to each frame of panoramic image data between the first key frame and the second key frame; and then playing the video corresponding to the panoramic image data between the first key frame and the second key frame according to the interpolation posture data.

For example, there are three frames between the first key frame and the second key frame, and these three frames are respectively determined as frame A, frame B, and frame C in time order. The control device can calculate the posture data corresponding to frame A, frame B, and frame C according to the first edited posture data and the second edited posture data, thereby obtaining the interpolated posture data corresponding to frame A, frame B, and frame C respectively. .

Then the viewing angles corresponding to frame A, frame B, and frame C are determined based on the interpolation gesture data, and finally the pictures corresponding to frame A, frame B, and frame C are played based on the viewing angles. Thereby, the video corresponding to the panoramic image data between the first key frame and the second key frame is played according to the interpolated posture data.

In some implementations, for panoramic image data outside the preset time range, preview gesture data extracted from the panoramic image data can be used to play the corresponding video.

For example, for panoramic image data 5 seconds before the first key frame, the video corresponding to the panoramic image data can be played through the preview posture data.

Through the above method, the attitude data of the entire panoramic image data can be smoothed, so that when playing the video corresponding to the panoramic image data, it can have a smoother viewing angle transition playback effect without the occurrence of lag between video frames. sudden phenomenon.

306. If there is no second key frame within the preset time range corresponding to the first key frame, determine the preset time range. Assume the first video frame corresponding to the time starting point and the second video frame corresponding to the time end point within the time range.

For example, the preset time range corresponding to the first key frame is the first five seconds before the first key frame and the last five seconds after the first key frame. If there is no second key frame within the preset time range corresponding to the first key frame, , at this time, the control device determines the first video frame corresponding to 5 seconds before the first key frame, and the second video frame corresponding to 5 seconds after the first key frame.

307. Perform interpolation calculation on the posture data corresponding to the first key frame according to the preview posture data corresponding to the first video frame and the preview posture data corresponding to the second video frame, to obtain the first interpolation posture data corresponding to the first key frame.

For example, the control device first obtains the preview posture data corresponding to the first video frame and the preview posture data corresponding to the second video frame from the panoramic image data, and then obtains the preview posture data corresponding to the first video frame and the preview posture data corresponding to the second video frame. The posture data performs interpolation calculation on the posture data corresponding to the first key frame, thereby calculating the first interpolation posture data corresponding to the first key frame.

For the first edited posture data edited by the user on the first keyframe, it is not used.

308. Display the picture corresponding to the first key frame according to the first interpolation posture data and the panoramic image data corresponding to the first key frame.

For example, the control device determines the viewing angle corresponding to the first key frame by using the first interpolation posture data, then determines the panoramic image data corresponding to the first key frame in the panoramic image data, and finally displays the viewing angle corresponding to the first key frame. A frame of panoramic image data corresponding to a key frame.

In the embodiment of the present application, after the control device obtains the panoramic image data, it can play the video corresponding to the panoramic image data at the angle when the user previews the panoramic image according to the preview posture data in the panoramic image data. If the user edits the preview posture data in the panoramic image data, the video corresponding to the panoramic image data can be played together based on the edited posture data and the preview posture data edited by the user.

In some implementations, if the edited panoramic image data needs to be played on other devices, the file format of the edited panoramic image data can be converted to a common format, such as mp4 format, and then other devices can play the edited panoramic image data. When viewing panoramic image data, you can directly play the video corresponding to the panoramic image data through the angle corresponding to the edited posture data and preview posture data.

In this embodiment of the present application, the control device obtains the preview posture data and then determines the difference between the first preview posture data corresponding to the current video frame and the second preview posture data corresponding to the previous video frame. If the difference is within the preset If the difference value is within the range, then stop sending the first preview gesture data to the recording device. If the difference is outside the preset difference range, the first preview posture data is sent to the recording device, and then the panoramic image data sent by the recording device according to the preview posture data is obtained, and the edited posture data generated after editing the preview posture data is obtained. , and finally play the video corresponding to the panoramic image data based on the edited posture data and preview posture data. Thereby, the video corresponding to the panoramic image data is played through the angle corresponding to the edited posture data and the preview posture data.

Please refer to Figure 6. Figure 6 is a first flow diagram of a data processing method provided by an embodiment of the present application. The data processing method is directed to a recording device. The data processing method may include the following steps:

410. Collect panoramic image data.

The recording device is equipped with a 360-degree panoramic camera, and the panoramic image data is captured through the panoramic camera, and the panoramic image data is native data.

The recording device can convert the panoramic image data into preview panoramic image data with lower definition, and then send the preview panoramic image data to the control device, so that the user can preview the panoramic picture corresponding to the preview panoramic image data through the control device.

420. Receive preview gesture data sent by the control device.

In some implementations, the recording device can also obtain preview gesture data sent from the control device, and the preview gesture data contains corresponding time information. For example, preview posture data at different times corresponds to a timestamp.

430. Correlate the panoramic image data and preview posture data at the same time to generate panoramic image data.

In some embodiments, the recording device can associate the panoramic image data and the preview posture data at the same time according to the time information of the preview posture data and the time information of the panoramic image data, and generate an associated file; set the associated file in the panoramic image data At the tail end, panoramic image data is generated.

440. Send the panoramic image data to the control device.

The recording device can send panoramic image data to the control device through wireless communication.

In the embodiment of this application, the recording device collects panoramic image data and then receives control The preview posture data sent by the device is then associated with the panoramic image data and preview posture data at the same time to generate panoramic image data, and finally the panoramic image data is sent to the control device. Thus, the user's preview posture data and the panoramic image data are associated.

Please refer to FIG. 7 , which is a schematic structural diagram of a video processing device provided by an embodiment of the present application. The video processing device 500 may include:

The sending module 510 is used to obtain corresponding preview posture data when the user previews the panoramic picture, and send the preview posture data to the recording device.

The sending module 510 is also used to determine the difference between the first preview gesture data corresponding to the current video frame and the second preview gesture data corresponding to the previous video frame;

If the difference is outside the preset difference range, the first preview gesture data is sent to the recording device;

If the difference is within the preset difference range, stop sending the first preview gesture data to the recording device.

The sending module 510 is also used to convert the first Euler angle data of the current video frame into a first space vector, and convert the second Euler angle data of the previous video frame into a second space vector;

The difference between the first space vector and the second space vector is determined.

The acquisition module 520 is used to acquire panoramic image data sent by the recording device. The panoramic image data is generated by the recording device based on the recorded panoramic image data and preview posture data.

The extraction module 530 is used to extract preview gesture data and panoramic image data from panoramic image data.

The play module 540 is used to play the video corresponding to the panoramic image data according to the preview gesture data.

The playback module 540 is also used to obtain the edited posture data generated after editing the preview posture data; and play the video corresponding to the panoramic image data according to the edited posture data and the preview posture data.

The playback module 540 is also used to determine the first key frame corresponding to the panoramic image data and the first target preview posture data corresponding to the first key frame;

Obtain the first edited posture data generated after editing the first target preview posture data.

The playback module 540 is also used to determine whether there is a second key frame within the preset time range corresponding to the first key frame;

If there is a second keyframe within the preset time range corresponding to the first keyframe, obtain the second keyframe The second edited posture data corresponding to the frame;

Play the video corresponding to the panoramic image data between the first key frame and the second key frame based on the first edited posture data and the second edited posture data, and play the video corresponding to the panoramic image data outside the preset time range based on the preview posture data. video.

The playback module 540 is also used to perform interpolation calculation on the posture data corresponding to each frame of panoramic image data between the first key frame and the second key frame based on the first edited posture data and the second edited posture data, to obtain the third Interpolated posture data corresponding to each frame of panoramic image data between the first key frame and the second key frame;

Play the video corresponding to the panoramic image data between the first key frame and the second key frame according to the interpolated posture data.

After determining whether there is a second key frame within the preset time range corresponding to the first key frame, the playback module 540 is also configured to determine if there is no second key frame within the preset time range corresponding to the first key frame. The first video frame corresponding to the time starting point and the second video frame corresponding to the time end point within the time range;

Perform interpolation calculation on the posture data of the first key frame according to the preview posture data corresponding to the first video frame and the preview posture data corresponding to the second video frame to obtain the first interpolation posture data corresponding to the first key frame;

Display the picture corresponding to the first key frame according to the first interpolation posture data and the panoramic image data corresponding to the first key frame.

Please refer to FIG. 8 , which is a schematic structural diagram of a data processing device provided by an embodiment of the present application. The data processing device 600 may include:

Collection module 610 is used to collect panoramic image data.

The receiving module 620 is used to receive preview gesture data sent by the control device.

The correlation module 630 is used to correlate panoramic image data and preview posture data at the same time to generate panoramic image data.

The association module 630 is also used to associate the initial panoramic image data and preview posture data at the same time, and generate an association file;

Set the associated file at the end of the panoramic image data to generate the panoramic image data.

The data sending module 640 is used to send panoramic image data to the control device.

In the embodiment of this application, the recording device collects panoramic image data, then receives the preview posture data sent by the control device, then associates the panoramic image data and preview posture data at the same time to generate panoramic image data, and finally converts the panoramic image data into sent to the control device. Thus, the user's preview posture data and the initial panoramic image data are associated.

Correspondingly, embodiments of the present application also provide an electronic device. As shown in Figure 9 , the electronic device may include a memory 701 with one or more computer-readable storage media, an input unit 702, a display unit 703, a sensor 704, It includes a processor 705 with one or more processing cores, a power supply 706 and other components. Those skilled in the art can understand that the structure of the electronic device shown in FIG. 9 does not constitute a limitation on the electronic device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange different components. in:

The memory 701 can be used to store software programs and modules. The processor 705 executes various functional applications and data processing by running the software programs and modules stored in the memory 701 . The memory 701 may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the storage data area may store a program based on Data created by the use of electronic devices (such as audio data, phone books, etc.), etc. In addition, the memory 701 may include high-speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory 701 may also include a memory controller to provide the processor 705 and the input unit 702 with access to the memory 701 .

The input unit 702 may be used to receive input numeric or character information, and to generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control. Specifically, in a specific embodiment, the input unit 702 may include a touch-sensitive surface as well as other input equipment. A touch-sensitive surface, also known as a touch display or trackpad, can collect the user's touch operations on or near it (such as the user using a finger, stylus, or any suitable object or accessory on or near the touch-sensitive surface). operations near the surface), and drive the corresponding connection device according to the preset program. Optionally, the touch-sensitive surface may include two parts: a touch detection device and a touch controller. Among them, the touch detection device detects the user's touch orientation, detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into contact point coordinates, and then sends it to the touch controller. to the processor 705, and can receive commands sent by the processor 705 and execute them. In addition, touch-sensitive surfaces can be implemented using a variety of types including resistive, capacitive, infrared, and surface acoustic waves. In addition to touch-sensitive surfaces, input unit 702 may also include other input devices. Specifically, other input devices may include, but are not limited to, one or more of a physical keyboard, function keys (such as volume control keys, switch keys, etc.), trackball, mouse, joystick, etc.

The display unit 703 may be used to display information input by the user or information provided to the user as well as various graphical user interfaces of the electronic device. These graphical user interfaces may be composed of graphics, text, icons, videos, and any combination thereof. The display unit 703 may include a display panel. Optionally, the display panel may be configured in the form of a liquid crystal display (LCD, Liquid Crystal Display), organic light-emitting diode (OLED, Organic Light-Emitting Diode), etc. Further, the touch-sensitive surface can cover the display panel. When the touch-sensitive surface detects a touch operation on or near the touch-sensitive surface, it is sent to the processor 705 to determine the type of the touch event. The processor 705 then displays the display panel according to the type of the touch event. Corresponding visual output is provided on the panel. Although in FIG. 9 , the touch-sensitive surface and the display panel are used as two independent components to implement the input and input functions, in some embodiments, the touch-sensitive surface and the display panel can be integrated to implement the input and output functions.

The electronic device may also include at least one sensor 704, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor. The ambient light sensor may adjust the brightness of the display panel according to the brightness of the ambient light. The proximity sensor may close the display panel and/or when the electronic device moves to the ear. Backlight. As a kind of motion sensor, the gravity acceleration sensor can detect the magnitude of acceleration in various directions (usually three axes). It can detect the magnitude and direction of gravity when stationary. It can be used to identify applications of electronic device posture (such as horizontal and vertical screen switching, related games, magnetometer attitude calibration), vibration recognition related functions (such as pedometer, tapping), etc.; as for electronic equipment, it can also be configured with gyroscope, barometer, hygrometer, etc. Other sensors such as thermometers, thermometers, and infrared sensors will not be described in detail here.

The processor 705 is the control center of the electronic device, using various interfaces and lines to connect various parts of the entire electronic device, by running or executing software programs and/or modules stored in the memory 701, and calling data stored in the memory 701 , perform various functions of the electronic device and process data, thereby overall monitoring the electronic device. Optionally, the processor 705 may include one or more processing cores; preferably, the processor 705 may integrate an application processor and a modem processor, where the application processor mainly processes the operating system, user interface, application programs, etc. , the modem processor mainly handles wireless communications. It can be understood that the above-mentioned modem processor may not be integrated into the processor 705.

The electronic device also includes a power supply 706 (such as a battery) that supplies power to various components. Preferably, the power supply can be logically connected to the processor 705 through a power management system, so that functions such as charging, discharging, and power consumption management can be implemented through the power management system. Power supply 706 may also include one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

Although not shown, the electronic device may also include a camera, a Bluetooth module, etc., which will not be described again here. Specifically, in this embodiment, the processor 705 in the electronic device loads the computer program stored in the memory 701. The processor 705 loads the computer program to implement various functions:

Extract preview pose data and panoramic image data from panoramic image data;

Those of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructions, or by controlling relevant hardware through instructions. The instructions can be stored in a computer-readable storage medium, and loaded and executed by the processor.

To this end, embodiments of the present application provide a computer-readable storage medium in which a plurality of instructions are stored, and the instructions can be loaded by the processor to execute the steps in any video processing method provided by the embodiments of the present application. . For example, this command can perform the following steps:

Extract preview pose data and panoramic image data from panoramic image data;

For the specific implementation of each of the above operations, please refer to the previous embodiments and will not be described again here.

Among them, the storage medium may include: read-only memory (ROM, Read Only Memory), random access memory (RAM, Random Access Memory), magnetic disk or optical disk, etc.

Since the instructions stored in the storage medium can execute the steps in any video processing method provided by the embodiments of this application, therefore, it is possible to achieve what can be achieved by any video processing method provided by the embodiments of this application. The beneficial effects can be found in the previous embodiments for details and will not be described again here.

The above has introduced in detail a video processing method, device, electronic equipment and storage medium provided by the embodiments of the present application. This article uses specific examples to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only It is used to help understand the methods and core ideas of this application; at the same time, for those skilled in the art, there will be changes in the specific implementation and application scope based on the ideas of this application. In summary, this specification The contents should not be construed as limitations on this application.

Claims

A video processing method, characterized by including:

Obtain corresponding preview posture data when the user previews the panoramic picture, and send the preview posture data to the recording device;

Obtain panoramic image data sent by the recording device, and the panoramic image data is generated by the recording device based on the recorded panoramic image data and the preview posture data;

Extract the preview gesture data and the panoramic image data from the panoramic image data;

Play the video corresponding to the panoramic image data according to the preview gesture data.
The video processing method according to claim 1, characterized in that, obtaining the corresponding preview gesture data when the user previews the panoramic picture, and sending the preview gesture data to the recording device, includes:

Determine a difference between the first preview gesture data corresponding to the current video frame and the second preview gesture data corresponding to the previous video frame;

If the difference is outside the preset difference range, send the first preview gesture data to the recording device;

If the difference is within the preset difference range, stop sending the first preview gesture data to the recording device.
The video processing method according to claim 2, wherein the preview posture data includes Euler angle data, and the determining of the first preview posture data corresponding to the current video frame and the second preview posture corresponding to the previous video frame Differences between data, including:

Convert the first Euler angle data of the current video frame into a first space vector, and convert the second Euler angle data of the previous video frame into a second space vector;

A difference between the first space vector and the second space vector is determined.
The video processing method according to claim 1, wherein playing the video corresponding to the panoramic image data according to the preview gesture data includes:

Obtain edited posture data generated after editing the preview posture data;

Play the video corresponding to the panoramic image data according to the edited posture data and the preview posture data.
The video processing method according to claim 4, characterized in that: obtaining the The edited posture data generated after editing the preview posture data includes:

Determine the first key frame corresponding to the panoramic image data and the first target preview posture data corresponding to the first key frame;

Obtain the first edited posture data generated after editing the first target preview posture data.
The video processing method according to claim 5, wherein playing the video corresponding to the panoramic image data according to the edited posture data and the preview posture data includes:

Determine whether there is a second key frame within the preset time range corresponding to the first key frame;

If there is a second key frame within the preset time range corresponding to the first key frame, obtain the second edited posture data corresponding to the second key frame;

The video corresponding to the panoramic image data between the first key frame and the second key frame is played according to the first edited posture data and the second edited posture data, and the video corresponding to the panoramic image data between the first key frame and the second key frame is played according to the preview posture data. The video corresponding to the panoramic image data outside the preset time range.
The video processing method according to claim 6, wherein the first key frame and the second key frame are played according to the first edited posture data and the second edited posture data. The video corresponding to the panoramic image data, including:

According to the first edited posture data and the second edited posture data, interpolation calculation is performed on the posture data corresponding to each frame of panoramic image data between the first key frame and the second key frame to obtain Interpolated posture data corresponding to each frame of panoramic image data between the first key frame and the second key frame;

Play the video corresponding to the panoramic image data between the first key frame and the second key frame according to the interpolation gesture data.
The video processing method according to claim 6, wherein after determining whether there is a second key frame within the preset time range corresponding to the first key frame, the method further includes:

If there is no second key frame within the preset time range corresponding to the first key frame, then determine the first video frame corresponding to the time starting point and the second video frame corresponding to the time end point within the preset time range;

Interpolation calculation is performed on the posture data of the first key frame according to the preview posture data corresponding to the first video frame and the preview posture data corresponding to the second video frame to obtain the first interpolation value corresponding to the first key frame. attitude data;

The picture corresponding to the first key frame is displayed according to the first interpolation posture data and the panoramic image data corresponding to the first key frame.
A video processing device, characterized by including:

A sending module, used to obtain corresponding preview posture data when the user previews the panoramic picture, and send the preview posture data to the recording device;

An acquisition module, configured to acquire panoramic image data sent by the recording device, where the panoramic image data is generated by the recording device based on the recorded panoramic image data and the preview gesture data;

An extraction module, configured to extract the preview gesture data and the panoramic image data from the panoramic image data;

A playback module, configured to play the video corresponding to the panoramic image data according to the preview gesture data.
An electronic device, characterized by including:

A memory storing executable program code, a processor coupled to the memory;

The processor calls the executable program code stored in the memory to execute the steps in the video processing method according to any one of claims 1 to 8.
A storage medium, characterized in that the storage medium stores a plurality of instructions, and the instructions are suitable for loading by a processor to execute the steps of the video processing method described in any one of claims 1 to 10.