WO2022222113A1 - Procédé, appareil et système de traitement de vidéo, et support de stockage - Google Patents

Procédé, appareil et système de traitement de vidéo, et support de stockage Download PDF

Info

Publication number
WO2022222113A1
WO2022222113A1 PCT/CN2021/089088 CN2021089088W WO2022222113A1 WO 2022222113 A1 WO2022222113 A1 WO 2022222113A1 CN 2021089088 W CN2021089088 W CN 2021089088W WO 2022222113 A1 WO2022222113 A1 WO 2022222113A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
frame
image sequence
motion
frames
Prior art date
Application number
PCT/CN2021/089088
Other languages
English (en)
Chinese (zh)
Inventor
李兵
周游
吴哲
Original Assignee
深圳市大疆创新科技有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 深圳市大疆创新科技有限公司 filed Critical 深圳市大疆创新科技有限公司
Priority to PCT/CN2021/089088 priority Critical patent/WO2022222113A1/fr
Publication of WO2022222113A1 publication Critical patent/WO2022222113A1/fr

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules

Definitions

  • Embodiments of the present invention relate to the technical field of image processing, and in particular, to a video processing method, device, system, and storage medium.
  • video stabilization technology refers to increasing the stability of video during the display process.
  • the video stabilization technology can only eliminate the video jitter caused by rotation, but cannot effectively eliminate the video jitter caused by the displacement change.
  • the embodiments of the present invention provide a video processing method, device, system, and storage medium, which can accurately perform frame extraction processing on the shaking direction, and then perform stabilization processing based on the above-mentioned extracted image frames, which is beneficial to ensure the image or The display quality and effect of the video.
  • a first aspect of the embodiments of the present invention is to provide a video processing method, including:
  • frame extraction processing is performed on the image sequence frame.
  • a second aspect of the embodiments of the present invention is to provide a video processing method, including:
  • a third aspect of the embodiments of the present invention is to provide a video processing apparatus, including:
  • a processor for running a computer program stored in the memory to achieve:
  • frame extraction processing is performed on the image sequence frame.
  • a fourth aspect of the embodiments of the present invention is to provide a video processing apparatus, including:
  • a processor for running a computer program stored in the memory to achieve:
  • a fifth aspect of the embodiments of the present invention is to provide a computer-readable storage medium, where the storage medium is a computer-readable storage medium, and program instructions are stored in the computer-readable storage medium, and the program instructions are used for the first The video processing method described in the aspect.
  • a sixth aspect of the embodiments of the present invention is to provide a computer-readable storage medium, where the storage medium is a computer-readable storage medium, and program instructions are stored in the computer-readable storage medium, and the program instructions are used for the second The video processing method described in the aspect.
  • a seventh aspect of the embodiments of the present invention is to provide a video processing system, including:
  • an image acquisition device for generating image sequence frames
  • the video processing device is connected in communication with the image acquisition device.
  • An eighth aspect of the embodiments of the present invention is to provide a video processing system, including:
  • an image acquisition device for generating image sequence frames
  • the video processing device is connected in communication with the image acquisition device.
  • the shaking direction corresponding to the image sequence frame is determined based on the shooting track, and the shaking direction corresponding to the image sequence frame is determined during the shooting.
  • the motion trajectory located in the shaking direction is extracted from the trajectory, and then the image sequence frame is processed based on the motion trajectory, thereby effectively realizing that the image frame to be selected located in the shaking direction can be accurately determined.
  • video stabilization processing can be performed based on the image frames to be selected, that is, the jitter of the image frames to be selected can be effectively eliminated, so that the video information after the video stabilization processing can be obtained, thereby ensuring the display of images or videos. quality and effect.
  • FIG. 1 is a schematic flowchart of a video processing method according to an embodiment of the present invention.
  • FIG. 2 is a schematic diagram of the principle of a video processing method provided by an embodiment of the present invention.
  • FIG. 3 is a schematic flowchart of determining a main motion direction corresponding to the image sequence frame based on the shooting track provided by an embodiment of the present invention
  • FIG. 4 is a schematic diagram of segmenting the shooting track to obtain a plurality of segmented tracks corresponding to the shooting track according to an embodiment of the present invention
  • FIG. 5 is a schematic flowchart of determining a shaking direction corresponding to the image sequence frame based on the main motion direction according to an embodiment of the present invention
  • FIG. 6 is a schematic diagram of reference motion trajectories corresponding to two reference directions according to an embodiment of the present invention.
  • FIG. 7 is a schematic flowchart of determining image frames to be selected included in the image sequence frames according to the motion trajectory according to an embodiment of the present invention.
  • FIG. 8 is a schematic diagram 1 of determining the motion trend of the image sequence frame in the shaking direction according to an embodiment of the present invention
  • FIG. 9 is a second schematic diagram of determining the motion trend of the image sequence frame in the shaking direction according to an embodiment of the present invention.
  • FIG. 10 is a schematic diagram of determining an image frame to be selected included in the image sequence frame according to an embodiment of the present invention.
  • FIG. 11 is a schematic flowchart of determining the motion trend based on the displacement information of the image sequence frame in the shaking direction according to an embodiment of the present invention
  • FIG. 12 is a schematic diagram of determining maximum displacement information and minimum displacement information corresponding to a time period provided by an embodiment of the present invention
  • FIG. 13 is a schematic flowchart 1 of performing frame extraction processing on the image sequence frame based on the image frame to be selected according to an embodiment of the present invention
  • FIG. 14 is a schematic flowchart 2 of performing frame extraction processing on the image sequence frame based on the image frame to be selected according to an embodiment of the present invention
  • 15 is a schematic diagram of performing segmentation processing on a shooting track according to an application embodiment of the present invention.
  • 16 is a schematic diagram of extracting the main motion direction of each path section provided by an application embodiment of the present invention.
  • 17 is a schematic diagram between a main motion direction and other motion directions provided by an application embodiment of the present invention.
  • 19 is a schematic diagram of processing a selected image frame according to an application embodiment of the present invention.
  • FIG. 20 is a schematic flowchart of another video processing method provided by an embodiment of the present invention.
  • FIG. 21 is a schematic structural diagram of a video processing apparatus according to an embodiment of the present invention.
  • FIG. 22 is a schematic structural diagram of another video processing apparatus provided by an embodiment of the present invention.
  • FIG. 23 is a schematic structural diagram of a video processing system according to an embodiment of the present invention.
  • FIG. 24 is a schematic structural diagram of another video processing system according to an embodiment of the present invention.
  • large-scale mobile time-lapse also known as high dynamic time-lapse
  • a large-scale move time-lapse requires the camera to move a long distance.
  • Hyperlapse is a video spliced by extracting video frames, it can be considered as a timed photo (such as taking a picture in 5s). Even if there is a stabilization system, such as mechanical gimbal or electronic stabilization, etc., the camera also It does not work very well, and the current video processing technology is to import the video into the personal computer PC after the video is acquired. However, it is more troublesome to use the personal computer to process the video stabilization.
  • a stabilization system such as mechanical gimbal or electronic stabilization, etc.
  • the current video stabilization technology can only eliminate the video jitter caused by rotation, but cannot effectively eliminate the video jitter caused by the displacement change. , there will be obvious up and down jitter, and this kind of up and down jitter cannot be effectively eliminated by video stabilization technology.
  • a mechanical device in order to eliminate the jitter caused by the displacement, a mechanical device can be used to perform stabilization processing. However, this requires an additional mechanical structure to perform video stabilization processing, resulting in complex structure, heavy weight, and inconvenience to use.
  • FIG. 1 is a schematic flowchart of a video processing method provided by an embodiment of the present invention
  • FIG. 2 is a schematic schematic diagram of a principle of a video processing method provided by an embodiment of the present invention
  • a video processing method is provided, and the execution body of the video processing method can be a video processing device. It can be understood that the video processing device can be implemented as software or a combination of software and hardware. Specifically, the video processing method can include: The following steps:
  • Step S101 Acquire an image sequence frame and a shooting track corresponding to the image sequence frame.
  • Step S102 Determine the shaking direction corresponding to the image sequence frame based on the shooting track.
  • Step S103 From the photographing trajectory, extract the motion trajectory in the shaking direction.
  • Step S104 Perform frame extraction processing on the image sequence frame based on the motion track.
  • Step S101 Acquire an image sequence frame and a shooting track corresponding to the image sequence frame.
  • the image sequence frame refers to multiple image frames that need to undergo stabilization processing/frame extraction processing. This embodiment does not limit the specific implementation of acquiring the image sequence frame. Those skilled in the art can use the specific application scenarios or applications. need to be set.
  • the video processing device may be provided with an image capture device (camera, camera, etc.) for capturing image frames. At this time, the image capture operation can be performed directly through the image capture device on the video processing device. , to obtain an image sequence frame.
  • the image sequence frame can be stored in a preset area, and the image sequence frame can be obtained by accessing the preset area.
  • a photographing device for capturing image frames is preset, and the photographing device can be set independently of the video processing device.
  • the photographing device is communicatively connected to the video processing device, and then the image capturing operation can be performed by the photographing device.
  • the photographing device can be mounted on a movable platform to perform image photographing operations, and then obtain image sequence frames through the movable platform, wherein the movable platform can include at least one of the following: a pan/tilt, an unmanned aerial vehicle, an unmanned vehicle, a mobile robot Or unmanned boats, etc.
  • the above-mentioned unmanned aerial vehicle can be a rotary-wing unmanned aerial vehicle, such as a quad-rotor unmanned aerial vehicle, a six-rotor unmanned aerial vehicle, an eight-rotor unmanned aerial vehicle, or a fixed-wing unmanned aerial vehicle;
  • the gimbal can include a hand-held gimbal, Or a gimbal that can be mounted on unmanned aerial vehicles, unmanned vehicles or unmanned boats.
  • the shooting track corresponding to the image sequence frame may include at least one of the following: position information corresponding to each of the multiple image frames in the image sequence frame, and shooting device posture information corresponding to each of the multiple image frames in the image sequence frame.
  • this embodiment does not limit the specific implementation of acquiring the shooting track corresponding to the image sequence frame.
  • Those skilled in the art can set according to specific application scenarios or application requirements, and use a video processing device to shoot to obtain an image sequence.
  • a global positioning system Global Positioning System, referred to as GPS
  • an inertial measurement unit Inertial Measurement Unit, referred to as IMU
  • GPS Global Positioning System
  • IMU Inertial Measurement Unit
  • the position information corresponding to the multiple image frames in the image sequence frame can be obtained through GPS
  • the photographing device attitude information corresponding to the multiple image frames in the image sequence frame can be obtained through the IMU.
  • GPS Global Positioning System
  • IMU Inertial Measurement Unit
  • the shooting trajectory corresponding to the image sequence frame may include two-dimensional trajectory information, for example, the horizontal direction is the X direction, the horizontal direction perpendicular to the X direction is the Y direction, and the direction perpendicular to the XY plane is Taking the Z direction as an example, the shooting trajectory corresponding to the image sequence frame may include two-dimensional trajectory information corresponding to the X direction and the Y direction.
  • the shooting track corresponding to the image sequence frame may be a three-dimensional shooting track.
  • the shooting track corresponding to the image sequence frame may include three-dimensional track information corresponding to the X, Y, and Z directions.
  • Step S102 Determine the shaking direction corresponding to the image sequence frame based on the shooting track.
  • the shooting device In the process of acquiring the shooting track corresponding to the image sequence frame, the shooting device needs to move along a preset direction (for example: horizontal X direction, horizontal Y direction, vertical Z direction, oblique direction, etc.), so that Image sequence frames and shooting trajectories corresponding to the image sequence frames can be obtained.
  • the photographing device may generate a certain degree of shaking. For example, when the photographing device moves along the horizontal X direction, the photographing device will move in the horizontal Y direction or the vertical Z direction.
  • the shooting trajectory can be analyzed and processed to determine the shaking direction corresponding to the image sequence frame.
  • the shaking direction is different from the above preset. Other directions with different directions, for example: when the preset direction is the horizontal direction, the shaking direction can be the vertical direction; when the preset direction is the vertical direction, the shaking direction can be the horizontal direction and so on.
  • determining the shaking direction corresponding to the image sequence frame based on the photographing trajectory in this embodiment may include: determining the shaking direction corresponding to the image sequence frame based on the photographing trajectory.
  • the shooting track since the shooting track includes the track information corresponding to the main movement direction, after the shooting track is obtained, the shooting track can be analyzed and processed to determine the main movement direction corresponding to the image sequence frame.
  • the direction is the main motion direction corresponding to the image sequence frame obtained by shooting, for example: when the photographing device moves along the horizontal X direction to perform the image shooting operation, the horizontal X direction is the main motion direction corresponding to the image sequence frame; When the photographing device moves along the horizontal Y direction to perform the image photographing operation, the horizontal Y direction is the main movement direction corresponding to the image sequence frame.
  • the main movement direction is perpendicular to the shaking direction, after the main movement direction is obtained, the main movement direction can be analyzed and processed to determine the shaking direction corresponding to the image sequence frame, which effectively realizes The exact reliability of the determination of the direction of shaking corresponding to the frames of the image sequence.
  • Step S103 From the photographing trajectory, extract the motion trajectory in the shaking direction.
  • the shooting track includes not only the track information corresponding to the main movement direction, but also the track information corresponding to the shaking direction. Therefore, after the shooting track and the shaking direction are obtained, it can be extracted from the shooting track. Motion trajectory in the direction of shaking.
  • Step S104 Perform frame extraction processing on the image sequence frame based on the motion track.
  • performing frame extraction processing on the image sequence frame based on the motion track may include: determining the image frame to be selected included in the image sequence frame according to the motion track; based on the to-be-selected image frame, Perform frame extraction processing on image sequence frames.
  • the motion trajectory corresponds to a plurality of image frames
  • the motion trajectory can be analyzed and processed to determine the image frame to be selected included in the image sequence frame, the image frame to be selected.
  • the frame is the image frame obtained after the frame extraction process in the image sequence frame. Therefore, after the image frame to be selected is obtained, the image sequence frame can be extracted based on the image frame to be selected.
  • the above-mentioned image frames to be selected are extracted from the frames, thereby effectively realizing the frame extraction operation on the image sequence frames according to the shaking direction.
  • the method in this embodiment may further include: performing image stabilization processing based on the image sequence frames after the frame extraction processing, to obtain a video corresponding to the image sequence frames information.
  • image stabilization processing can be performed based on the image sequence frame after the frame extraction process, so that the video information corresponding to the image sequence frame can be obtained.
  • the video information obtained after processing is stabilized, thereby effectively ensuring the display quality and effect of the video information.
  • the image sequence frame (P1, P2, P3, P4 and P5) is obtained by moving the photographing device along the horizontal X direction to perform the image capturing operation as an example for description.
  • the shooting track corresponding to the image sequence frame can be obtained, and the shooting track can be in the form of a sine wave.
  • the shaking direction corresponding to the image sequence frame can be determined based on the shooting track. For example, the Z-axis direction perpendicular to the horizontal X-direction may be determined as the shaking direction.
  • the motion trajectory in the shaking direction can be extracted from the shooting trajectory, and then the image sequence frame can be extracted based on the motion trajectory in the shaking direction.
  • the image frames to be selected can be determined to be P1, P3 and P5 based on the motion trajectory in the Z-axis direction, and then the above-mentioned image frames to be selected can be extracted, and a video can be generated based on the extracted image frames to be selected.
  • the video information is the video after stabilization processing, thereby effectively ensuring the display quality and effect of the video information.
  • the shaking direction corresponding to the image sequence frame is determined based on the shooting track, and the shaking direction is extracted from the shooting track. Then the image sequence frame is processed based on the motion trajectory, which effectively realizes that the image frame to be selected in the direction of shaking can be accurately determined, and then video stabilization can be performed based on the image frame to be selected. Processing can effectively eliminate the jitter of the image frame to be selected, so that the video information after the video stabilization processing can be obtained, thereby ensuring the display quality and effect of the image or video.
  • FIG. 3 is a schematic flowchart of determining a main motion direction corresponding to an image sequence frame based on a shooting track provided by an embodiment of the present invention; with reference to FIG. 3 , this embodiment provides a method for determining a main motion direction corresponding to an image sequence frame.
  • the implementation of the main movement direction, specifically, in this embodiment, based on the shooting track, determining the main movement direction corresponding to the image sequence frame may include:
  • Step S301 Segment the shooting track to obtain a plurality of segmented tracks corresponding to the shooting track.
  • segmenting the shooting track in this embodiment and obtaining multiple segmented tracks corresponding to the shooting track may include: acquiring curvature information corresponding to each shooting track point in the shooting track; based on the curvature information The shooting track is segmented to obtain a plurality of segmented tracks corresponding to the shooting track.
  • the shooting track may be composed of multiple shooting track points, and the curvature information corresponding to different shooting track points in the shooting track may be the same or different.
  • the curvature information corresponding to each shooting track point in the shooting track can be obtained, and then the curvature information is analyzed and processed to perform segmentation processing on the shooting track based on the curvature information, so that the corresponding shooting track can be obtained.
  • the respective corresponding track directions and track lengths of the multiple segmented trajectories may be the same or different, as shown in FIG. 4 .
  • Step S302 Determine the main motion direction corresponding to the image sequence frame based on the plurality of segmented trajectories.
  • the multiple segmented trajectories can be analyzed and processed to determine the main motion direction corresponding to the image sequence frame.
  • the corresponding main motion directions may include: using principal component analysis to process multiple segmented trajectories to determine the main motion directions corresponding to the image sequence frames, wherein each segmented trajectory corresponds to a main motion direction, and different segments
  • the main motion directions corresponding to the segment trajectories are the same or different.
  • the shooting track by segmenting the shooting track, multiple segmented tracks corresponding to the shooting track are obtained, and then the main motion direction corresponding to the image sequence frame is determined based on the multiple segmented tracks, which not only ensures the The accuracy and reliability of determining the main motion direction are also improved, and the accuracy of determining the shaking direction based on the main motion direction is also improved, which further improves the stability and reliability of the video processing method.
  • FIG. 5 is a schematic flowchart of determining a shaking direction corresponding to an image sequence frame based on a main motion direction provided by an embodiment of the present invention; with reference to FIG. 5 , this embodiment provides a method for determining a shaking direction corresponding to an image sequence frame
  • the implementation of the shaking direction, specifically, in this embodiment, based on the main motion direction, determining the shaking direction corresponding to the image sequence frame may include:
  • Step S501 Acquire two reference directions perpendicular to the main movement direction.
  • the shooting track corresponding to the image sequence frame may include track information corresponding to the X-axis direction, the Y-axis direction, and the Z-axis direction
  • the main motion direction after the main motion direction is acquired, it is possible to acquire the main motion direction.
  • Two vertical reference directions for example, when the main motion direction is the X-axis direction, the two reference directions perpendicular to the main motion direction can be the Y-axis direction and the Z-axis direction; when the main motion direction is the Z-axis direction , the two reference directions perpendicular to the main motion direction can be the X-axis direction and the Y-axis direction.
  • Step S502 Based on the shooting trajectory, determine the reference motion trajectory corresponding to each of the two reference directions.
  • the shooting track and the two reference directions can be analyzed and processed to determine the reference motion tracks corresponding to the reference directions.
  • the reference motion trajectories corresponding to the two reference directions can be obtained, that is, A reference motion trajectory corresponding to the Z-axis direction and a reference motion trajectory corresponding to the Y-axis direction can be obtained.
  • Step S503 Determine the shaking direction corresponding to the image sequence frame according to the respective reference motion trajectories corresponding to the two reference directions.
  • the reference motion trajectories corresponding to the two reference directions can be analyzed and processed to determine the shaking direction corresponding to the image sequence frame.
  • determining the shaking direction corresponding to the image sequence frame according to the reference motion trajectories corresponding to the two reference directions may include: comparing the reference motion trajectories corresponding to the two reference directions, and determining the target with a large motion range Reference motion track; the reference direction corresponding to the target reference motion track is determined as the shaking direction corresponding to the image sequence frame.
  • the reference motion corresponding to the two reference directions can be Comparing the motion trajectories, that is, to analyze and compare the reference motion trajectory corresponding to the Y-axis direction and the reference motion trajectory corresponding to the Z-axis direction.
  • the reference motion trajectory corresponding to the Z-axis direction is determined as the target reference motion trajectory.
  • the reference direction corresponding to the target reference motion trajectory can be determined as the shaking direction corresponding to the image sequence frame, that is, the Z-axis direction can be determined as the shaking direction corresponding to the image sequence frame , which effectively realizes that by comparing the reference motion trajectories corresponding to the two reference directions, the reference direction with the larger vibration amplitude is determined as the shaking direction, thereby effectively ensuring the accuracy and reliability of determining the shaking direction.
  • determining the shaking direction corresponding to the image sequence frame according to the reference motion trajectories corresponding to the two reference directions may include: synthesizing the two reference directions based on the reference motion trajectories corresponding to the two reference directions respectively. , obtain the composite direction; determine the composite direction as the shaking direction corresponding to the image sequence frame.
  • the two reference directions can be synthesized based on the reference motion trajectories, so that the synthesized direction can be obtained, and then the synthesized direction can be determined as the corresponding image sequence frame.
  • the shaking direction is effectively realized by combining the two reference directions, so that the combined direction can be determined as the shaking direction corresponding to the image sequence frame, which further improves the flexibility and reliability of determining the shaking direction.
  • two reference directions perpendicular to the main motion direction are obtained, and then the reference motion trajectories corresponding to the two reference directions are determined based on the shooting trajectories, and the reference motion trajectories corresponding to the two reference directions are determined according to the respective reference motion trajectories of the two reference directions.
  • the shaking direction corresponding to the image sequence frame effectively ensures the accuracy and reliability of determining the shaking direction, and further improves the accuracy of processing image data or video data.
  • FIG. 7 is a schematic flowchart of determining image frames to be selected in an image sequence frame according to a motion trajectory provided by an embodiment of the present invention; with reference to FIG. The implementation of the included image frames to be selected. Specifically, this embodiment provides a method for determining the image frames to be selected included in the image sequence frame according to the motion trajectory, which may include:
  • Step S701 Based on the motion trajectory, determine the motion trend of the image sequence frame in the shaking direction.
  • the motion trajectory can be analyzed and processed to determine the motion trend of the image sequence frame in the shaking direction.
  • the motion trend of the image sequence frame in the shaking direction is determined. It may include: determining the displacement information of the image sequence frame in the shaking direction based on the motion trajectory; determining the motion trend based on the displacement information of the image sequence frame in the shaking direction.
  • the motion trajectory can be analyzed and processed to determine the image sequence frame in the Z-axis direction. Then, the movement trend can be determined based on the displacement information of the image sequence frame in the Z-axis direction.
  • the movement area can be a relatively stable movement amplitude.
  • the movement trend determined in Figure 8 is in the Z-axis direction. A straight line L in the direction and relatively stable amplitude. It can be understood that the movement trend of the image sequence frame in the shaking direction may not only be a straight line, but also an inclined line L, as shown in FIG. 9 .
  • Step S702 Determine the image frames to be selected included in the image sequence frames based on the motion track and motion trend.
  • the motion track and motion trend may be analyzed and processed to determine the image frames to be selected included in the image sequence frames.
  • determining the image frames to be selected included in the image sequence frame based on the motion track and the motion trend may include: acquiring the image frame intersection existing between the motion track and the motion trend; The image frames included in the set are determined to be image frames to be selected.
  • the intersection of image frames existing between the motion trajectory and the motion trajectory can be obtained, and the intersection of image frames includes multiple image frames.
  • the image frames included in the frame intersection lie both in the motion track and in the motion trend.
  • the image frames included in the image frame intersection can be determined as the image frames to be selected, thereby effectively ensuring that the determination of the image frames to be selected included in the image sequence frame is accurate and reliable. sex.
  • FIG. 11 is a schematic flowchart of determining a motion trend based on displacement information of an image sequence frame in a shaking direction provided by an embodiment of the present invention; with reference to FIG. 11 , this embodiment provides an implementation method for determining a motion trend.
  • the determination of the motion trend based on the displacement information of the image sequence frame in the shaking direction may include:
  • Step S1101 Based on the displacement information of the image sequence frame in the shaking direction, determine the maximum displacement information and the minimum displacement information corresponding to a time period.
  • the displacement information can be analyzed and compared. Specifically, a plurality of displacement information located in a time period can be analyzed and compared, and then the comparison with a time period can be determined. The corresponding maximum position information and minimum position information.
  • the specific time length of the time period is not limited in this embodiment, and those skilled in the art can set it according to specific application scenarios and application requirements. For example, the time period can be 5min, 10min, or 15min, etc.
  • Step S1102 Determine a plurality of displacement change processes composed of the minimum displacement information and the maximum displacement information.
  • a plurality of displacement change processes composed of the minimum displacement information and the maximum displacement information can be obtained, wherein the displacement change process may include a peak process and a trough process, and the above-mentioned peak process may be The process of changing from the minimum displacement information to the maximum displacement information, and the trough process can be the process of changing from the maximum displacement information to the minimum displacement information.
  • Step S1103 Determine a motion trend based on multiple displacement change processes.
  • determining the movement trend based on the multiple displacement change processes may include: calculating a median point corresponding to each of the multiple displacement change processes; and determining the movement trend based on the respective median points corresponding to the multiple displacement change processes.
  • the multiple displacement change processes can be analyzed and processed to calculate the respective median points corresponding to the multiple displacement change processes. Therefore, the multiple displacement change processes corresponding to the multiple displacement change processes can be obtained. Multiple median points, and then the respective median points corresponding to multiple displacement change processes can be analyzed and processed to determine the movement trend.
  • determining the motion trend based on the respective median points of the multiple displacement change processes may include: performing a fitting process on the respective median points corresponding to the multiple displacement change processes to obtain a fitting line corresponding to the median point. ; Identify the fitted line as a motion trend.
  • the displacement information of the image sequence frame in the shaking direction is analyzed and processed to determine the maximum displacement information and the minimum displacement information corresponding to a time period, and then based on the determination of the maximum displacement information and the maximum displacement information A plurality of displacement change processes are formed, and the movement trend is determined based on the plurality of displacement change processes, thereby effectively ensuring the accuracy and reliability of the determination of the movement trend.
  • FIG. 13 is a schematic flow chart 1 of performing frame extraction processing on image sequence frames based on image frames to be selected according to an embodiment of the present invention; with reference to FIG. 13 , this embodiment provides a method for performing frame extraction on image sequence frames.
  • the implementation manner of the processing, specifically, in this embodiment, based on the image frame to be selected, performing frame extraction processing on the image sequence frame may include:
  • Step S1301 Obtain the time interval between adjacent image frames to be selected.
  • Step S1302 Based on the time interval and the image frames to be selected, perform frame extraction processing on the image sequence frames.
  • the adjacent image frames to be selected can be analyzed and processed to obtain the time interval corresponding to the adjacent image frames to be selected.
  • the time intervals between the image frames to be selected may be the same or different, for example: the image frames to be selected sequentially include: image frame P1, image frame P2, image frame P3, and image frame P4, wherein adjacent image frames P1
  • the time interval between the image frame P2 and the image frame P2 is 5 frames
  • the time interval between the adjacent image frame P2 and the image frame P3 is 7 frames
  • the time interval between the adjacent image frame P3 and the image frame P4 is 8 frames.
  • the time interval and the image frames to be selected may be analyzed and processed, so as to implement frame extraction processing on the image sequence frames.
  • performing frame extraction processing on the image sequence frames may include: when the time interval is greater than a time threshold, generating a compensation image frame corresponding to the time interval, and the compensation image frame is located at Between adjacent image frames to be selected corresponding to the time interval; based on the image frames to be selected and the compensation image frame, the frame extraction process is performed on the image sequence frames.
  • the time interval After obtaining the time interval between adjacent image frames to be selected, the time interval can be analyzed and compared with a preset time threshold. When the time interval is greater than the time threshold, the above-mentioned adjacent images to be selected are indicated.
  • the time interval corresponding to the frame is relatively large.
  • a compensation image frame corresponding to the time interval can be generated, and the compensation image frame is located in the adjacent corresponding to the time interval. between image frames to be selected.
  • generating the compensation image frame corresponding to the time interval may include: acquiring adjacent image frames to be selected corresponding to the time interval; determining image change information existing between adjacent image frames to be selected; information to generate compensated image frames corresponding to time intervals.
  • the adjacent image frames to be selected corresponding to the time interval can be obtained, and then the image change information existing between the adjacent image frames to be selected can be determined, After the image change information is acquired, the compensation image frame corresponding to the time interval can be generated based on the image change information, thereby effectively ensuring the quality and effect of generating the compensation image frame.
  • the image frame to be selected and the compensation image frame can be analyzed and processed, so that the image sequence frame can be subjected to stable and effective frame extraction processing.
  • performing frame extraction processing on the image sequence frames may include: when the time interval is less than a time threshold, then processing adjacent image frames to be selected corresponding to the time interval Screening is performed to obtain the screened image frames; based on the screened image frames, frame extraction processing is performed on the image sequence frames.
  • the time interval can be analyzed and compared with a preset time threshold, and when the time interval is less than the time threshold, the above-mentioned adjacent images to be selected are indicated
  • the time interval corresponding to the frame is relatively small.
  • the adjacent image frames to be selected corresponding to the time interval can be screened, so as to obtain the filtered image frame;
  • screening the adjacent image frames to be selected corresponding to the time interval, and obtaining the filtered image frames may include: removing any image frames to be selected corresponding to the time interval from the adjacent image frames to be selected image frame, obtain the screened image frame; after the screened image frame is obtained, the image sequence frame can be processed based on the screened image frame, so as to effectively realize the frame extraction processing of video data or image data, and further The stability and reliability of the video processing method are guaranteed.
  • FIG. 14 is a schematic diagram 2 of a flowchart of performing frame extraction processing on an image sequence frame based on an image frame to be selected according to an embodiment of the present invention; with reference to FIG. 14 , this embodiment provides another method for extracting an image sequence frame.
  • the implementation manner of frame processing, specifically, in this embodiment, based on the image frame to be selected, performing frame extraction processing on the image sequence frame may include:
  • Step S1401 Obtain the average interval of image frames between adjacent image frames to be selected.
  • Step S1402 Determine the frame sampling multiple based on the average interval of image frames.
  • Step S1403 Perform frame extraction processing on the image sequence frame based on the frame extraction multiple.
  • the adjacent image frames to be selected can be analyzed and processed to obtain the average interval of image frames between the adjacent image frames to be selected, and then the image frame average interval can be obtained.
  • the average interval of the image frames is analyzed to determine the frame sampling multiple.
  • determining the frame sampling multiple based on the image frame average interval may include: determining an integer multiple of the image average interval as the frame sampling multiple.
  • the image sequence obtained by moving the handheld gimbal is taken as an example for illustration.
  • This application embodiment provides a video stabilization method for large-scale mobile time-lapse photography, which can eliminate the video jitter caused by rotation and translation. , specifically, the method may include the following steps:
  • Step 1 Calculate and record the shooting trajectory.
  • the image sequence frame when the image sequence frame is obtained by shooting with a handheld PTZ, the image sequence frame includes multiple image frames.
  • the handheld PTZ is equipped with GPS, IMU and visual mileage calculation unit VIO module, and then uses GPS, IMU and VIO to calculate And record the shooting track of the image sequence frame obtained by shooting; specifically, the shooting track corresponding to the image sequence frame can be recorded by the combination of IMU and GPS or the combination of IMU and VIO, and the shooting track can include the corresponding image frame.
  • Corresponding position information, PTZ attitude information, moving speed information, etc., the above-mentioned moving speed information can be determined by the change information of the photographed object in multiple image frames.
  • Step 2 Segment the shooting track.
  • the shooting track can be segmented. Specifically, the shooting track can be segmented according to the length of the track, or a fourth-order Bezier curve fitting algorithm can be used to realize the segment processing; Alternatively, the shooting track can be segmented according to the curvature of the track. That is, there are many ways to perform segment processing on the shooting track. For the convenience of understanding, this application example provides a method based on the Douglas-Puck algorithm ( Douglas–Peucker algorithm, also known as Lamer-Douglas-Pucker algorithm, iterative adaptive point algorithm, split and merge algorithm) to perform segmentation processing. Segmentation can include the following steps:
  • Step 3 Determine the main motion direction corresponding to the frame of the image sequence.
  • PCA principal component analysis
  • a movement direction is used to identify the direction within a time period, wherein each path corresponds to a main movement direction, which is denoted as the x-axis direction, then the y-axis direction and the z-axis direction constitute a direction perpendicular to the main movement direction ( x-axis direction), as shown in Figure 17.
  • Step 4 Determine the shaking direction (may be referred to as the secondary motion direction) corresponding to the frame of the image sequence.
  • the main motion direction determines the motion in two directions on the plane perpendicular to the main motion direction, that is, the motion in the y-axis direction and the z-axis direction perpendicular to the x-axis direction can be obtained; after acquiring the y-axis direction After the movement in the direction and the movement in the z-axis direction, a direction with a larger movement can be determined, and then the direction can be determined as the shaking direction corresponding to the image sequence frame; alternatively, the y-axis direction and the z-axis direction can be combined into one vector, and then the synthesized vector is determined as the shaking direction corresponding to the image sequence frame, thereby effectively realizing that the shaking direction corresponding to the image sequence frame can be determined through the motion trajectory.
  • Step 5 frame extraction processing.
  • the motion trajectory of the shaking direction can be extracted based on the shooting trajectory.
  • the main movement direction is the forward movement (x-axis direction)
  • the turbulence of the picture mainly comes from the user's walking.
  • displacement in the vertical direction (z-axis direction) occurred during the alternating footsteps, which caused vibration and affected the stability of the picture.
  • the position changes in the vertical direction were recorded.
  • the median value of each peak and trough can be calculated, as shown in the dots in Figure 18, and then the random sample consensus algorithm (RANSAC) can be used to fit a straight line , the straight line is the line with the most intersection with the motion trajectory and the most uniform distribution.
  • the straight line is used to represent the motion trend in the direction of shaking.
  • the motion trend can be a relatively stable linear motion. It is understandable that the motion The trend is not necessarily a straight line (k is not necessarily equal to 1), it may also be a diagonal line (corresponding to uphill/stairs).
  • the point where the motion trajectory and the motion trend line intersect can be determined as the selected image frame, and after the selected image frame is obtained, the frame can be drawn process.
  • a video can be assembled based on the selected image frames, thereby causing the feeling of time fast-forwarding.
  • the fast-forward multiple arbitrarily, such as: 2x speed, 3x speed.
  • the user-selectable frame extraction multiple relationship is 7 times, 14 times, 21 times... .
  • the selected image frame may be further analyzed and processed to improve the quality and effect of the selected image frame. Specifically, the time interval between adjacent image frames to be selected is obtained, and the time interval is analyzed and compared with a preset threshold.
  • the time interval is greater than the time threshold, it means that during the frame extraction process, the distance between adjacent image frames is If the time interval is too large, a compensation image frame corresponding to the time interval can be generated, and the compensation image frame is located between the adjacent image frames to be selected corresponding to the time interval; when the time interval is less than the time threshold, then The adjacent image frames to be selected corresponding to the time interval are screened to obtain the screened image frames, which effectively guarantees the quality and effect of the selected frames.
  • This application example uses the computer vision image processing method to determine the image frame to be extracted by estimating the motion trend and the motion trajectory, and then based on the image frame to be extracted, the video jitter can be effectively suppressed, so that a relatively smooth video can be obtained.
  • this method can be applied to smart devices, so as to quickly realize video stabilization, which can solve the situation of video ups and downs caused by the photographer walking without using a mechanical device, which further improves the use of this video stabilization method. stable reliability.
  • the video stabilization processing method in this application embodiment may include the following steps:
  • Step 11 Acquire image sequence frames.
  • the user starts to perform hyperlapse on the preset application program, and then starts to take pictures at intervals to obtain a series of pictures (ie, image sequence frames).
  • a series of pictures ie, image sequence frames.
  • the above-mentioned video processing method can be directly used to extract frames to extract pictures, so as to obtain image sequence frames.
  • the above-mentioned video stabilization algorithm can be called once every time a picture is taken.
  • the image can be converted. It is a 1080p image, or you can convert a color image to a grayscale image, just convert the color image to YUV format, and then directly use the image information of the Y channel.
  • Step 12 Acquire the gimbal attitude information corresponding to the image frame in the image sequence frame.
  • the current gimbal attitude of the current image frame and the previous gimbal attitude of the previous image frame are obtained, and the relative rotation relationship of the current image frame with respect to the previous image frame is calculated, which is denoted as R gyro, i .
  • the gyroscope in the gimbal IMU can obtain the above-mentioned relative rotation relationship.
  • Step 13 Analyze and process the image frames in the image sequence frames to obtain image feature points.
  • the sparse representation function sparse can be used to analyze and process the image frame.
  • the feature points of the image can be extracted first.
  • the point can be used as the feature point of the image, wherein, the corner point can be determined by using the corner point detection algorithm (Corner Detection Algorithm), and the above-mentioned corner point detection algorithm can include at least one of the following: the characteristic algorithm of the accelerated segment test ( features from accelerated segment test (FAST for short), Smallest Univalue Segment Assimilating Nucleus (SUSAN for short) and Harris corner detection, etc.
  • Harris Corner detection Harris corner detection
  • a matrix A When analyzing and processing an image frame, a matrix A can be defined as a construction tensor. Specifically, the tensor matrix A can be expressed as:
  • I x and I y are the gradient information of a certain pixel on the image frame in the x and y directions, respectively.
  • Mc the eigenvalue function Mc of the pixel
  • det(A) is the determinant of matrix A
  • trace(A) is the trace of matrix A
  • ⁇ is the parameter for adjusting the sensitivity
  • the threshold Mth is set.
  • Step 14 After the image feature points are acquired, the optical flow tracking operation can be performed by using the Kanade-Lucas-Tomasi feature tracker (KLT for short) to calculate its movement (light flow).
  • KLT Kanade-Lucas-Tomasi feature tracker
  • h can be used as the offset between the two image frames before and after, that is, the offset between the image frame F(x) and the image frame G(x) can be F(x+h), and then for each image frame
  • the optical flow tracking operation can be realized by the following formula:
  • the displacement h of the image feature points between the front and rear image frames can be obtained by iteration.
  • the latter image be F(x) and the previous image be G(x)
  • h -h ⁇
  • Step 15 Construct the reprojection error constraint.
  • the camera internal parameter matrix K is obtained, and then the reprojection error constraint is constructed based on the optical flow vector solved in the previous step, and R gyro, i is used to construct the error constraint measured by the IMU gyroscope.
  • R gyro, i is used to construct the error constraint measured by the IMU gyroscope.
  • the relative translation t is a variable, and the beam adjustment method is used to optimize the solution:
  • R gyro, i is the relative rotation relationship of the current image frame relative to the previous image frame, That is, R gyro, the transpose of i , R is the actual relative rotation of the two frames before and after, t is the relative translation of the two frames before and after, I is the identity matrix of identiy, and P i is the three-dimensional point in the camera coordinate system of the previous frame. three-dimensional coordinates.
  • the parameters (objectives) to be optimized in the arg function are R, t, P i .
  • ⁇ represents the projection function
  • P i is the 3D coordinate of the 3D point in the camera coordinate system of the previous frame
  • pi is the 3D point in the previous frame
  • R and t represent the rotation and translation transformation of the current frame relative to the previous frame
  • w is the weight (specifically, it can be an engineering experience value), which can be adjusted as needed.
  • Step 16 Smoothly rotate and stabilize the video.
  • the gimbal pose R i ′ corresponding to the rear image; Perform image affine transformation on each frame of the original image to obtain a new re-rendered image, namely:
  • p is the pixel point on the original image
  • the corresponding p' is the point on the corrected new image
  • Step 17 Synthesize the new image to stabilize the video.
  • the new image After obtaining the corrected new image, the new image can be synthesized into a stabilized video.
  • the above-mentioned image sequence frame may be a frame-extracted image obtained by performing frame extraction processing on the shaking direction, thereby not only effectively realizing the stabilization effect of the video, but also avoiding the influence of the video on the shaking direction. It should be noted that every time an image is acquired, a real-time image processing operation can be performed on the image, so that a new image corresponding to the image can be obtained. The new image is used for video synthesis operation, so that the final stabilized video can be obtained.
  • the motion trend of shooting is estimated by computer vision algorithm, and the motion trajectory can be smoothed, and then the stabilized and corrected image is calculated, so as to complete the stabilization processing of the video stream, and then combined with the attitude information of the gimbal itself to quickly
  • the video processing operation is completed online, so that the quality and effect of the video processing can be ensured without the need to perform the video processing operation through other equipment in the later stage, and the stability and reliability of the use of the method can be further improved.
  • FIG. 20 is a schematic flowchart of another video processing method provided by an embodiment of the present invention. with reference to FIG. 20 , this embodiment provides another video processing method, and the execution body of the video processing method may be a video processing device , it can be understood that the video processing apparatus can be implemented as software, or a combination of software and hardware. Specifically, the video processing method can include the following steps:
  • Step S2001 Identify the shaking direction of the image acquisition device based on the image sequence frames recorded by the image acquisition device.
  • Step S2002 Determine the frame sampling period in the time-lapse photography based on the motion trajectory of the image acquisition device in the direction of shaking, and extract the frames of the image sequence according to the frame sampling period to obtain the time-lapse photography video, so as to eliminate the time-lapse photography video. jitter in the jitter direction.
  • Step S2001 Identify the shaking direction of the image acquisition device based on the image sequence frames recorded by the image acquisition device.
  • the video processing device is communicatively connected with an image acquisition device, and the image acquisition device can perform an image recording operation, thereby generating an image sequence frame, and the image sequence frame includes a plurality of image frames. After the image sequence frame recorded by the image acquisition device is acquired After that, the image sequence frame can be analyzed and processed to identify the shaking direction of the image acquisition device.
  • Step S2002 Determine the frame sampling period in the time-lapse photography based on the motion trajectory of the image acquisition device in the direction of shaking, and extract the frames of the image sequence according to the frame sampling period to obtain the time-lapse photography video, so as to eliminate the time-lapse photography video. jitter in the jitter direction.
  • the shaking direction of the image acquisition device After the shaking direction of the image acquisition device is obtained, the shaking direction of the image acquisition device can be analyzed and processed to extract the movement trajectory of the image acquisition device in the shaking direction, and then the movement trajectory of the image acquisition device in the shaking direction can be determined based on the movement trajectory of the image acquisition device in the shaking direction.
  • the frame sampling period in time-lapse photography. After the frame extraction period is obtained, the image sequence frames can be processed according to the frame extraction period, so as to obtain a time-lapse photography video, which can eliminate the shaking in the direction of shaking in the time-lapse photography video, and further ensure quality and effect of video processing.
  • the shaking direction of the image capturing device is identified based on the image sequence frames recorded by the image capturing device, and then the frame sampling period in time-lapse photography is determined based on the motion trajectory of the image capturing device in the shaking direction.
  • the frame period of the image sequence frame is extracted, which effectively ensures the accuracy and reliability of the time-lapse video, further eliminates the jitter in the direction of the jitter in the time-lapse video, and improves the quality and quality of video processing. Effect.
  • identifying the shaking direction of the image capture device based on the image sequence frames recorded by the image capture device may include: based on the shooting trajectory corresponding to the image sequence frames recorded by the image capture device; The main motion direction of ; based on the main motion direction, the shaking direction corresponding to the image sequence frame is determined.
  • determining the shaking direction corresponding to the image sequence frame based on the main motion direction may include: acquiring two reference directions perpendicular to the main motion direction; determining the reference motion corresponding to each of the two reference directions based on the shooting track Trajectory; according to the respective reference motion trajectories corresponding to the two reference directions, determine the shaking direction corresponding to the image sequence frame.
  • determining the shaking direction corresponding to the image sequence frame according to the reference motion trajectories corresponding to the two reference directions may include: comparing the reference motion trajectories corresponding to the two reference directions, and determining the target with a large motion range Reference motion track; the reference direction corresponding to the target reference motion track is determined as the shaking direction corresponding to the image sequence frame.
  • determining the shaking direction corresponding to the image sequence frame according to the reference motion trajectories corresponding to the two reference directions may include: synthesizing the two reference directions based on the reference motion trajectories corresponding to the two reference directions respectively, Obtain the composite direction; determine the composite direction as the shaking direction corresponding to the image sequence frame.
  • FIG. 21 is a schematic structural diagram of a video processing apparatus provided by an embodiment of the present invention. with reference to FIG. 21 , this embodiment provides a video processing apparatus, which is used to execute the video shown in FIG. 1 above.
  • the processing method specifically, the video processing apparatus may include:
  • a first memory 12 for storing computer programs
  • the first processor 11 is used for running the computer program stored in the first memory 12 to realize:
  • the motion track located in the shaking direction is extracted
  • the first processor 11 is further configured to execute all or part of the steps in at least some of the embodiments shown in FIG. 1 to FIG. 19 .
  • the structure of the electronic device may further include a first communication interface 13 for the electronic device to communicate with other devices or a communication network.
  • the first processor 11 determines the shaking direction corresponding to the image sequence frame based on the shooting track
  • the first processor 11 is configured to: determine the main motion direction corresponding to the image sequence frame based on the shooting track ; Based on the main motion direction, determine the shaking direction corresponding to the frame of the image sequence.
  • the first processor 11 determines the main movement direction corresponding to the image sequence frame based on the shooting track
  • the first processor 11 is configured to: segment the shooting track, and obtain a moving direction corresponding to the shooting track. Multiple segmented trajectories; based on the multiple segmented trajectories, determine the main direction of motion corresponding to the frame of the image sequence.
  • the first processor 11 when the first processor 11 segments the shooting trajectory to obtain multiple segmented trajectories corresponding to the shooting trajectory, the first processor 11 is configured to: obtain the location of each shooting trajectory point in the shooting trajectory. Corresponding curvature information; segment the shooting track based on the curvature information, and obtain a plurality of segmented tracks corresponding to the shooting track.
  • the first processor 11 determines the main motion direction corresponding to the image sequence frame based on the plurality of segmented trajectories
  • the first processor 11 is configured to: analyze the plurality of segmented trajectories by using principal component analysis Processing is performed to determine the dominant direction of motion corresponding to the frames of the image sequence.
  • each segmented trajectory corresponds to a main movement direction
  • the main movement directions corresponding to different segmented trajectories are the same or different.
  • the first processor 11 determines the shaking direction corresponding to the image sequence frame based on the main motion direction
  • the first processor 11 is configured to: obtain two reference directions perpendicular to the main motion direction; Shooting the track, determine the reference motion track corresponding to the two reference directions; determine the shaking direction corresponding to the image sequence frame according to the reference motion track corresponding to the two reference directions.
  • the first processor 11 determines the shaking direction corresponding to the image sequence frame according to the reference motion trajectories corresponding to the two reference directions
  • the first processor 11 is configured to: correspond to the two reference directions respectively The reference motion trajectory of the target is compared, and the target reference motion trajectory with a large movement range is determined; the reference direction corresponding to the target reference motion trajectory is determined as the shaking direction corresponding to the image sequence frame.
  • the first processor 11 determines the shaking direction corresponding to the image sequence frame according to the respective reference motion trajectories corresponding to the two reference directions
  • the first processor 11 is configured to: based on the respective corresponding two reference directions
  • the two reference directions are synthesized to obtain the synthesized direction; the synthesized direction is determined as the shaking direction corresponding to the image sequence frame.
  • the first processor 11 when the first processor 11 performs frame extraction processing on the image sequence frame based on the motion trajectory, the first processor 11 is configured to: determine the image frame to be selected included in the image sequence frame according to the motion trajectory ; Based on the image frame to be selected, the frame extraction process is performed on the image sequence frame.
  • the first processor 11 determines the image frames to be selected included in the image sequence frames according to the motion trajectory
  • the first processor 11 is configured to: determine, based on the motion trajectory, that the image sequence frames are in the shaking direction based on the motion trajectory and the motion trend, determine the image frames to be selected included in the image sequence frames.
  • the first processor 11 determines the motion trend of the image sequence frame in the shaking direction based on the motion trajectory
  • the first processor 11 is configured to: determine the displacement of the image sequence frame in the shaking direction based on the motion trajectory information; based on the displacement information of the image sequence frame in the direction of shaking, determine the motion trend.
  • the first processor 11 determines the motion trend based on the displacement information of the image sequence frames in the shaking direction
  • the first processor 11 is configured to: based on the displacement information of the image sequence frames in the shaking direction, determine and The maximum displacement information and the minimum displacement information corresponding to a time period; determine multiple displacement change processes formed by the minimum displacement information and the maximum displacement information; determine the movement trend based on the multiple displacement change processes.
  • the first processor 11 determines the movement trend based on multiple displacement change processes
  • the first processor 11 is configured to: calculate the median point corresponding to each of the multiple displacement change processes; based on the multiple displacement change processes The corresponding median points are used to determine the movement trend.
  • the first processor 11 determines the motion trend based on the respective median points of the multiple displacement change processes
  • the first processor 11 is configured to: perform a calculation on the respective median points corresponding to the multiple displacement change processes.
  • the fitting process is performed to obtain a fitting line corresponding to the median point; the fitting line is determined as a motion trend.
  • the first processor 11 determines the image frames to be selected included in the image sequence frame based on the motion track and the motion trend
  • the first processor 11 is configured to: obtain the difference between the motion track and the motion trend.
  • the existing image frame intersection; the image frames included in the image frame intersection are determined as the image frames to be selected.
  • the first processor 11 when the first processor 11 performs frame extraction processing on the image sequence frames based on the image frames to be selected, the first processor 11 is configured to: acquire the time interval between adjacent image frames to be selected. ; Based on the time interval and the image frame to be selected, the frame extraction process is performed on the image sequence frame.
  • the first processor 11 when the first processor 11 performs frame extraction processing on the image sequence frames based on the time interval and the image frames to be selected, the first processor 11 is configured to: when the time interval is greater than the time threshold, generate a Compensation image frames corresponding to the time interval, and the compensated image frames are located between adjacent image frames to be selected corresponding to the time interval; based on the image frames to be selected and the compensation image frames, frame extraction processing is performed on the image sequence frames.
  • the first processor 11 when the first processor 11 generates the compensation image frame corresponding to the time interval, the first processor 11 is configured to: acquire adjacent image frames to be selected corresponding to the time interval; The image change information existing between the image frames is selected; based on the image change information, the compensation image frame corresponding to the time interval is generated.
  • the first processor 11 when the first processor 11 performs frame extraction processing on the image sequence frames based on the time interval and the image frames to be selected, the first processor 11 is configured to: when the time interval is less than the time threshold, perform a comparison between the time interval and the time threshold. The adjacent image frames to be selected corresponding to the time interval are screened to obtain the screened image frames; based on the screened image frames, the frame selection processing is performed on the image sequence frames.
  • the first processor 11 when the first processor 11 filters adjacent image frames to be selected corresponding to the time interval to obtain filtered image frames, the first processor 11 is configured to: In the frame, any image frame to be selected corresponding to the time interval is removed to obtain a filtered image frame.
  • the first processor 11 when the first processor 11 performs frame extraction processing on the image sequence frames based on the image frames to be selected, the first processor 11 is configured to: acquire image frames between adjacent image frames to be selected. Average interval; determine the frame sampling multiple based on the average interval of the image frames; perform frame sampling processing on the image sequence frame based on the frame sampling multiple.
  • the first processor 11 determines the frame sampling multiple based on the average interval of the image frames
  • the first processor 11 is configured to: determine an integer multiple of the image average interval as the frame sampling multiple.
  • the first processor 11 is configured to: perform image stabilization processing based on the image sequence frame after the frame extraction process, to obtain video information corresponding to the image sequence frame.
  • the parachute shown in FIG. 21 can perform the method of the embodiment shown in FIG. 1 to FIG. 19 .
  • an embodiment of the present invention provides a computer storage medium for storing computer software instructions used by an electronic device, which contains the video processing methods involved in executing at least some of the embodiments shown in FIG. 1 to FIG. 19 above. program of.
  • FIG. 22 is a schematic structural diagram of another video processing apparatus provided by an embodiment of the present invention. with reference to FIG. 22 , this embodiment provides another video processing apparatus, and the video processing apparatus is used to execute the above-mentioned video processing apparatus shown in FIG. 20 .
  • the video processing method specifically, the video processing device may include:
  • the second processor 21 is used for running the computer program stored in the second memory 22 to realize:
  • the second processor 21 is further configured to execute all or part of the steps in at least some of the embodiments shown in FIG. 20 .
  • the structure of the electronic device may further include a second communication interface 23 for the electronic device to communicate with other devices or a communication network.
  • the second processor 21 when the second processor 21 identifies the shaking direction of the image capture device based on the image sequence frames recorded by the image capture device, the second processor 21 is configured to: capture images corresponding to the image sequence frames recorded by the image capture device track; based on the shooting track, determine the main motion direction corresponding to the image sequence frame; based on the main motion direction, determine the shaking direction corresponding to the image sequence frame.
  • the second processor 21 determines the shaking direction corresponding to the image sequence frame based on the main motion direction
  • the second processor 21 is configured to: obtain two reference directions perpendicular to the main motion direction; Shooting the track, determine the reference motion track corresponding to the two reference directions; determine the shaking direction corresponding to the image sequence frame according to the reference motion track corresponding to the two reference directions.
  • the second processor 21 determines the shaking direction corresponding to the image sequence frame according to the reference motion trajectories corresponding to the two reference directions
  • the second processor 21 is configured to: correspond to the two reference directions respectively The reference motion trajectory of the target is compared, and the target reference motion trajectory with a large movement range is determined; the reference direction corresponding to the target reference motion trajectory is determined as the shaking direction corresponding to the image sequence frame.
  • the second processor 21 determines the shaking direction corresponding to the image sequence frame according to the reference motion trajectories corresponding to the two reference directions
  • the second processor 21 is configured to: based on the corresponding corresponding two reference directions
  • the two reference directions are synthesized to obtain the synthesized direction; the synthesized direction is determined as the shaking direction corresponding to the image sequence frame.
  • the parachute shown in FIG. 22 can perform the method of the embodiment shown in FIG. 20 .
  • the parts not described in detail in this embodiment reference may be made to the relevant description of the embodiment shown in FIG. 20 .
  • an embodiment of the present invention provides a computer storage medium for storing computer software instructions used by an electronic device, which includes a program for executing the video processing method in at least some of the embodiments shown in FIG. 20 .
  • FIG. 23 is a schematic structural diagram of a video processing system provided by an embodiment of the present invention. with reference to Figure 23, the present embodiment provides a video processing system, which may include:
  • the image acquisition device 31 is used to generate an image sequence frame, wherein the image acquisition device 31 can be a camera equipped with a camera, a mobile phone, glasses, etc., such as virtual reality (VR, virtual reality) glasses or a first-person perspective (FPV, first person view). )Glasses.
  • VR virtual reality
  • FMV first-person perspective
  • the video processing device 32 in the above-mentioned FIG. 21 is connected in communication with the image acquisition device 31 .
  • the video processing system may include a movable platform, and the image capture device 31 can be mounted on the movable platform.
  • the movable platform may include a gimbal, unmanned aerial vehicle, unmanned vehicle, or unmanned watercraft.
  • the unmanned aerial vehicle can be a rotary-wing drone, such as a quad-rotor drone, a six-rotor drone, an eight-rotor drone, or a fixed-wing drone;
  • the gimbal can include a handheld drone capable of carrying a camera A gimbal, or a gimbal that can be mounted on an unmanned aerial vehicle, unmanned vehicle, or unmanned boat.
  • FIG. 24 is a schematic structural diagram of another video processing system provided by an embodiment of the present invention. with reference to FIG. 24, the present embodiment provides another video processing system, and the video processing system may include:
  • the image acquisition device 41 is used to generate an image sequence frame, wherein the image acquisition device 31 can be a camera equipped with a camera, a mobile phone, glasses, etc., such as virtual reality (VR, virtual reality) glasses or a first-person perspective (FPV, first person view). )Glasses.
  • VR virtual reality
  • FMV first-person perspective
  • the video processing device 42 shown in FIG. 22 above, the video processing device 42 is connected in communication with the image acquisition device 41 .
  • the video processing system may include a movable platform, and the image capture device 41 can be mounted on the movable platform.
  • the movable platform may include a gimbal, unmanned aerial vehicle, unmanned vehicle, or unmanned watercraft.
  • the unmanned aerial vehicle can be a rotary-wing drone, such as a quad-rotor drone, a six-rotor drone, an eight-rotor drone, or a fixed-wing drone;
  • the gimbal can include a handheld drone capable of carrying a camera A gimbal, or a gimbal that can be mounted on an unmanned aerial vehicle, unmanned vehicle, or unmanned boat.
  • the disclosed related detection apparatus and method may be implemented in other manners.
  • the embodiments of the detection apparatus described above are only illustrative.
  • the division of the modules or units is only a logical function division.
  • the shown or discussed mutual coupling or direct coupling or communication connection may be through some interfaces, and the indirect coupling or communication connection of the detection device or unit may be in electrical, mechanical or other forms.
  • the units described as separate components may or may not be physically separated, and components displayed as units may or may not be physical units, that is, may be located in one place, or may be distributed to multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution in this embodiment.
  • each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically alone, or two or more units may be integrated into one unit.
  • the above-mentioned integrated units may be implemented in the form of hardware, or may be implemented in the form of software functional units.
  • the integrated unit if implemented in the form of a software functional unit and sold or used as an independent product, may be stored in a computer-readable storage medium.
  • the technical solution of the present invention is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium , including several instructions for causing a computer processor (processor) to perform all or part of the steps of the methods described in the various embodiments of the present invention.
  • the aforementioned storage medium includes: U disk, mobile hard disk, Read-Only Memory (ROM, Read-Only Memory), Random Access Memory (RAM, Random Access Memory), magnetic disk or optical disk and other media that can store program codes.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Studio Devices (AREA)

Abstract

Procédé, appareil et système de traitement de vidéo, et support de stockage. Le procédé de traitement de vidéo consiste à : acquérir une trame de séquence d'image et une trajectoire de photographie correspondant à la trame de séquence d'image ; sur la base de la trajectoire de photographie, déterminer une direction d'agitation correspondant à la trame de séquence d'image ; extraire une trajectoire de mouvement dans la direction d'agitation à partir de la trajectoire de photographie ; et effectuer un traitement d'extraction de trame sur la trame de séquence d'image sur la base de la trajectoire de mouvement. Au moyen de la solution technique fournie dans le présent mode de réalisation, une trame de séquence d'image et une trajectoire de photographie sont acquises, une direction d'agitation est déterminée sur la base de la trajectoire de photographie, une trajectoire de mouvement dans la direction d'agitation est extraite de la trajectoire de photographie, puis un traitement d'extraction de trame est effectué sur la trame de séquence d'image sur la base de la trajectoire de mouvement, ce qui fait en sorte que la détermination précise d'une trame d'image à extraire dans la direction d'agitation peut être efficacement réalisée, un traitement de stabilisation peut être effectué sur la base de la trame d'image déterminée devant être extraite, et de cette manière, des informations vidéo qui ont été soumises à un traitement de stabilisation peuvent être obtenues, ce qui permet d'assurer la qualité et l'effet d'affichage d'une image ou d'une vidéo.
PCT/CN2021/089088 2021-04-22 2021-04-22 Procédé, appareil et système de traitement de vidéo, et support de stockage WO2022222113A1 (fr)

Priority Applications (1)

Application Number Priority Date Filing Date Title
PCT/CN2021/089088 WO2022222113A1 (fr) 2021-04-22 2021-04-22 Procédé, appareil et système de traitement de vidéo, et support de stockage

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/CN2021/089088 WO2022222113A1 (fr) 2021-04-22 2021-04-22 Procédé, appareil et système de traitement de vidéo, et support de stockage

Publications (1)

Publication Number Publication Date
WO2022222113A1 true WO2022222113A1 (fr) 2022-10-27

Family

ID=83723612

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2021/089088 WO2022222113A1 (fr) 2021-04-22 2021-04-22 Procédé, appareil et système de traitement de vidéo, et support de stockage

Country Status (1)

Country Link
WO (1) WO2022222113A1 (fr)

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20130177251A1 (en) * 2012-01-11 2013-07-11 Samsung Techwin Co., Ltd. Image adjusting apparatus and method, and image stabilizing apparatus including the same
CN104601897A (zh) * 2015-02-13 2015-05-06 浙江宇视科技有限公司 一种摄像机防抖装置及防抖方法
CN104869285A (zh) * 2015-05-20 2015-08-26 南方电网科学研究院有限责任公司 一种基于主运动方向识别的运动补偿方法
CN105791705A (zh) * 2016-05-26 2016-07-20 厦门美图之家科技有限公司 适用于移动式延时摄影的视频防抖方法、系统及拍摄终端
CN108293123A (zh) * 2015-12-22 2018-07-17 三星电子株式会社 用于产生缩时图像的方法和设备
CN109314744A (zh) * 2017-12-29 2019-02-05 深圳市大疆创新科技有限公司 拍摄设备的图像处理方法、拍摄设备及可移动平台
JP2019149717A (ja) * 2018-02-27 2019-09-05 キヤノン株式会社 画像処理装置及び画像処理方法、撮像装置、プログラム、記憶媒体

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20130177251A1 (en) * 2012-01-11 2013-07-11 Samsung Techwin Co., Ltd. Image adjusting apparatus and method, and image stabilizing apparatus including the same
CN104601897A (zh) * 2015-02-13 2015-05-06 浙江宇视科技有限公司 一种摄像机防抖装置及防抖方法
CN104869285A (zh) * 2015-05-20 2015-08-26 南方电网科学研究院有限责任公司 一种基于主运动方向识别的运动补偿方法
CN108293123A (zh) * 2015-12-22 2018-07-17 三星电子株式会社 用于产生缩时图像的方法和设备
CN105791705A (zh) * 2016-05-26 2016-07-20 厦门美图之家科技有限公司 适用于移动式延时摄影的视频防抖方法、系统及拍摄终端
CN109314744A (zh) * 2017-12-29 2019-02-05 深圳市大疆创新科技有限公司 拍摄设备的图像处理方法、拍摄设备及可移动平台
JP2019149717A (ja) * 2018-02-27 2019-09-05 キヤノン株式会社 画像処理装置及び画像処理方法、撮像装置、プログラム、記憶媒体

Similar Documents

Publication Publication Date Title
CN106550174B (zh) 一种基于单应性矩阵的实时视频稳像方法
EP2847991B1 (fr) Système de mélange ou de composition en temps réel, objets tridimensionnels (3d) générés par ordinateur et source vidéo provenant d'une caméra cinématographique
JP5179398B2 (ja) 画像処理装置、画像処理方法、画像処理プログラム
CN110799921A (zh) 拍摄方法、装置和无人机
US10687046B2 (en) Trajectory smoother for generating multi-view interactive digital media representations
EP3296952B1 (fr) Procédé et dispositif pour corriger un objet virtuel dans une video
US9171379B2 (en) Hybrid precision tracking
US9025859B2 (en) Inertial sensor aided instant autofocus
US9088772B2 (en) Image-capturing apparatus
CN105678809A (zh) 手持式自动跟拍装置及其目标跟踪方法
JP6736362B2 (ja) 画像処理装置、画像処理方法、プログラム
CN106331480A (zh) 基于图像拼接的视频稳像方法
WO2023071790A1 (fr) Procédé et appareil de détection de pose pour un objet cible, dispositif et support de stockage
US11503209B2 (en) Image alignment using a virtual gyroscope model
Kamali et al. Stabilizing omnidirectional videos using 3d structure and spherical image warping
CN105635808B (zh) 一种基于贝叶斯理论的视频拼接方法
CN116170689A (zh) 视频生成方法、装置、计算机设备和存储介质
CN111712857A (zh) 图像处理方法、装置、云台和存储介质
CN114143459A (zh) 一种适用于大变焦摄像头的视频消抖方法
WO2022222113A1 (fr) Procédé, appareil et système de traitement de vidéo, et support de stockage
Sánchez et al. Motion smoothing strategies for 2D video stabilization
Wang et al. Adaptive video stabilization based on feature point detection and full-reference stability assessment
JP7164873B2 (ja) 画像処理装置及びプログラム
Li et al. Real Time and Robust Video Stabilization Based on Block-Wised Gradient Features
Favorskaya et al. Warping techniques in video stabilization

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21937351

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 21937351

Country of ref document: EP

Kind code of ref document: A1