WO2023216822A1 - 图像校正方法、装置、电子设备及存储介质 - Google Patents

图像校正方法、装置、电子设备及存储介质 Download PDF

Info

Publication number
WO2023216822A1
WO2023216822A1 PCT/CN2023/089112 CN2023089112W WO2023216822A1 WO 2023216822 A1 WO2023216822 A1 WO 2023216822A1 CN 2023089112 W CN2023089112 W CN 2023089112W WO 2023216822 A1 WO2023216822 A1 WO 2023216822A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
virtual
sample
physical
axis
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2023/089112
Other languages
English (en)
French (fr)
Inventor
陈誉中
焦少慧
吴垚垚
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing ByteDance Network Technology Co Ltd
Original Assignee
Beijing ByteDance Network Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing ByteDance Network Technology Co Ltd filed Critical Beijing ByteDance Network Technology Co Ltd
Priority to US18/865,668 priority Critical patent/US20250317654A1/en
Publication of WO2023216822A1 publication Critical patent/WO2023216822A1/zh
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G03—PHOTOGRAPHY; CINEMATOGRAPHY; ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ELECTROGRAPHY; HOLOGRAPHY
    • G03B—APPARATUS OR ARRANGEMENTS FOR TAKING PHOTOGRAPHS OR FOR PROJECTING OR VIEWING THEM; APPARATUS OR ARRANGEMENTS EMPLOYING ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ACCESSORIES THEREFOR
    • G03B37/00—Panoramic or wide-screen photography; Photographing extended surfaces, e.g. for surveying; Photographing internal surfaces, e.g. of pipe
    • G03B37/04—Panoramic or wide-screen photography; Photographing extended surfaces, e.g. for surveying; Photographing internal surfaces, e.g. of pipe with cameras or projectors providing touching or overlapping fields of view
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/698—Control of cameras or camera modules for achieving an enlarged field of view, e.g. panoramic image capture
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/80—Geometric correction
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/50—Depth or shape recovery
    • G06T7/55—Depth or shape recovery from multiple images
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/80—Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/90—Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20068—Projection on vertical or horizontal image axis

Definitions

  • Embodiments of the present disclosure relate to the field of image processing technology, such as an image correction method, device, electronic device, and storage medium.
  • Free-angle video is a popular video form nowadays. It provides users with the function of interactively selecting viewing angles, giving fixed two-dimensional (2D) videos a viewing experience of "changing scenes as you move", thereby giving users It brought a strong three-dimensional impact.
  • Embodiments of the present disclosure provide an image correction method, device, electronic device, and storage medium.
  • an image correction method which may include:
  • the projection axes on each target image after correction are consistent, and the virtual center is located on multiple shooting machines.
  • the virtual plane corresponding to the device, and corresponding to the physical optical center of multiple shooting devices are projected onto the target image to obtain the projection axis.
  • an image correction device which may include:
  • the target image acquisition module is configured to acquire the target image captured by the shooting device for each of the multiple shooting devices
  • a target image correction module configured to correct the target image based on correction parameters corresponding to the shooting device
  • the projection axes on each target image after correction are consistent, and the virtual center is located on multiple shooting machines.
  • the virtual plane corresponding to the device, and corresponding to the physical optical center of multiple shooting devices are projected onto the target image to obtain the projection axis.
  • embodiments of the present disclosure also provide an electronic device, which may include:
  • processors one or more processors
  • memory configured to store one or more programs
  • the one or more processors are caused to implement the image correction method provided by any embodiment of the present disclosure.
  • embodiments of the present disclosure also provide a computer-readable storage medium on which a computer program is stored. When executed by a processor, the computer program can implement the image correction method provided by any embodiment of the present disclosure.
  • Figure 1 is a flow chart of an image correction method in an embodiment of the present disclosure
  • Figure 2 is a flow chart of another image correction method in an embodiment of the present disclosure.
  • Figure 3 is a schematic diagram of an example of another image correction method in an embodiment of the present disclosure.
  • Figure 4 is a flow chart of yet another image correction method in an embodiment of the present disclosure.
  • FIG. 5 is a schematic diagram of an example of yet another image correction method in an embodiment of the present disclosure.
  • Figure 6 is a structural block diagram of an image correction device in an embodiment of the present disclosure.
  • FIG. 7 is a schematic structural diagram of an electronic device in an embodiment of the present disclosure.
  • embodiments of the present disclosure provide an image correction method, device, electronic device, and storage medium.
  • the term “include” and its variations are open-ended, ie, “including but not limited to.”
  • the term “based on” means “based at least in part on.”
  • the term “one embodiment” means “at least one embodiment”; the term “another embodiment” means “at least one additional embodiment”; and the term “some embodiments” means “at least some embodiments”. Relevant definitions of other terms will be given in the description below.
  • Figure 1 is a flow chart of an image correction method provided in an embodiment of the present disclosure.
  • This embodiment can perform image correction, and can correct target images captured by multiple shooting devices.
  • This method can be performed by the image correction device provided by the embodiment of the present disclosure.
  • the device can be implemented in the form of software and/or hardware.
  • the device can be integrated on an electronic device, and the electronic device can be various terminal devices or servers.
  • the method according to the embodiment of the present disclosure includes the following steps:
  • the multiple shooting devices may be multiple electronic devices with shooting functions, such as cameras, camcorders or cameras, etc.
  • these shooting devices can be used for free-angle shooting or light field shooting, which are not specifically limited here; for another example, these shooting devices can be deployed in a circular shape around the subject being photographed. , in order to synchronously collect videos or images of the photographed object, thereby giving users a smooth viewing experience of spatial videos or spatial images.
  • the target image can be an image shot (i.e., collected) by any one of multiple shooting devices or a certain video frame (i.e., video picture) in the video.
  • the video can be a recorded video or a live video, etc., in This is not specifically limited.
  • S120 Calibrate the target image based on the correction parameters corresponding to the shooting device. Among them, when at least two axis points on the rotation axis including the virtual center are projected onto the target image to obtain the projection axis, the projection axes on each target image after correction are consistent, and the virtual center is located on multiple shooting machines. set up On the virtual plane corresponding to the equipment, and corresponding to the physical optical center of multiple shooting equipment.
  • Each photographing device corresponds to its own correction parameter, which is used to correct the target image captured by the corresponding photographing device.
  • it may be a geometric correction or an affine transformation correction parameter.
  • correction parameters can be represented in various ways, such as correction matrices, correction vectors, correction tensors or correction images, etc.
  • the target image captured by the shooting device is corrected based on the correction parameter corresponding to the shooting device. On this basis, for example, when the target image is a video frame, the corrected video frame can be stored offline or played in real time.
  • the virtual plane can be understood as the same plane, that is, the plane where the shooting equipment that meets the construction expectations is located.
  • the virtual center can be a virtual center on the virtual plane corresponding to the physical optical centers of multiple shooting devices, that is, when the construction positions of each building device meet expectations, the center where their optical axes converge, at this time each The physical optical center lies on a standard circle.
  • the corrected The projection axes are consistent, that is, each corrected target image corresponds to the same rotation axis.
  • the spatial points on it have been converted from the respective shooting equipment coordinate system to the fixed-axis coordinate system (that is, different axis points have been transformed
  • the target image is corrected to the target image with the same axis point), so each corrected target image will no longer have jitter due to perspective change.
  • the target image captured by the shooting device is corrected through the correction parameters corresponding to the shooting device.
  • the virtual center on the virtual plane corresponding to the multiple shooting devices and corresponding to the physical optical center of the multiple shooting devices at least two axis points on the rotation axis including the virtual center are projected onto the target image.
  • the projection axes on each corrected target image are consistent, which means that each corrected target image corresponds to the same rotation axis, that is, the spatial points on them have been transformed by their respective shooting device coordinate systems.
  • the corrected target images will no longer jitter due to perspective changes, thereby eliminating the image jitter caused by adjacent perspective changes, and because the images can be corrected synchronously during the normal shooting process , reducing the accuracy requirements for the construction of multiple shooting equipment.
  • FIG. 2 is a flow chart of another image correction method provided in an embodiment of the present disclosure. This embodiment is adjusted based on the above embodiment.
  • the correction parameters are obtained in advance through the following steps: determine the virtual plane and obtain the virtual center on the virtual plane; determine the rotation axis including the virtual center, Project at least two axis points on the rotation axis onto the sample images shot by each shooting device to obtain the sample axis; use the sample image shot by the main shooting device among multiple shooting devices in each sample image as the main sample image , and the sample image captured by the auxiliary shooting device as the auxiliary sample image; for each auxiliary sample image, the auxiliary sample image is corrected so that the sample axis on the corrected auxiliary sample image is consistent with the sample axis on the main sample image. Consistent; get the calibration parameters of each shooting device based on the calibration results.
  • the explanation of terms that are the same as or corresponding to the above embodiments will not be repeated here.
  • the method of this embodiment may include the following steps:
  • S210 For multiple shooting devices, determine virtual planes corresponding to the multiple shooting devices, and obtain virtual centers on the virtual plane corresponding to the physical optical centers of the multiple shooting devices.
  • multiple shooting devices correspond to the same virtual plane
  • multiple physical optical centers correspond to the same virtual center.
  • the virtual plane and the virtual center have been explained above and will not be described again here.
  • the rotation axis can be an axis perpendicular to the virtual plane, or an axis at a certain inclination angle to the virtual plane, which is not specifically limited here.
  • the at least two pivot points on the rotation axis may be manually selected or automatically determined; the at least two pivot points may or may not include a virtual center, which is not specifically limited here.
  • These axis points on any sample image constitute their respective projection axes.
  • the projection axis It can be represented by projected lines, projected rays or projected line segments. These sample images may be images captured synchronously by these shooting devices, such as the same video frame in the captured video; or they may not be images captured synchronously, which is not specifically limited here.
  • each sample image the sample image captured by the main photography device among the plurality of photography devices is used as the main sample image, and the sample image captured by the auxiliary photography device is used as the auxiliary sample image.
  • the main shooting device is determined from multiple shooting devices, and the sample image taken by the main shooting device is used as the main sample image; on this basis, the remaining shooting devices other than the main shooting device are used as auxiliary shooting devices, and The sample image captured by the auxiliary shooting device is used as the auxiliary sample image.
  • the number of main shooting devices can be one, so that in combination with subsequent steps, the auxiliary sample images captured by the remaining auxiliary shooting devices are corrected based on the main sample image, thereby ensuring that each Consistency of the sample axis on the sample image.
  • the correction parameters can be obtained according to the correction process, that is, when the sample image is captured
  • the correction parameters corresponding to the auxiliary shooting equipment are obtained, thereby obtaining the correction parameters of each auxiliary shooting equipment.
  • the correction parameters corresponding to the main shooting device can be understood as parameters that will not cause distortion in the main sample image; otherwise, the correction parameters corresponding to the main sample image can be understood as The correction process of the sample image obtains the corresponding correction parameters.
  • S260 For each shooting device, obtain the target image captured by the shooting device, and correct the target image based on the correction parameters corresponding to the shooting device.
  • the sample axis is obtained by projecting at least two axis points on the rotation axis including the virtual center onto the sample images captured by each shooting device; further, for each sample image captured by multiple shooting devices
  • the auxiliary sample image taken by each auxiliary shooting device in the auxiliary sample image is corrected so that the sample axis on the corrected auxiliary sample image is consistent with the main sample image in each sample image taken by the main shooting device among the multiple shooting devices.
  • the correction parameters of each shooting device are obtained according to the correction results, that is, the correction parameters of each shooting device are obtained through geometric projection operations.
  • the above image correction method may further include: determining the posture parameters of each shooting device respectively; on this basis, determining the virtual plane and obtaining the virtual center on the virtual plane, It may include: fitting the optical center position of each physical optical center to obtain a plane equation, and obtaining a virtual plane based on the plane equation; for each shooting device, according to the plane equation and the pose parameters of the shooting device, the corresponding to the shooting device The optical center position is projected onto the virtual plane to obtain the projection position. After obtaining each projection position, the virtual position is obtained by fitting each projection position, and the virtual center is obtained based on the virtual position.
  • the pose parameters of each shooting device are calibrated separately.
  • the plane equation is obtained by fitting the optical center positions of each physical optical center, and then the virtual plane is obtained based on the plane equation.
  • the least squares algorithm can be used to calculate the plane coefficients, thereby obtaining the plane equation; for another example, the above-mentioned optical center position can be expressed by optical center coordinates.
  • the corresponding optical center position can be projected onto the virtual plane according to the plane equation and the pose parameters of the shooting device to obtain the projection position.
  • the pose parameter is one of the important reference factors in the process of determining the correction parameters, which ensures the accuracy of the determination of the correction parameters.
  • pose parameters can also be used in axis point projection.
  • projecting at least two axis points on the rotation axis onto sample images captured by each shooting device may include: for each shooting device, based on the pose parameters of the shooting device, projecting at least two axis points on the rotation axis The axis point is projected onto the sample image captured by the shooting device.
  • determining the pose parameters of each shooting device separately may include: obtaining the target image sequences captured by each shooting device respectively, and determining the feature matching relationship between the target image sequences; and obtaining the target image sequence according to the feature matching relationship.
  • the pose parameters of each shooting device That is, by obtaining the collected multi-view videos and calculating the feature matching relationship between the multi-view videos, the pose parameters (i.e., calibration results) of each shooting device are obtained.
  • the pose parameters It can be represented by external parameters (such as rotation matrix and translation matrix, etc.).
  • the above calibration process is a self-calibration process. There is no need for a calibration board during the calibration process. The calibration can be completed through the recorded video, which greatly reduces the calibration time and improves the calibration efficiency.
  • determining the rotation axis including the virtual center may include: obtaining the plane equation of the virtual plane and normalizing the plane equation to obtain the plane normal vector of the virtual plane;
  • the axis containing the plane normal vector of the virtual center is used as the axis of rotation.
  • the plane normal vector of the virtual plane can be obtained. Since the virtual plane contains at least one plane normal vector, the plane normal vector passing through the virtual center among the at least one plane normal vector is used as the rotation axis, thereby obtaining a rotation axis that is perpendicular to the virtual plane.
  • the virtual plane is usually parallel to the ground, and the subject is usually standing on the ground (that is, perpendicular to the ground), then when the rotation axis is perpendicular to the virtual plane (that is, the ground), corresponding to the rotation
  • the sample image after axis correction is more in line with the user's visual experience and improves the user experience.
  • the sample axis is represented by a sample line segment
  • the auxiliary sample image is corrected so that the sample axis on the corrected auxiliary sample image is consistent with the sample axis on the main sample image.
  • the rotation operation can make the sample line segments on the corrected auxiliary sample image parallel to the sample line segments on the main sample image
  • the scaling operation can make the sample line segments on the corrected auxiliary sample image have the same length as the sample line segments on the main sample image.
  • the translation operation can make the calibration
  • the relative position of the sample line segment on the corrected auxiliary sample image on the corrected auxiliary sample image is the same as the relative position of the sample line segment on the main sample image on the main sample image, thereby ensuring that the corrected auxiliary sample image
  • the consistency of the sample line segment and the sample line segment on the main sample image that is, the consistency of the axis points on the two.
  • the rotation operation there is no specific requirement for the execution order of the rotation operation, the scaling operation and the translation operation, because they can all be attributed to the correction parameters.
  • the rotation matrix corresponding to the rotation operation the scaling matrix corresponding to the scaling operation, and the translation matrix corresponding to the translation operation can be combined to obtain a correction matrix.
  • the above image correction method may further include: rotating the main sample image so that the sample axis on the rotated main sample image Parallel to the target axis of the sample image; update the main sample image according to the rotation result, and obtain the correction parameters of the main shooting device; obtain the correction parameters of each shooting device according to the correction results, which may include: obtain the correction parameters of the auxiliary shooting device according to the correction results .
  • the target axis can be the horizontal axis or the vertical axis of the sample image. In practical applications, for example, it can be the vertical axis.
  • the subject when the subject is photographed horizontally, the subject The rotated main sample image stands vertically on the ground, which is more consistent with the user's visual experience. Furthermore, the main sample image is updated according to the rotation result, so that the rotated main sample image is used as a reference to correct the remaining auxiliary sample images, and the correction parameters of the auxiliary shooting device are obtained according to the correction result; at the same time, the correction parameters of the auxiliary shooting device can also be obtained according to the rotation result. Calibration parameters of the main shooting device.
  • the target image and the sample image are both video frames in the video captured by the shooting device.
  • the correction matrix determination process can be understood as a preprocessing process, and the correction matrix application process can occur during the video playback process.
  • input synchronized video frames that is, synchronized sample images
  • the virtual plane is obtained by fitting the optical center coordinates of the physical optical center of each camera, and combined with the pose parameters, each physical optical center is projected onto the virtual plane, and each projection result is fitted to obtain the virtual center.
  • Calculate the plane normal vector of the virtual plane take the plane normal vector passing through the virtual center as the rotation axis, and determine the two axis points on the rotation axis. Project these two axis points onto each video frame to obtain the projected line segment on each video frame. Based on the projected line segments on each video frame, the affine transformation matrix (i.e., correction matrix) of the corresponding camera is calculated, the correction matrix is output, and the correction matrix is matched one-to-one with the camera number of the corresponding camera. At this point, the preprocessing process is completed.
  • the affine transformation matrix i.e., correction matrix
  • FIG. 4 is a flow chart of yet another image correction method provided in an embodiment of the present disclosure. This embodiment is adjusted based on the above embodiment.
  • the above-mentioned image correction method may also include: for the first shooting device and the second shooting device that are adjacent in placement among the multiple shooting devices, The target image captured by the first shooting device is used as the first physical image and the target image captured by the second shooting device is used as the second physical image; based on the first physical image and the second physical image, a virtual image is generated, wherein the virtual image is located The virtual perspective of is located between the physical perspective of the first physical image and the physical perspective of the second physical image.
  • the explanations of terms that are the same as or corresponding to the above embodiments will not be repeated here.
  • the method of this embodiment may include the following steps:
  • S320 Calibrate the target image based on the correction parameters corresponding to the shooting device. Among them, when at least two axis points on the rotation axis including the virtual center are projected onto the target image to obtain the projection axis, the projection axes on each target image after correction are consistent, and the virtual center is located on multiple shooting machines. On the virtual plane corresponding to the device, and corresponding to the physical optical center of multiple shooting devices.
  • the first photographing device and the second photographing device may be two photographing devices that are placed adjacent to each other. Since they are actual electronic devices, the target images captured by them may be called physical images.
  • the target image captured by the first shooting device is taken as the first physical image and the second The target image captured by the shooting device serves as the second physical image.
  • S340 Generate a virtual image based on the first physical image and the second physical image.
  • the virtual viewing angle where the virtual image is located is between the physical viewing angle where the first physical image is located and the physical viewing angle where the second physical image is located.
  • a virtual image is generated based on the first physical image and the second physical image, and the virtual viewing angle of the virtual image is located between the physical viewing angle of the first physical image and the physical viewing angle of the second physical image, that is, by automatically synthesizing adjacent Virtual images under virtual perspectives between physical perspectives to achieve the effect of video frame insertion.
  • the number of shooting equipment can be reduced by generating virtual images (that is, images from a virtual perspective), thereby avoiding It is a lightweight free-angle acquisition solution that eliminates a series of situations caused by too many shooting devices.
  • corresponding physical images are generated based on the two.
  • Virtual images from a virtual perspective between perspectives reduce the number of shooting devices by generating virtual images, thus avoiding the high hardware costs, difficulty in uniformity, and calibration time caused by too many shooting devices. Too long.
  • generating a virtual image based on the first physical image and the second physical image may include: determining the first depth of field of the first shooting device and the second depth of field of the second shooting device, The first physical image and the second physical image are matched, point cloud reconstruction is performed based on the first depth of field, the second depth of field and the matching result, and a virtual image is obtained based on the point cloud reconstruction result.
  • the matching process of the first physical image and the second physical image can be realized based on algorithms such as stereo matching, and then point cloud reconstruction is performed based on the first depth of field, the second depth of field and the matching results, thereby generating a virtual perspective. Point clouds are used to obtain virtual images, thereby achieving the effect of video frame insertion.
  • generating a virtual image based on the first physical image and the second physical image may include: calculating the optical flow using the first physical image and the second physical image as video files, and Generate virtual images based on optical flow.
  • the free angle of view is the result of shooting from multiple angles in space at the same point in time.
  • Another way of thinking it can also be understood as the result of shooting at multiple spatial positions (i.e. multiple angles) based on the same shooting device. This is the time area. Therefore, the first physical image and the second physical image can be used as video files to calculate the optical flow, and then generate a virtual image based on the optical flow, thereby achieving the effect of video frame insertion.
  • generating a virtual image based on the first physical image and the second physical image may include: inputting the first physical image and the second physical image into a pre-trained In the video frame interpolation deep learning model, virtual images are generated based on the output results of the video frame interpolation deep learning model.
  • the video frame interpolation deep learning model can be understood as an end-to-end deep learning model used to implement video frame interpolation. After inputting the first physical image and the second physical image into it, the virtual image between the two physical perspectives can be obtained.
  • the virtual image under the viewing angle can, for example, generate two virtual images based on two physical images, thereby achieving the effect of video frame insertion.
  • the collected multi-channel videos can be processed as shown in Figure 5: the collected multi-channel videos are input into the calibration system to obtain the pose parameters of each camera; then, the multi-channel videos are and the corresponding pose parameters are input into the fixed-axis system to convert the spatial points of each video into the fixed-axis coordinate system to obtain the corresponding fixed-axis perspective (that is, the physical perspective under the fixed axis); then, based on the phase
  • the virtual perspective is generated from the adjacent fixed-axis perspective, so that the free-view video can be obtained based on the fixed-axis perspective and the virtual perspective. This is a lightweight free-view acquisition solution.
  • FIG. 6 is a structural block diagram of an image correction device provided in an embodiment of the present disclosure.
  • the device is used to execute the image correction method provided in any of the above embodiments.
  • This device has the same concept as the image correction method in each of the above embodiments.
  • the device may include: a target image acquisition module 410 and a target image correction module 420 . in,
  • the target image acquisition module 410 is configured to acquire, for each of the plurality of shooting devices, the target image captured by the shooting device;
  • the target image correction module 420 is configured to correct the target image based on the correction parameters corresponding to the shooting device;
  • the projection axes on each target image after correction are consistent, and the virtual center is located on multiple shooting devices.
  • the corresponding virtual plane, and corresponding to the physical optical centers of multiple shooting devices are projected onto the target image to obtain the projection axis.
  • the correction parameters are predetermined by the following module:
  • the virtual center obtaining module is set to determine the virtual plane and obtain the virtual center on the virtual plane;
  • the sample axis obtaining module is configured to determine the rotation axis including the virtual center, and project at least two axis points on the rotation axis onto the sample images captured by each shooting device to obtain the sample axis;
  • the auxiliary sample image obtaining module is configured to obtain each sample image by the main shooting device among multiple shooting devices.
  • the sample image captured by the device is used as the main sample image, and the sample image captured by the auxiliary shooting device is used as the auxiliary sample image;
  • An auxiliary sample image correction module is configured to correct the auxiliary sample image for each auxiliary sample image so that the sample axis on the corrected auxiliary sample image is consistent with the sample axis on the main sample image;
  • the first correction parameter obtaining module is configured to obtain the correction parameters of each shooting device according to the correction results.
  • the above-mentioned image correction device may also include:
  • the pose parameter acquisition module is configured to determine the pose parameters of each shooting device respectively;
  • Virtual centers get modules that can include:
  • the virtual plane obtaining unit is set to obtain the plane equation by fitting the optical center position of each physical optical center, and obtain the virtual plane based on the plane equation;
  • the virtual center obtaining unit is set to project the optical center position corresponding to the shooting device onto the virtual plane according to the plane equation and the pose parameters of the shooting device for each shooting device to obtain the projection position, so as to obtain each projection position. Afterwards, the virtual position is obtained by fitting each projection position, and the virtual center is obtained based on the virtual position;
  • the sample axis obtaining module may include:
  • the axis point projection unit is configured to, for each shooting device, project at least two axis points on the rotation axis to the sample image captured by the shooting device based on the posture parameters of the shooting device.
  • the pose parameter acquisition module may include:
  • the feature matching relationship determination unit is configured to respectively obtain the sample image sequences captured by each shooting device and determine the feature matching relationship between each sample image sequence
  • the pose parameter obtaining unit is configured to obtain the pose parameters of each shooting device according to the feature matching relationship.
  • the sample axis obtaining module may include:
  • the plane normal vector obtaining unit is set to obtain the plane equation of the virtual plane, and normalize the plane equation to obtain the plane normal vector of the virtual plane;
  • the axis of rotation gets the unit and is set to use the axis where the normal vector of the plane containing the virtual center is located as the axis of rotation.
  • the sample axis is represented by a sample line segment
  • the auxiliary sample image correction module may include:
  • the auxiliary sample image correction unit is configured to perform a correction operation on the auxiliary sample image, so that the sample line segments on the corrected auxiliary sample image are parallel to the sample line segments on the main sample image and have the same length, and the corrected auxiliary sample image
  • the relative position of the sample line segment on the corrected auxiliary sample image is the same as the relative position of the sample line segment on the main sample image on the main sample image, where the correction operation includes Including rotation operations, scaling operations and translation operations.
  • the above image correction device may also include:
  • the main sample image rotation module is configured to rotate the main sample image before correcting the auxiliary sample image so that the sample axis on the rotated main sample image is parallel to the target axis of the main sample image;
  • the second correction parameter obtaining module is configured to update the main sample image according to the rotation result, and obtain the correction parameters of the main shooting device;
  • the first correction parameter obtaining module is configured to: obtain the correction parameters of the auxiliary shooting device according to the correction result.
  • the above image correction device may also include:
  • the second physical image obtaining module is configured to, after correcting the target image, for the first shooting device and the second shooting device that are adjacent in the placement position among the plurality of shooting devices, obtain the target captured by the first shooting device.
  • the image is used as the first physical image
  • the target image captured by the second shooting device is used as the second physical image
  • the virtual image generation module is configured to generate a virtual image based on the first physical image and the second physical image, wherein the virtual perspective of the virtual image is between the physical perspective of the first physical image and the physical perspective of the second physical image.
  • the virtual image generation module may include:
  • the first virtual image generating unit is configured to determine the first depth of field of the first shooting device and the second depth of field of the second shooting device, and match the first physical image and the second physical image, according to the first depth of field, the second depth of field And the matching results are used for point cloud reconstruction, and the virtual image is obtained based on the point cloud reconstruction results.
  • the virtual image generation module may include:
  • the second virtual image generating unit is configured to calculate the optical flow using the first physical image and the second physical image as video files, and generate a virtual image based on the optical flow.
  • the virtual image generation module may include:
  • the third virtual image generation unit is configured to input the first physical image and the second physical image into the pre-trained video frame interpolation deep learning model, and generate a virtual image according to the output result of the video frame interpolation deep learning model.
  • the image correction device provided by the embodiment of the present disclosure, through the cooperation of the target image acquisition module and the target image correction module, for each of the multiple shooting devices, through the correction parameters corresponding to the shooting device, the shooting device is The captured target image is corrected.
  • the above device is directed to a virtual center corresponding to the physical optical center of the plurality of photographing devices on a virtual plane corresponding to the plurality of photographing devices.
  • the projection axes on each target image after correction are consistent, which means that the projection axes on each target image after correction are Corresponding to the same rotation axis, that is, the spatial points on them have been converted from the respective shooting equipment coordinate system to the fixed-axis coordinate system, so the corrected target images will no longer have jitter due to perspective change, thereby eliminating The image jitter caused by adjacent viewing angle changes, and because synchronous image correction can be performed during the normal shooting process, the requirements for the construction accuracy of multiple shooting equipment are reduced.
  • the image correction device provided by the embodiments of the present disclosure can execute the image correction method provided by any embodiment of the present disclosure, and has functional modules and beneficial effects corresponding to the execution method.
  • FIG. 7 a schematic structural diagram of an electronic device (such as the terminal device or server in FIG. 7 ) 500 suitable for implementing embodiments of the present disclosure is shown.
  • Electronic devices in embodiments of the present disclosure may include, but are not limited to, mobile phones, notebook computers, digital broadcast receivers, personal digital assistants (Personal Digital Assistant, PDA), tablet computers (PAD), portable multimedia players (Portable Media Player , PMP), mobile terminals such as vehicle-mounted terminals (such as vehicle-mounted navigation terminals), and fixed terminals such as digital televisions (Television, TV), desktop computers, etc.
  • PDA Personal Digital Assistant
  • PMP portable multimedia players
  • mobile terminals such as vehicle-mounted terminals (such as vehicle-mounted navigation terminals)
  • fixed terminals such as digital televisions (Television, TV), desktop computers, etc.
  • the electronic device shown in FIG. 7 is only an example and should not impose any limitations on the functions and scope of use of the embodiments of the present disclosure.
  • the electronic device 500 may include a processing device (such as a central processing unit, a graphics processor, etc.) 501, which may process data according to a program stored in a read-only memory (Read-Only Memory, ROM) 502 or from a storage device. 508 loads the program in the random access memory (Random Access Memory, RAM) 503 to perform various appropriate actions and processes. In the RAM 503, various programs and data required for the operation of the electronic device 500 are also stored.
  • the processing device 501, ROM 502 and RAM 503 are connected to each other via a bus 504.
  • An input/output (I/O) interface 505 is also connected to bus 504.
  • I/O interface 505 input devices 506 including, for example, a touch screen, touch pad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; including, for example, a Liquid Crystal Display (LCD) , an output device 507 such as a speaker, a vibrator, etc.; a storage device 508 including a magnetic tape, a hard disk, etc.; and a communication device 509.
  • Communication device 509 may allow electronic device 500 to communicate wirelessly or wiredly with other devices to exchange data.
  • Electronic device 500 has various means, but it should be understood that implementation or having all illustrated means is not required. More or fewer means may alternatively be implemented or provided.
  • embodiments of the present disclosure include a computer program product including a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the method illustrated in the flowchart.
  • the computer program may be downloaded and installed from the network via communication device 509, or from storage device 508, or from ROM 502.
  • the processing device 501 When the computer program is executed by the processing device 501, the above-mentioned functions defined in the method of the embodiment of the present disclosure are performed.
  • the computer-readable medium mentioned above in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the above two.
  • the computer-readable storage medium may be, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or any combination thereof.
  • Examples of computer readable storage media may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard drive, random access memory (RAM), read only memory (ROM), erasable programmable read only memory Memory (Erasable Programmable Read-Only Memory, EPROM) or flash memory, optical fiber, portable compact disk read-only memory (Compact Disc Read-Only Memory, CD-ROM), optical storage device, magnetic storage device, or any suitable combination of the above .
  • a computer-readable storage medium may be any tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device.
  • a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code therein. Such propagated data signals may take many forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above.
  • a computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device .
  • Program code contained on a computer-readable medium can be transmitted using any appropriate medium, including but not limited to: wires, optical cables, radio frequency (Radio Frequency, RF), etc., or any suitable combination of the above.
  • the client and server can communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and can communicate with digital data in any form or medium.
  • Communications e.g., communications network
  • Examples of communication networks include Local Area Network (LAN), Wide Area Network (WAN), the Internet (e.g., the Internet), and end-to-end networks (e.g., ad hoc end-to-end networks). peer network), and any network currently known or developed in the future.
  • LAN Local Area Network
  • WAN Wide Area Network
  • the Internet e.g., the Internet
  • end-to-end networks e.g., ad hoc end-to-end networks
  • peer network any network currently known or developed in the future.
  • the above-mentioned computer-readable medium may be included in the above-mentioned electronic device; it may also exist independently without being assembled into the electronic device.
  • the above-mentioned computer-readable medium carries one or more programs.
  • the electronic device executes the above-mentioned one or more programs.
  • the projection axes on each target image after correction are consistent, and the virtual center is located on multiple shooting devices.
  • the corresponding virtual plane, and corresponding to the physical optical centers of multiple shooting devices are projected onto the target image to obtain the projection axis.
  • the storage medium may be a non-transitory storage medium.
  • Computer program code for performing the operations of the present disclosure may be written in one or more programming languages, including but not limited to object-oriented programming languages—such as Java, Smalltalk, C++, and Includes conventional procedural programming languages—such as "C” or similar programming languages.
  • the program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server.
  • the remote computer can be connected to the user's computer through any kind of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (such as an Internet service provider through Internet connection).
  • LAN local area network
  • WAN wide area network
  • Internet service provider such as an Internet service provider through Internet connection
  • each block in the flowchart or block diagram may represent a module, segment, or portion of code that contains one or more logic functions that implement the specified executable instructions.
  • the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown one after another may actually execute substantially in parallel, or they may sometimes execute in the reverse order, depending on the functionality involved.
  • each block of the block diagram and/or flowchart illustration, and combinations of blocks in the block diagram and/or flowchart illustration can be implemented by special purpose hardware-based systems that perform the specified functions or operations. , or can be implemented using a combination of specialized hardware and computer instructions.
  • the units involved in the embodiments of the present disclosure can be implemented in software or hardware. Among them, the name of the unit does not constitute a reference to the unit itself under certain circumstances.
  • the target image acquisition module can also be described as "a module that acquires the target image captured by the shooting device for each of the multiple shooting devices.”
  • exemplary types of hardware logic components include: field programmable gate array (Field Programmable Gate Array, FPGA), application specific integrated circuit (Application Specific Integrated Circuit, ASIC), application specific standard product (Application Specific Standard Product (ASSP), System on Chip (SOC), Complex Programmable Logic Device (CPLD), etc.
  • a machine-readable medium may be a tangible medium that may contain or store a program for use by or in connection with an instruction execution system, apparatus, or device.
  • the machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium.
  • Machine-readable media may include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or devices, or any suitable combination of the foregoing.
  • machine-readable storage media examples include one or more wire-based electrical connections, laptop disks, hard drives, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM) ) or flash memory, optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination of the foregoing.
  • RAM random access memory
  • ROM read only memory
  • EPROM erasable programmable read only memory
  • flash memory optical fiber
  • CD-ROM portable compact disk read-only memory
  • magnetic storage device or any suitable combination of the foregoing.
  • Example 1 provides an image correction method, which may include:
  • the projection axes on each target image after correction are consistent, and the virtual center is located on multiple shooting devices.
  • the corresponding virtual plane, and corresponding to the physical optical centers of multiple shooting devices are projected onto the target image to obtain the projection axis.
  • Example 2 provides the method of Example 1, and the correction parameters can be predetermined through the following steps:
  • the sample image captured by the main capturing device among the plurality of capturing devices is used as the main sample image
  • the sample image captured by the auxiliary capturing device is used as the auxiliary sample image
  • auxiliary sample image For each auxiliary sample image, correct the auxiliary sample image so that the corrected auxiliary sample image
  • the sample axis on is consistent with the sample axis on the main sample image
  • Example 3 provides the method of Example 2.
  • the above image correction method may also include:
  • the plane equation is obtained by fitting the optical center position of each physical optical center, and the virtual plane is obtained based on the plane equation;
  • the optical center position corresponding to the shooting device is projected onto the virtual plane to obtain the projection position.
  • the projection position is calculated by The virtual position is obtained by fitting, and the virtual center is obtained based on the virtual position.
  • Example 4 provides the method of Example 3 to respectively determine the pose parameters of each shooting device, which may include:
  • the pose parameters of each shooting device are obtained respectively.
  • Example 5 provides the method of Example 2. Determining the rotation axis including the virtual center may include:
  • the axis containing the normal vector of the plane containing the virtual center is used as the axis of rotation.
  • Example 6 provides the method of Example 2.
  • the sample axis is represented by a sample line segment, and the auxiliary sample image is corrected so that the corrected auxiliary sample image
  • the sample axes are consistent with the sample axes on the main sample image and can include:
  • the relative position on the sample image is the same as the relative position of the sample line segment on the main sample image on the main sample image, where the correction operation includes a rotation operation, a scaling operation and a translation operation.
  • Example 7 provides the method of Example 2. Before correcting the auxiliary sample image, the above image correction method may also include:
  • Calibration parameters for each shooting device are obtained based on the calibration results, which may include:
  • the correction parameters of the auxiliary shooting equipment are obtained according to the correction results.
  • Example 8 provides the method of Example 1. After correcting the target image, the above image correction method may also include:
  • the target image captured by the first shooting device is used as the first physical image and the target image captured by the second shooting device is used as the third physical image.
  • a virtual image is generated, wherein the virtual viewing angle of the virtual image is located between the physical viewing angle of the first physical image and the physical viewing angle of the second physical image.
  • Example 9 provides the method of Example 8, which generates a virtual image based on the first physical image and the second physical image, which may include:
  • Example 10 provides the method of Example 8, which generates a virtual image based on the first physical image and the second physical image, which may include:
  • the optical flow is calculated using the first physical image and the second physical image as video files, and a virtual image is generated based on the optical flow.
  • Example 11 provides the method of Example 8, which generates a virtual image based on the first physical image and the second physical image, which may include:
  • the first physical image and the second physical image are input into the pre-trained video frame interpolation deep learning model, and a virtual image is generated based on the output result of the video frame interpolation deep learning model.
  • Example 12 provides an image correction device, which may include:
  • the target image acquisition module is configured to acquire the target image captured by the shooting device for each of the multiple shooting devices
  • a target image correction module configured to correct the target image based on correction parameters corresponding to the shooting device
  • the projection axes on each target image after correction are consistent, and the virtual center is located On the virtual plane corresponding to multiple shooting devices, and corresponding to the physical optical center of the multiple shooting devices.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Studio Devices (AREA)

Abstract

本公开实施例公开了一种图像校正方法、装置、电子设备及存储介质。该方法可包括：针对多台拍摄设备中的每台拍摄设备，获取拍摄设备拍摄的目标图像；基于与拍摄设备对应的校正参数，对目标图像进行校正；其中，将包含虚拟中心的旋转轴上的至少两个轴点投影到目标图像上后得到投影轴的情况下，校正后的每个目标图像上的投影轴相一致，虚拟中心位于多台拍摄设备对应的虚拟平面上，并且与多台拍摄设备的物理光心对应。

Description

图像校正方法、装置、电子设备及存储介质
本申请要求在2022年05月13日提交中国专利局、申请号为202210523446.7的中国专利申请的优先权，以上申请的全部内容通过引用结合在本申请中。
技术领域
本公开实施例涉及图像处理技术领域，例如涉及一种图像校正方法、装置、电子设备及存储介质。
背景技术
自由视角视频是时下热门的一种视频形式，其是通过向用户提供交互选择观看角度的功能，赋予了固定二维(two dimensional，2D)视频“移步换景”的观看体验，从而给用户带来了强烈的立体冲击。
发明内容
本公开实施例提供了一种图像校正方法、装置、电子设备及存储介质。
第一方面，本公开实施例提供了一种图像校正方法，可以包括：
针对多台拍摄设备中的每台拍摄设备，获取拍摄设备拍摄的目标图像；
基于与拍摄设备对应的校正参数，对目标图像进行校正；
其中，在将包含虚拟中心的旋转轴上的至少两个轴点投影到目标图像上后得到投影轴的情况下，校正后的每个目标图像上的投影轴相一致，虚拟中心位于多台拍摄设备对应的虚拟平面上，并且与多台拍摄设备的物理光心对应。
第二方面，本公开实施例还提供了一种图像校正装置，可以包括：
目标图像获取模块，设置为针对多台拍摄设备中的每台拍摄设备，获取拍摄设备拍摄的目标图像；
目标图像校正模块，设置为基于与拍摄设备相对应的校正参数，对目标图像进行校正；
其中，在将包含虚拟中心的旋转轴上的至少两个轴点投影到目标图像上后得到投影轴的情况下，校正后的每个目标图像上的投影轴相一致，虚拟中心位于多台拍摄设备对应的虚拟平面上，并且与多台拍摄设备的物理光心对应。
第三方面，本公开实施例还提供了一种电子设备，可以包括：
一个或多个处理器；
存储器，设置为存储一个或多个程序，
当一个或多个程序被一个或多个处理器执行，使得一个或多个处理器实现本公开任意实施例所提供的图像校正方法。
第四方面，本公开实施例还提供了一种计算机可读存储介质，其上存储有计算机程序，该计算机程序被处理器执行时可实现本公开任意实施例所提供的图像校正方法。
附图说明
贯穿附图中，相同或相似的附图标记表示相同或相似的元素。应当理解附图是示意性的，原件和元素不一定按照比例绘制。
图1是本公开实施例中的一种图像校正方法的流程图；
图2是本公开实施例中的另一种图像校正方法的流程图；
图3是本公开实施例中的另一种图像校正方法中一类示例的示意图；
图4是本公开实施例中的再一种图像校正方法的流程图；
图5是本公开实施例中的再一种图像校正方法中一类示例的示意图；
图6是本公开实施例中的一种图像校正装置的结构框图；
图7是本公开实施例中的一种电子设备的结构示意图。
具体实施方式
自由视角视频的制作过程需要多角度多相机拍摄的环境，考虑到视角转动时图像(如视频帧)的顺滑连续性，这对相机部署(如相机的摆放位置和俯仰朝向等)提出了较高的要求。目前，主要是通过人工部署相机来满足上述要求。
但是，人工部署耗时耗力且精度较低(即无法真正满足上述要求)，这使得相邻视角变换时，图像会出现明显抖动现象，如上下视差、左右视差以及图像缩放等，从而影响了用户的视频观看体验。
考虑到上述情况，本公开实施例提供了一种图像校正方法、装置、电子设备及存储介质。
下面将参照附图描述本公开的实施例。虽然附图中显示了本公开的某些实施例，然而应当理解的是，本公开可以通过各种形式来实现，而且不应该被解释为限于这里阐述的实施例，相反提供这些实施例是为了更加透彻和完整地理解本公开。应当理解的是，本公开的附图及实施例仅用于示例性作用，并非用于限制本公开的保护范围。
应当理解，本公开的方法实施方式中记载的各个步骤可以按照不同的顺序 执行，和/或并行执行。此外，方法实施方式可以包括附加的步骤和/或省略执行示出的步骤。本公开的范围在此方面不受限制。
本文使用的术语“包括”及其变形是开放性包括，即“包括但不限于”。术语“基于”是“至少部分地基于”。术语“一个实施例”表示“至少一个实施例”；术语“另一实施例”表示“至少一个另外的实施例”；术语“一些实施例”表示“至少一些实施例”。其他术语的相关定义将在下文描述中给出。
需要注意，本公开中提及的“第一”、“第二”等概念仅用于对不同的装置、模块或单元进行区分，并非用于限定这些装置、模块或单元所执行的功能的顺序或者相互依存关系。
需要注意，本公开中提及的“一个”、“多个”的修饰是示意性而非限制性的，本领域技术人员应当理解，除非在上下文另有明确指出，否则应该理解为“一个或多个”。
本公开实施方式中的多个装置之间所交互的消息或者信息的名称仅用于说明性的目的，而并不是用于对这些消息或信息的范围进行限制。
图1是本公开实施例中所提供的一种图像校正方法的流程图。本实施例可以进行图像校正，可以对多台拍摄设备拍摄的目标图像进行校正。该方法可以由本公开实施例提供的图像校正装置来执行，该装置可以由软件和/或硬件的方式实现，该装置可以集成在电子设备上，该电子设备可以是各种终端设备或服务器。
参见图1，本公开实施例的方法包括如下步骤：
S110、针对于多台拍摄设备中的每台拍摄设备，获取拍摄设备拍摄的目标图像。
其中，多台拍摄设备可以是多台具有拍摄功能的电子设备，如相机、摄像机或摄像头等。在实际应用中，例如，这些拍摄设备可被用于自由视角拍摄或是光场拍摄等，在此未做具体限定；再例如，这些拍摄设备可以呈圆环形环绕部署在被拍摄对象的周围，以便对该被拍摄对象进行视频或是图像的同步采集，从而给用户带来平滑的空间视频或空间图像的观看体验。目标图像可以是多台拍摄设备中的任一台拍摄设备拍摄(即采集)的图像或是视频中的某个视频帧(即视频画面)，该视频可以是录播视频或直播视频等，在此未做具体限定。
S120、基于与拍摄设备对应的校正参数，对目标图像进行校正。其中，在将包含虚拟中心的旋转轴上的至少两个轴点投影到该目标图像上后得到投影轴的情况下，校正后的各目标图像上的投影轴相一致，虚拟中心位于多台拍摄设 备对应的虚拟平面上，并且与多台拍摄设备的物理光心对应。
其中，每个拍摄设备均对应有各自的校正参数，其是用于对相应拍摄设备拍摄的目标图像进行校正，例如可以是几何校正或是仿射变换校正的参数。在实际应用中，例如，校正参数可以通过多种方式进行表示，如校正矩阵、校正向量、校正张量或校正图像等。针对于每个拍摄设备，基于与该拍摄设备对应的校正参数，对该拍摄设备拍摄的目标图像进行校正。在此基础上，例如，当目标图像是视频帧时，校正后的视频帧可供离线存储或是实时播放。
需要说明的是，人工在搭建多台拍摄设备时，通常期望将它们搭建到同一平面(即同一高度)上，事实上很难将它们真正搭建到同一平面上，而与多台拍摄设备对应的虚拟平面可以理解为该同一平面，即满足搭建期望的各拍摄设备所在的平面。虚拟中心可以是该虚拟平面上的与多台拍摄设备的物理光心对应的虚拟出来的中心，即当各台搭建设备的搭建位置满足期望时，它们的光轴所汇聚的中心，此时各物理光心位于一个标准圆上。在此基础上，针对于包含虚拟中心的旋转轴上的至少两个轴点，在将该至少两个轴点投影到目标图像上后得到投影轴的情况下，校正后的各目标图像上的投影轴相一致，即该校正后的各目标图像对应于同一旋转轴，换言之，其上的空间点均已由各自的拍摄设备坐标系转换到定轴坐标系下(即已将不同轴点的目标图像校正为相同轴点的目标图像)，因此校正后的各目标图像不会再因为视角变换而存在抖动现象。需要强调的是，校正前后的目标图像上不一定有投影轴，上文只是通过在目标图像上投影出投影轴来表示校正效果。
本公开实施例，针对多台拍摄设备中的每台拍摄设备，通过该拍摄设备对应的校正参数，对该拍摄设备拍摄的目标图像进行校正。上述实施例，针对多台拍摄设备对应的虚拟平面上的与多台拍摄设备的物理光心对应的虚拟中心，在将包含虚拟中心的旋转轴上的至少两个轴点投影到目标图像上后得到投影轴的情况下，校正后的各目标图像上的投影轴相一致，这说明校正后的各目标图像对应于同一旋转轴，即它们上的空间点均已由各自的拍摄设备坐标系转换到定轴坐标系下，因此校正后的各目标图像不会再因为视角变换而存在抖动现象，从而消除了相邻视角变换带来的图像抖动现象，并且因为可在正常拍摄过程中同步校正图像，降低了对于多台拍摄设备的搭建精度的要求。
图2是本公开实施例中提供的另一种图像校正方法的流程图。本实施例以上述实施例为基础进行调整。本实施例中，校正参数通过如下步骤预先得到：确定虚拟平面，并得到虚拟平面上的虚拟中心；确定包含虚拟中心的旋转轴， 将旋转轴上的至少两个轴点投影到每台拍摄设备分别拍摄的样本图像上，得到样本轴；将各样本图像中由多台拍摄设备中的主拍摄设备拍摄的样本图像作为主样本图像、以及辅拍摄设备拍摄的样本图像作为辅样本图像；针对于每个辅样本图像，对辅样本图像进行校正，以使校正后的辅样本图像上的样本轴与主样本图像上的样本轴相一致；根据校正结果得到各拍摄设备的校正参数。与上述各实施例相同或相应的术语的解释在此不再赘述。
相应的，如图2所示，本实施例的方法可以包括如下步骤：
S210、针对多台拍摄设备，确定与多台拍摄设备对应的虚拟平面，并得到虚拟平面上的与多台拍摄设备的物理光心对应的虚拟中心。
其中，多台拍摄设备对应于同一虚拟平面，多个物理光心对应于同一虚拟中心，虚拟平面和虚拟中心已在上文中进行解释，在此不再赘述。
S220、确定包含虚拟中心的旋转轴，并将旋转轴上的至少两个轴点投影到每台拍摄设备分别拍摄的样本图像上，得到样本轴。
其中，旋转轴上的一个轴点是虚拟中心，该旋转轴可以是与虚拟平面相互垂直的轴，也可以是与虚拟平面呈一定倾斜角度的轴，在此未做具体限定。该旋转轴上的至少两个轴点可以是手动选择出来的，也可以是自动确定出来的；该至少两个轴点可以包括虚拟中心，也可以未包括虚拟中心，在此未做具体限定。将这些轴点投影到各拍摄设备分别拍摄的样本图像上，即每个样本图像上都包含投影出来的这些轴点，任一样本图像上的这些轴点构成了各自的投影轴，该投影轴可以通过投影直线、投影射线或投影线段等进行表示。这些样本图像可以是这些拍摄设备同步拍摄的图像，如拍摄的视频中的同一视频帧；也可以不是同步拍摄的图像，在此未做具体限定。
需要说明的是，样本图像和目标图像的本质均是图像，这里只是为了区分是校正参数确定过程中拍摄的图像还是校正参数应用过程中拍摄的图像而进行的不同命名，并非是对它们的本质含义的具体限定。
S230、将各样本图像中由多台拍摄设备中的主拍摄设备拍摄的样本图像作为主样本图像、以及辅拍摄设备拍摄的样本图像作为辅样本图像。
其中，从多台拍摄设备中确定主拍摄设备，并且将主拍摄设备拍摄的样本图像作为主样本图像；在此基础上，将主拍摄设备之外的其余的拍摄设备分别作为辅拍摄设备，并且将辅拍摄设备拍摄的样本图像作为辅样本图像。在实际应用中，例如，主拍摄设备的数量可以是一个，以便结合后续步骤，令其余的辅拍摄设备拍摄的辅样本图像均以主样本图像为基准进行校正，从而保证了各 样本图像上的样本轴的一致性。
S240、针对于每个辅样本图像，对辅样本图像进行校正，以使校正后的辅样本图像上的样本轴与主样本图像上的样本轴相一致。
S250、根据校正结果得到各拍摄设备的校正参数。
其中，针对于每个辅样本图像，当对其进行校正，以使其上的样本轴与主样本图像上的样本轴相一致时，可以根据校正过程得到校正参数，即与拍摄出该样本图像的辅拍摄设备对应的校正参数，从而得到各个辅拍摄设备的校正参数。另外，如果在与主样本图像进行比较之前，未对主样本图像进行过任何校正，则与主拍摄设备对应的校正参数可以理解为不会让主样本图像出现畸变的参数；否则，可以根据主样本图像的校正过程得到相应的校正参数。
S260、针对每台拍摄设备，获取拍摄设备拍摄的目标图像，并基于与拍摄设备对应的校正参数，对目标图像进行校正。
本公开实施例，通过将包含有虚拟中心的旋转轴上的至少两个轴点投影到每台拍摄设备分别拍摄的样本图像上，得到样本轴；进而，针对各样本图像中由多台拍摄设备中的每台辅拍摄设备拍摄的辅样本图像，对其进行校正，以使校正后的辅样本图像上的样本轴与各样本图像中由多台拍摄设备中的主拍摄设备拍摄的主样本图像相一致，从而根据校正结果得到各拍摄设备的校正参数，即通过几何投影运算的方式，得到每个拍摄设备的校正参数。
一种实施例，在上述实施例的基础上，上述图像校正方法，还可包括：分别确定各拍摄设备的位姿参数；在此基础上，确定虚拟平面，并得到虚拟平面上的虚拟中心，可以包括：根据各物理光心的光心位置拟合得到平面方程，并基于平面方程得到虚拟平面；针对于每个拍摄设备，根据平面方程和拍摄设备的位姿参数，将与拍摄设备对应的光心位置投影到虚拟平面上，得到投影位置，以在得到各投影位置后，通过对各投影位置进行拟合得到虚拟位置，并基于虚拟位置得到虚拟中心。
其中，分别标定出每个拍摄设备的位姿参数。在此基础上，根据各个物理光心的光心位置拟合得到平面方程，进而基于平面方程得到虚拟平面。在实际应用中，例如，可以通过采用最小二乘算法来计算平面系数，从而得到平面方程；再例如，上述光心位置可以通过光心坐标进行表示。针对于每个拍摄设备，可以根据平面方程和该拍摄设备的位姿参数，将相应的光心位置投影到虚拟平面上，得到投影位置，以在得到各个光心位置的投影位置之后，通过对这些投影位置进行拟合来得到虚拟位置，如基于最小二乘算法进行拟合，然后基于虚 拟位置得到虚拟中心。由此可知，位姿参数是校正参数的确定过程中的重要参考因素之一，其保证了校正参数的确定精度。
在实际应用中，例如，位姿参数除了在光心位置投影方面有所应用外，还可在轴点投影方面有所应用。例如，将旋转轴上的至少两个轴点投影到各拍摄设备分别拍摄的样本图像上，可以包括：针对于每个拍摄设备，基于拍摄设备的位姿参数，将旋转轴上的至少两个轴点投影到拍摄设备拍摄的样本图像。
在此基础上，例如，分别确定各拍摄设备的位姿参数，可以包括：分别获取各拍摄设备拍摄的目标图像序列，并确定各目标图像序列之间的特征匹配关系；根据特征匹配关系分别得到各拍摄设备的位姿参数。即通过获取采集到的多视角视频，并通过计算出该多视角视频之间的特征匹配关系，从而得到各拍摄设备的位姿参数(即标定结果)，实际应用中，例如，该位姿参数可以通过外参参数(如旋转矩阵和平移矩阵等)进行表示。需要说明的，上述标定过程是一种自标定过程，在标定过程中无需标定板，通过录制的视频即可完成标定，由此大大降低了标定时间，从而提高了标定效率。
另一实施例，在上述实施例的基础上，确定包含虚拟中心的旋转轴，可包括：获取虚拟平面的平面方程，并对平面方程进行归一化处理，得到虚拟平面的平面法向量；将包含虚拟中心的平面法向量所在的轴，作为旋转轴。其中，对平面方程进行归一化处理后，可以得到虚拟平面的平面法向量。由于虚拟平面上包含有至少一条平面法向量，将该至少一条平面法向量中穿过虚拟中心的平面法向量作为旋转轴，由此得到了与虚拟平面相互垂直的旋转轴。在实际应用中，例如，虚拟平面通常与地面平行，而且被拍摄对象通常是站在地面上(即与地面相互垂直)的，那么当旋转轴垂直于虚拟平面(即地面)时，对应于旋转轴的校正后的样本图像更加符合用户的视觉感受，提升了用户体验。
另一实施例，在上述实施例的基础上，样本轴通过样本线段进行表示，对辅样本图像进行校正，以使校正后的辅样本图像上的样本轴与主样本图像上的样本轴相一致，可以包括：对辅样本图像执行校正操作，以使校正后的辅样本图像上的样本线段与主样本图像上的样本线段相平行并且长度相同，而且校正后的辅样本图像上的样本线段在校正后的辅样本图像上的相对位置、与主样本图像上的样本线段在主样本图像上的相对位置相同，其中，校正操作包括旋转操作、缩放操作和平移操作。其中，旋转操作可以让校正后的辅样本图像上的样本线段与主样本图像上的样本线段相平行，缩放操作可以让校正后的辅样本图像上的样本线段与主样本图像上的样本线段长度相同，且平移操作可以让校 正后的辅样本图像上的样本线段在校正后的辅样本图像上的相对位置、与主样本图像上的样本线段在主样本图像上的相对位置相同，从而保证了校正后的辅样本图像上的样本线段与主样本图像上的样本线段的一致性，即二者上的轴点的一致性。需要说明的是，旋转操作、缩放操作和平移操作的先后执行顺序没有具体要求，因为它们都可以被归结到校正参数中。示例性的，以校正参数通过校正矩阵进行表示为例，可以将旋转操作对应的旋转矩阵、缩放操作对应的缩放矩阵、及平移操作对应的平移矩阵进行合并，从而得到校正矩阵。
另一实施例，在上述实施例的基础上，在对辅样本图像进行校正之前，上述图像校正方法，还可包括：对主样本图像进行旋转，以使旋转后的主样本图像上的样本轴与样本图像的目标轴相平行；根据旋转结果更新主样本图像，并且得到主拍摄设备的校正参数；根据校正结果得到各拍摄设备的校正参数，可包括：根据校正结果得到辅拍摄设备的校正参数。其中，目标轴可以是样本图像的水平轴或是竖直轴，在实际应用中，例如，其可以是竖直轴，这样一来，在对被拍摄对象进行水平拍摄的情况下，被拍摄对象在旋转后的主样本图像上是垂直站立在地面上的，与用户的视觉感受更为匹配。进而，根据旋转结果更新主样本图像，以便将旋转后的主样本图像作为基准来校正其余的辅样本图像，并根据校正结果得到辅拍摄设备的矫正参数；与此同时，还可以根据旋转结果得到主拍摄设备的校正参数。
为了从整体上更好地理解上述各实施例，下面通过结合示例来进行示例性说明。示例性的，目标图像和样本图像均是拍摄设备拍摄到的视频中的视频帧，校正矩阵确定过程可以理解为预处理过程，且校正矩阵应用过程可以发生在视频播放过程中。在此基础上，参见图3，输入同步视频帧(即同步的各样本图像)，并基于这些视频帧进行多相机自标定，得到各相机的位姿参数。根据各相机的物理光心的光心坐标拟合得到虚拟平面，并结合位姿参数，将各物理光心投影到该虚拟平面上，对各投影结果进行拟合，得到虚拟中心。计算虚拟平面的平面法向量，并将穿过虚拟中心的平面法向量作为旋转轴，确定该旋转轴上的两个轴点。将这两个轴点投影到各视频帧上，得到每个视频帧上的投影线段。基于各视频帧上的投影线段进行相应相机的仿射变换矩阵(即校正矩阵)的计算，输出校正矩阵，并将校正矩阵和相应相机的相机编号一一对应。至此，预处理过程完成。在视频播放过程中，针对于获取的某视频帧(即目标图像)，输入该视频帧对应的相机编号，并查询与该相机编号对应的校正矩阵，然后基于该校正矩阵校正该视频帧，最后输出校正后的视频帧，从而可以基于该校正后的视 频帧实现离线存储或是实时播放。
在介绍本公开下述实施例前，先对其的应用场景进行示例性说明：以自由视角拍摄为例，为了能够拍摄到自由视角的多视角视频，采集端往往需要搭建多台拍摄设备来同步采集视频，以便在播放端构建平滑的空间视频观看体验。但是，多台拍摄设备的搭建方案存在如下状况：1)硬件成本较高；2)拍摄设备需要做到白平衡和亮度等设备参数的一致性、以及时间同步的一致性，越多数量的拍摄设备，越难以统一一致性；3)需要标定每个拍摄设备的位姿参数，越多数量的拍摄设备会导致标定复杂度的显著上升，从而导致标定时间过长。需要说明的是，下述实施例是以自由视角拍摄的应用场景为例进行阐述，但这并非意味着下述实施例只能应用在该应用场景中，其余应用场景(如光场拍摄的应用场景)下的图像校正过程依然适用。
图4是本公开实施例中提供的再一种图像校正方法的流程图。本实施例以上述实施例为基础进行调整。在本实施例中，例如，在对目标图像进行校正之后，上述图像校正方法，还可以包括：针对于多台拍摄设备中在摆放位置上相邻的第一拍摄设备和第二拍摄设备，将第一拍摄设备拍摄的目标图像作为第一物理图像并且将第二拍摄设备拍摄的目标图像作为第二物理图像；基于第一物理图像和第二物理图像，生成虚拟图像，其中，虚拟图像所在的虚拟视角位于第一物理图像所在的物理视角以及第二物理图像所在的物理视角之间。其中，与上述各实施例相同或是相应的术语的解释在此不再赘述。
相应的，如图4所示，本实施例的方法可以包括如下步骤：
S310、针对于多台拍摄设备中的每台拍摄设备，获取拍摄设备拍摄的目标图像。
S320、基于与拍摄设备对应的校正参数，对目标图像进行校正。其中，在将包含虚拟中心的旋转轴上的至少两个轴点投影到该目标图像上后得到投影轴的情况下，校正后的各目标图像上的投影轴相一致，虚拟中心位于多台拍摄设备对应的虚拟平面上，并且与多台拍摄设备的物理光心对应。
S330、针对多台拍摄设备中在摆放位置上相邻的第一拍摄设备和第二拍摄设备，将第一拍摄设备拍摄的目标图像作为第一物理图像并且将第二拍摄设备拍摄的目标图像作为第二物理图像。
其中，第一拍摄设备和第二拍摄设备可以是在摆放位置上相邻的两台拍摄设备，由于它们是实际存在的电子设备，因此可以将它们拍摄的目标图像称为物理图像。这里将第一拍摄设备拍摄的目标图像作为第一物理图像并且将第二 拍摄设备拍摄的目标图像作为第二物理图像。
S340、基于第一物理图像以及第二物理图像，生成虚拟图像。其中，虚拟图像所在的虚拟视角位于第一物理图像所在的物理视角以及第二物理图像所在的物理视角之间。
其中，基于第一物理图像和第二物理图像生成虚拟图像，该虚拟图像所在的虚拟视角位于第一物理图像所在的物理视角和第二物理图像所在的物理视角之间，即通过自动合成相邻物理视角之间的虚拟视角下的虚拟图像来达到视频插帧的效果。这样一来，为了达到同样的空间视频观看体验，或是说同等或者更佳的自由视角视频效果，可以通过生成虚拟图像(即虚拟视角下的图像)的方式来降低拍摄设备的数量，从而避免了因拍摄设备过多而带来的一系列情况，是一种轻量化的自由视角采集方案。
本公开实施例，针对于多台拍摄设备中的在摆放位置上相邻的第一拍摄设备拍摄的第一物理图像和第二拍摄设备拍摄的第二物理图像，根据这二者生成相应物理视角之间的虚拟视角下的虚拟图像，其通过生成虚拟图像的方式来降低拍摄设备的数量，从而避免了因拍摄设备过多而带来的硬件成本较高、难以统一一致性、以及标定时间过长的情况。
一种实施例，在上述实施例的基础上，基于第一物理图像和第二物理图像，生成虚拟图像，可包括：确定第一拍摄设备的第一景深和第二拍摄设备的第二景深，且对第一物理图像和第二物理图像进行匹配，根据第一景深、第二景深和匹配结果进行点云重建，并基于点云重建结果得到虚拟图像。其中，第一物理图像以及第二物理图像的匹配过程可以基于立体匹配(stereo matching)等算法来实现，然后基于第一景深、第二景深和匹配结果进行点云重建，从而生成虚拟视角下的点云，得到虚拟图像，由此达到了视频插帧的效果。
另一实施例，在上述实施例的基础上，基于第一物理图像和第二物理图像，生成虚拟图像，可以包括：将第一物理图像以及第二物理图像作为视频文件来计算光流，并根据光流生成虚拟图像。其中，自由视角是同一时间点下在空间上的多视角拍摄的结果，换个思路，其也可以理解为基于同一拍摄设备在多个空间位置(即多个视角)上的拍摄结果，这就是时域。因此，可以将第一物理图像和第二物理图像作为视频文件来计算光流，然后根据光流生成虚拟图像，由此达到了视频插帧的效果。
另一实施例，在上述实施例的基础上，基于第一物理图像和第二物理图像，生成虚拟图像，可以包括：将第一物理图像和第二物理图像输入到预先训练完 成的视频插帧深度学习模型中，根据视频插帧深度学习模型的输出结果，生成虚拟图像。其中，视频插帧深度学习模型可以理解为端到端的用于实现视频插帧的深度学习模型，在将第一物理图像和第二物理图像输入到其中之后，可以得到两个物理视角中间的虚拟视角下的虚拟图像，例如可以在两个物理图像的基础上生成两个虚拟图像，由此达到了视频插帧的效果。
为了从整体上更好地理解上述各实施例，下面通过结合示例来进行示例性说明。示例性的，通过在舞台、场馆或演播厅等现场内的多个物理视角下环绕部署相机，将现场(即采集端)采集到的多路视频通过网络回传至视频云端进行处理，然后将处理后的自由视角视频实时传送给播放端进行播放，以让用户的观看体验更加身临其境。在视频云端上，可以对采集的多路视频进行如图5所示的处理过程：将采集的多路视频输入到标定系统中，得到各相机的位姿参数；进而，再将该多路视频和相应的位姿参数输入到定轴系统中，以将各路视频的空间点均转换到定轴坐标系下，得到相应的定轴视角(即定轴下的物理视角)；然后，基于相邻的定轴视角生成虚拟视角，从而可以基于定轴视角和虚拟视角得到自由视角视频，这是一种轻量化的自由视角采集方案。
图6为本公开实施例中提供的图像校正装置的结构框图，该装置用于执行上述任意实施例所提供的图像校正方法。该装置与上述各个实施例的图像校正方法属于同一个构思，在图像校正装置的实施例中未详尽描述的细节内容，可参考上述图像校正方法的实施例。参见图6，该装置可以包括：目标图像获取模块410和目标图像校正模块420。其中，
目标图像获取模块410，设置为针对多台拍摄设备中的每台拍摄设备，获取拍摄设备拍摄的目标图像；
目标图像校正模块420，设置为基于与拍摄设备对应的校正参数，对该目标图像进行校正；
其中，在将包含虚拟中心的旋转轴上的至少两个轴点投影到目标图像上后得到投影轴的情况下，校正后的各目标图像上的投影轴相一致，虚拟中心位于多台拍摄设备对应的虚拟平面上，并且与多台拍摄设备的物理光心对应。
在一实施例中，校正参数通过如下模块预先确定：
虚拟中心得到模块，设置为确定虚拟平面，并得到虚拟平面上的虚拟中心；
样本轴得到模块，设置为确定包含虚拟中心的旋转轴，并将旋转轴上的至少两个轴点投影到每台拍摄设备分别拍摄的样本图像上，得到样本轴；
辅样本图像得到模块，设置为将各样本图像中由多台拍摄设备中的主拍摄 设备拍摄的样本图像作为主样本图像、及辅拍摄设备拍摄的样本图像作为辅样本图像；
辅样本图像校正模块，设置为针对每个辅样本图像，对辅样本图像进行校正，以使校正后的辅样本图像上的样本轴与主样本图像上的样本轴相一致；
校正参数第一得到模块，设置为根据校正结果得到各拍摄设备的校正参数。
在此基础上，在一实施例中，上述图像校正装置，还可以包括：
位姿参数获取模块，设置为分别确定各拍摄设备的位姿参数；
虚拟中心得到模块，可以包括：
虚拟平面得到单元，设置为根据各物理光心的光心位置拟合得到平面方程，并基于平面方程得到虚拟平面；
虚拟中心得到单元，设置为针对于每个拍摄设备，根据平面方程和拍摄设备的位姿参数，将与拍摄设备对应的光心位置投影到虚拟平面上，得到投影位置，以在得到各投影位置之后，通过对各投影位置进行拟合得到虚拟位置，并基于虚拟位置得到虚拟中心；
在此基础上，在一实施例中，样本轴得到模块，可以包括：
轴点投影单元，设置为针对每个拍摄设备，基于拍摄设备的位姿参数，将旋转轴上的至少两个轴点投影到拍摄设备拍摄的样本图像。
在此基础上，在一实施例中，位姿参数获取模块，可以包括：
特征匹配关系确定单元，设置为分别获取各拍摄设备拍摄的样本图像序列，确定各样本图像序列之间的特征匹配关系；
位姿参数得到单元，设置为根据特征匹配关系得到各拍摄设备的位姿参数。
另一实施例中，样本轴得到模块，可以包括：
平面法向量得到单元，设置为获取虚拟平面的平面方程，并对平面方程进行归一化处理，得到虚拟平面的平面法向量；
旋转轴得到单元，设置为将包含虚拟中心的平面法向量所在的轴作为旋转轴。
另一实施例中，样本轴是通过样本线段进行表示的，辅样本图像校正模块，可以包括：
辅样本图像校正单元，设置为对辅样本图像执行校正操作，以使校正后的辅样本图像上的样本线段与主样本图像上的样本线段相平行并且长度相同，而且校正后的辅样本图像上的样本线段在校正后的辅样本图像上的相对位置、与主样本图像上的样本线段在主样本图像上的相对位置相同，其中，校正操作包 括旋转操作、缩放操作和平移操作。
另一实施例中，上述图像校正装置，还可以包括：
主样本图像旋转模块，设置为在对辅样本图像进行校正之前，对主样本图像进行旋转，以使旋转后的主样本图像上的样本轴与主样本图像的目标轴相平行；
校正参数第二得到模块，设置为根据旋转结果更新主样本图像，并且得到主拍摄设备的校正参数；
校正参数第一得到模块，设置为：根据校正结果得到辅拍摄设备的校正参数。
另一实施例中，上述图像校正装置，还可以包括：
第二物理图像得到模块，设置为在对目标图像进行校正之后，针对于多台拍摄设备中在摆放位置上相邻的第一拍摄设备和第二拍摄设备，将第一拍摄设备拍摄的目标图像作为第一物理图像、并且将第二拍摄设备拍摄的目标图像作为第二物理图像；
虚拟图像生成模块，设置为基于第一物理图像以及第二物理图像，生成虚拟图像，其中，虚拟图像所在的虚拟视角位于第一物理图像所在的物理视角以及第二物理图像所在的物理视角之间。
在此基础上，另一实施例中，虚拟图像生成模块，可以包括：
虚拟图像第一生成单元，设置为确定第一拍摄设备的第一景深和第二拍摄设备的第二景深，并且对第一物理图像和第二物理图像进行匹配，根据第一景深、第二景深以及匹配结果进行点云重建，并基于点云重建结果得到虚拟图像。
另一实施例中，虚拟图像生成模块，可以包括：
虚拟图像第二生成单元，设置为将第一物理图像以及第二物理图像作为视频文件来计算光流，并根据光流生成虚拟图像。
另一实施例中，虚拟图像生成模块，可以包括：
虚拟图像第三生成单元，设置为将第一物理图像和第二物理图像输入到预先训练完成的视频插帧深度学习模型中，根据视频插帧深度学习模型的输出结果，生成虚拟图像。
本公开实施例所提供的图像校正装置，通过目标图像获取模块和目标图像校正模块相互配合，针对多台拍摄设备中的每台拍摄设备，通过与该拍摄设备对应的校正参数，对该拍摄设备拍摄的目标图像进行校正。上述装置，针对与多台拍摄设备对应的虚拟平面上的与多台拍摄设备的物理光心对应的虚拟中 心，在将包含虚拟中心的旋转轴上的至少两个轴点投影到目标图像上后得到投影轴的情况下，校正后的各目标图像上的投影轴一致，这说明校正后的各目标图像对应于同一旋转轴，即它们上的空间点均已由各自的拍摄设备坐标系转换到定轴坐标系下，因此校正后的各目标图像不会再因为视角变换而存在抖动现象，从而消除了相邻视角变换带来的图像抖动现象，并且因为可在正常拍摄过程中进行同步的图像校正，降低了对于多台拍摄设备的搭建精度的要求。
本公开实施例提供的图像校正装置可执行本公开任意实施例所提供的图像校正方法，具备执行方法相应的功能模块和有益效果。
值得注意的是，上述图像校正装置的实施例中，所包括的各个单元和模块只是按照功能逻辑进行划分的，但并不局限于上述的划分，只要能够实现相应的功能即可；另外，各功能单元的具体名称也只是为了便于相互区分，并不用于限制本公开的保护范围。
下面参考图7，其示出了适于用来实现本公开实施例的电子设备(例如图7中的终端设备或服务器)500的结构示意图。本公开实施例中的电子设备可以包括但不限于诸如移动电话、笔记本电脑、数字广播接收器、个人数字助理(Personal Digital Assistant，PDA)、平板电脑(PAD)、便携式多媒体播放器(Portable Media Player，PMP)、车载终端(例如车载导航终端)等等的移动终端以及诸如数字电视(Television，TV)、台式计算机等等的固定终端。图7示出的电子设备仅仅是一个示例，不应对本公开实施例的功能和使用范围带来任何限制。
如图7所示，电子设备500可以包括处理装置(例如中央处理器、图形处理器等)501，其可以根据存储在只读存储器(Read-Only Memory，ROM)502中的程序或者从存储装置508加载到随机访问存储器(Random Access Memory，RAM)503中的程序而执行各种适当的动作和处理。在RAM 503中，还存储有电子设备500操作所需的各种程序和数据。处理装置501、ROM 502以及RAM503通过总线504彼此相连。输入/输出(Input/Output，I/O)接口505也连接至总线504。
通常，以下装置可以连接至I/O接口505：包括例如触摸屏、触摸板、键盘、鼠标、摄像头、麦克风、加速度计、陀螺仪等的输入装置506；包括例如液晶显示器(Liquid Crystal Display，LCD)、扬声器、振动器等的输出装置507；包括例如磁带、硬盘等的存储装置508；以及通信装置509。通信装置509可以允许电子设备500与其他设备进行无线或有线通信以交换数据。虽然图7中示出了 具有各种装置的电子设备500，但是应理解的是，并不要求实施或具备所有示出的装置。可以替代地实施或具备更多或更少的装置。
特别地，根据本公开的实施例，上文参考流程图描述的过程可以被实现为计算机软件程序。例如，本公开的实施例包括一种计算机程序产品，其包括承载在非暂态计算机可读介质上的计算机程序，该计算机程序包含用于执行流程图所示的方法的程序代码。在这样的实施例中，该计算机程序可以通过通信装置509从网络上被下载和安装，或者从存储装置508被安装，或者从ROM 502被安装。在该计算机程序被处理装置501执行时，执行本公开实施例的方法中限定的上述功能。
需要说明的是，本公开上述的计算机可读介质可以是计算机可读信号介质或者计算机可读存储介质或者是上述两者的任意组合。计算机可读存储介质例如可以是——但不限于——电、磁、光、电磁、红外线、或半导体的系统、装置或器件，或者任意以上的组合。计算机可读存储介质的例子可以包括但不限于：具有一个或多个导线的电连接、便携式计算机磁盘、硬盘、随机访问存储器(RAM)、只读存储器(ROM)、可擦式可编程只读存储器(Erasable Programmable Read-Only Memory，EPROM)或闪存、光纤、便携式紧凑磁盘只读存储器(Compact Disc Read-Only Memory，CD-ROM)、光存储器件、磁存储器件、或者上述的任意合适的组合。在本公开中，计算机可读存储介质可以是任何包含或存储程序的有形介质，该程序可以被指令执行系统、装置或者器件使用或者与其结合使用。而在本公开中，计算机可读信号介质可以包括在基带中或者作为载波一部分传播的数据信号，其中承载了计算机可读的程序代码。这种传播的数据信号可以采用多种形式，包括但不限于电磁信号、光信号或上述的任意合适的组合。计算机可读信号介质还可以是计算机可读存储介质以外的任何计算机可读介质，该计算机可读信号介质可以发送、传播或者传输用于由指令执行系统、装置或者器件使用或者与其结合使用的程序。计算机可读介质上包含的程序代码可以用任何适当的介质传输，包括但不限于：电线、光缆、射频(Radio Frequency，RF)等等，或者上述的任意合适的组合。
在一些实施方式中，客户端、服务器可以利用诸如HTTP(HyperText Transfer Protocol，超文本传输协议)之类的任何当前已知或未来研发的网络协议进行通信，并且可以与任意形式或介质的数字数据通信(例如，通信网络)互连。通信网络的示例包括局域网(Local Area Network，LAN)，广域网(Wide Area Network，WAN)，网际网(例如，互联网)以及端对端网络(例如，ad hoc端 对端网络)，以及任何当前已知或未来研发的网络。
上述计算机可读介质可以是上述电子设备中所包含的；也可以是单独存在，而未装配入该电子设备中。
上述计算机可读介质承载有一个或者多个程序，当上述一个或者多个程序被该电子设备执行时，使得该电子设备：
针对多台拍摄设备中的每台拍摄设备，获取拍摄设备拍摄的目标图像；
基于与拍摄设备对应的校正参数，对目标图像进行校正；
其中，在将包含虚拟中心的旋转轴上的至少两个轴点投影到目标图像上后得到投影轴的情况下，校正后的各目标图像上的投影轴相一致，虚拟中心位于多台拍摄设备对应的虚拟平面上，并且与多台拍摄设备的物理光心对应。
存储介质可以是非暂态(non-transitory)存储介质。
可以以一种或多种程序设计语言或其组合来编写用于执行本公开的操作的计算机程序代码，上述程序设计语言包括但不限于面向对象的程序设计语言—诸如Java、Smalltalk、C++，还包括常规的过程式程序设计语言—诸如“C”语言或类似的程序设计语言。程序代码可以完全地在用户计算机上执行、部分地在用户计算机上执行、作为一个独立的软件包执行、部分在用户计算机上部分在远程计算机上执行、或者完全在远程计算机或服务器上执行。在涉及远程计算机的情形中，远程计算机可以通过任意种类的网络——包括局域网(LAN)或广域网(WAN)—连接到用户计算机，或者，可以连接到外部计算机(例如利用因特网服务提供商来通过因特网连接)。
附图中的流程图和框图，图示了按照本公开各种实施例的系统、方法和计算机程序产品的可能实现的体系架构、功能和操作。在这点上，流程图或框图中的每个方框可以代表一个模块、程序段、或代码的一部分，该模块、程序段、或代码的一部分包含一个或多个用于实现规定的逻辑功能的可执行指令。也应当注意，在有些作为替换的实现中，方框中所标注的功能也可以以不同于附图中所标注的顺序发生。例如，两个接连地表示的方框实际上可以基本并行地执行，它们有时也可以按相反的顺序执行，这依所涉及的功能而定。也要注意的是，框图和/或流程图中的每个方框、以及框图和/或流程图中的方框的组合，可以用执行规定的功能或操作的专用的基于硬件的系统来实现，或者可以用专用硬件与计算机指令的组合来实现。
描述于本公开实施例中所涉及到的单元可以通过软件的方式实现，也可以通过硬件的方式来实现。其中，单元的名称在某种情况下并不构成对该单元本 身的限定，例如，目标图像获取模块还可被描述为“针对多台拍摄设备中的每台拍摄设备，获取拍摄设备拍摄的目标图像的模块”。
本文中以上描述的功能可以至少部分地由一个或多个硬件逻辑部件来执行。例如，非限制性地，可以使用的示范类型的硬件逻辑部件包括：现场可编程门阵列(Field Programmable Gate Array，FPGA)、专用集成电路(Application Specific Integrated Circuit，ASIC)、专用标准产品(Application Specific Standard Product，ASSP)、片上系统(System on Chip，SOC)、复杂可编程逻辑设备(Complex Programmable Logic Device，CPLD)等等。
在本公开的上下文中，机器可读介质可以是有形的介质，其可以包含或存储以供指令执行系统、装置或设备使用或与指令执行系统、装置或设备结合地使用的程序。机器可读介质可以是机器可读信号介质或机器可读储存介质。机器可读介质可以包括但不限于电子的、磁性的、光学的、电磁的、红外的、或半导体系统、装置或设备，或者上述内容的任何合适组合。机器可读存储介质的示例会包括基于一个或多个线的电气连接、便携式计算机盘、硬盘、随机存取存储器(RAM)、只读存储器(ROM)、可擦除可编程只读存储器(EPROM)或快闪存储器、光纤、便捷式紧凑盘只读存储器(CD-ROM)、光学储存设备、磁储存设备、或上述内容的任何合适组合。
根据本公开的一个或多个实施例，【示例一】提供了一种图像校正方法，该方法可以包括：
针对多台拍摄设备中的每台拍摄设备，获取拍摄设备拍摄的目标图像；
基于与拍摄设备对应的校正参数，对目标图像进行校正；
其中，在将包含虚拟中心的旋转轴上的至少两个轴点投影到目标图像上后得到投影轴的情况下，校正后的各目标图像上的投影轴相一致，虚拟中心位于多台拍摄设备对应的虚拟平面上，并且与多台拍摄设备的物理光心对应。
根据本公开的一个或是多个实施例，【示例二】提供了示例一的方法，校正参数可以通过如下步骤预先确定：
确定虚拟平面，并得到虚拟平面上的虚拟中心；
确定包含虚拟中心的旋转轴，并将旋转轴上的至少两个轴点投影到每台拍摄设备分别拍摄的样本图像上，得到样本轴；
将各样本图像中的由多台拍摄设备中的主拍摄设备拍摄的样本图像作为主样本图像、以及辅拍摄设备拍摄的样本图像作为辅样本图像；
针对每个辅样本图像，对辅样本图像进行校正，以使校正后的辅样本图像 上的样本轴与主样本图像上的样本轴相一致；
根据校正结果得到各拍摄设备的校正参数。
根据本公开的一个或是多个实施例，【示例三】提供了示例二的方法，上述图像校正方法，还可以包括：
分别确定各拍摄设备的位姿参数；
确定虚拟平面，并得到虚拟平面上的虚拟中心，可以包括：
根据各物理光心的光心位置拟合得到平面方程，并基于平面方程得到虚拟平面；
针对每个拍摄设备，根据平面方程和拍摄设备的位姿参数，将与拍摄设备对应的光心位置投影到虚拟平面上，得到投影位置，以在得到各投影位置后，通过对各投影位置进行拟合得到虚拟位置，并基于虚拟位置得到虚拟中心。
根据本公开的一个或是多个实施例，【示例四】提供了示例三的方法，分别确定各拍摄设备的位姿参数，可以包括：
分别获取各拍摄设备拍摄的样本图像序列，并确定各样本图像序列之间的特征匹配关系；
根据特征匹配关系分别得到各拍摄设备的位姿参数。
根据本公开的一个或是多个实施例，【示例五】提供了示例二的方法，确定包含虚拟中心的旋转轴，可以包括：
获取虚拟平面的平面方程，并对平面方程进行归一化处理，得到虚拟平面的平面法向量；
将包含虚拟中心的平面法向量所在的轴，作为旋转轴。
根据本公开的一个或多个实施例，【示例六】提供了示例二的方法，样本轴是通过样本线段进行表示的，则对辅样本图像进行校正，以使校正后的辅样本图像上的样本轴与主样本图像上的样本轴相一致，可以包括：
对辅样本图像执行校正操作，以使校正后的辅样本图像上的样本线段与主样本图像上的样本线段相平行且长度相同，而且校正后的辅样本图像上的样本线段在校正后的辅样本图像上的相对位置与主样本图像上的样本线段在主样本图像上的相对位置相同，其中，校正操作包括旋转操作、缩放操作和平移操作。
根据本公开的一个或是多个实施例，【示例七】提供了示例二的方法，在对辅样本图像进行校正之前，上述图像校正方法，还可以包括：
对主样本图像进行旋转，以使旋转后的主样本图像上的样本轴与主样本图像的目标轴相平行；
根据旋转结果更新主样本图像，并且得到主拍摄设备的校正参数；
根据校正结果得到各拍摄设备的校正参数，可以包括：
根据校正结果得到辅拍摄设备的校正参数。
根据本公开的一个或是多个实施例，【示例八】提供了示例一的方法，在对目标图像进行校正之后，上述图像校正方法，还可以包括：
针对多台拍摄设备中在摆放位置上相邻的第一拍摄设备和第二拍摄设备，将第一拍摄设备拍摄的目标图像作为第一物理图像并且将第二拍摄设备拍摄的目标图像作为第二物理图像；
基于第一物理图像和第二物理图像，生成虚拟图像，其中，虚拟图像所在的虚拟视角位于第一物理图像所在的物理视角及第二物理图像所在的物理视角之间。
根据本公开的一个或是多个实施例，【示例九】提供了示例八的方法，基于第一物理图像和第二物理图像，生成虚拟图像，可以包括：
确定第一拍摄设备的第一景深和第二拍摄设备的第二景深，且对第一物理图像和第二物理图像进行匹配，并根据第一景深、第二景深以及匹配结果进行点云重建，并基于点云重建结果得到虚拟图像。
根据本公开的一个或是多个实施例，【示例十】提供了示例八的方法，基于第一物理图像和第二物理图像，生成虚拟图像，可以包括：
将第一物理图像以及第二物理图像作为视频文件来计算光流，并根据光流生成虚拟图像。
根据本公开的一个或多个实施例，【示例十一】提供了示例八的方法，基于第一物理图像和第二物理图像，生成虚拟图像，可以包括：
将第一物理图像和第二物理图像输入到预先训练完成的视频插帧深度学习模型中，根据视频插帧深度学习模型的输出结果，生成虚拟图像。
根据本公开的一个或多个实施例，【示例十二】提供了一种图像校正装置，该装置可以包括：
目标图像获取模块，设置为针对多台拍摄设备中的每台拍摄设备，获取拍摄设备拍摄的目标图像；
目标图像校正模块，设置为基于与拍摄设备相对应的校正参数，对目标图像进行校正；
其中，在将包含虚拟中心的旋转轴上的至少两个轴点投影到目标图像上后得到投影轴的情况下，校正后的各目标图像上的投影轴相一致，虚拟中心位于 多台拍摄设备对应的虚拟平面上，并且与多台拍摄设备的物理光心对应。
本领域技术人员应当理解，本公开中所涉及的公开范围，并不限于上述技术特征的特定组合而成的实施例，同时也应涵盖在不脱离上述公开构思的情况下，由上述技术特征或其等同特征进行任意组合而形成的其它实施例。例如上述特征与本公开中公开的(但不限于)具有类似功能的技术特征进行互相替换而形成的实施例。
此外，虽然采用特定次序描绘了各操作，但是这不应当理解为要求这些操作以所示出的特定次序或以顺序次序执行来执行。在一定环境下，多任务和并行处理可能是有利的。同样地，虽然在上面论述中包含了若干具体实现细节，但是这些不应当被解释为对本公开的范围的限制。在单独的实施例的上下文中描述的某些特征还可以组合地实现在单个实施例中。相反地，在单个实施例的上下文中描述的各种特征也可以单独地或以任何合适的子组合的方式实现在多个实施例中。
尽管已经采用特定于结构特征和/或方法逻辑动作的语言描述了本主题，但是应当理解所附权利要求书中所限定的主题未必局限于上面描述的特定特征或动作。相反，上面所描述的特定特征和动作仅仅是实现权利要求书的示例形式。

Claims (14)

  1. 一种图像校正方法，包括：
    针对多台拍摄设备中的每台所述拍摄设备，获取所述拍摄设备拍摄的目标图像；
    基于与所述拍摄设备对应的校正参数，对所述目标图像进行校正；
    其中，在将包含虚拟中心的旋转轴上的至少两个轴点投影到所述目标图像上后得到投影轴的情况下，校正后的每个所述目标图像上的所述投影轴相一致，所述虚拟中心位于所述多台拍摄设备对应的虚拟平面上，并且与所述多台拍摄设备的物理光心对应。
  2. 根据权利要求1所述的方法，其中，所述校正参数通过如下步骤预先确定：
    确定所述虚拟平面，并得到所述虚拟平面上的所述虚拟中心；
    确定包含所述虚拟中心的旋转轴，并将所述旋转轴上的所述至少两个轴点投影到每台所述拍摄设备分别拍摄的样本图像上，得到样本轴；
    将所述样本图像中由所述多台拍摄设备中的主拍摄设备拍摄的所述样本图像作为主样本图像、以及辅拍摄设备拍摄的所述样本图像作为辅样本图像；
    针对每个所述辅样本图像，对所述辅样本图像进行校正，以使校正后的所述辅样本图像上的所述样本轴与所述主样本图像上的所述样本轴相一致；
    根据校正结果得到所述多台拍摄设备的所述校正参数。
  3. 根据权利要求2所述的方法，还包括：
    分别确定每台所述拍摄设备的位姿参数；
    所述确定所述虚拟平面，并得到所述虚拟平面上的所述虚拟中心，包括：
    根据所述多台拍摄设备的所述物理光心的光心位置拟合得到平面方程，并基于所述平面方程得到所述虚拟平面；
    针对每个所述拍摄设备，根据所述平面方程和所述拍摄设备的位姿参数，将与所述拍摄设备对应的所述光心位置投影到所述虚拟平面上，得到投影位置，以在得到每个所述投影位置后，通过对每个所述投影位置进行拟合得到虚拟位置，并基于所述虚拟位置得到所述虚拟中心。
  4. 根据权利要求3所述的方法，其中，所述分别确定每台所述拍摄设备的位姿参数，包括：
    分别获取每台所述拍摄设备拍摄的样本图像序列，确定每个所述样本图像序列之间的特征匹配关系；
    根据所述特征匹配关系分别得到每台所述拍摄设备的位姿参数。
  5. 根据权利要求2所述的方法，其中，所述确定包含所述虚拟中心的旋转轴，包括：
    获取所述虚拟平面的平面方程，并对所述平面方程进行归一化处理，得到所述虚拟平面的平面法向量；
    将包含所述虚拟中心的所述平面法向量所在的轴，作为旋转轴。
  6. 根据权利要求2所述的方法，其中，所述样本轴是通过样本线段进行表示的，所述对所述辅样本图像进行校正，以使校正后的所述辅样本图像上的所述样本轴与所述主样本图像上的所述样本轴相一致，包括：
    对所述辅样本图像执行校正操作，以使校正后的所述辅样本图像上的所述样本线段与所述主样本图像上的所述样本线段相平行并且长度相同，而且所述校正后的所述辅样本图像上的所述样本线段在所述校正后的所述辅样本图像上的相对位置、与所述主样本图像上的所述样本线段在所述主样本图像上的相对位置相同，其中，所述校正操作包括旋转操作、缩放操作和平移操作。
  7. 根据权利要求2所述的方法，在所述对所述辅样本图像进行校正之前，还包括：
    对所述主样本图像进行旋转，以使旋转后的所述主样本图像上的所述样本轴与所述主样本图像的目标轴相平行；
    根据旋转结果更新所述主样本图像，并且得到所述主拍摄设备的所述校正参数；
    所述根据校正结果得到所述多台拍摄设备的所述校正参数，包括：
    根据校正结果得到所述辅拍摄设备的所述校正参数。
  8. 根据权利要求1所述的方法，在所述对所述目标图像进行校正之后，还包括：
    针对所述多台拍摄设备中在摆放位置上相邻的第一拍摄设备以及第二拍摄设备，将所述第一拍摄设备拍摄的所述目标图像作为第一物理图像并且将所述第二拍摄设备拍摄的所述目标图像作为第二物理图像；
    基于所述第一物理图像和所述第二物理图像，生成虚拟图像，其中，所述虚拟图像所在的虚拟视角位于所述第一物理图像所在的物理视角以及所述第二物理图像所在的物理视角之间。
  9. 根据权利要求8所述的方法，其中，所述基于所述第一物理图像和所述第二物理图像，生成虚拟图像，包括：
    确定所述第一拍摄设备的第一景深和所述第二拍摄设备的第二景深，并且 对所述第一物理图像和所述第二物理图像进行匹配，根据所述第一景深、所述第二景深以及匹配结果进行点云重建，并基于点云重建结果得到虚拟图像。
  10. 根据权利要求8所述的方法，其中，所述基于所述第一物理图像和所述第二物理图像，生成虚拟图像，包括：
    将所述第一物理图像以及所述第二物理图像作为视频文件来计算光流，并根据所述光流生成虚拟图像。
  11. 根据权利要求8所述的方法，其中，所述基于所述第一物理图像和所述第二物理图像，生成虚拟图像，包括：
    将所述第一物理图像和所述第二物理图像输入到预先训练完成的视频插帧深度学习模型中，根据所述视频插帧深度学习模型的输出结果，生成虚拟图像。
  12. 一种图像校正装置，包括：
    目标图像获取模块，设置为针对多台拍摄设备中的每台所述拍摄设备，获取所述拍摄设备拍摄的目标图像；
    目标图像校正模块，设置为基于与所述拍摄设备相对应的校正参数，对所述目标图像进行校正；
    其中，在将包含虚拟中心的旋转轴上的至少两个轴点投影到所述目标图像上后得到投影轴的情况下，校正后的每个所述目标图像上的所述投影轴相一致，所述虚拟中心位于所述多台拍摄设备对应的虚拟平面上，并且与所述多台拍摄设备的物理光心对应。
  13. 一种电子设备，包括：
    一个或多个处理器；
    存储器，设置为存储一个或多个程序；
    当所述一个或多个程序被所述一个或多个处理器执行，使得所述一个或多个处理器实现如权利要求1-11中任一所述的图像校正方法。
  14. 一种计算机可读存储介质，所述计算机可读存储介质上存储有计算机程序，所述计算机程序被处理器执行时实现如权利要求1-11中任一所述的图像校正方法。
PCT/CN2023/089112 2022-05-13 2023-04-19 图像校正方法、装置、电子设备及存储介质 Ceased WO2023216822A1 (zh)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US18/865,668 US20250317654A1 (en) 2022-05-13 2023-04-19 Image correction method and apparatus, electronic device, and storage medium

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202210523446.7A CN115002345B (zh) 2022-05-13 2022-05-13 一种图像校正方法、装置、电子设备及存储介质
CN202210523446.7 2022-05-13

Publications (1)

Publication Number Publication Date
WO2023216822A1 true WO2023216822A1 (zh) 2023-11-16

Family

ID=83027857

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2023/089112 Ceased WO2023216822A1 (zh) 2022-05-13 2023-04-19 图像校正方法、装置、电子设备及存储介质

Country Status (3)

Country Link
US (1) US20250317654A1 (zh)
CN (1) CN115002345B (zh)
WO (1) WO2023216822A1 (zh)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115002345B (zh) * 2022-05-13 2024-02-13 北京字节跳动网络技术有限公司 一种图像校正方法、装置、电子设备及存储介质

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR101915729B1 (ko) * 2017-06-20 2018-11-06 주식회사 아이닉스 360도 전방향 뷰 영상 생성 장치 및 방법
CN112312111A (zh) * 2020-10-30 2021-02-02 北京字节跳动网络技术有限公司 虚拟图像的显示方法、装置、电子设备及存储介质
CN112581369A (zh) * 2020-12-24 2021-03-30 中国银联股份有限公司 图像拼接方法以及装置
CN112689135A (zh) * 2021-03-19 2021-04-20 深圳市火乐科技发展有限公司 投影校正方法、装置、存储介质及电子设备
CN115002345A (zh) * 2022-05-13 2022-09-02 北京字节跳动网络技术有限公司 一种图像校正方法、装置、电子设备及存储介质

Family Cites Families (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110785789A (zh) * 2017-06-27 2020-02-11 索尼公司 图像处理装置、图像处理方法和程序
WO2021106027A1 (ja) * 2019-11-25 2021-06-03 日本電信電話株式会社 カメラパラメータ導出装置、カメラパラメータ導出方法及びカメラパラメータ導出プログラム
CN113129241B (zh) * 2019-12-31 2023-02-07 RealMe重庆移动通信有限公司 图像处理方法及装置、计算机可读介质、电子设备
CN112150638B (zh) * 2020-09-14 2024-01-26 北京百度网讯科技有限公司 虚拟对象形象合成方法、装置、电子设备和存储介质
CN114092668B (zh) * 2021-10-29 2025-10-28 深圳市慧鲤科技有限公司 虚实融合方法、装置、设备及存储介质
CN114187366B (zh) * 2021-12-10 2025-04-29 北京有竹居网络技术有限公司 一种相机安装校正方法、装置、电子设备及存储介质
CN114332224A (zh) * 2021-12-29 2022-04-12 北京字节跳动网络技术有限公司 3d目标检测样本的生成方法、装置、设备及存储介质
CN114095662B (zh) * 2022-01-20 2022-07-05 荣耀终端有限公司 拍摄指引方法及电子设备

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR101915729B1 (ko) * 2017-06-20 2018-11-06 주식회사 아이닉스 360도 전방향 뷰 영상 생성 장치 및 방법
CN112312111A (zh) * 2020-10-30 2021-02-02 北京字节跳动网络技术有限公司 虚拟图像的显示方法、装置、电子设备及存储介质
CN112581369A (zh) * 2020-12-24 2021-03-30 中国银联股份有限公司 图像拼接方法以及装置
CN112689135A (zh) * 2021-03-19 2021-04-20 深圳市火乐科技发展有限公司 投影校正方法、装置、存储介质及电子设备
CN115002345A (zh) * 2022-05-13 2022-09-02 北京字节跳动网络技术有限公司 一种图像校正方法、装置、电子设备及存储介质

Also Published As

Publication number Publication date
CN115002345A (zh) 2022-09-02
CN115002345B (zh) 2024-02-13
US20250317654A1 (en) 2025-10-09

Similar Documents

Publication Publication Date Title
JP7223449B2 (ja) 撮影に基づく３ｄモデリングシステム
CN114419300B (zh) 风格化图像生成方法、装置、电子设备及存储介质
CN108462838B (zh) 一种全景视频防抖方法、装置及便携式终端
WO2021036353A1 (zh) 基于拍照的3d建模系统及方法、自动3d建模装置及方法
US20240291938A1 (en) Video fusion method, apparatus, electronic device and storage medium
CN115690382B (zh) 深度学习模型的训练方法、生成全景图的方法和装置
CN114869528B (zh) 扫描数据处理方法、装置、设备及介质
CN113873264A (zh) 显示图像的方法、装置、电子设备及存储介质
US20240314289A1 (en) Method and device for processing three-dimensional video, and storage medium
WO2024016828A2 (zh) 一种基于虚拟相机的图像采集方法和相关装置
WO2023226628A1 (zh) 图像展示方法、装置、电子设备及存储介质
WO2018101652A1 (ko) 실감형 미디어 영상을 제공하는 장치
US20240037856A1 (en) Walkthrough view generation method, apparatus and device, and storage medium
CN114187366A (zh) 一种相机安装校正方法、装置、电子设备及存储介质
CN113643346B (zh) 场景重建方法和扫描设备
WO2023216822A1 (zh) 图像校正方法、装置、电子设备及存储介质
CN109961395B (zh) 深度图像的生成及显示方法、装置、系统、可读介质
WO2023088104A1 (zh) 视频的处理方法、装置、电子设备和存储介质
CN113038262B (zh) 一种全景直播方法和装置
CN116193158A (zh) 子弹时间视频生成方法、装置、电子设备及存储介质
CN115022613B (zh) 一种视频重建方法、装置、电子设备及存储介质
CN110602480A (zh) 一种采用增强现实分享场景的方法、装置及系统
KR102101771B1 (ko) 컨텐츠를 제공하는 장치 및 방법
WO2024217237A1 (zh) 数据生成方法、装置、电子设备、计算机可读介质
CN120689489A (zh) 三维场景生成方法、装置和电子设备

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 23802597

Country of ref document: EP

Kind code of ref document: A1

WWE Wipo information: entry into national phase

Ref document number: 18865668

Country of ref document: US

NENP Non-entry into the national phase

Ref country code: DE

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 21.03.2025)

WWP Wipo information: published in national office

Ref document number: 18865668

Country of ref document: US

122 Ep: pct application non-entry in european phase

Ref document number: 23802597

Country of ref document: EP

Kind code of ref document: A1