WO2020082722A1 - 基于混合相机的视频成像方法、系统、设备及存储介质 - Google Patents

基于混合相机的视频成像方法、系统、设备及存储介质 Download PDF

Info

Publication number
WO2020082722A1
WO2020082722A1 PCT/CN2019/085966 CN2019085966W WO2020082722A1 WO 2020082722 A1 WO2020082722 A1 WO 2020082722A1 CN 2019085966 W CN2019085966 W CN 2019085966W WO 2020082722 A1 WO2020082722 A1 WO 2020082722A1
Authority
WO
WIPO (PCT)
Prior art keywords
resolution
image
low
camera
images
Prior art date
Application number
PCT/CN2019/085966
Other languages
English (en)
French (fr)
Inventor
方璐
戴琼海
朱天奕
Original Assignee
清华-伯克利深圳学院筹备办公室
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 清华-伯克利深圳学院筹备办公室 filed Critical 清华-伯克利深圳学院筹备办公室
Publication of WO2020082722A1 publication Critical patent/WO2020082722A1/zh

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/95—Computational photography systems, e.g. light-field imaging systems
    • H04N23/951—Computational photography systems, e.g. light-field imaging systems by using two or more images to influence resolution, frame rate or aspect ratio
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/698—Control of cameras or camera modules for achieving an enlarged field of view, e.g. panoramic image capture

Definitions

  • Embodiments of the present disclosure relate to the field of image recognition technology, for example, to a hybrid camera-based video imaging method, system, device, and storage medium.
  • the common practice is to increase the image resolution by increasing the number of image sensors.
  • the information dimension of the resulting image is red, green and blue similar to the human eye (Red, Green , Blue, RGB) (three primary colors that can be captured by the human eye, the same below) three-dimensional information, the higher dimensional information at the spectral level of the light rays captured cannot be captured by the current system.
  • Embodiments of the present disclosure provide a video imaging method, system, device, and storage medium based on a hybrid camera to achieve the technical effect of improving the resolution of a video image.
  • the present disclosure provides a hybrid camera-based video imaging method, which includes:
  • the present disclosure provides a video imaging system based on a hybrid camera array, the system includes:
  • An image acquisition module configured to acquire at least three low-resolution images captured by a wide-field camera, and acquire at least one high-resolution image captured by a narrow-field camera;
  • An image stitching processing module which is configured to obtain a low-resolution ring image by stitching the at least three low-resolution images
  • An image fusion processing module configured to fuse the low-resolution ring image with the at least one high-resolution image, and determine the high-resolution ring image
  • the video image processing module is configured to obtain a high-resolution video image by encoding and integrating multiple high-resolution ring images.
  • the present disclosure also provides a device, the device includes:
  • One or more processors are One or more processors;
  • Storage device configured to store one or more programs
  • the one or more programs are executed by the one or more processors, so that the one or more processors implement the method described in any embodiment of the present disclosure.
  • the present disclosure also provides a storage medium containing computer-executable instructions, which when executed by a computer processor are used to perform the method described in any embodiment of the present disclosure.
  • FIG. 1 is a video imaging method based on a hybrid camera provided by Embodiment 1 of the present disclosure
  • Embodiment 2 is a placement position of the hybrid camera provided by Embodiment 1 of the present disclosure
  • Embodiment 3 is a video imaging method based on a hybrid camera provided by Embodiment 2 of the present disclosure
  • Embodiment 4 is a video imaging system based on a hybrid camera array provided by Embodiment 3 of the present disclosure
  • FIG. 5 is a schematic structural diagram of a device according to Embodiment 4 of the present disclosure.
  • FIG. 1 is a schematic flowchart of a hybrid camera-based video imaging method according to Embodiment 1 of the present disclosure. The method may be performed by a hybrid camera array-based video imaging system, which may be implemented in the form of software and / or hardware .
  • the method of this embodiment includes:
  • a wide-field camera can be understood as a camera with a short-focus lens.
  • the lens aperture of the camera is large and the image quality of the captured image is good.
  • the camera does not have a clear image. That is to say, wide-field cameras can take close-up images.
  • the resolution of the captured images is relatively low but the captured images have a wide field of view.
  • a narrow field of view camera can be understood as a telephoto lens camera.
  • the lens aperture of the camera is small, and it can take pictures of distant objects.
  • the effect of the image captured is not as good as that of a wide field camera.
  • the wide field of view camera and the narrow field of view camera proposed in the technical solutions of the embodiments of the present disclosure are relative, so the corresponding low-resolution images and high-resolution images are also relative.
  • the number of at least three low-resolution images may be three or more, which may be determined according to the number of wide-field cameras, because at least three low-resolution images are respectively captured by at least three wide-field cameras.
  • the number of wide-field cameras is three
  • at least three low-resolution images may be low-resolution images captured by three wide-field cameras at the same time.
  • the low-resolution images of multiple wide-field cameras taken at the same time can be combined to cover a 360-degree scene and each phase The area shot between two adjacent wide-field cameras overlaps. The advantage of this setting is that one of the scenes will not be missed when processing low-resolution images.
  • At least three low-resolution images were captured by at least three wide-field cameras, and the area captured between each adjacent two wide-field cameras overlapped, and each wide-field camera captured at the same time
  • the low-resolution images are combined to cover a set angle, for example, a 360-degree circular image.
  • the number of the at least three wide-field cameras can be three or more, and the user can set according to the actual situation.
  • the at least three wide-field cameras can be placed in the environment where the user wants to take video images, that is to say
  • the setting of the wide field of view camera can be selected according to the actual situation. Under certain conditions, it has great freedom, that is, the setting of the camera is unstructured.
  • a disc base that is, a circular table 201 is set in the environment where the video image is to be captured, and at least three wide-field cameras, for example, five wide-field cameras 202 are placed on the circular table 201 on.
  • the round table 201 may use materials that are not easily vibrated and deformed, and at the same time, in order to ensure the distance between each adjacent two wide-field cameras 202, the base is not easy to be too large.
  • the shooting angles of the lenses of the wide-field camera 202 are not the same.
  • the shooting angle of the camera lens may be 60 degrees, 90 degrees, or 120 degrees, etc. When the shooting angle of the camera lens is higher or lower, it may cause image distortion. In order to avoid distortion of the image, the lens shooting degree of the wide-field camera 202 may be 90 degrees.
  • the wide field of view camera has a lens shooting degree of 90 degrees, in order to ensure that the low resolution images taken by multiple wide field of view cameras 202 at the same time can be combined to cover a 360 degree panoramic image, and the two adjacent wide fields of view
  • the images taken by the camera 202 have overlapping areas, and the number of wide-field cameras 202 may be at least five.
  • Each wide-field camera 202 can take the center of the circular table 201 as the center of the circle, and arrange the lens to extend the radius to capture a low-resolution image.
  • each wide-field camera has the same parameters and the same model, that is to say, each wide-field camera is the same.
  • the gimbal 203 is also provided on the upper surface of the round table 201, and the gimbal 203 can be placed at the center of the round table 201, that is, the center of the circle on the upper surface of the round table 201, and the gimbal 203 itself has a certain height. And the gimbal 203 can be rotated, and the height can be adjusted in the vertical direction.
  • the narrow field of view camera 204 may be disposed on the gimbal 203, that is, the narrow field of view camera 204 is placed at the center position of the round table 201, and the height in the vertical direction is higher than that of the wide field of view camera 202.
  • the gimbal 203 itself has a certain height, that is to say, the narrow-field camera 204 installed on the gimbal 203 and the The wide-field camera 202 on the round table 201 has a certain height difference in the vertical direction.
  • the vertical distance difference between the wide-field camera 202 and the narrow-field camera 204 cannot be too large. The advantage of this setting is that it is narrow.
  • the field-of-view camera 204 can take a high-resolution image in a 360-degree range during the rotation of the captured image, and the horizontal distance to the wide-field camera 202 as close as possible can reduce the wide-field camera 202 and the narrow-field camera 204 Negative effects caused by poor viewing angle.
  • the low-resolution ring image may be an image obtained by stitching the low-resolution images taken by multiple wide-field cameras at the same moment. Since the images taken by two adjacent cameras in a wide field of view camera have a certain overlapping area, and the images taken by multiple wide field of view cameras are stitched together to obtain a 360-degree panoramic image, the low-resolution ring image covers 360 Circular panoramic image.
  • At least one low-resolution image captured by at least three wide-field cameras can be stitched together.
  • at least three low-resolution images can be resolved according to a pre-stored camera matrix
  • the high-resolution image is stitched, that is, the pixels of the coincidence area are identified according to the pre-calculated at least three low-resolution images, and the placement position of the wide-field camera is obtained to obtain a low-resolution ring image.
  • At least three low-resolution images captured by at least three wide-field cameras are acquired; a ring stitching algorithm is used to stitch at least three low-resolution images, and each adjacent two are identified from the stitched images
  • the shooting range of each wide-field camera; the camera matrix is determined according to the camera position and shooting range of each wide-field camera, and the internal parameter matrix and distortion parameter matrix of the camera are stored, wherein the internal parameter matrix and distortion parameter of the camera
  • the matrix is used to stitch at least three low-resolution images to obtain a low-resolution ring image.
  • the camera matrix can be understood as a camera parameter matrix.
  • the camera parameters include internal parameters, external parameters, and distortion parameters.
  • At least one wide-field camera may be placed first to ensure that the images taken by the at least one wide-field camera can be combined to cover the device.
  • At least one wide-field camera takes at least one low-resolution preview image for processing, and when there are multiple wide-field cameras, determine each two adjacent wide-field cameras in the multiple wide-field cameras The coincidence area between the low-resolution images taken at the same time and the calculated placement position of each camera are stored in the computer.
  • a pre-stored camera matrix can be used for stitching to obtain a low-resolution ring image.
  • a ring stitching algorithm is used to stitch at least three low-resolution images to determine the stitching process.
  • the camera matrix that is to determine the coincident pixels when stitching at least three low-resolution images and the placement position of each wide-field camera, the user can set according to the actual situation in the actual application process.
  • the comparison between the low-resolution ring image and each high-resolution image may be performed first, Determine the similar blocks of each high-resolution image in the low-resolution ring image, that is to say, the ring low-resolution image can be cut into at least one area, and determine in which area each high-resolution image is divided. If similar blocks are determined, high-resolution recovery and matching algorithms can be used to fuse at least one high-resolution image into a low-resolution ring image to obtain a high-resolution ring image.
  • the program code that can be stored in the computer is used to determine whether the current frame image to be generated is the first frame image. If the frame image is the first frame image in the video image, the low-resolution ring image and at least one high-resolution image are fused into a high-resolution ring image; if the current frame image to be generated is not the first in the video image
  • One frame image, the low-resolution ring image, at least one high-resolution image, and at least two frame images before the current frame image are combined to determine a high-resolution ring image; wherein, one of at least two frame images
  • the frame image is continuous with the current frame image.
  • the current frame image to be generated may be understood as a current high-resolution ring image obtained by fusing a low-resolution ring image with at least one high-resolution image. If the current frame image to be generated is the first frame image, the similarity area of each high-resolution image on the ring low-resolution image can be determined according to the comparison between each high-resolution image and the ring low-resolution image Block, fuse each high-resolution image into a ring-shaped low-resolution image to obtain a high-resolution ring-shaped image; if the current frame image to be generated is not the first frame image, you can use the low-resolution ring-shaped image, at least one A high-resolution image and at least two consecutive frame images before the current frame image are combined to determine a high-resolution ring image.
  • the at least two consecutive frame images before the current frame image can be understood as a frame image that is before the current frame image and continuous with the current frame image. At least two frames of images can be understood as two frames of images, three frames of images, etc.
  • the user can set the program code in the computer according to the specific requirements when processing the images. You can determine the similar block of the high-resolution image on the low-resolution image based on the currently formed low-resolution ring image and at least two consecutive frames before the current frame image to be generated.
  • the advantage of this setting is to avoid When the images taken by the wide-field camera and the narrow-field camera are particularly similar, the problem of inaccuracy when determining the position in the low-resolution ring image directly from the high-resolution image.
  • the processing method may be: for each high-resolution image, determine the high-resolution image in the low-resolution ring image At least two candidate fusion positions; perform position recognition on at least two frame images of high-resolution images, and screen at least two candidate fusion positions according to the recognition result to determine the target fusion position; according to the target fusion position, at least one A high-resolution image and a low-resolution ring image are fused to determine a high-resolution ring image.
  • At least two candidate fusion positions of the high-resolution image in the low-resolution ring image are determined, that is, at least two fusion positions are determined on the low-resolution ring image to fuse the high-resolution image; in order to determine the fusion high
  • the first three frames of images can be acquired and processed to determine the fusion position of the high-resolution image until the target fusion position of the high-resolution image in the circular low-resolution image is selected.
  • each high-resolution image is fused to a similar block of the low-resolution ring image, that is, at the target fusion position, a high-resolution ring image is obtained.
  • the resulting high-resolution ring image has a higher resolution and a wider spectrum of dimensions.
  • a scene in the high-resolution ring image is a leaf and the color is green
  • the image captured by the camera in the related art may be close to green, but with the technical solution of the embodiment of the present disclosure, the image captured by the wide field of view camera and the narrow field of view camera can be processed together to obtain a high resolution
  • the ring image achieves a resolution of one billion pixels, that is, the resulting image has a wider spectral dimension and higher resolution, which is closer to the true color of the image.
  • S140 Encoding and integrating multiple high-resolution ring images to obtain a high-resolution video image.
  • Encoding and integrating the obtained high-resolution ring image that is, processing the obtained high-resolution ring image using a video encoding algorithm in the related art to obtain a high-resolution video image.
  • Encoding and integrating images with a resolution of one billion pixels can produce video images with a resolution of one billion pixels.
  • the technical solution of the embodiments of the present disclosure acquires at least three low-resolution images captured by a wide-field camera, and acquires high-resolution images captured by a narrow-field camera, and performs stitching processing on at least three low-resolution images, Obtain a low-resolution ring image, combine the low-resolution ring image with the high-resolution image to determine it as a high-resolution ring image, and obtain a high-resolution video image by encoding and integrating multiple high-resolution ring images , It solves that the images taken by the high-resolution imaging system or the dual-camera high-resolution imaging system in the related art are similar to the three primary colors captured by the human eye, and higher-dimensional image information cannot be obtained, that is, the image
  • the problem of low resolution is achieved by combining a high-resolution image and a low-resolution image captured by a narrow-field camera and a wide-field camera and using a series of algorithms to process one billion pixels
  • the video image with high resolution achieve
  • the method further includes: determining the current frame image according to the high-resolution ring image; Use the weight algorithm to process the low-resolution ring image and the current frame image, determine the interest weight of the next high-resolution image, and generate the first control signal according to the interest weight; control the next high-resolution image according to the first control signal
  • the shooting height and shooting angle of the resolution image wherein, the camera that shoots the high-resolution image is set on the gimbal; the shooting height of the camera on the gimbal and the angle between the camera and the horizontal plane are controlled according to the first control information.
  • the shooting height of the camera on the gimbal can be understood as the angle between the shooting azimuth of the camera and the horizontal plane.
  • the current frame image corresponding to the high-resolution ring image can be obtained by processing the high-resolution ring image.
  • the high-resolution ring image is processed using a video encoding algorithm.
  • the next frame image needs to be determined, that is, the low-resolution image and the high-resolution image need to be acquired.
  • the position of the wide-field camera shooting the low-resolution image does not change during the process of capturing the image, and the position and shooting angle of the high-resolution image captured by the narrow-field camera are determined according to the weight of interest in the entire image of.
  • interest weights include contribution weights and cost weights.
  • the interest weight reflects the demand for at least one area of the low-resolution ring image for a high-resolution image, which can be understood as which area in the low-resolution image is more clear.
  • the cost weight represents the cost that the narrow field of view camera consumes when turning from the current position to the position where the next high-resolution image is taken when taking a high-resolution image.
  • the optimal position for taking the next high-resolution image can be determined according to the weight of interest.
  • the optimal position for the narrow-field camera taking the next high-resolution image can be:
  • ⁇ gain represents the weight of contribution weights in interest weights
  • ⁇ cost represents the weight of cost weights in interest weights
  • F gain ( ⁇ pos ) represents the contribution weight
  • F cost ( ⁇ pos ) represents the cost weight.
  • the size of the contribution weight F gain ( ⁇ pos ) depends on the single-image super-resolution algorithm's demand for super-resolution images, the hyperspectral image deep learning recognition algorithm's demand for low-resolution ring images of interest areas, and forward frames The requirements of the image for high-resolution ring images are comprehensively determined; the size of the cost weight F cost ( ⁇ pos ) is caused by the narrow-field camera's azimuth movement loss and high-frequency neighborhood when shooting different high-resolution images It is determined by the loss of information and the distance traveled by the current shooting azimuth from the next shooting.
  • the interest weight of the next high-resolution image can be determined by processing the low-resolution ring image, the current frame image, and the frame image before the current frame.
  • the pre-stored program code or pre-stored program in the computer can be based on the shooting
  • the interest weight of a high-resolution image generates a first control signal.
  • the first control signal may be a signal for adjusting the shooting height and shooting angle of the next high-resolution image.
  • the narrow field of view camera can be set on the gimbal.
  • the height of the camera on the gimbal and the angle between the camera and the horizontal plane can be adjusted.
  • the gimbal can communicate with the computer wirelessly or electrically, and can receive the first control signal sent by the computer, and adjust the shooting height of the camera on the gimbal and the angle between the camera and the horizontal plane according to the first control signal.
  • the pan-tilt head is set on the narrow-field camera, when the pan-tilt adjusts the angle between the height of the narrow-field camera and the horizontal plane according to the first control signal, the corresponding narrow-field camera on the pan-tilt can be adjusted according to The rotation of the gimbal and the height adjustment of the camera adjust the height and angle of the next high-resolution image.
  • FIG. 3 is a video imaging method based on a hybrid camera provided in Embodiment 2 of the present disclosure. The method includes:
  • the at least three wide-field cameras may be three or more, for example, five wide-field cameras, and the lens shooting degree of each camera is 90 degrees.
  • a low-resolution preview image can be captured by the five wide-field cameras.
  • the wide-field camera before taking a low-resolution image based on a wide-field camera and processing it, the wide-field camera may be adjusted and set.
  • the wide-field camera ’s The parameter may be the performance parameter of adjusting the wide-field camera, and the adjusting position may be that multiple wide-field cameras are placed on the same horizontal plane. In an embodiment, see FIG.
  • the images taken by the field of view camera can be combined to cover 360 degrees. At the same time, there are overlapping areas between the low-resolution images taken by each adjacent two wide field of view cameras.
  • the advantage is that the wide field of view camera is adjusted in advance to Ensure that the resulting video images meet the needs of users.
  • the low-resolution image can be processed using a ring stitching algorithm.
  • the pixels within the overlapping range of the images taken by two adjacent wide-field cameras can be determined from the stitched image, and then the camera matrix can be determined according to the pixels within the overlapping range and the position of each wide-field camera.
  • the camera matrix can be stored in the calculation so that it can be directly recalled when processing low-resolution images.
  • the low-resolution image When the low-resolution image is captured by the wide-field camera, the low-resolution image can be processed by using a pre-stored camera matrix to obtain a low-resolution ring image.
  • the computer can obtain the low-resolution image and call the pre-stored camera matrix to stitch the low-resolution image to obtain the stitched low-resolution ring image. Since the images taken by multiple wide-field cameras at the same time are combined to cover a 360-degree circular panoramic image, the stitched low-resolution ring image can also cover a full 360-degree image.
  • the narrow field of view camera can be fixed on the gimbal.
  • the gimbal can rotate and adjust the angle between the camera and the horizontal plane. Therefore, when the gimbal rotates and adjusts between the camera and the horizontal plane The angle of the camera can change the position of the narrow field of view camera.
  • the narrow-field camera also captures high-resolution images. If the high-resolution image captured by the narrow-field camera is the first high-resolution image, the computer can obtain high-resolution images. High-resolution image; if the high-resolution image captured by the narrow-field camera is not the first high-resolution image, the height of the high-resolution image captured and the angle between it and the horizontal plane can be combined before the current frame image to be generated
  • the feedback data of the frame image and the low-resolution ring image determine the weight of interest in taking a high-resolution image, that is, the height, elevation, or depression angle of the narrow-field camera when taking the low-resolution image.
  • the determination of the weight of interest in taking a high-resolution image can be processed by a computer. After determining the interest weights, a first control signal corresponding to the interest weights will be generated, and the first control signal will be sent to the gimbal. The gimbal receives the first control signal, adjusts the shooting height of the camera and the camera and the horizontal plane The angle between them makes the narrow field of view camera take high-resolution images.
  • the current frame image to be generated can be understood as an image after the fusion of the high-resolution image and the low-resolution ring image. Determine whether the current frame image to be generated is the first frame image, if it is the first frame image, you can directly merge the high-resolution image and the low-resolution ring image, that is, execute S306; if it is not the first frame image, then combine The frame image before the current frame image and the high-resolution image use the low-resolution ring image to determine the target fusion position of the high-resolution image on the ring low-resolution image, and fuse them S307.
  • the similar blocks of the high-resolution image on the low-resolution ring image can be determined according to the comparable comparison between the high-resolution image and the low-resolution ring image, that is, the high-resolution image fusion To the target fusion position on the low-resolution ring image. Then, high-resolution restoration and matching algorithms are used to fuse high-resolution images into low-resolution ring images to obtain high-resolution ring images.
  • a high-resolution restoration and matching algorithm is used to fuse the frame image, the high-resolution image, and the low-resolution ring image before the generated current frame image to obtain a high-resolution ring image.
  • At least two candidate fusion positions can be determined according to the similarity, ie similarity, between the low-resolution ring image and the high-resolution image; Generate at least two frames before the current frame image, and position the high-resolution image in at least two frames for image recognition. If the target fusion position is determined, the high-resolution can be adjusted according to the high-resolution recovery and matching algorithm The image is fused into a low-resolution ring image; if the target fusion position is still not determined, the fusion position of the target image can be determined according to the first three frames of the current frame image to be generated until the high-resolution image is determined at the low resolution The target fusion position in the ring image.
  • the high-resolution recovery and matching algorithm can be used to obtain the high-resolution ring image.
  • the high-resolution restoration and matching algorithm is a CC-Net + SS-Net mode image fusion restoration algorithm based on deep convolutional neural network proposed by Haitian Zheng et al. In 2017.
  • the obtained high-resolution ring image may be cached and compressed, and the video encoding algorithm may be used to process the high-resolution ring image to obtain a high-resolution video image.
  • the video imaging method based on the hybrid camera the previous frame image or the previous frame images of the current frame image are used, so each frame image can be returned to S304 and S307.
  • the technical solution of the embodiments of the present disclosure acquires at least three low-resolution images captured by a wide-field camera, and acquires high-resolution images captured by a narrow-field camera, and performs stitching processing on at least three low-resolution images, Obtain a low-resolution ring image, combine the low-resolution ring image with the high-resolution image to determine it as a high-resolution ring image, and obtain a high-resolution video image by encoding and integrating multiple high-resolution ring images , It solves that the images taken by the high-resolution imaging system or the dual-camera high-resolution imaging system in the related art are similar to the three primary colors captured by the human eye, and higher-dimensional image information cannot be obtained, that is, the image
  • the problem of low resolution is achieved by combining a high-resolution image and a low-resolution image captured by a narrow-field camera and a wide-field camera and using a series of algorithms to process one billion pixels
  • the video image with high resolution achieve
  • FIG. 4 is a schematic structural diagram of a video imaging system based on a hybrid camera array according to Embodiment 3 of the present disclosure.
  • the system includes: an image acquisition module 410, an image stitching processing module 420, an image fusion processing module 430, and a video image processing module 440.
  • the image acquisition module 410 is set to acquire at least three low-resolution images captured by a wide field of view camera and at least one high resolution image captured by a narrow field of view camera;
  • the image stitching processing module 420 is set to By stitching at least three of the low-resolution images, a low-resolution ring image is obtained;
  • an image fusion processing module 430 is set to fuse the low-resolution ring image with the at least one high-resolution image Is determined to be a high-resolution ring image;
  • the video image processing module 440 is set to obtain a high-resolution video image by encoding and integrating multiple high-resolution ring images.
  • the low-resolution ring image is a ring-shaped panoramic image covering 360 degrees.
  • the at least three low-resolution images are respectively captured by at least three wide-field cameras, and the areas captured between each two adjacent wide-field cameras overlap, and at least three wide-field images
  • the low-resolution images taken by the field-of-view camera at the same time are combined into a circular image covering a set angle.
  • the system further includes: a pre-processing module configured to obtain at least three low-resolution images respectively captured by at least three wide-field cameras; and a ring stitching algorithm module configured to use a pair of ring stitching algorithms Perform stitching on at least two low-resolution images and identify the shooting overlap range of each adjacent two wide-field cameras from the stitched image; the camera matrix determination module is set to match the camera position and shooting coincidence of each wide-field camera The range determines the camera matrix, and stores the internal parameter matrix and distortion parameter matrix of the camera, wherein the internal parameter matrix and distortion parameter matrix of the camera are used to stitch the at least three low-resolution images to obtain a low resolution Ring image.
  • a pre-processing module configured to obtain at least three low-resolution images respectively captured by at least three wide-field cameras
  • a ring stitching algorithm module configured to use a pair of ring stitching algorithms Perform stitching on at least two low-resolution images and identify the shooting overlap range of each adjacent two wide-field cameras from the stitched image
  • the image fusion processing module is further configured to, if the current frame image to be generated is the first frame image in the video image, combine the low-resolution ring image with the at least one high-resolution image
  • the resolution image fusion process is a high-resolution ring image, and the high-resolution ring image is used as the current frame image to be generated; if the current frame image to be generated is not the first frame image in the video image, then The low-resolution ring image, the at least one high-resolution image, and at least two consecutive frame images before the current frame image are combined to determine a high-resolution ring image, and the high-resolution ring image is used as the The current frame image to be generated; wherein one frame image of the at least two frame images is continuous with the current frame image.
  • the image fusion processing module is further configured to: determine at least two candidate fusion positions of each high-resolution image in the low-resolution ring image;
  • the high-resolution image performs position recognition in the at least two frame images, and screens the at least two candidate fusion positions according to the recognition result to determine the target fusion position; according to the target fusion position, the at least one A high-resolution image is merged with the low-resolution ring image to determine a high-resolution ring image.
  • the system further includes: An interest weight determination module; the interest weight determination module includes a current frame image determination unit, an interest weight determination unit, and an adjustment unit; the current frame image determination unit is set to determine the current frame image based on the high-resolution ring image; interest The weight determination unit is configured to process the low-resolution ring image and the current frame image using a weight algorithm to determine the interest weight for shooting the next high-resolution image, and generate a first according to the interest weight A control signal; an adjustment unit configured to control the shooting height and shooting angle of the next high-resolution image according to the first control signal; wherein, the camera that takes the high-resolution image is set on the pan / tilt; according to the The first control signal controls the shooting height of the camera on the gimbal and the angle between the camera and the horizontal plane.
  • the image fusion processing module is further configured to determine each high-resolution image based on the comparability between the low-resolution ring image and each high-resolution image A similar block in the low-resolution ring image; using a high-resolution restoration and matching algorithm, the at least one high-resolution image is fused into a similar block in the low-resolution ring image to obtain a high-resolution Resolution ring image.
  • the technical solution of the embodiments of the present disclosure acquires at least three low-resolution images captured by a wide-field camera, and acquires high-resolution images captured by a narrow-field camera, and performs stitching processing on at least three low-resolution images, Obtain a low-resolution ring image, combine the low-resolution ring image with the high-resolution image to determine it as a high-resolution ring image, and obtain a high-resolution video image by encoding and integrating multiple high-resolution ring images , It solves that the images taken by the high-resolution imaging system or the dual-camera high-resolution imaging system in the related art are similar to the three primary colors captured by the human eye, and cannot obtain higher-dimensional image information, that is, images
  • the problem of low resolution is achieved by combining a high-resolution image and a low-resolution image captured by a narrow-field camera and a wide-field camera and using a series of algorithms to process one billion pixels
  • the video image with high resolution achieves the
  • FIG. 5 is a schematic structural diagram of a device according to Embodiment 4 of the present disclosure.
  • FIG. 5 shows a block diagram of an exemplary device 50 suitable for implementing embodiments of the present disclosure.
  • the device 50 shown in FIG. 5 is only an example, and should not bring any limitation to the functions and use scope of the embodiments of the present disclosure.
  • the device 50 is represented in the form of a general-purpose computing device.
  • the components of the device 50 may include, but are not limited to, one or more processors or processing units 501, a system memory 502, and a bus 503 connecting different system components (including the system memory 502 and the processing unit 501).
  • the device 50 is provided with at least three wide field of view cameras and narrow field of view cameras, and is provided on a round table, such as the structure shown in FIG. 2.
  • the bus 503 represents one or more of several types of bus structures, including a memory bus or a memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any of a variety of bus structures.
  • these architectures include, but are not limited to, Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MCA) bus, Enhanced ISA bus, Video Electronics Standards Association (Video Electronics Standards Association) , VESA) local area bus and peripheral component interconnect (Peripheral Component Interconnect, PCI) bus.
  • Device 50 includes a variety of computer system readable media. These media may be any available media that can be accessed by the device 50, including volatile and nonvolatile media, removable and non-removable media.
  • the system memory 502 may include computer system readable media in the form of volatile memory, such as random access memory (Random Access Memory, RAM) 504 and / or cache memory 505.
  • the device 50 may further include other removable / non-removable, volatile / nonvolatile computer system storage media.
  • the storage system 506 may be configured to read and write non-removable, non-volatile magnetic media (not shown in FIG. 5 and is generally referred to as a "hard disk drive").
  • each drive may be connected to the bus 503 through one or more data media interfaces.
  • the memory 502 may include at least one program product having a set (eg, at least one) of program modules configured to perform the functions of one or more embodiments of the present disclosure.
  • a program / utility tool 508 having a set of (at least one) program modules 507 may be stored in, for example, the memory 502.
  • Such program modules 507 include but are not limited to an operating system, one or more application programs, other program modules, and program data Each of these examples or some combination may include the implementation of a network environment.
  • the program module 507 generally performs the functions and / or methods in the embodiments described in the present disclosure.
  • the device 50 may also communicate with one or more external devices 509 (eg, keyboard, pointing device, display 510, etc.), and may also communicate with one or more devices that enable a user to interact with the device 50, and / or
  • the device 50 can communicate with any device (eg, network card, modem, etc.) that communicates with one or more other computing devices. Such communication may be performed through an input / output (Input / Output, I / O) interface 511.
  • the device 50 can also communicate with one or more networks (such as a local area network (Local Area Network, LAN), wide area network (Wide Area Network, WAN), and / or a public network, such as the Internet) through the network adapter 512.
  • networks such as a local area network (Local Area Network, LAN), wide area network (Wide Area Network, WAN), and / or a public network, such as the Internet
  • the network adapter 512 communicates with other modules of the device 50 via the bus 503. It should be understood that although not shown in FIG. 5, other hardware and / or software modules may be used in conjunction with the device 50, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, disk arrays (Redundant Arrays) of Independent Drives (RAID) systems, tape drives and data backup storage systems, etc.
  • the processing unit 501 executes one or more functional applications and data processing by running a program stored in the system memory 502, for example, to implement a hybrid camera-based video imaging method provided by an embodiment of the present disclosure.
  • Embodiment 5 of the present disclosure also provides a storage medium containing computer-executable instructions, which when executed by a computer processor are used to perform a hybrid camera-based video imaging method.
  • the computer storage media of the embodiments of the present disclosure may adopt any combination of one or more computer-readable media.
  • the computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium.
  • the computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any combination of the above.
  • Computer-readable storage media include (non-exhaustive list): electrical connection with one or more wires, portable computer disk, hard disk, RAM, read-only memory (Read-Only Memory, ROM), erasable programmable only Read memory (Erasable Programmable Read Only Memory, EPROM) or flash memory, optical fiber, CD-ROM, optical storage device, magnetic storage device, or any suitable combination of the above.
  • the computer-readable storage medium may be any tangible medium that contains or stores a program, and the program may be used by or in combination with an instruction execution system, apparatus, or device.
  • the computer-readable signal medium may include a data signal that is propagated in baseband or as part of a carrier wave, in which computer-readable program code is carried. This propagated data signal can take many forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above.
  • the computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, and the computer-readable medium may send, propagate, or transmit a program for use by or in combination with an instruction execution system, apparatus, or device. .
  • the program code contained on the computer-readable medium may be transmitted on any appropriate medium, including but not limited to wireless, wire, optical cable, radio frequency (Radio Frequency, RF), etc., or any suitable combination of the foregoing.
  • the computer program code for performing the operations of the embodiments of the present disclosure may be written in one or more programming languages or a combination thereof, and the programming languages include object-oriented programming languages such as Java, Smalltalk, C ++, and also include Conventional procedural programming language-such as "C" language or similar programming language.
  • the program code may execute entirely on the user's computer, partly on the user's computer, as an independent software package, partly on the user's computer and partly on a remote computer, or entirely on the remote computer or server.
  • the remote computer may be connected to the user's computer through any kind of network, including LAN or WAN, or may be connected to an external computer (eg, using an Internet service provider to connect through the Internet).

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Computing Systems (AREA)
  • Theoretical Computer Science (AREA)
  • Studio Devices (AREA)

Abstract

本文公开了一种基于混合相机的视频成像方法、系统、设备及存储介质，基于混合相机的视频成像方法包括：获取通过宽视场相机拍摄的至少三幅低分辨率图像，以及获取通过窄视场相机拍摄的至少一幅高分辨率图像；通过对所述至少三幅低分辨率图像进行拼接处理，得到低分辨率环形图像；将所述低分辨率环形图像与所述至少一幅高分辨率图像融合处理，确定为高分辨率环形图像；通过对多幅所述高分辨率环形图像进行编码整合处理，得到高分辨率的视频图像。

Description

基于混合相机的视频成像方法、系统、设备及存储介质
本申请要求在2018年10月24日提交中国专利局、申请号为201811241672.6的中国专利申请的优先权，该申请的全部内容通过引用结合在本申请中。
技术领域
本公开实施例涉及图像识别技术领域，例如涉及一种基于混合相机的视频成像反方、系统、设备及存储介质。
背景技术
随着计算摄像学、相机产业及人工智能产业技术的发展，计算机视觉领域在视频采集、视频目标跟踪、实时视频识别上都取得了性能突破和数据规模的重大突破，即相关技术中的视频识别技术的识别准确度可以超过人眼。为了进一步的实现效果更好的现代人工智能算法以及帮助相关领域的研究，同时在高清监控、遥感识别乃至军事应用中达到更为出色的视频识别效果，如何有效提高当前视频在信息空间的维度，则成为了一个势必要解决的关键问题。
为了提高图像的分辨率，常见的做法是通过增加大量的图像传感器数量来实现图像分辨率的提高。目前已经存在基于参考图像的相机阵列实现分辨率提高的系统，并被用于安防、便携高分辨视频采集等领域。然而，无论是视角更大的环形超高分辨成像系统，亦或是便携版的双相机高分辨成像系统，其所成的像的信息维度均为与人眼类似的红绿蓝(Red、Green、Blue，RGB)(为人眼所能捕获的三原色，下同)三维信息，所拍摄到的光线在光谱层面的更高维度的信息无法被当前系统所捕获。
近些年来，虽然超分辨率算法、高光谱相机以及基于高光谱图像的人工智能技术已经逐渐趋于成熟，但是在相关技术中的高分辨率的视频采集系统的基础上，仍然没有解决在像素级别上增加更高维度的信息量，视频图像分辨率低的问题。
发明内容
本公开实施例提供一种基于混合相机的视频成像方法、系统、设备及存储 介质，以实现提高视频图像分辨率的技术效果。
在一实施例中，本公开提供了一种基于混合相机的视频成像方法，该方法包括：
获取通过宽视场相机拍摄的至少三幅低分辨率图像，以及获取通过窄视场相机拍摄的至少一幅高分辨率图像；
通过对所述至少三幅低分辨率图像进行拼接处理，得到低分辨率环形图像；
将所述低分辨率环形图像与所述至少一幅高分辨率图像融合处理，确定为高分辨率环形图像；
通过对多幅所述高分辨率环形图像进行编码整合处理，得到高分辨率的视频图像。
在一实施例中，本公开提供了一种基于混合相机阵列的视频成像系统，该系统包括：
获取图像模块，设置为获取通过宽视场相机拍摄的至少三幅低分辨率图像，以及获取通过窄视场相机拍摄的至少一幅高分辨率图像；
图像拼接处理模块，设置为通过对所述至少三幅低分辨率图像进行拼接处理，得到低分辨率环形图像；
图像融合处理模块，设置为将所述低分辨率环形图像与所述至少一幅高分辨率图像融合处理，确定为高分辨率环形图像；
视频图像处理模块，设置为通过对多幅所述高分辨率环形图像进行编码整合处理，得到高分辨率的视频图像。
在一实施例中，本公开还提供了一种设备，所述设备包括：
一个或多个处理器；
存储装置，设置为存储一个或多个程序，
所述一个或多个程序被所述一个或多个处理器执行，使得所述一个或多个处理器实现本公开任一实施例所述的方法。
在一实施例中，本公开还提供一种包含计算机可执行指令的存储介质，所述计算机可执行指令在由计算机处理器执行时用于执行本公开任一实施例所述 的方法。
附图说明
图1为本公开实施例一所提供的一种基于混合相机的视频成像方法；
图2为本公开实施例一所提供的混合相机的摆放位置；
图3为本公开实施例二所提供的一种基于混合相机的视频成像方法；
图4为本公开实施例三所提供的一种基于混合相机阵列的视频成像系统；
图5为本公开实施例四所提供的一种设备结构示意图。
具体实施方式
下面结合附图和实施例对本公开作进一步的详细说明。可以理解的是，此处所描述的实施例仅仅用于解释本公开，而非对本公开的限定。为了便于描述，附图中仅示出了与本公开相关的部分而非全部结构。
实施例一
图1为本公开实施例一所提供的一种基于混合相机的视频成像方法流程示意图，该方法可以由基于混合相机阵列的视频成像系统来执行，该系统可以通过软件和/或硬件的形式实现。
如图1所述，本实施例的方法包括：
S110、获取通过宽视场相机拍摄的至少三幅低分辨率图像，以及获取通过窄视场相机拍摄的至少一幅高分辨率图像。
其中，宽视场相机可以理解为短焦镜头的相机，该相机的镜头光圈较大、拍摄的图像画质较好，但是当拍摄远处细小的物体时，拍摄的图像不清楚的相机，也就是说宽视场相机可以拍摄近景图像，拍摄的图像分辨率相对来说较低但是拍摄的图像视野较广。窄视场相机可以理解为长焦镜头的相机，相机的镜头光圈较小，可以拍清楚较远的物体，当拍摄近景图像时拍摄的图像效果没有 宽视场相机拍摄的效果好，即能够将远处的某一场景或者物体拍摄清楚的相机。本公开实施例技术方案中提出的宽视场相机和窄视场相机是相对而言的，那么相对应的低分辨率图像与高分辨率图像也是相对而言的。
至少三幅低分辨率图像的数量可以是三张或者多张，可以根据宽视场相机的数量来确定，这是因为至少三幅低分辨率图像由至少三个宽视场相机分别拍摄。示例性的，若宽视场相机的数量为3个，至少三幅低分辨率图像可以是三个宽视场相机在同一时刻拍摄的低分辨率图像。为了确保能够获取到窄视场相机与宽视场相机所处整个环境的视频图像，多个宽视场相机的在同一时刻拍摄的低分辨率图像，组合起来可以覆盖360度场景，以及每相邻两个宽视场相机之间拍摄的区域存在重叠，这样设置的好处在于在对低分辨率图像进行处理时不会漏掉其中的某一个场景。也就是说至少三幅低分辨率图像由至少三个宽视场相机分别拍摄，每相邻两个宽视场相机之间拍摄的区域存在重叠，且每个宽视场相机在同一时间拍摄的低分辨率图像组合在一起覆盖设定角度，例如覆盖360度的环形图像。
其中，至少三个宽视场相机的数量可以是三个或者多个，用户可以根据实际情况进行设置，可以将至少三个宽视场相机放置在用户所要拍摄视频图像的环境中，也就是说宽视场相机的设置，可以根据实际情况选择设置的位置，在满足一定条件的情况下，具有很大的自由性，即相机的设置非结构化。在一实施例中，参见图2，在所要拍摄视频图像的环境中设置一个圆盘底座，即圆台201，将至少三个宽视场相机，例如，五个宽视场相机202放置在圆台201上。为了保证拍摄图像的稳定性，圆台201可以选用不易振动、不易变形的材料，同时为了确保每相邻两个宽视场相机202之间的距离，底座不易过大。在一实施例中，宽视场相机202的镜头拍摄角度是不尽相同的。在一实施例中，相机 镜头的拍摄角度可以是60度、90度或者120度等，当相机镜头拍摄拍摄角度较高或者较低时都有可能引起图像的畸变。为了避免图像的畸变，宽视场相机202的镜头拍摄度数可以是90度。若宽视场相机的镜头拍摄度数为90度，为了保证多个宽视场相机202在同一时刻拍摄的低分辨率图像组合在一起可以覆盖360度全景图像，并且相邻的两个宽视场相机202拍摄的图像存在重合区域，宽视场相机202的数量可以是至少5个。每个宽视场相机202可以以圆台201的中心为圆心，镜头往半径延伸的摆放来拍摄低分辨率图像。
在一实施例中，每个宽视场相机的参数相同、型号相同，也就是说每个宽视场相机是相同的。
继续参见图2，在圆台201的上表面还设置了云台203，且云台203可以放置在圆台201的中心位置处，即圆台201上表面的圆心处，云台203自身具有一定的高度，并且云台203是可以旋转，以及在竖直方向上可以调节高度。窄视场相机204可以设置在云台203上，即窄视场相机204摆放在圆台201的中心位置处，且在竖直方向上的高度高于宽视场相机202。
为了避免在窄视场相机204拍摄高分辨率图像时被宽视场相机202遮挡视线，云台203自身具有一定的高度，也就是说设置在云台203上的窄视场相机204与设置在圆台201上的宽视场相机202在竖直方向上具有一定的高度差，当然宽视场相机202与窄视场相机204竖直方向上的距离差也不能过大，这样设置的好处在于窄视场相机204在转动拍摄图像的过程中可以拍摄360度范围内的高分辨率图像，并且与宽视场相机202水平方向上的距离尽可能近可以降低宽视场相机202和窄视场相机204视角差带来的负面影响。
S120、通过对所述至少三幅低分辨率图像进行拼接处理，得到低分辨率环形图像。
其中，低分辨率环形图像可以是将多个宽视场相机在同时一刻拍摄的低分辨率图像进行拼接处理后得到的一幅图像。由于宽视场相机中每相邻两个相机拍摄的图像具有一定的重合区域，并且多个宽视场相机拍摄的图像拼接在一起可以得到360度全景图像，所以低分辨率环形图像是覆盖360度的环形全景图像。
为了得到低分辨率环形图像，可以对至少三个宽视场相机分别拍摄至少一幅低分辨率图像进行拼接处理，在一实施例中，可以是根据预先存储的相机矩阵对至少三幅低分辨率图像进行拼接处理，也即是根据预先计算得到的至少三幅低分辨率图像进行拼接处理时识别重合区域的像素点，以及宽视场相机的摆放位置，得到低分辨率环形图像。
在一实施例中，获取至少三个宽视场相机分别拍摄的至少三幅低分辨率图像；采用环形拼接算法对至少三幅低分辨图像进行拼接处理，并从拼接图像中识别每相邻两个宽视场相机的拍摄重合范围；根据每个宽视场相机的相机位置和拍摄重合范围确定相机矩阵，并存储所述相机的内参矩阵和畸变参数矩阵，其中，相机的内参矩阵和畸变参数矩阵用于对至少三幅低分辨率图像进行拼接处理，得到低分辨率环形图像。在一实施例中，相机矩阵可以理解为相机参数矩阵，相机参数包括内参、外参以及畸变参数。
在一实施例中，在获取至少三幅低分辨率图像以及高分辨率图像之前，可以先摆放至少一个宽视场相机，以确保至少一个宽视场相机拍摄的图像组合在一起可以覆盖设定角度的环形全景图像。再对至少一个宽视场相机分别拍摄至少一幅低分辨率预览图像进行处理，在存在多个宽视场相机的情况下，确定多个宽视场相机中每相邻两个宽视场相机在同一时刻拍摄的低分辨率图像之间的重合区域以及计算得到每个相机的摆放位置，并将其存储在计算机中。当对至 少一幅低分辨率图像进行拼接处理时可以采用预先存储的相机矩阵进行拼接处理，得到低分辨率环形图像。
当然，在实际应用的过程中，也可以在多个宽视场相机在同一时刻分别拍摄一幅低分辨率图像时，采用环形拼接算法对至少三幅低分辨率图像进行拼接处理确定拼接处理时的相机矩阵，即确定至少三幅低分辨率图像进行拼接时的重合像素点以及每个宽视场相机的摆放位置，用户在实际应用的过程中用户可以根据实际情况进行设置。
S130、将所述低分辨率环形图像与所述至少一幅高分辨率图像融合处理，确定为高分辨率环形图像。
将所述低分辨率图像与至少一幅高分辨率图像融合处理时，为了提高制备视频图像的分辨率，可以先根据低分辨率环形图像与每幅高分辨率图像之间的可比拟性，确定每幅高分辨率图像在低分辨率环形图像中的相似区块，也就是说可以将环形低分辨率图像切割为至少一个区域，确定每幅高分辨率图像在分割的哪一个区域中。若确定了相似区块，可以采用高分辨率恢复与匹配算法，将至少一幅高分辨率图像融合到低分辨率环形图像中，得到高分辨率环形图像。
在对低分辨率图像与高分辨率图像进行融合处理得到高分辨率环形图像时，计算机内可以存储的程序代码用来判断待生成的当前帧图像是否为第一帧图像，若待生成的当前帧图像为视频图像中的第一帧图像，则将低分辨率环形图像与至少一幅高分辨率图像融合处理为高分辨率环形图像；若待生成的当前帧图像不为视频图像中的第一帧图像，则将低分率环形图像、至少一幅高分辨率图像以及当前帧图像之前至少两个帧图像结合在一起，确定高分辨率环形图像；其中，至少两个帧图像中的一个帧图像与当前帧图像相连续。
在一实施例中，待生成的当前帧图像可以理解为低分辨率环形图像与至少 一幅高分辨率图像进行融合后得到的当前高分辨率环形图像。若待生成的当前帧图像为第一帧图像，则可以根据每幅高分辨率图像与环形低分辨率图像之间的可比拟性，确定每幅高分辨图像在环形低分辨图像上的相似区块，将每幅高分辨率图像融合到环形低分辨率图像上，得到高分辨率环形图像；若待生成的当前帧图像不为第一帧图像，则可以根据低分辨率环形图像、至少一幅高分辨率图像以及当前帧图像之前连续的至少两个帧图像结合在一起确定高分辨率环形图像。
其中，当前帧图像之前连续的至少两个帧图像可以理解为，在当前帧图像之前并与当前帧图像相连续的帧图像。至少两帧图像可以理解为，两帧图像、三帧图像等，用户可以根据在处理图像时的具体需求对计算机内的程序代码进行设置。可以根据，当前形成的低分辨率环形图像，以及待生成的当前帧图像之前至少连续的两帧图像，确定高分辨率图像在低分辨率图像上的相似区块，这样设置的好处在于避免了宽视场相机和窄视场相机拍摄的图像特别相近时，直接根据高分辨率图像确定在低分辨率环形图像中的位置时不准确的问题。
在一实施例中，若待生成的当前帧图像不为视频图像中的第一帧图像，处理方法可以是：对于每幅高分辨率图像，确定高分辨率图像在低分率环形图像中的至少两个候选融合位置；将高分辨率图像在至少两个帧图像中进行位置识别，根据识别结果对至少两个候选融合位置进行筛选，以确定目标融合位置；根据目标融合位置，将至少一幅高分辨率图像与低分率环形图像进行融合处理，确定高分辨率环形图像。
可以理解为，确定高分辨率图像在低分辨率环形图像中的至少两个候选融合位置，也就是在低分辨率环形图像上确定至少两个融合位置来融合高分辨率图像；为了确定融合高分辨率图像的目标融合区位置可以再根据待生成的当前 帧图像之前的两帧图像筛选高分辨率图像在低分辨率环形图像上的融合位置，并将最终筛选到的融合位置作为目标融合位置；若没有确定目标融合位置，则可以再获取前三帧图像进行处理确定高分辨率图像的融合位置，直至筛选出高分辨率图像在环形低分辨率图像中的目标融合位置。根据目标融合位置并采用高分辨率回复与匹配算法，将每幅高分辨率图像融合到低分辨率环形图像的相似区块，即目标融合位置处，得到高分辨率环形图像。
在上述技术方案的基础上，得到的高分辨率环形图像的分辨率较高并且光谱的维度较广，示例性的，若高分辨率环形图像中的某一个场景为一片树叶，颜色为绿色，采用相关技术中的相机拍摄到的图像可能就是接近绿色，但是采用本公开实施例技术方案，通过将宽视场相机和窄视场相机拍摄的图像组合在一起进行处理可以是得到的高分辨率环形图像达到十亿像素级别的分辨率，即最终得到的图像光谱维度更广分辨率更高，更接近图像的真实颜色。
S140、通过对多幅所述高分辨率环形图像进行编码整合处理，得到高分辨率的视频图像。
将得到的高分辨率环形图像进行编码整合处理，也就是利用相关技术中的视频编码算法将得到的高分辨率环形进行处理，得到高分辨率视频图像。将十亿像素级别分辨率的图像进行编码整合可以得到十亿像素级分辨率的视频图像。
本公开实施例的技术方案获取通过宽视场相机拍摄的至少三幅低分辨率图像，以及获取通过窄视场相机拍摄的高分辨率图像，通过对至少三幅低分辨率图像进行拼接处理，得到低分辨率环形图像，将低分辨率环形图像与高分辨率图像融合处理，确定为高分辨率环形图像，通过对多幅高分辨率环形图像进行编码整合处理，得到高分辨率的视频图像，解决了相关技术中不论采用的是高分辨率成像系统或者是双相机高分辨率成像系统拍摄的图像均为与人眼捕捉到 的三原色类似，不能获取到更高维度的图像信息，即图像的分辨率不高的问题，实现了通过将窄视场相机与宽视场相机拍摄到的高分辨率图像以及低分辨率图像结合在一起并采用一系列算法对其进行处理得到了十亿像素级分辨率的视频图像，实现了提高视频图像的分辨率以及光谱维度的技术效果，并且在实际应用的过程中宽视场相机与窄视场相机可以直接摆放在所要拍摄的视频环境中，不需要复杂的安装、通用性比较强，节省成本的技术效果。
在上述技术方案的基础上，在将低分辨率环形图像与至少一幅高分辨率环形图像融合处理，确定为高分辨率环形图像之后，还包括：根据高分辨率环形图像确定当前帧图像；采用权值算法对低分辨率环形图像以及当前帧图像进行处理，确定拍摄下一幅高分辨图像的兴趣权值，根据兴趣权值产生第一控制信号；根据第一控制信号控制拍摄下一个高分辨率图像的拍摄高度和拍摄角度；其中，拍摄高分辨率图像的相机设置在云台上；根据第一控制信息控制云台上相机的拍摄高度及相机与水平面之间的夹角。在一实施例中，云台上相机的拍摄高度可以理解为相机的拍摄方位角与水平面夹角。
在一实施例中，对高分辨率环形图像进行处理就可以得到与高分辨率环形图像相对应的当前帧图像，可选的，采用视频编码算法对高分辨率环形图像处理。当然在确定了当前帧图像之后还需要确定下一帧图像，也就是还需要获取低分辨率图像和高分辨率图像。其中，在拍摄图像的过程中宽视场相机拍摄低分辨率图像的位置是不发生变化的，窄视场相机拍摄高分辨率图像的位置以及拍摄角度是根据对整幅图像的兴趣权值决定的。
其中，兴趣权值包括贡献权值和代价权值。兴趣权值反映了低分辨率环形图像的至少一个区域对高分辨率图像的需求，可以理解为低分辨率图像中需要哪一个区域更加清楚。代价权值表示窄视场相机在拍摄高分辨率图像时从当前 位置转动到拍摄下一张高分辨率图像的位置时的所要消耗的代价。根据兴趣权值可以确定拍摄下一张高分辨率图像的最优位置，拍摄下一张高分辨率图像的窄视场相机最优位置可以是:
Figure PCTCN2019085966-appb-000001
其中，λgain表示贡献权值在兴趣权值中所占的权重，λcost表示代价权值在兴趣权值中所占的权重，
Figure PCTCN2019085966-appb-000002
表示最终得到的最优拍摄位置参数。F gain(θ pos)表示贡献权值；F cost(θ pos)表示代价权值。
贡献权值F gain(θ pos)的大小取决于单张图像超分辨率算法对于超分辨图像的需求程度、高光谱图像深度学习识别算法对于低分辨率环形图像兴趣区域的需求程度、前向帧图像对于高分辨率环形图像的需求来综合决定；代价权值F cost(θ pos)的大小是由窄视场相机在拍摄不同高分辨率图像时窄视场相机方位移动损耗、邻域高频信息损失以及当前拍摄方位距离下一次拍摄时移动的距离等决定。
通过对低分辨率环形图像、当前帧图像以及当前帧之前的帧图像进行处理可以确定拍摄下一个高分辨率图像的兴趣权值，计算机内预先存储的程序代码或者预先存储的程序可以根据拍摄下一张高分辨率图像的兴趣权值产生第一控制信号。其中，第一控制信号可以是调节拍摄下一张高分辨率图像的拍摄高度和拍摄角度的信号。
在一实施例中，窄视场相机可以设置在云台上，云台上相机的高度以及相机与水平面之间的夹角是可以调节的。云台可以与计算机无线通信或者电连接，可以接收计算机发出的第一控制信号，并根据第一控制信号调节云台上相机的拍摄高度以及相机与水平面之间的夹角。由于窄视场相机设置的云台上，因此当云台根据第一控制信号调节窄视场相机的高度与水平面之间的夹角时，相应 的设置在云台上的窄视场相机可以根据云台的旋转以及相机高度调节调节拍摄下一张高分辨率图像的高度和角度。
实施例二
作为上述实施例的一可选实施例，图3为本公开实施例二所提供的一种基于混合相机的视频成像方法，所述方法包括：
S301、基于至少三个宽视场相机分别获取至少三幅低分辨预览图像。
至少三个宽视场相机可以是三个或者多个，例如，五个宽视场相机，每个相机的镜头拍摄度数为90度。在对宽视场相机拍摄的低分辨率图像进行处理之前可以先通过五个宽视场相机分辨拍摄一幅低分辨率预览图像。在一实施例中，在对基于宽视场相机拍摄低分辨率图像并对其进行处理之前，可以先对宽视场相机进行参数调节和设置，在一实施例中，调节宽视场相机的参数可以是调节宽视场相机的性能参数，调节摆放位置可以是多个宽视场相机放置在同一水平面上，在一实施例中，参见图2，圆台201上，并且同一时刻五个宽视场相机拍摄的图像组合在一起可以覆盖360度，同时每相邻两个宽视场相机拍摄的低分辨率图像之间存在重叠的区域，好处在于，预先对宽视场相机进行调节，以确保最终得到的视频图像满足用户的需求。
S302、采用环形拼接算法对所述至少三幅低分辨率图像进行环形拼接处理，从拼接图像中确定每相邻两个宽视场相机的拍摄重合范围，根据每个宽视场相机的相机位置和拍摄重合范围确定相机矩阵，并存储。
在获取到低分辨率预览图像之后可以采用环形拼接算法对低分辨图像进行处理。可以从拼接图像中确定每相邻两个宽视场相机拍摄图像重合范围内的像素点，进而根据重合范围内的像素点以及每个宽视场相机的位置确定相机矩阵。可以将所述相机矩阵存储到计算中，以便对低分辨率图像进行处理时可以直接 调用。
S303、宽视场相机拍摄到低分辨率图像时，可以采用预先存储的相机矩阵对低分辨率图像进行处理，得到低分辨率环形图像。
预处理完成之后，若宽视场相机拍摄到低分辨图像，计算机可以获取低分辨率图像，并调用预先存储的相机矩阵对低分辨率图像进行拼接处理，得到拼接后的低分辨率环形图像，由于多个宽视场相机在同一时刻拍摄的图像组合在一起是可以覆盖360度的环形全景图像，因此拼接后的低分辨率环形图像也是可以覆盖360度的全图像。
S304、基于窄视场相机拍摄高分辨率图像。
在一实施例中，窄视场相机可以固定设置在云台上，云台是可以发生旋转、调节相机与水平面之间的夹角的，因此当云台在发生旋转以及调节相机与水平面之间的夹角时，可以带动窄视场相机的位置发生变化。
在宽视场相机拍摄低分辨率图像的同时窄视场相机也在拍摄高分辨率图像，若窄视场相机拍摄的高分辨率图像为第一张高分辨率图像，则计算机可以获取高分辨率图像；若窄视场相机拍摄的高分辨率图像不为第一张高分辨率图像，此时拍摄高分辨率图像的高度和与水平面之间的夹角可以结合待生成的当前帧图像之前的帧图像的反馈数据以及低分辨率环形图像确定拍摄高分辨率图像的兴趣权值，也就是窄视场相机拍摄低分辨率图像时的高度、仰角或者俯角。
在一实施例中，确定拍摄高分辨率图像的兴趣权值可以由计算机进行处理。在确定兴趣权值之后，会产生与兴趣权值相对应的第一控制信号，并将第一控制信号发送至云台，云台接收第一控制信号，调节相机的拍摄高度以及相机与水平面之间的夹角，从而使窄视场相机拍摄高分辨率图像。
S305、判断待生成的当前帧图像是否为第一帧图像，若是，则执行S306； 若否，则执行S307。
待生成的当前帧图像可以理解为高分辨率图像与低分辨率环形图像融合后的图像。判断待生成的当前帧图像是否为第一帧图像，若是第一帧图像则可以直接对高分辨率图像与低分辨率环形图像进行融合，也就是执行S306；若不是第一帧图像，则结合当前帧图像之前的帧图像、高分辨率图像以低分辨率环形图像来确定高分辨率图像在环形低分辨率图像上的目标融合位置，并对其进行融合S307。
S306、采用高分辨率恢复与匹配算法对低分辨率环形图像与基于窄视场相机拍摄的高分辨率图像融合处理，得到高分辨率环形图像。
在采用高分辨率恢复与匹配算法之前可以根据高分辨率图像与低分辨率环形图像的可比比拟性确定高分辨率图像在低分辨率环形图像上的相似区块，也就是高分辨率图像融合到低分辨率环形图像上的目标融合位置。再采用高分辨率恢复与匹配算法将高分辨率图像融合到低分辨率环形图像中，得到高分辨率环形图像。
S307、采用高分辨率恢复与匹配算法对待生成的当前帧图像之前的帧图像、高分辨率图像以及低分辨率环形图像进行融合处理，得到高分辨率环形图像。
若待生成的当前帧图像不为第一帧图像，则可以根据低分辨率环形图像与高分辨率图像之间的可比拟性，即相似性，确定出至少两个候选融合位置；再获取待生成的当前帧图像前至少两个帧图像，并将高分辨率图像在至少两个帧图像中进行位置识别，若是确定了目标融合位置，则可以根据高分辨率恢复与匹配算法将高分辨率图像融合到低分辨率环形图像中；若依然没有确定目标融合位置，则可以根据待生成当前帧图像的前三帧图像来确定目标图像的融合位置，直至确定出高分辨率图像在低分辨率环形图像中的目标融合位置。若确定 了高分辨率图像在低分辨率环形图像上的融合位置可以采用高分辨率恢复与匹配算法得到高分辨环形图像。其中，高分辨率恢复与匹配算法是2017年Haitian Zheng等人提出的基于深度卷积神经网络的CC-Net+SS-Net模式图像融合恢复算法。
S308、采用视频编码算法对高分辨率环形图像进行处理，得到高分辨率视频图像。
在一实施例中，可以将得到的高分辨率环形图像进行缓存与压缩，利用视频编码算法将高分辨率环形图像处理得到高分辨率视频图像。其中，在基于混合相机的视频成像方法中，会利用到当前帧图像的前一帧图像或者前几帧图像，因此可以将每一帧图像返回至S304和S307。
本公开实施例的技术方案获取通过宽视场相机拍摄的至少三幅低分辨率图像，以及获取通过窄视场相机拍摄的高分辨率图像，通过对至少三幅低分辨率图像进行拼接处理，得到低分辨率环形图像，将低分辨率环形图像与高分辨率图像融合处理，确定为高分辨率环形图像，通过对多幅高分辨率环形图像进行编码整合处理，得到高分辨率的视频图像，解决了相关技术中不论采用的是高分辨率成像系统或者是双相机高分辨率成像系统拍摄的图像均为与人眼捕捉到的三原色类似，不能获取到更高维度的图像信息，即图像的分辨率不高的问题，实现了通过将窄视场相机与宽视场相机拍摄到的高分辨率图像以及低分辨率图像结合在一起并采用一系列算法对其进行处理得到了十亿像素级分辨率的视频图像，实现了提高视频图像的分辨率以及光谱维度的技术效果，并且在实际应用的过程中宽视场相机与窄视场相机可以直接摆放在所要拍摄的视频环境中，不需要复杂的安装、通用性比较强，节省成本的技术效果。
实施例三
图4为本公开实施例三提供的一种基于混合相机阵列的视频成像系统结构示意图，该系统包括：获取图像模块410、图像拼接处理模块420、图像融合处理模块430以及视频图像处理模块440。
其中，获取图像模块410，设置为获取通过宽视场相机拍摄的至少三幅低分辨率图像，以及获取通过窄视场相机拍摄的至少一幅高分辨率图像；图像拼接处理模块420，设置为通过对至少三幅所述低分辨率图像进行拼接处理，得到低分辨率环形图像；图像融合处理模块430，设置为将所述低分辨率环形图像与所述至少一幅高分辨率图像融合处理，确定为高分辨率环形图像；视频图像处理模块440，设置为通过对多幅所述高分辨率环形图像进行编码整合处理，得到高分辨率的视频图像。
在上述技术方案的基础上，所述低分辨率环形图像是覆盖360度的环形全景图像。
在上述技术方案的基础上，所述至少三幅低分辨率图像由至少三个宽视场相机分别拍摄，每相邻两个宽视场相机之间拍摄的区域存在重叠，且至少三个宽视场相机在同一时间拍摄的低分辨率图像组合在一起为覆盖设定角度的环形图像。
在上述技术方案的基础上，所述系统还包括：预处理模块，设置为获取至少三个宽视场相机分别拍摄的至少三幅低分辨图像；环形拼接算法模块，设置为采用环形拼接算法对至少两幅低分辨图像进行拼接处理，并从拼接图像中识别每相邻两个宽视场相机的拍摄重合范围；相机矩阵确定模块，设置为根据每个宽视场相机的相机位置和拍摄重合范围确定相机矩阵，并存储所述相机的内参矩阵和畸变参数矩阵，其中，所述相机的内参矩阵和畸变参数矩阵用于对所述至少三幅低分辨率图像进行拼接处理，得到低分辨率环形图像。
在上述技术方案的基础上，所述图像融合处理模块还设置为若待生成的当前帧图像为视频图像中的第一帧图像，则将所述低分辨率环形图像与所述至少一幅高分辨率图像融合处理为高分辨率环形图像，将所述高分辨率环形图像作为所述待生成的当前帧图像；若待生成的当前帧图像不为视频图像中的第一帧图像，则将所述低分率环形图像、所述至少一幅高分辨率图像以及当前帧图像之前连续的至少两个帧图像结合在一起，确定高分辨率环形图像，将所述高分辨率环形图像作为所述待生成的当前帧图像；其中，所述至少两个帧图像中的一个帧图像与所述当前帧图像相连续。
在上述技术方案的基础上，所述图像融合处理模块，还设置为：确定每幅所述高分辨率图像在所述低分率环形图像中的至少两个候选融合位置；将所述每幅高分辨率图像在所述至少两个帧图像中进行位置识别，根据识别结果对所述至少两个候选融合位置进行筛选，以确定目标融合位置；根据所述目标融合位置，将所述至少一幅高分辨率图像与所述低分率环形图像进行融合处理，确定高分辨率环形图像。
在上述技术方案的基础上，在所述图像融合处理模块设置为将所述低分辨率环形图像与所述高分辨率图像融合处理，确定为高分辨率环形图像之后，所述系统还包括：兴趣权值确定模块；所述兴趣权值确定模块包括当前帧图像确定单元、兴趣权值确定单元以及调节单元；当前帧图像确定单元设置为根据所述高分辨率环形图像确定当前帧图像；兴趣权值确定单元，设置为采用权值算法对所述低分辨率环形图像以及所述当前帧图像进行处理，确定拍摄下一幅高分辨图像的兴趣权值，根据所述兴趣权值产生第一控制信号；调节单元，设置为根据所述第一控制信号控制拍摄下一个高分辨率图像的拍摄高度和拍摄角度；其中，拍摄所述高分辨率图像的相机设置在云台上；根据所述第一控制信号控 制所述云台上相机的拍摄高度及相机与水平面之间的夹角。
在上述技术方案的基础上，所述图像融合处理模块，还设置为基于所述低分率环形图像与所述每幅高分辨图像之间的可比拟性，确定所述每个高分辨率图像在所述低分辨率环形图像中的相似区块；采用高分辨率恢复与匹配算法，将所述至少一幅高分辨率图像融合到所述低分辨率环形图像的相似区块中，得到高分辨率环形图像。
本公开实施例的技术方案获取通过宽视场相机拍摄的至少三幅低分辨率图像，以及获取通过窄视场相机拍摄的高分辨率图像，通过对至少三幅低分辨率图像进行拼接处理，得到低分辨率环形图像，将低分辨率环形图像与高分辨率图像融合处理，确定为高分辨率环形图像，通过对多幅高分辨率环形图像进行编码整合处理，得到高分辨率的视频图像，解决了相关技术中不论采用的是高分辨率成像系统或者是双相机高分辨率成像系统拍摄的图像均为与人眼捕捉到的三原色类似，不能获取到更高维度的图像信息，即图像的分辨率不高的问题，实现了通过将窄视场相机与宽视场相机拍摄到的高分辨率图像以及低分辨率图像结合在一起并采用一系列算法对其进行处理得到了十亿像素级分辨率的视频图像，实现了提高视频图像的分辨率以及光谱维度的技术效果，并且在实际应用的过程中宽视场相机与窄视场相机可以直接摆放在所要拍摄的视频环境中，不需要复杂的安装、通用性比较强，节省成本的技术效果。本公开实施例所提供的基于混合相机阵列的视频成像系统可执行本公开任意实施例所提供的基于混合相机的视频成像方法，具备执行方法相应的功能模块和效果。
上述系统所包括的多个单元和模块只是按照功能逻辑进行划分的，但并不局限于上述的划分，只要能够实现相应的功能即可；另外，每个功能单元的具体名称也只是为了便于相互区分。
实施例四
图5为本公开实施例四提供的一种设备的结构示意图。图5示出了适于用来实现本公开实施例实施方式的示例性设备50的框图。图5显示的设备50仅仅是一个示例，不应对本公开实施例的功能和使用范围带来任何限制。
如图5所示，设备50以通用计算设备的形式表现。设备50的组件可以包括但不限于：一个或者多个处理器或者处理单元501，系统存储器502，连接不同系统组件(包括系统存储器502和处理单元501)的总线503。该设备50设置有至少三个宽视场相机和窄视场相机，且设置在圆台上，例如图2所示的结构。
总线503表示几类总线结构中的一种或多种，包括存储器总线或者存储器控制器，外围总线，图形加速端口，处理器或者使用多种总线结构中的任意总线结构的局域总线。举例来说，这些体系结构包括但不限于工业标准体系结构(Industry Standard Architecture，ISA)总线，微通道体系结构(MicroChannel Architecture，MCA)总线，增强型ISA总线、视频电子标准协会(Video Electronics Standards Association，VESA)局域总线以及外围组件互连(Peripheral Component Interconnect，PCI)总线。
设备50包括多种计算机系统可读介质。这些介质可以是任何能够被设备50访问的可用介质，包括易失性和非易失性介质，可移动的和不可移动的介质。
系统存储器502可以包括易失性存储器形式的计算机系统可读介质，例如随机存取存储器(Random Access Memory，RAM)504和/或高速缓存存储器505。设备50可以进一步包括其它可移动/不可移动的、易失性/非易失性计算机系统存储介质。仅作为举例，存储系统506可以设置为读写不可移动的、非易失性磁介质(图5未显示，通常称为“硬盘驱动器”)。尽管图5中未示出，可以提 供用于对可移动非易失性磁盘(例如“软盘”)读写的磁盘驱动器，以及对可移动非易失性光盘(例如便携式紧凑磁盘只读存储器(Compact Disc Read-Only Memory，CD-ROM)，数字视盘(Digital Video Disc-Read Only Memory，DVD-ROM)或者其它光介质)读写的光盘驱动器。在这些情况下，每个驱动器可以通过一个或者多个数据介质接口与总线503相连。存储器502可以包括至少一个程序产品，该程序产品具有一组(例如至少一个)程序模块，这些程序模块被配置以执行本公开一个或多个实施例的功能。
具有一组(至少一个)程序模块507的程序/实用工具508，可以存储在例如存储器502中，这样的程序模块507包括但不限于操作系统、一个或者多个应用程序、其它程序模块以及程序数据，这些示例中的每一个或某种组合中可能包括网络环境的实现。程序模块507通常执行本公开所描述的实施例中的功能和/或方法。
设备50也可以与一个或多个外部设备509(例如键盘、指向设备、显示器510等)通信，还可与一个或者多个使得用户能与该设备50交互的设备通信，和/或与使得该设备50能与一个或多个其它计算设备进行通信的任何设备(例如网卡，调制解调器等等)通信。这种通信可以通过输入/输出(Input/Output，I/O)接口511进行。并且，设备50还可以通过网络适配器512与一个或者多个网络(例如局域网(Local Area Network，LAN)，广域网(Wide Area Network，WAN)和/或公共网络，例如因特网)通信。如图所示，网络适配器512通过总线503与设备50的其它模块通信。应当明白，尽管图5中未示出，可以结合设备50使用其它硬件和/或软件模块，包括但不限于：微代码、设备驱动器、冗余处理单元、外部磁盘驱动阵列、磁盘阵列(Redundant Arrays of Independent Drives，RAID)系统、磁带驱动器以及数据备份存储系统等。
处理单元501通过运行存储在系统存储器502中的程序，从而执行一种或多种功能应用以及数据处理，例如实现本公开实施例所提供的基于混合相机的视频成像方法。
实施例五
本公开实施例五还提供一种包含计算机可执行指令的存储介质，所述计算机可执行指令在由计算机处理器执行时用于执行基于混合相机的视频成像方法。
本公开实施例的计算机存储介质，可以采用一个或多个计算机可读的介质的任意组合。计算机可读介质可以是计算机可读信号介质或者计算机可读存储介质。计算机可读存储介质例如可以是——但不限于——电、磁、光、电磁、红外线、或半导体的系统、装置或器件，或者任意以上的组合。计算机可读存储介质包括(非穷举的列表)：具有一个或多个导线的电连接、便携式计算机磁盘、硬盘、RAM、只读存储器(Read-Only Memory，ROM)、可擦式可编程只读存储器(Erasable Programmable Read Only Memory，EPROM)或闪存、光纤、CD-ROM、光存储器件、磁存储器件、或者上述的任意合适的组合。在本文件中，计算机可读存储介质可以是任何包含或存储程序的有形介质，该程序可以被指令执行系统、装置或者器件使用或者与其结合使用。
计算机可读的信号介质可以包括在基带中或者作为载波一部分传播的数据信号，其中承载了计算机可读的程序代码。这种传播的数据信号可以采用多种形式，包括但不限于电磁信号、光信号或上述的任意合适的组合。计算机可读的信号介质还可以是计算机可读存储介质以外的任何计算机可读介质，该计算机可读介质可以发送、传播或者传输用于由指令执行系统、装置或者器件使用或者与其结合使用的程序。
计算机可读介质上包含的程序代码可以用任何适当的介质传输，包括—— 但不限于无线、电线、光缆、射频(Radio Frequency，RF)等等，或者上述的任意合适的组合。
可以以一种或多种程序设计语言或其组合来编写用于执行本公开实施例操作的计算机程序代码，所述程序设计语言包括面向对象的程序设计语言—诸如Java、Smalltalk、C++，还包括常规的过程式程序设计语言——诸如“C”语言或类似的程序设计语言。程序代码可以完全地在用户计算机上执行、部分地在用户计算机上执行、作为一个独立的软件包执行、部分在用户计算机上部分在远程计算机上执行、或者完全在远程计算机或服务器上执行。在涉及远程计算机的情形中，远程计算机可以通过任意种类的网络——包括LAN或WAN—连接到用户计算机，或者，可以连接到外部计算机(例如利用因特网服务提供商来通过因特网连接)。

Claims (11)

  1. 一种基于混合相机的视频成像方法，包括：
    获取通过宽视场相机拍摄的至少三幅低分辨率图像，以及获取通过窄视场相机拍摄的至少一幅高分辨率图像；
    通过对所述至少三幅低分辨率图像进行拼接处理，得到低分辨率环形图像；
    将所述低分辨率环形图像与所述至少一幅高分辨率图像融合处理，确定为高分辨率环形图像；
    通过对多幅所述高分辨率环形图像进行编码整合处理，得到高分辨率的视频图像。
  2. 根据权利要求1所述的方法，其中，所述低分辨率环形图像是覆盖360度的环形全景图像。
  3. 根据权利要求1所述的方法，其中，所述至少三幅低分辨率图像由至少三个宽视场相机分别拍摄，每相邻两个所述宽视场相机之间拍摄的区域存在重叠，且所述至少三个宽视场相机在同一时间拍摄的低分辨率图像组合在一起为覆盖设定角度的环形图像。
  4. 根据权利要求3所述的方法，其中，通过对所述至少三幅低分辨率图像进行拼接处理，得到低分辨率环形图像，包括：
    采用环形拼接算法对所述至少三幅低分辨图像进行拼接处理，并从拼接图像中识别每相邻两个所述宽视场相机的拍摄重合范围；
    根据每个所述宽视场相机的相机位置和拍摄重合范围确定相机矩阵，并存储所述相机的内参矩阵和畸变参数矩阵，其中，所述相机的内参矩阵和畸变参数矩阵用于对所述至少三幅低分辨率图像进行拼接处理，得到低分辨率环形图像。
  5. 根据权利要求1所述的方法，其中，将所述低分辨率环形图像与所述至 少一幅高分辨率图像融合处理，确定为高分辨率环形图像，包括：
    响应于待生成的当前帧图像为视频图像中的第一帧图像，将所述低分辨率环形图像与所述至少一幅高分辨率图像融合处理为所述高分辨率环形图像，将所述高分辨率环形图像作为所述待生成的当前帧图像；
    响应于待生成的当前帧图像不为视频图像中的第一帧图像，将所述低分率环形图像、所述至少一幅高分辨率图像以及所述当前帧图像之前连续的至少两个帧图像结合在一起，确定所述高分辨率环形图像，将所述高分辨率环形图像作为所述待生成的当前帧图像；
    其中，所述至少两个帧图像中的一个帧图像与所述当前帧图像相连续。
  6. 根据权利要求5所述的方法，其中，将所述低分辨率环形图像、所述至少一幅高分辨率图像以及当前帧图像之前至少两个帧图像结合在一起，确定所述高分辨率环形图像，包括：
    确定每幅所述高分辨率图像在所述低分率环形图像中的至少两个候选融合位置；
    将所述每幅高分辨率图像在所述至少两个帧图像中进行位置识别，根据识别结果对所述至少两个候选融合位置进行筛选，以确定目标融合位置；
    根据所述目标融合位置，将所述至少一幅高分辨率图像与所述低分率环形图像进行融合处理，确定所述高分辨率环形图像。
  7. 根据权利要求1所述的方法，在将所述低分辨率环形图像与所述至少一幅高分辨率图像融合处理，确定为高分辨率环形图像之后，还包括：
    根据所述高分辨率环形图像确定当前帧图像；
    采用权值算法对所述低分辨率环形图像以及所述当前帧图像进行处理，确定拍摄下一幅高分辨图像的兴趣权值，根据所述兴趣权值产生第一控制信号；
    根据所述第一控制信号控制拍摄下一幅高分辨率图像的拍摄高度和拍摄角度；
    其中，拍摄所述高分辨率图像的相机设置在云台上；根据所述第一控制信号控制所述云台上相机的拍摄高度及所述相机与水平面之间的夹角。
  8. 根据权利要求1所述的方法，其中，将所述低分辨率环形图像与所述至少一幅高分辨率图像融合处理，确定为高分辨率环形图像，还包括：
    基于所述低分率环形图像与每幅所述高分辨图像之间的可比拟性，确定所述每幅高分辨率图像在所述低分辨率环形图像中的相似区块；
    采用高分辨率恢复与匹配算法，将所述每幅高分辨率图像融合到所述低分辨率环形图像的相似区块中，得到所述高分辨率环形图像。
  9. 一种基于混合相机阵列的视频成像系统，包括：
    获取图像模块，设置为获取通过宽视场相机拍摄的至少三幅低分辨率图像，以及获取通过窄视场相机拍摄的至少一幅高分辨率图像；
    图像拼接处理模块，设置为通过对所述至少三幅低分辨率图像进行拼接处理，得到低分辨率环形图像；
    图像融合处理模块，设置为将所述低分辨率环形图像与所述至少一幅高分辨率图像融合处理，确定为高分辨率环形图像；
    视频图像处理模块，设置为通过对多幅所述高分辨率环形图像进行编码整合处理，得到高分辨率的视频图像。
  10. 一种设备，包括：
    一个或多个处理器；
    存储装置，设置为存储一个或多个程序，
    所述一个或多个程序被所述一个或多个处理器执行，使得所述一个或多个 处理器实现如权利要求1-8中任一项所述的方法。
  11. 一种包含计算机可执行指令的存储介质，所述计算机可执行指令在由计算机处理器执行时用于执行如权利要求1-8中任一项所述的方法。
PCT/CN2019/085966 2018-10-24 2019-05-08 基于混合相机的视频成像方法、系统、设备及存储介质 WO2020082722A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201811241672.6A CN109167924B (zh) 2018-10-24 2018-10-24 基于混合相机的视频成像方法、系统、设备及存储介质
CN201811241672.6 2018-10-24

Publications (1)

Publication Number Publication Date
WO2020082722A1 true WO2020082722A1 (zh) 2020-04-30

Family

ID=64878927

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2019/085966 WO2020082722A1 (zh) 2018-10-24 2019-05-08 基于混合相机的视频成像方法、系统、设备及存储介质

Country Status (2)

Country Link
CN (1) CN109167924B (zh)
WO (1) WO2020082722A1 (zh)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112862732A (zh) * 2021-01-19 2021-05-28 华中科技大学 多分辨率图像融合方法、装置、设备、介质及产品
CN114302031A (zh) * 2021-11-15 2022-04-08 华能国际电力股份有限公司上海石洞口第二电厂 基于5g通讯的相机阵列传输方法及系统

Families Citing this family (14)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109167924B (zh) * 2018-10-24 2021-05-11 清华-伯克利深圳学院筹备办公室 基于混合相机的视频成像方法、系统、设备及存储介质
WO2020189223A1 (ja) 2019-03-15 2020-09-24 ソニー株式会社 動画配信システム、動画配信方法、及び表示端末
CN109951637B (zh) * 2019-03-19 2020-09-11 河北川谷信息技术有限公司 基于大数据的安防监控探头分析处理方法
CN111161369B (zh) * 2019-12-20 2024-04-23 上海联影智能医疗科技有限公司 图像重建存储方法、装置、计算机设备和存储介质
CN111553972B (zh) * 2020-04-27 2023-06-30 北京百度网讯科技有限公司 用于渲染增强现实数据的方法、装置、设备及存储介质
CN112468832A (zh) * 2020-10-22 2021-03-09 北京拙河科技有限公司 一种十亿级像素全景视频直播方法、装置、介质及设备
CN112767243B (zh) * 2020-12-24 2023-05-26 深圳大学 一种高光谱图像超分辨率的实现方法及系统
CN113079325B (zh) * 2021-03-18 2023-01-06 北京拙河科技有限公司 暗光条件下的十亿级像素成像方法、装置、介质及设备
CN112927142B (zh) * 2021-04-02 2022-11-11 中国人民解放军国防科技大学 一种基于时域插值的高速高分辨率视频生成方法和装置
CN113066011B (zh) * 2021-04-07 2022-11-11 合肥英睿系统技术有限公司 一种图像处理方法、装置、系统、介质和电子设备
CN113963175A (zh) * 2021-05-13 2022-01-21 北京市商汤科技开发有限公司 一种图像处理方法、装置、计算机设备以及存储介质
CN113689550B (zh) * 2021-08-03 2023-06-06 南昌威爱信息科技有限公司 一种vr超高清三维数字化建模系统
CN113873201B (zh) * 2021-09-27 2023-09-15 北京环境特性研究所 一种超视距制高点反向观察系统及方法
CN113630517B (zh) * 2021-10-08 2022-01-25 清华大学 光电感算一体光场智能成像方法及装置

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR20120000736A (ko) * 2010-06-28 2012-01-04 서울대학교산학협력단 영상의 분광 및 공간특성이 반영된 융합계수를 이용하여 고해상도 위성영상을 융합하는 방법
CN107659774A (zh) * 2017-09-30 2018-02-02 深圳市未来媒体技术研究院 一种基于多尺度相机阵列的视频成像系统及视频处理方法
CN107959805A (zh) * 2017-12-04 2018-04-24 深圳市未来媒体技术研究院 基于混合相机阵列的光场视频成像系统及视频处理方法
CN108038893A (zh) * 2017-12-06 2018-05-15 深圳市未来媒体技术研究院 一种基于混合相机阵列的十亿像素视频的生成方法
CN108347577A (zh) * 2018-02-05 2018-07-31 清华-伯克利深圳学院筹备办公室 一种成像系统和方法
CN108399600A (zh) * 2018-02-23 2018-08-14 清华-伯克利深圳学院筹备办公室 一种全景成像系统及方法
CN109167924A (zh) * 2018-10-24 2019-01-08 清华-伯克利深圳学院筹备办公室 基于混合相机的视频成像方法、系统、设备及存储介质

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR20120000736A (ko) * 2010-06-28 2012-01-04 서울대학교산학협력단 영상의 분광 및 공간특성이 반영된 융합계수를 이용하여 고해상도 위성영상을 융합하는 방법
CN107659774A (zh) * 2017-09-30 2018-02-02 深圳市未来媒体技术研究院 一种基于多尺度相机阵列的视频成像系统及视频处理方法
CN107959805A (zh) * 2017-12-04 2018-04-24 深圳市未来媒体技术研究院 基于混合相机阵列的光场视频成像系统及视频处理方法
CN108038893A (zh) * 2017-12-06 2018-05-15 深圳市未来媒体技术研究院 一种基于混合相机阵列的十亿像素视频的生成方法
CN108347577A (zh) * 2018-02-05 2018-07-31 清华-伯克利深圳学院筹备办公室 一种成像系统和方法
CN108399600A (zh) * 2018-02-23 2018-08-14 清华-伯克利深圳学院筹备办公室 一种全景成像系统及方法
CN109167924A (zh) * 2018-10-24 2019-01-08 清华-伯克利深圳学院筹备办公室 基于混合相机的视频成像方法、系统、设备及存储介质

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112862732A (zh) * 2021-01-19 2021-05-28 华中科技大学 多分辨率图像融合方法、装置、设备、介质及产品
CN112862732B (zh) * 2021-01-19 2024-03-19 华中科技大学 多分辨率图像融合方法、装置、设备、介质及产品
CN114302031A (zh) * 2021-11-15 2022-04-08 华能国际电力股份有限公司上海石洞口第二电厂 基于5g通讯的相机阵列传输方法及系统

Also Published As

Publication number Publication date
CN109167924B (zh) 2021-05-11
CN109167924A (zh) 2019-01-08

Similar Documents

Publication Publication Date Title
WO2020082722A1 (zh) 基于混合相机的视频成像方法、系统、设备及存储介质
KR102500759B1 (ko) 심 없는 이미지 스티칭(Seamless image stitching)
WO2021227359A1 (zh) 一种无人机投影方法、装置、设备及存储介质
US9811910B1 (en) Cloud-based image improvement
WO2019161814A2 (zh) 一种全景成像系统及方法
US7990421B2 (en) Arrangement and method relating to an image recording device
WO2020103020A1 (zh) 一种测绘采样点的规划方法、装置、控制终端及存储介质
US11258949B1 (en) Electronic image stabilization to improve video analytics accuracy
WO2022000300A1 (zh) 图像处理方法、图像获取装置、无人机、无人机系统和存储介质
WO2014023231A1 (zh) 宽视场超高分辨率光学成像系统及方法
US20210127059A1 (en) Camera having vertically biased field of view
CN111062234A (zh) 一种监控方法、智能终端及计算机可读存储介质
US11587259B2 (en) Fixed pattern calibration for multi-view stitching
JP2018007150A (ja) 撮像装置、撮像装置の制御方法、及び、プログラム
WO2023041013A1 (zh) 一种无人机拍摄方法、装置、无人机及存储介质
EP3472667B1 (en) System and method for capturing horizontal disparity stereo panorama
WO2020103021A1 (zh) 一种测绘采样点的规划方法、装置、控制终端及存储介质
WO2022040868A1 (zh) 全景拍摄方法、电子设备及存储介质
TW202011349A (zh) 全景圖形成方法及系統
WO2018196854A1 (zh) 一种拍照方法、拍照装置及移动终端
JP7424076B2 (ja) 画像処理装置、画像処理システム、撮像装置、画像処理方法およびプログラム
WO2023165535A1 (zh) 图像处理方法、装置及设备
KR100890710B1 (ko) 전방위 영상 장치
US10051192B1 (en) System and apparatus for adjusting luminance levels of multiple channels of panoramic video signals
WO2015141185A1 (ja) 撮像制御装置、撮像制御方法および記録媒体

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19875033

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 19875033

Country of ref document: EP

Kind code of ref document: A1

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 15/09/2021)