TWI696147B - Method and system for rendering a panoramic image - Google Patents

Abstract

The disclosure is related to a method and a system for rendering a panoramic image. A camera is used to capture multiple images of a scene. Pixel features can be extracted from the images. The pixel features are used to determine if the images have edge image features and corner image features so as to determine the overlapped areas. The images can be composed at this early stage and transformed to an equirectangular projection image. An object detection is performed on the equirectangular projection image for determining if the scene is an indoor scene or outdoor scene. The mentioned pixel features, the edge image features and/or the corner image features with respect to the overlapped areas are incorporated to render the panoramic image. A learning model of neural network can be used to optimize the image.

Description

Method and system for forming panoramic image

The disclosure book relates to a method for generating a panoramic image, especially a method and system that can form a panoramic image for indoor and outdoor scenes with different characteristics.

Generally, cameras, mobile phones and other tools for shooting images have a certain Field of View (FOV) according to the design of the lens, and can only shoot images within a certain field of view at a time. The conventional technology can use computer-assisted tools to stitch together. Image to get a wide-angle photo or panoramic photo.

Taking a panorama as an example, a panorama, or a panoramic view, is a wide-angle image that covers a field of view of 360 degrees to the left and right of the panorama, and 180 degrees up and down. The general way of formation is to first take a picture with a camera. For images of a specific field of view in a scene, if the camera is equipped with a gyroscope, a sensor that can obtain the shooting azimuth to record the azimuth of continuously shooting multiple images, and finally stitch the multiple images according to the azimuth information between each image Get up and get a panoramic picture.

In the prior art, in order to take a panoramic image, someone once set multiple cameras on a carrier device and pointed them in multiple different directions to shoot, and then generated the panoramic image by stitching. There are also technologies that use sensors such as gyroscopes in mobile devices to direct the user to hold the device in a certain direction to take a picture through a software program running on the mobile device, and then instruct the user to move the shooting angle Take a shot in the second direction, and get a panoramic image after multiple shots. The software The program then combines these images into a panorama.

The disclosure discloses a method for forming a panoramic image, which can be applied to generate a panoramic image of augmented real or virtual reality scenes, and can be photographed using general camera equipment. An embodiment of the method for forming a panoramic image includes first obtaining a scene with a photographing device For the multiple images of, the lens is preferably a fisheye lens, and the multiple images are images shot at different shooting angles.

Then, a camera or other computer device uses the image processing technology to capture the pixel characteristics of multiple images, and based on the pixel characteristics, determine the boundary image feature and the corner image feature, or one of them, to obtain the multiple The overlapping area of images. At this time, multiple images are preliminarily synthesized and converted into an equidistant histogram. After comparing the images in the library, object detection is performed, and the properties of the equidistant histogram can be obtained based on the detected objects. , For example, determine whether the image is an outdoor image or an indoor image. Then, according to the attributes of the equidistant columnar projection image, it can be judged to synthesize a panoramic image by using the pixel feature, the boundary image feature and the corner image feature of the multiple overlapping regions, or one of them.

Further, in the step of determining whether the image has boundary image features and corner image features, first extract features from multiple images, for example, use boundary and corner detection method to determine whether there are boundary image features or corner image features , And then use random sampling consensus algorithm (RANSAC) to filter features to find overlapping areas.

Furthermore, the method of judging the objects in the image can use a deep integrative neural network (Convolutional Neural Network, CNN) to identify the objects in the isometric histogram, and then determine the attributes of the scene, such as indoor or Outdoor classification.

In one embodiment, in the above method, when multiple images of the scene are obtained, each image can be shot in a high dynamic range (HDR) imaging mode to obtain multiple sub-images with different exposure values corresponding to each image. And based on the indoor deep integration The CNN-like learning data obtains the exposure value and saturation value of the panoramic image, and the high dynamic range imaging (HDR) sub-image is synthesized according to the numerical value.

The disclosure also discloses a system for forming a panoramic image. The main component of the system is a camera. The camera can be mounted on a rotatable carrier. The carrier decides to shoot each image according to the FOV of the camera lens. The rotation angle of, to shoot multiple images of a scene in multiple shooting directions through a wide-angle or fisheye lens, including a processing unit to perform the above-mentioned panoramic image forming method.

In order to further understand the technology, methods and effects of the present invention to achieve the established objectives, please refer to the following detailed descriptions and drawings about the present invention. I believe that the objectives, features and characteristics of the present invention can be thoroughly and concretely obtained. It is understood that, however, the accompanying drawings are only provided for reference and illustration, and are not intended to limit the present invention.

10‧‧‧Internet

12‧‧‧Servo system

14‧‧‧Host

11‧‧‧Camera

15‧‧‧Lens

13‧‧‧Carrier device

135‧‧‧Rotating mechanism

20‧‧‧Carrier device

201‧‧‧controller

203‧‧‧Rotary drive unit

205‧‧‧Carrier Unit

22‧‧‧Camera

221‧‧‧Processing unit

223‧‧‧Memory Unit

225‧‧‧Image capture unit

227‧‧‧lens unit

229‧‧‧Communication Unit

301‧‧‧The first wide-angle image

302‧‧‧The second wide-angle image

303‧‧‧The third wide-angle image

Steps S401~S407‧‧‧Preparation process for shooting panorama

Steps S501~S515‧‧‧Method flow of shooting panorama

Steps S601~S611‧‧‧Method flow of shooting panorama

Fig. 1 shows a schematic diagram of an embodiment of a related system and equipment for shooting a panoramic view; Fig. 2 shows a schematic diagram of a circuit embodiment of a shooting device and its carrying device; Fig. 3 shows a schematic diagram of an embodiment of synthesizing an isometric columnar projection diagram; Fig. 4 Shown to describe an embodiment of the preparation process for shooting a panoramic image; Fig. 5 shows one of the flowcharts of an embodiment of the method for forming a panoramic image; Fig. 6 shows the second flowchart of an embodiment of the method for forming a panoramic image.

The specification discloses a method for forming a panoramic image. The panoramic image (panorama) is a wide-angle image covering a panoramic view of 360 degrees left and right and 180 degrees up and down. One of its applications can be used in augmented reality (AR) or Virtual reality (VR) scenes allow users to freely browse the scene in a 360-degree view of left and right and 180 degrees up and down when wearing a specific virtual reality device.

In this embodiment, the terminal device includes a shooting device 11 for shooting images. The shooting device 11 may be a camera, preferably equipped with a fisheye lens capable of shooting ultra-wide-angle images; it may be a mobile phone, and the camera may not have fisheyes. The ability of the head, but it can be achieved through the external lens 15.

If you want to take a panoramic view of the entire scene, you need to cover the left and right 360 degrees and the top and bottom 180 degrees of field of view. In this example, the camera 11 is mounted on a carrier device 13 that can drive the rotation to take pictures of the entire scene, and the rotation mechanism 135 carries the camera The device 11 is provided with a motor that can drive the camera 11 to rotate, such as a stepping motor.

The carrying device 13 is a programmable device, and the rotation angle of each image can be determined according to the coverage of the field of view of the lens 15 of the camera 11. For example, when the lens 15 is a lens that can cover a 180-degree field of view, in order to take a panoramic view covering a 360-degree field of view from the left to the right and a 180-degree field of view, at least after the first shot, it is necessary to rotate 180 degrees and then take the second shot. , So that we can get a panoramic view covering the left and right 360 degrees and the top and bottom 180 degrees. Or, according to the field of view covered by the lens 15, multiple shots can be taken after several rotations. Each shot only covers a specific angle of view, and multiple images will include overlapping features such as borders or corners. The basis for splicing images.

After the shooting device 11 cooperates with the carrier device 13 to complete the panorama shooting, the image data can not only be stitched to form a panorama through the processing capabilities of the shooting device 11 and related software programs, but also can be sent to the servo system 12 or the host 14 via the network 10 , The image processing is executed by the servo system 12 or the host 14 to complete the stitching to form a panoramic image. Finally, the formed panoramic image can be stored or shared through the servo system 12. The servo system 12 can realize the service of providing virtual reality or augmented reality panorama images for users to download.

It should be mentioned that the embodiment described in FIG. 1 is only one of the embodiments for shooting a panoramic image, and is not used to limit the implementation scope of the method for forming a panoramic image disclosed in the disclosure.

Figure 2 is then shown as a circuit embodiment diagram of the photographing device and its carrying device. The circuit elements in the carrying device 20 and the photographing device 22 are described in the above. The main purpose of the carrying device 20 is to adjust the rotation angle of each shot according to the ability of the photographing device 22 to take a panoramic view of the scene, and to complete the acquisition of panoramic images by rotating and photographing.

The carrying device 20 is provided with a controller 201 that manages its operation. The rotating drive unit 203 drives the photographing device 22 through the carrying unit 205. The carrying device 20 will decide to shoot each frame according to the field of view (FOV) of the lens of the photographing device 22 The rotation angle of the image. The controller 201 obtains an instruction for controlling the rotation driving unit 203 to set the rotation angle of the rotation driving unit 203 for each shot.

The shooting device 22, such as a mobile phone, a tablet computer, a camera, and other electronic devices, shoots multiple images of a scene in multiple shooting directions through a lens, and includes a processing unit 221 for processing image signals and commands, and a storage image signal and other information The memory unit 223, the image capturing unit 225 and the lens unit 227 for performing shooting and image processing, and includes a communication unit 229 for transmitting messages to the outside, and the communication unit 229 executes communication protocols such as WiFi ^TM and Bluetooth ^TM .

In an embodiment, the carrier device 20 and the camera device 22 can communicate. For example, when the angle of rotation of the carrier device 20 is set each time, when the rotation drive unit 203 obtains the rotation to the set angle, the controller 201 may generate a shooting instruction and transmit it to it through the communication unit 229 of the shooting device 22 for processing. The unit 221 indicates the timing of the shooting; and after the shooting is completed, the processing unit 221 generates a shooting completion instruction, which is transmitted to the controller 201 of the carrier device 20 via the communication unit 229, and the controller 201 controls the rotation driving unit 203 to rotate to the next shooting Angle, or stop turning after shooting. After the shooting device 22 receives the shooting instruction, the processing unit 221 controls the image capture unit 225 to perform shooting, and the acquired image signal can be stored in the memory unit 223 first, and can be transmitted through the communication unit 229.

Through the above shooting method, multiple wide-angle images can be obtained. If it is a fisheye lens, the image shot is generally a circular image with a deformed periphery. A schematic diagram of the embodiment is shown in Figure 3. These circular images can be combined into an equidistant Equirectangular Projection.

In this schematic diagram, it is necessary to stitch multiple wide-angle images to take a panoramic view of a certain scene with a camera, such as the first wide-angle image 301, the second wide-angle image 302, and the third wide-angle image 303. These three wide-angle images are respectively Cover images from three shooting angles of a certain scene, and then perform stitching through image processing technology, including correction, to form a photo suitable for human eyes. In this example, the first wide-angle image 301, the second wide-angle image 302, and the second wide-angle image are formed. The three wide-angle images 303 are calibrated and spliced to obtain an equidistant columnar projection map covering the left and right 360 degrees and the vertical 180 degrees field of view.

According to the panoramic image forming method proposed in the disclosure, before the panoramic image is formed, the embodiment of the preparation process for shooting the panoramic image shown in FIG. 4 may be performed first.

In this preparation process, the system obtains the camera field of view (FOV) of the camera (step S401) and the panoramic shooting range (step S403) through the software program in the camera or the software program on the server side, thereby determining each For the direction of the second shooting, if the carrying device of the above embodiment drives the shooting device to shoot, the field of view determines the rotation angle of each shooting (step S405), and then the system determines the direction or rotation angle of each shooting to perform shooting, and Since the system knows the direction of each shooting and the field of view of the lens, the system can accurately obtain information about the overlap of each image (step S407).

In the shooting process, in addition to single-frame shooting, the shooting device can assist in obtaining high-quality images with High Dynamic Range Imaging (HDRI or HDR). In the field of digital imaging, high dynamic range imaging is used to obtain more One of the purposes of the bit-matrix image technology shot by Zhang is to increase the dynamic range of exposure. One of the purposes is to correctly represent the image. In the panoramic image forming method proposed in the disclosure, multiple images taken by high dynamic range imaging can be processed individually, and finally combined into a single panoramic image.

For the main steps of the method embodiment of forming a panoramic image, refer to the flow shown in FIG. 5.

In this panoramic image forming method, at the beginning, as in step S501, a certain scene is photographed by a photographing device, wherein the method can use the device shown in FIG.

In step S503, since the system knows the camera lens to obtain the view of the image Field, that is, the shooting range of each image is known. According to the requirements of the panorama, the software program in the system (executed on the server or in the shooting device) can obtain specific image characteristics between adjacent images to determine the overlap area. For example, preliminary judgment of the objects in the features such as color scale and texture. The image processing method is used for each image, or each image obtained through the aforementioned high dynamic range imaging (HDR), for example, multiple images with a certain shooting direction image -1, 0, +1 exposure value , Analyze, process and process each image to meet the visual needs of the panorama and temporarily store it in the system's memory (such as the memory in the shooting device).

Then, according to an embodiment, in step S505, after feature extraction, an edge and corner detection method can be applied to determine the edge image feature and corner image feature. feature), or one of them, and then in step S507, each image is randomly sampled using a specific algorithm. These boundary and corner features are mainly used to determine the location of walls and floors in the scene (especially indoors). In other cases, it may be judged that there is no boundary image feature or corner image feature in the image, which can generally be judged as an outdoor scene.

It is worth mentioning that when the boundary image features and corner image features in the image are obtained, the random sampling is performed by a random sample consensus (RANSAC) algorithm from multiple images. This random sampling consensus algorithm mainly uses the iterative method to estimate the parameters of the mathematical model from a set of data that contains outliers and other data that is not suitable for modeling, and calculates until the outliers do not affect the estimation Value. The outliers can refer to noise, so the random sampling uniform algorithm can obtain parameters that are not interfered by noise and can be used to describe a specific mathematical model. Therefore, in the panoramic shooting method proposed in the disclosure book, the random sampling step adopts the random sampling consensus algorithm to obtain features that can be used to describe the panoramic image without being disturbed by noise.

According to one of the embodiments, the method of deriving corners and edges according to image features is as follows. For the corner detection method (corner detection), several conventional techniques can be used. For example, in a Moravec corner detection algorithm, First define the "corner" in the image, which is the point in the image with low degree of similarity to each other. You can check all the pixels in the image and compare the similarity between each pixel and its surrounding pixels. The similarity is corresponding to the two ranges of images The sum of squared differences of pixels, the smaller the sum of squared differences, the higher the similarity. In this way, a pixel in a certain range without corners has a high similarity to its surrounding pixels; conversely, a pixel in a certain range has a low similarity to its surrounding pixels, indicating that the pixels in the range are not similar. When the whole image has been calculated, some of the most dissimilar areas can be regarded as "corners."

Another algorithm for finding corners is a Harris & Stephens method. This method includes first obtaining an image in the scene, seeking the pixel gradient change along a specific direction in the image, and applying a window function, such as a Gaussian Gaussian window function, for the gradient near the center of the image, and calculates the sum of squared errors of each image data. After calculating the gradients in both directions, if the two values are large enough, it means that the gradients have two main The direction means that there are two straighter edges, and it is judged that there is a corner.

For the edge detection method, generally speaking, it is to find the part of the image where the gray-scale pixels have drastic changes. Take a two-dimensional image as an example, calculate the gray-scale change gradient of the image, and when it is found that a certain pixel is in a certain When there is a high gradient change in each direction (compared to a threshold), it will be judged as a boundary. Common boundary detection algorithms include Sobel, Prewitt, Laplacian, etc. To cite an example, in the operation, the noise can be eliminated first (such as using Laplace Filter), and then the Sobel operator can be used for edge detection, and after obtaining the approximate value of the horizontal and vertical brightness difference, determine the horizontal or vertical boundary.

After detecting the boundary and corner features, because the overlap area between each image can be obtained and the features of the overlapped image can be obtained with the feature extraction, in step S509, a preliminary synthesis is performed, for example, the wide-angle image shown in FIG. 3 Convert into equidistant histogram (Equirectangular Projection), if shooting with high dynamic range imaging, it is necessary to convert the high dynamic range imaging images one by one (step S511). At this time, the composite image The field of view has covered the horizon +/- 180° each, and the vertical +/- 90° each.

For example, if you use a circular fisheye lens to shoot a scene, first look forward and shoot once to get a single image with a field of view (FOV) of 180 degrees, and then shoot 2 to Three images can be synthesized into a panoramic image of the isometric columnar projection type through the software program. The splicing and synthesis of multiple images can avoid the problems of peripheral light and shadow and image fusion defects caused by the defects of the shooting device and the lens, which involves optics. The variables of lens and image processing capabilities will not be repeated here.

In step S513, the preliminary synthesized image is compared with the images in a gallery in order to obtain the attributes of the aforementioned equidistant histogram, which is also one of the purposes of preliminary synthesis. After that, according to the attributes of the equidistant columnar projection image, the system can determine the pixel feature, the boundary image feature and the corner image feature of the multiple overlapping regions, or one of them, to synthesize a panoramic image, as in step S515.

According to one of the embodiments, in the step of comparing the images in the library to obtain the attributes of the equidistant histogram, a deep gyro-integral neural network (Convolutional Neural Network, CNN) may be used to identify the equidistant histogram. The inner object is used to determine the attributes of the isometric columnar projection map, for example, it is determined that the scene is an indoor scene or an outdoor scene, and includes the corrected image. It is mentioned here that the deep-back integral neural network (CNN) is an image deep learning technology, which can be used for image recognition by constructing an image deep learning model. Therefore, the disclosure book proposes to form a panoramic image In the method, the deep integration-like neural network can be used to optimize the corner and boundary images in the figure, and can optimize the exposure value and saturation value.

At this time, the implementation example shown in FIG. 6 is the process of the embodiment of the method for forming a panoramic image. The synthesis can be performed according to the determined scene attributes, such as step S601. In this software program, an isometric histogram is obtained, such as step S603. After comparing the image library, it is determined whether the isometric histogram is indoor or outdoor (step S605).

If it is judged that the equidistant histogram is indoor, the image is synthesized by the above-mentioned judgment result of detecting the corners and boundaries of each wide-angle image, as in step S607, it can be optimized and synthesized based on the indoor deep-integrated neural network (CNN) learning data Image of In step S611, the exposure value and the saturation value of the panoramic image are obtained using the CNN learning data, and the exposure value and the saturation value of the image synthesized by the high dynamic range imaging (HDR) sub-image are corrected.

Conversely, if it is determined that the equidistant columnar projection image is outdoor, step S609 is executed to synthesize the panoramic image according to the pixel features in the multiple overlapping regions (such as the determined object, and the features such as color scale and texture in the image).

Among them, it is worth mentioning that if each image is shot with high dynamic range imaging, multiple sub-images with different exposure values corresponding to each image are obtained. In the synthesis step, these sub-images are captured and overlapped respectively. The pixel features of the area, the boundary image features or corner image features are converted into equidistant columnar projection images and the judgment attributes, and the final panorama image is synthesized from multiple sub-images with different exposure values.

To sum up, the panoramic image formation method disclosed in the disclosure book first obtains multiple images of a certain scene. When stitching these images, it is necessary to capture the pixel features in multiple overlapping areas between multiple images, and then according to the pixels Features, the boundary image features and corner image features are judged, and then the attributes of the scene can be judged. Therefore, in the synthesis process, a better synthetic image can be obtained for the features such as color scale and texture.

However, the above descriptions are only the preferred and feasible embodiments of the present invention. Therefore, the patent scope of the present invention is not limited. Therefore, all equivalent structural changes made by using the description of the present invention and the contents of the diagrams are included in the present invention in the same way. Within the scope, together to Chen Ming.

S501‧‧‧Shooting a scene

S503‧‧‧Capturing pixel features

S505‧‧‧Detect corners and borders

S507‧‧‧random sampling

S509‧‧‧Preliminary synthesis

S511‧‧‧Converted to equidistant columnar projection

S513‧‧‧Judging attributes

S515‧‧‧Composite panorama

Claims (7)

A method for forming a panoramic image includes: obtaining multiple images of a scene; capturing pixel features in multiple overlapping regions between the multiple images; and judging boundary image features and corners in the multiple overlapping regions according to the pixel features Image feature, or one of them, or it is judged that there is no boundary image feature or corner image feature; the multiple images are preliminarily synthesized and converted into an equidistant histogram; a deep-integrated neural network is used Identify the objects in the equidistant histogram and compare the images in a library to obtain the properties of the equidistant histogram; and according to the properties of the equidistant histogram, determine that the scene is an indoor scene , Or an outdoor scene, and judging the pixel features, boundary image features, and corner image features of the multiple overlapping areas, or one of them, to synthesize a panoramic image; wherein, if the scene is judged to be the indoor scene, according to the multiple The boundary image feature and the corner image feature in the overlapping area, or one of the two, synthesize the panoramic image; if the scene is judged to be the outdoor scene, synthesize the panoramic image according to the pixel characteristics in the multiple overlapping areas. The panoramic image forming method according to claim 1, wherein a camera is used to shoot the scene through a lens to obtain the plurality of images, and the overlapping area is between two adjacent images in the plurality of images. The panoramic image forming method according to claim 2, wherein, in the step of determining whether the multiple overlapping regions have boundary image features and corner image features, firstly extract features from the multiple images, and determine whether there are Boundary image feature or corner image feature. The panoramic image forming method according to claim 3, wherein sampling is performed by a random sampling consensus algorithm to obtain features that overlap pixel features and obtain features that can be used to describe the panoramic image without being disturbed by noise . The panoramic image forming method according to any one of claims 1 to 4, wherein when multiple images of the scene are obtained, each image is shot in a high dynamic range imaging mode, and the multiple images corresponding to each image are obtained. Sub-images with different exposure values; optimizing the exposure value and saturation value of the panoramic image with the learning data of the deep integrator-like neural network, and then synthesize multiple sub-images obtained by imaging with the high dynamic range. A system for forming a panoramic image includes: a photographing device installed on a rotatable carrier device, and multiple images of a scene are photographed in multiple photographing directions through a lens, including a processing unit for executing a panoramic image The forming method includes: obtaining multiple images of the scene; capturing pixel features in multiple overlapping regions between the multiple images; judging boundary image features and corner images in the multiple overlapping regions according to the pixel features Features, or one of them, or it is judged that there are no boundary image features or corner image features; the multiple images are preliminarily synthesized and converted into an equidistant histogram; identified by a deep-back integral neural network Compare the objects in the equidistant histogram projection map and compare the images in a library to obtain the properties of the equidistant histogram projection map; and according to the properties of the equidistant histogram projection map, determine that the scene is an indoor scene, Or an outdoor scene, and judging the pixel features, boundary image features, and corner image features of the multiple overlapping areas, or one of them, to synthesize a panoramic image; wherein, if the scene is judged to be the indoor scene, based on the multiple overlapping areas The boundary image feature and the corner image feature in the area, or one of the two, synthesize the panoramic image; if the scene is judged to be the room In the external scene, the panoramic image is synthesized according to the pixel features in the multiple overlapping regions. The system for forming a panoramic image according to claim 6, wherein the supporting device determines the rotation angle of each image according to the shooting field of view of the lens of the shooting device.

Images