TWI569641B - Image capturing method and electronic apparatus - Google Patents

Description

Image capturing method and electronic device

The present disclosure relates to an image capture method, and more particularly to an image capture method for generating a panoramic image.

Recently, electronic devices such as digital cameras, smart phones, and tablet computers have become popular items in modern life, so that users can conveniently record scenes or important events around them. However, the photographic lens on the electronic device usually has certain restrictions (for example, the wide-angle end can only accommodate a certain imaging range). When users want to take spectacular natural scenery, huge buildings, crowds in circular squares, large-scale artistic creations, or when taking group photos, they often encounter the inability to accommodate the entire picture in the same photo.

In high-end photographic equipment, it may be possible to replace the super wide-angle lens or use a special fisheye lens to capture a larger angle of view, but the general digital camera, smart phone and tablet cannot change the lens or camera module at will. .

Recently, a solution for realizing panoramic photography (PANORAMA IMAGE) photography by software processing has been developed. Generally, when entering the panoramic image shooting mode, a plurality of images are manually taken by a user, and then multiple images are joined to form a panoramic image.

At present, it is common practice for the user to adjust multiple image viewing positions by themselves, and then image processing and joining by a photographing device or a computer to generate a panoramic image. However, current users need certain shooting skills. The user decides how many angles should be rotated and then shoots the next image. Otherwise, the processing of image bonding is difficult (for example, it is impossible to judge between two images to be joined). The relative relationship), or the overall effect of the image after bonding is not as expected (for example, the image joint is obviously unnatural or has errors).

According to an embodiment of the present disclosure, an image capture method for an electronic device is disclosed. The image capture method includes: calculating a plurality of feature points in a first image frame captured by an electronic device; The distribution of the feature points calculates a maximum rotation angle when the electronic device rotates in one direction; and automatically captures the second image frame when the electronic device has rotated the maximum rotation angle in the direction.

According to another embodiment of the present disclosure, a method for generating a panoramic image for an electronic device is disclosed. The method for generating a panoramic image includes: capturing a first image frame; calculating a plurality of feature points in the first image frame; a distribution of the feature points in an image frame, and calculating a maximum rotation angle when the electronic device rotates in one direction; when the electronic device is to the side When the rotation has reached the maximum rotation angle, the second image frame is automatically captured, and the first image frame and the second image frame are used to be joined as a panoramic image.

According to another embodiment of the present disclosure, an electronic device includes a dynamic sensor, an image capturing unit, a display unit, and a processing unit. The dynamic sensor is used to detect the axial direction of the electronic device. The display unit is used to display the user interface. The processing unit is coupled to the dynamic sensor, the image capturing unit, and the display unit.

When the image capturing unit captures the first image frame, the processing unit is configured to calculate a plurality of feature points in the first image frame, and calculate a maximum when the electronic device rotates in one direction according to the distribution of the feature points in the first image frame. Rotate the angle and indicate the relative position of the maximum rotation angle on the user interface.

When the dynamic sensor detects that the electronic device has rotated to the maximum rotation angle in the direction, the processing unit is configured to drive the image capturing unit to capture the second image frame, and the first image frame and the second image frame are used to be combined into the panoramic image. .

As described in the above embodiments, the present disclosure provides an image capturing method, a panoramic image generating method, and an electronic device, which can calculate a distribution of feature points according to a current image frame, and obtain a feature point distribution according to each direction boundary. Knowing the maximum rotation angle of each direction, the user can quickly rotate the electronic device to the optimal position, ensure the effect of image bonding, and effectively use less image capturing times to form a panoramic image covering the maximum viewing angle range.

In order to make the disclosure more obvious, the attached symbols are as follows:

100‧‧‧Image acquisition method

200‧‧‧Electronic devices

220‧‧‧Dynamic sensor

240‧‧‧Image capture unit

260‧‧‧ display unit

280‧‧‧Processing unit

ZONE1‧‧‧right screen

ZONE2‧‧‧left screen

IMG1‧‧‧ first image screen

IMG2R, IMG2L, IMG2T, IMG2B‧‧‧ second image screen

FP‧‧‧ feature points

AX0‧‧‧ initial axial direction

AX1, AX2, AX3, AX4‧‧‧ axial

OL1, OL2, OL3, OL4‧‧‧ overlap

θ 1, θ 2, θ 3, θ 4‧‧‧ maximum rotation angle

TO, TR, TL, TT, TB‧‧‧ focus frame

S102~S116‧‧‧Steps

In order to make the present invention more obvious and understandable, the description of the drawings is as follows: FIG. 1 is a flowchart of a method for capturing an image according to an embodiment of the present disclosure; FIG. 2 is a diagram showing a method according to the present disclosure. A functional block diagram of an electronic device in an embodiment; FIG. 3A is a schematic diagram of a first image frame captured by an electronic device according to an embodiment of the present disclosure; and FIG. 3B is a first image of FIG. 3A A schematic diagram of feature points in the picture; FIG. 4 is a schematic diagram showing a second image frame rotated and photographed in different directions; and FIG. 5 is a schematic diagram showing a second image frame rotated and photographed in different directions.

The spirit and scope of the present disclosure will be apparent from the following description of the preferred embodiments of the present disclosure. Modifications do not depart from the spirit and scope of the disclosure.

Please refer to FIG. 1 , which is a flowchart of a method for capturing an image 100 according to an embodiment of the present disclosure. In this embodiment, the image capturing method 100 can be used to generate a panoramic image. That is to say, the plurality of images captured by the image capturing method 100 can be stitched to each other as a panoramic image corresponding to a larger viewing angle.

The image capturing method 100 in this embodiment can be applied to an electronic device. Referring to FIG. 2, a functional block diagram of an electronic device 200 according to an embodiment of the present disclosure is shown. As shown in FIG. 2 , the electronic device 200 includes a dynamic sensor 220 , an image capturing unit 240 , a display unit 260 , and a processing unit 280 . The processing unit 280 is coupled to the dynamic sensor 220, the image capturing unit 240, and the display unit 260.

The image capturing unit 240 can include a lens, an optical module, and an image sensing unit, which are well known to those skilled in the art. The image capturing unit 240 can be disposed on one side surface of the electronic device 200. , shoot the image in the direction of the framing. The electronic device 200 can be a digital camera, a digital SLR camera, a digital cameraless camera, a camera phone, a smart phone, or other equivalent electronic device having an image capture unit 240.

The dynamic sensor 220 is configured to detect an axial direction of the electronic device 200, for example, a horizontal deflection vector of the body of the electronic device 200, a vertical deflection vector, and a deflection angle of at least one of the magnetic north. In other words, the dynamic sensor 220 can know the framing axis that the image capturing unit 240 faces when capturing the image frame; in addition, the dynamic sensor 220 can also detect the instant image during the framing process. The image capturing unit 240 is in the immediate axial direction of the pair. In practical applications, the dynamic sensor 220 includes at least one of a gyro sensor, an electronic compass, and a gravity sensor (G-sensor).

The display unit 260 is configured to display a user interface (UI), and the user interface can display the image captured by the image capturing unit 240, the live view screen during the framing process, and the related information of the shooting process (eg, Such as aperture, shutter, power, time, focus, etc.) and other tips for use.

As shown in the embodiment of FIG. 1 , the image capturing method 100 first performs step S102 , and the image capturing unit 240 captures the first image frame. In this embodiment, when the first image frame is captured, the image capturing method 100 executes step S104, and uses the dynamic sensor 220 to obtain the initial axial direction of the electronic device 200, and the three-dimensional axis of the electronic device 200 when the first image frame is captured. to.

Next, the image capturing method 100 executes step S106 to calculate a plurality of feature points in the captured first image frame. Please refer to FIG. 3A and FIG. 3B together. FIG. 3A is a schematic diagram of a first image frame IMG1 captured by the electronic device 200 according to an embodiment of the present disclosure, and FIG. 3B is a first diagram of FIG. 3A. A schematic diagram of a feature point FP in the image screen IMG1. As shown in FIG. 3A, the first image frame IMG1 captured in step S106 may contain a plurality of objects such as a character, a background, a vehicle, a tree or various objects, shapes, color patches, and the like. The processing unit 280 can perform image processing according to the content of the first image frame IMG1 to find a plurality of feature points FP in the first image frame IMG1, and the first image frame IMG1 includes a plurality of feature points FP. The distribution of the point FP is as shown in Fig. 3B.

In the practical application of image processing, there are many ways to identify feature points in image content, such as face feature recognition, fingerprint recognition, edge detection, corner detection, block detection, ridge detection, etc. The algorithm for extracting feature points in the image, in this embodiment, the feature point FP can be found by the first image frame IMG1 in combination with a plurality of algorithms, and the feature points FP depicted in the 3A and 3B are mainly Example of edge detection algorithm An illustrative description, but the disclosure is not limited to a particular algorithm.

In addition, the present disclosure exemplifies the position of the feature pattern obtained by the image processing by using the feature point FP, but the feature point FP is not limited to a single point or a single coordinate, and may also be an image composed of multiple pixels. Block or image feature range, etc.

The image capturing method 100 executes step S108 to calculate a maximum rotation angle when the electronic device 200 rotates in a specific direction according to the distribution of the feature points FP in the first image frame IMG1. The following paragraphs will explain in detail how to calculate the maximum rotation angle from the distribution of the feature points FP in the first image frame IMG1.

As shown in the examples shown in FIG. 3A and FIG. 3B, the right image ZONE1 of the original image frame IMG1 has less variation, and the objects are more dispersed or distinguishable; the opposite left side image ZONE2 changes more, and the object More concentrated or distinguishable features. The feature point FP density of the right picture ZONE1 in the first video picture IMG1 is low, and the feature point FP density of the left picture ZONE2 is high.

When the user rotates the electronic device 200 and tries to capture another image frame, and joins another image frame with the original image frame (ie, the first image frame IMG1), the two image frames are combined into a panoramic image with a wide viewing angle range. The user must decide how many angles the electronic device 200 needs to be rotated.

If the angle of the user's rotation is too large, and the portion where the other image frame overlaps with the original image frame is too small (even without any overlap), the electronic device 200 cannot find the correspondence between the two images through the fraction of the image processing. Therefore, effective image stitching cannot be performed. If If the angle of the rotation is too small, the portion where the other image frame overlaps with the original image frame is too large (even if the difference between the two image angles is not large), the electronic device 200 needs to take a new image multiple times to cover the new viewing angle. Therefore, The user may need to take a lot of images and repeat the image bonding many times, which will result in inefficient processing.

Please refer to FIG. 4 together, which shows a schematic diagram of the second image frames IMG2R, IMG2L rotated and photographed in different directions. In the above step S108, the calculation of the maximum rotation angle when rotating in a specific direction is based on the feature point density in the vicinity of the side boundary corresponding to the specific direction in the first video screen IMG1.

Assume that the user will rotate to the right to take the next image frame to engage with the first image frame IMG1. Step S108 further includes calculating the number of feature points FP covered by the overlapping portion OL1 of the first image frame IMG2R in the first image frame IMG1 (as shown in FIG. 4), and making the first image frame IMG1 The number of feature points FP covered by the overlapping portion OL1 of the second image frame IMG2R is above the threshold. It is assumed that the threshold value of the feature point FP is at least ten feature points in this embodiment, and the processing unit 280 of the electronic device 200 needs ten The feature points can perform effective image bonding with the second image frame IMG2R in the first image frame IMG1.

In this way, the processing unit 280 can determine how many overlapping portions OL1 that need to be large (horizontal width) in the vicinity of the side boundary in the right direction of the first image frame IMG1 to have ten feature points. Further, from the size of the overlapping portion OL1, the maximum rotation angle θ1 when rotating in the right direction and the second image screen IMG2R corresponding to the maximum rotation angle θ1 are obtained. Where the location is.

On the other hand, assume that the user will rotate to the left to take the next image frame to be engaged with the first image frame IMG1. Step S108 further includes calculating the number of feature points FP covered by the overlapping portion OL2 of the first image frame IMG2L in the first image frame IMG1 (as shown in FIG. 4), and making the first image frame IMG1 The number of feature points FP covered by the overlapping portion OL2 of the second image frame IMG2L is at least ten feature points.

In this way, the processing unit 280 can determine the number of overlapping portions OL2 that need to be large (horizontal width) in the vicinity of the side boundary in the left direction of the first image frame IMG1 to have ten feature points. Further, from the size of the overlapping portion OL2, the maximum rotation angle θ 2 when rotating in the right direction is obtained, and the position of the second image screen IMG2L corresponding to the maximum rotation angle θ 2 is estimated.

As shown in FIG. 4, the density of feature points near the side boundary of the first image frame IMG1 in the right direction is low. Therefore, the required overlap portion OL1 is large to meet the minimum threshold of image bonding, and therefore, step S108 is calculated. The maximum rotation angle θ 1 obtained to rotate to the right side is small. On the other hand, the density of the feature points near the side boundary of the first image frame IMG1 in the left direction is high. Therefore, the required overlapping portion OL2 is small to meet the minimum threshold of image bonding, and therefore, step S108 is calculated. The maximum rotation angle θ 2 of the left rotation is large. In this embodiment, the maximum rotation angles θ 1 and θ 2 are positively correlated with the feature point density near the right and left boundaries of the first image frame IMG1.

In the embodiment of FIG. 4 , the image capturing method 100 and the electronic device 200 can calculate different maximum rotation angles in the horizontal rotation direction (rotating to the right or to the left), and ensuring the horizontal direction. The overlapping portions of the two image frames IMG2R, IMG2L and the first image frame IMG1 have sufficient feature points FP, but the present disclosure is not limited to the horizontal direction. Please refer to FIG. 5 together, which shows a schematic diagram of the second image frames IMG2T and IMG2B rotated and photographed in different directions.

As shown in FIG. 5, the density of the feature points near the side boundary of the first image frame IMG1 toward the top direction is high. Therefore, the required overlapping portion OL3 is small to meet the minimum threshold of image bonding, and therefore, step S108 It is calculated that the maximum rotation angle θ 3 that is rotated upward is large. On the other hand, the density of the feature points near the side boundary of the first image frame IMG1 toward the bottom direction is low, and therefore, the required overlap portion OL4 is larger than the minimum threshold for image joint, and therefore, step S108 is calculated downward. The maximum rotation angle θ 4 of the rotation is small. In this embodiment, the maximum rotation angles θ 3 and θ 4 are positively related to the feature point density near the top and bottom boundaries of the first image frame IMG1.

In the embodiment of FIG. 4 and FIG. 5, the first image frame IMG1 includes a plurality of side boundaries, and the step S108 of the image capturing method 100 can calculate the left and right levels according to the feature point density near the side boundaries. The respective maximum rotation angles when the directions are rotated in the vertical direction of the upper and lower directions, and each of the maximum rotation angles is positively correlated with the density of the feature points near the respective side boundaries.

After the calculation of the maximum rotation angle is completed (ie, step S108 is completed). Thereafter, the image capturing method 100 can perform step S110, and through the display unit 260 of the electronic device 200, prompt the maximum rotation angle θ 1 , θ along a specific direction (such as right, left, up, or down) on the user interface. 2. θ 3 and/or θ 4 .

For example, when the electronic device 200 captures the first image frame IMG1 along the initial axis AX0, the center point of the first image frame IMG1 is located in the focus frame TO. After the calculation of the maximum rotation angles θ 1 , θ 2 , θ 3 , and θ 4 is completed, the target focus frames TR and TL corresponding to the maximum rotation angles θ 1 , θ 2 , θ 3 , and θ 4 are obtained (as shown in FIG. 4 Show) with TT, TB (as shown in Figure 5).

Then, the user can refer to the above prompt and rotate the axial direction of the electronic device 200 in the horizontal direction or the vertical direction. During the process of rotating the electronic device 200, the image capturing method 100 performs step S112 to detect the instantaneous axial direction of the electronic device 200 by using the dynamic sensor 220.

Then, the image capturing method 100 can perform step S114, using the difference between the instantaneous axial direction sensed by the dynamic sensor 220 and the initial axial direction (see step S104), thereby determining the electronic device 200 to a specific direction (eg, right, left, Whether the rotation of the up or down has reached the maximum rotation angles θ 1 , θ 2, θ 3 and/or θ 4 . For example, when the instantaneous axial direction sensed by the dynamic sensor 220 is the axial direction AX1, the difference between the instantaneous axial direction AX1 and the initial axial direction AX0 is used to determine that the electronic device 200 has been rotated to the right by the maximum rotation angle θ 1; It can be judged according to the difference between the axial direction AX2, the axial direction AX3, the axial direction AX4 and the initial axial direction AX0.

If it is determined in step S114 that the current axial direction of the electronic device 200 has not reached the maximum rotation angle θ 1 , θ 2 , θ 3 , and/or θ 4 in a specific direction, Then, returning to step S112, the monitoring of the instantaneous axial direction of the electronic device 200 is continued.

If it is determined in step S114 that the current axial direction of the electronic device 200 has reached the maximum rotation angles θ 1 , θ 2, θ 3 and/or θ 4 in the specific direction, the image capturing method 100 executes step S116 to capture the second image frame. In some embodiments, when the instantaneous axial direction has reached the maximum rotation angle in a specific direction, the electronic device 200 automatically triggers the shooting of the second image frame without additional user instructions, thereby improving the convenience of operation. As shown in Fig. 4, if the maximum rotation angle θ 1 has been reached, the second image frame IMG2R is automatically captured, and if the maximum rotation angle θ 2 has been reached, the second image frame IMG2L is automatically captured; as shown in Fig. 5, if When the maximum rotation angle θ 3 is reached, the second image frame IMG2T is automatically captured, and if the maximum rotation angle θ 4 has been reached, the second image frame IMG2B is automatically captured.

The first image frame IMG1 and the second image frame IMG2R, IMG2L, IMG2T or IMG2B may be used to be joined as a panoramic image.

In an embodiment, the image capturing method 100 can further join the second image frame IMG2R, IMG2L, IMG2T or IMG2B with the first image frame IMG1, thereby forming a panoramic image. In addition, the generation of the panoramic image is not limited to the joining of the two image images, and the image capturing method 100 can be cyclically performed to realize the joining of three, four or more images to form a panoramic image with a wider viewing angle, and the cycle is performed. The method can be known from the joining of the above two images, and will not be further described herein. It should be particularly noted that after the two joined images are formed, the image capturing method 100 may generate a new feature point distribution according to the combined image, by using the side borders of the new feature point distribution. The feature point density calculates the new maximum rotation angle of each party. In addition, the images captured multiple times are not limited to the same direction of rotation. For example, you can first grab the next image frame, then upload another image frame, and then drag it to the right. Image image.

In addition, the image capturing method 100 is not limited to performing image bonding immediately after capturing two image frames. In another embodiment, the image capturing method 100 for generating a panoramic image can continuously capture multiple image images (such as two image images, three image images or more image frames) or when the user images all the images. The image is joined once after the shooting is completed.

In addition, the dynamic sensor 220 will sense the immediate axial direction of the electronic device 200 as the user rotates the electronic device 200. At the same time, displaying the instant viewfinder screen on the user interface of the display unit 260 will display the center focus point on the instant axis. When the user rotates to the right to approach the maximum rotation angle θ 1 , the target focus frame TR will appear in the user interface of the display unit 260 , and the user only needs to focus the target focus frame TR on the user interface to center the focus point. By aligning the target focus frame TR, it is ensured that the electronic device 200 is accurately rotated to a predetermined maximum rotation angle θ 1 . Similarly, when rotating in other directions, the target focus frame TL, the target focus frame TT and the target focus frame TB can be used to prompt the user to accurately rotate the electronic device 200 to a predetermined maximum rotation angle θ 2, θ 3 and θ. 4.

In summary, the present disclosure provides an image capturing method, a panoramic image generating method, and an electronic device, which can calculate a distribution of feature points according to a current image frame, and learn from the distribution of feature points on each direction boundary. The maximum rotation angle of each direction is convenient for the user to quickly The electronic device rotates to the optimal position to ensure the image joint effect, and can effectively use the fewer image capture times to form a panoramic image covering the maximum viewing angle range.

Although the present invention has been disclosed in the above embodiments, it is not intended to limit the present case. Anyone skilled in the art can make various changes and refinements without departing from the spirit and scope of the present case. The scope defined in the patent application is subject to change.

100‧‧‧Image acquisition method

S102~S116‧‧‧Steps

Claims (10)

An image capture method for an electronic device, the image capture method includes: calculating a plurality of feature points in a first image frame captured by the electronic device; and distributing the feature points according to the first image frame Calculating a maximum rotation angle when the electronic device rotates in one direction; and automatically capturing a second image frame when the electronic device rotates in the direction to reach the maximum rotation angle. The image capturing method according to claim 1, wherein the calculation of the maximum rotation angle when rotating in the direction is based on a feature point density near a boundary of one side of the corresponding image in the first image frame. And the maximum rotation angle is positively related to the feature point density near the side boundary. The image capturing method of claim 2, wherein the calculating of the maximum rotation angle comprises calculating a feature point of the first image frame that overlaps with a predetermined portion of the second image frame that is captured. The number, and the number of feature points covered by the overlapping portion in the first image frame is above a threshold. The image capturing method of claim 1, wherein the first image frame includes a plurality of side boundaries, and the image capturing method calculates the left and right levels according to the feature point density near the side boundaries. The respective maximum rotation angles when the directions are rotated in the vertical direction of the upper and lower directions, and each of the maximum rotation angles is positively correlated with the density of the feature points near the respective side boundaries. The image capturing method of claim 1, further comprising: obtaining an initial axial direction of the electronic device when the first image is taken; detecting an immediate axial direction of the electronic device; and utilizing the instant A gap of the axial direction relative to the initial axis determines whether the electronic device has rotated in the direction to reach the maximum rotation angle. The image capturing method of claim 1, after the calculation of the maximum rotation angle is completed, further comprising: prompting the maximum rotation angle along the direction on a user interface of the electronic device. The image capture method of claim 1, wherein the first image frame and the second image frame are used to join a panoramic image. An electronic device includes: a dynamic sensor for detecting an axial direction of the electronic device; an image capturing unit; a display unit for displaying a user interface; and a processing unit coupled to the a dynamic sensor, the image capturing unit, and the display unit, wherein when the image capturing unit captures a first image frame, the processing unit is configured to calculate a plurality of feature points in the first image frame, according to the The distribution of the feature points in the first image frame calculates a maximum rotation angle when the electronic device rotates in one direction, and prompts the maximum on the user interface. The relative position of the rotation angle, wherein the processing unit is configured to drive the image capturing unit to capture a second image frame when the dynamic sensor detects that the electronic device has rotated the direction of the maximum rotation angle. An image frame and the second image frame are used to be joined as a panoramic image. The electronic device of claim 8 , wherein when the image capturing unit captures the first image frame, the processing unit obtains one of the electronic devices when the first image frame is captured by the dynamic sensor Initial axial direction and record the initial axial direction. The electronic device of claim 8, wherein the dynamic sensor detects an immediate axial direction of the electronic device, and the processing unit determines the electronic device by using a gap between the instantaneous axial direction and the initial axial direction. Whether the maximum rotation angle has been reached by rotating in this direction.