TWI839241B - Image processing device and method for viewing angle conversion - Google Patents

Abstract

An image processing device for viewing angle conversion is provided, which comprises an image capturing circuit, a memory and a processor. The processor is configured for processing instructions to perform following operations: performing feature extraction on a previous image and a current image to calculate an optical flow information matrix from a first feature set of the previous image and a second feature set of the current image; estimating positions in the current image corresponding to a second key point set in the previous image by using the optical flow information matrix to store as a third key point set; calculating a current homography matrix between a first key point set and the third key point set; and projecting the target image to a corresponding position on the current image by using the current homography matrix.

Description

Image processing device and method for perspective conversion

The present disclosure relates to an image processing device and method. Specifically, the present invention relates to an image processing device and method for perspective conversion.

In general image processing for perspective conversion, in order to find the correspondence between images of different perspectives, there are two current methods: manual and automatic.

However, the manual method requires manual pre-processing of different images, which may sometimes be inaccurate due to human errors (eg, hand shaking of the user), and the manual method is also very time-consuming and labor-intensive.

In addition, although the automatic method can currently use computer vision and deep learning to obtain the homography matrix. However, the current method is very time-consuming to process an image and consumes a lot of computing resources. Because it uses deep learning, it is impossible to fine-tune the internal parameters, which may result in an incorrect homography matrix when the light and shadow scenery of the application scene is not similar to the training data.

Furthermore, the above conventional methods often need to recalculate the view angle transformation relationship (ie, a new homography matrix) from the new image when the shooting position may change, which consumes more computing resources.

Therefore, how to greatly reduce the amount of calculation when the shooting position may change and solve the problem of rapid changes in viewing angle is a problem that technicians in this field are eager to solve.

The present disclosure provides an image processing device for viewing angle conversion, including an image capture circuit, a memory and a processor. The image capture circuit is used to capture a previous image and a current image; the memory is used to store a plurality of instructions, a target image in a field template, and a first key point set in the field template; and the processor is used to process the plurality of instructions to execute the following operations: performing feature extraction operations on the previous image and the current image to calculate an optical flow information matrix from a first feature set of the previous image and a second feature set of the current image; using the optical flow information matrix to infer the position of the second key point set in the previous image in the current image to store it as a third key point set; calculating a current homography matrix between the first key point set and the third key point set; and using the current homography matrix to project the target image to a corresponding position on the current image.

The present disclosure also proposes an image method for perspective conversion, which includes: shooting a previous image and a current image, and storing a target image in a field template and a first key point set in the field template; performing feature extraction operations on the previous image and the current image to calculate an optical flow information matrix from the first feature set of the previous image and the second feature set of the current image; using the optical flow information matrix to infer the position of the second key point set in the previous image in the current image to store it as a third key point set; calculating a current homography matrix between the first key point set and the third key point set; and using the current homography matrix to project the target image to a corresponding position on the current image.

Based on the above, the image device and method for view angle conversion provided by the present disclosure can use the optical flow method to update the homography matrix, so as to greatly reduce the amount of calculation for regenerating the homography matrix. In addition, the homography matrix can be updated in real time between different frames, and this approach greatly increases the smoothness after projecting onto continuous images.

In view of the current general image processing for perspective transformation, it may require a lot of manpower for pre-processing, resulting in a lot of manpower consumption or the training parameters cannot be fine-tuned in real time. In addition, since it relies on human judgment, it often causes errors in the image after perspective transformation processing.

To solve the above problems, the present disclosure proposes an image processing device and method, which greatly increases the accuracy of the calculated homography matrix and can make fine adjustments to the scene changes in real time through color gamut conversion, line search processing, line segment fusion, angle difference and distance difference methods and methods for calculating the number of overlaps and non-overlaps. In this way, this high-precision homography matrix can be used in image processing with various viewing angle changes. The above-mentioned technology of the present disclosure is specifically described by taking the following embodiments as examples.

Referring to FIG. 1 , FIG. 1 is a schematic diagram of an image processing device 100 in some embodiments of the present disclosure. As shown in FIG. 1 , the image processing device 100 includes an image capture circuit 110 , a memory 120 , and a processor 130 . The processor 130 is connected between the image capture circuit 110 and the memory 120 .

In some embodiments, the image processing device 100 may be any device having an image processor, or a device for providing image processing functions, such as a computer, a server, or a processing center, etc., which is not limited here.

In the present embodiment, the image capture circuit 110 is used to capture a field image IMG1 (e.g., an image of a tennis court or a football field). In some embodiments, the image capture circuit 110 can be implemented by various image capture devices, such as a photographic circuit or a photosensitive circuit, etc., to capture an image of any field (e.g., a tennis court, a football field, etc.) within a preset shooting range (field of view, FoV). In some embodiments, the field image IMG1 can be an image in an RGB color space. In some embodiments, the image capture circuit 110 can capture a field (e.g., a tennis court or a football field) from a specific shooting angle to generate the field image IMG1.

In addition, in this embodiment, the memory 120 is used to store multiple instructions, which are used to execute the detailed steps described in the subsequent paragraphs. In some embodiments, these instructions can be corresponding software or firmware instruction programs. In some embodiments, the memory 120 can be implemented by a memory unit, a flash memory, a read-only memory, a hard disk, or any equivalent storage component.

In addition, in this embodiment, the processor 130 is used to process these instructions and execute the detailed steps described in the following paragraphs. In some embodiments, the processor 130 can be implemented by a processing unit, a central processing unit, or a computing unit.

Referring to FIG. 2, FIG. 2 is a flowchart of an image processing method in some embodiments of the present disclosure. The image processing method is applicable to an electronic device, which may be the image processing device 100 in FIG. 1. The components in the image processing device 100 in FIG. 1 are used to execute steps S210 to S250 in the image processing method. As shown in FIG. 2, first, in step S210, a plurality of line segments are found from the field image IMG1.

In some embodiments, the field image IMG1 may be converted from an RGB color space to an image in an HSV color space, and the image in the HSV color space may be converted into a binary image according to a plurality of color intervals. Then, a line detection operation may be performed on the binary image to generate a plurality of line segments. In some embodiments, the line detection operation may be a Hough transform or a fast line detection (i.e., a HoughLines function or a FastLineDetector function in an openCV library).

In some embodiments, the plurality of color intervals include a hue interval, a saturation interval, and a brightness interval. In some embodiments, the binary image may be a black and white image. In some embodiments, a line detection operation may be performed on the binary image to generate the start point coordinates and the end point coordinates of the plurality of line segments. In some embodiments, the plurality of line segments may be marked on the field image IMG1 according to the start point coordinates and the end point coordinates of the plurality of line segments.

The generation of a binary image is described below with an actual example. Referring to FIG. 3 , FIG. 3 is a schematic diagram showing the generation of a binary image BIMG in some embodiments of the present disclosure. As shown in FIG. 3 , assuming that the hue range is 0-180, the saturation range is 0-100, and the brightness range is 180-255, the field image IMG1 can be first converted from the RGB color space to the HSV color space.

When the hue of a pixel in the image in the HSV color space is in the range of 0 to 180, the saturation of the pixel in the image in the HSV color space is in the range of 0 to 100, and the lightness of the pixel in the image in the HSV color space is in the range of 180 to 255, the pixel in the image in the HSV color space may be set to white.

On the contrary, when the hue of a pixel in the HSV color space image is not in the range of 0 to 180, or the saturation of a pixel in the HSV color space image is not in the range of 0 to 100, or the brightness of a pixel in the HSV color space image is not in the range of 180 to 255, the pixel in the HSV color space image can be set to black. In this way, the image in the HSV color space can be converted into a binary image BIMG (i.e., a black and white image).

The following is an actual example to illustrate the line segments generated on the field image IMG1. Referring to FIG. 4, FIG. 4 is a schematic diagram showing the generation of multiple line segments LS1-LS12 from the field image IMG1 in some embodiments of the present disclosure. As shown in FIG. 4, the line segments LS1-LS12 are generated from the field image IMG1 by the above-mentioned line detection operation, and the line segments LS1-LS12 are marked on the field image IMG1.

Furthermore, as shown in FIG. 2 , in step S220 , a fusion operation is performed based on a plurality of line segments to generate a plurality of fused lines.

In some embodiments, the fusion operation may include: calculating the segment angle and the maximum distance between one of the multiple line segments and another of the multiple line segments. Then, it may be determined whether the one of the multiple line segments and the other of the multiple line segments are fused according to the segment angle and the maximum distance.

In some embodiments, it may be determined whether the line segment angle is less than the angle threshold and the maximum distance is less than the distance threshold. When it is determined that the line segment angle is less than the angle threshold and the maximum distance is less than the distance threshold, one of the multiple line segments and another of the multiple line segments may be merged into one of the multiple merged lines.

Specifically, two of the multiple line segments can be randomly selected, and it is determined whether the two selected line segments can be merged. When they can be merged, a fusion operation can be performed to calculate the straight line equation of one of the merged lines. Taking FIG. 4 as an example, two of the line segments LS1 to LS12 can be randomly selected (for example, line segments LS1 and LS3 can be selected) to determine whether the two selected line segments can be merged to calculate a straight line equation. That is, 66 selections can be made on line segments LS1 to LS12 to determine whether they can be merged to calculate multiple straight line equations.

In some embodiments, one of the multiple line segments and another of the multiple line segments may be fused using any line segment fusion process. In some embodiments, the line segment fusion process may be a perceptually accurate line segment merging algorithm, a linear regression algorithm, or a line parameter averaging algorithm (e.g., calculating the average of the starting point coordinates (or end point coordinates) and the average of the slopes of two line segments to generate a line equation according to the average of the starting point coordinates (or end point coordinates) and the average of the slopes), etc.

The following is an actual example to illustrate the line segment angle and the maximum distance. Referring to FIG. 5 , FIG. 5 shows the line segment angle in some embodiments of the present disclosure. As shown in Figure 5, assume that line segments L1 and L2 are selected, and there is a line segment angle between the extended line EL of line segment L1 and line segment L2. . In this way, the angle between line segments can be determined. Is it less than the angle threshold?

In addition, to facilitate understanding of how the maximum distance of the present disclosure is calculated, please refer to FIG. 6. FIG. 6 is a schematic diagram of the maximum distance d in some embodiments of the present disclosure. It should be noted that any point on the line segment L2 to the line segment L1 corresponds to a vertical distance, wherein the vertical distance with the largest distance value is defined as the maximum distance d.

In this example, it is assumed that the processor 130 selects line segments L1-L2, and as shown in FIG6, there is a maximum distance d between line segment L1 and line segment L2. Thus, the processor 130 can determine whether the maximum distance d between line segment L1 and line segment L2 is less than a distance threshold.

The following describes multiple fusion lines using an actual example. Referring to FIG. 7 , FIG. 7 is a schematic diagram showing multiple fusion lines IL1 to IL10 in a field image IMG1 in some embodiments of the present disclosure. As shown in FIG. 7 , the fusion lines IL1 to IL10 can be marked on the field image IMG1 according to the straight line equations of the fusion lines IL1 to IL10.

Furthermore, as shown in FIG. 2 , in step S230 , a first group and a second group are distinguished by a plurality of fusion lines.

In some embodiments, a first group and a second group may be distinguished by a plurality of blend lines according to a slope threshold. In some embodiments, a blend line greater than the slope threshold may be used as a blend line for the first group, and a blend line not greater than the slope threshold may be used as a blend line for the second group.

In some embodiments, the minimum distance of the plurality of fused lines of the first group may be calculated based on the leftmost middle point of the field image IMG1, and the plurality of fused lines of the first group may be sorted according to the minimum distance of the plurality of fused lines of the first group. In some embodiments, the minimum distance of the plurality of fused lines of the second group may be calculated based on the uppermost middle point of the field image IMG1, and the plurality of fused lines of the second group may be sorted according to the minimum distance of the plurality of fused lines of the second group. Taking FIG. 7 as an example, as shown in FIG. 7, the fused lines IL1 to IL5 greater than the slope threshold and the fused lines IL6 to IL10 less than the slope threshold may be identified from the fused lines IL1 to IL10 according to the slope threshold. Thus, the fused lines IL1 to IL5 may be regarded as the first group, and the fused lines IL6 to IL10 may be regarded as the second group.

The following uses an actual example to illustrate the sorting of the first group and the sorting of the second group. Referring to FIG. 8 , FIG. 8 is a schematic diagram showing the sorting of the first group in some embodiments of the present disclosure. As shown in FIG. 8 , the leftmost middle point of the image IMG2 is used as the reference point LO, and the minimum distances LD1 to LD5 between the plurality of fusion lines VL1 to VL5 of the first group and the reference point LO are calculated. Thus, the fusion lines VL1 to VL5 can be arranged in sequence according to the minimum distances LD1 to LD5.

In addition, refer to FIG. 9, which is a schematic diagram showing the arrangement of the second group in some embodiments of the present disclosure. As shown in FIG. 9, the middle point of the uppermost side of the image IMG3 is used as the reference point UO, and the minimum distances UD1 to UD5 between the plurality of fusion lines HL1 to HL5 of the second group and the reference point UO are calculated. Thus, the fusion lines HL1 to HL5 can be arranged in sequence according to the minimum distances UD1 to UD5.

Furthermore, as shown in FIG. 2, in step S240, at least one first intersection point set is selected from the plurality of intersection points between the fusion lines in the first group and the second group, and at least one second intersection point set is selected from the scene template. In some embodiments, the scene template may be a scene map overlooking the scene, or an image generated by photographing the scene at a different viewing angle from the scene image IMG1.

In some embodiments, the step of selecting the first intersection point set and the second intersection point set includes: randomly selecting two of the first group and two of the second group, and using four intersection points between the two of the first group and the two of the second group as one of at least one first intersection point set. Then, two of the plurality of horizontal field lines and two of the plurality of vertical field lines may be randomly selected from the field template, and using four intersection points between the two of the plurality of horizontal field lines and the two of the plurality of vertical field lines as one of at least one second intersection point set.

In some embodiments, when one of the two in the first group is selected first, one of the vertical field lines can be selected from the field template according to the ranking of the one. When one of the two in the second group is selected first, one of the horizontal field lines can be selected from the field template according to the ranking of the one. In this way, the number of times of selecting the first intersection point and the second intersection point set can be greatly reduced, so as to reduce the amount of calculation.

For example, if one of the fusion lines in the first group is selected first, and the fusion line is ranked ahead (i.e., closer to the leftmost middle point of the field image IMG1), then the vertical field line close to the leftmost side of the field template will be selected, and the vertical field line close to the rightmost side of the field template will not be selected. On the contrary, if another fusion line in the first group is selected first, and the fusion line is ranked behind (i.e., farther from the leftmost middle point of the field image IMG1), then the vertical field line farther from the leftmost side of the field template will be selected, and the vertical field line close to the leftmost side of the field template will not be selected. In this way, the number of selections will be greatly reduced.

The horizontal field lines and vertical field lines in the field template are explained below with practical examples. Referring to FIG. 10, FIG. 10 is a schematic diagram of the field template TEM1 in some embodiments of the present disclosure. As shown in FIG. 10, the field template TEM1 is a field map of a tennis court, and a plurality of field lines are drawn on the field map. Then, the straight line equations of the plurality of horizontal field lines TL1~TL4 and the straight line equations of the plurality of vertical field lines TL5~TL9 can be calculated from the plurality of field lines in the field template TEM1, and the origin TO of the field template TEM1 can be set at the upper left corner of the field template TEM1, the horizontal field lines TL1~TL4 are all the horizontal lines in the plurality of field lines, and the vertical field lines TL5~TL9 are all the vertical lines in the plurality of field lines.

The following is an actual example to illustrate the selection of two fusion lines in the first group and two fusion lines in the second group. Referring to FIG. 11, FIG. 11 is a schematic diagram showing the selection of multiple fusion lines IL1, IL5, IL6, and IL9 in some embodiments of the present disclosure. As described in FIG. 11, continuing the example of FIG. 7, fusion lines IL1 and IL5 can be randomly selected from fusion lines IL1 to IL5 in the first group in FIG. 7, and fusion lines IL6 and IL9 can be randomly selected from fusion lines IL6 to IL10 in the second group. Then, the four intersections IP1 to IP4 between fusion lines IL1 and IL5 and fusion lines IL6 and IL9 can be used as one of the first intersection sets.

The following is an actual example to illustrate the selection of two horizontal field lines and two vertical field lines. Referring to FIG. 12, FIG. 12 is a schematic diagram showing the selection of two horizontal field lines TL1, TL4 and two vertical field lines TL5, TL6 in some embodiments of the present disclosure. As described in FIG. 12, continuing the example of FIG. 10, horizontal field lines TL1, IL4 can be randomly selected from the horizontal field lines TL1~TL4 in the field template TEM1 in FIG. 10, and vertical field lines TL5, TL6 can be randomly selected from the vertical field lines TL5~TL9. Then, the four intersection points TP1~TP4 between the horizontal field lines TL1, IL4 and the vertical field lines TL5, TL6 can be used as one of the second intersection point sets.

In other words, by the above method of randomly selecting two of the first group and two of the second group and randomly selecting two of the plurality of horizontal field lines and two of the plurality of vertical field lines from the field template, taking Figures 7 and 10 as examples, 6,000 selection results may be generated.

Furthermore, as shown in FIG. 2 , in step S250 , at least one homography matrix between at least one first intersection point set and at least one second intersection point set is calculated.

In some embodiments, at least one homography matrix is a homography matrix used to convert the coordinates of the field template to the coordinates of the field image IMG1. For example, the field image IMG1 can be an image of a tennis court shot from a television broadcast shooting angle, the field template can be a field map overlooking the tennis court, and the homography matrix can be a conversion matrix between the image and the field map. In addition, the homography matrix can be calculated by taking at least four points from the image (i.e., one of the first intersection points mentioned above) and taking at least four corresponding points from the field map (i.e., one of the corresponding second intersection points mentioned above). For example, taking FIG. 11 and FIG. 12 as examples, the homography matrix between the field image IMG1 and the field template TEM1 can be calculated based on the intersection points IP1-IP4 in the field image IMG1 and the intersection points TP1-TP4 in the field template TEM1.

It is worth noting that the homography matrix is a transformation matrix between two planar images generated by shooting the same scene from two different viewing angles in this field. Therefore, the calculation of the homography matrix will not be elaborated.

In some embodiments, after calculating the homography matrix, the accuracy operation can be performed on the homography matrix to generate a corresponding conversion score. Then, the template to be superimposed can be superimposed on the field image IMG1 using the homography matrix with the highest conversion score to generate a superimposed image. In some embodiments, the template to be superimposed can also be superimposed on the field image IMG1 using a plurality of homography matrices with the highest conversion scores (e.g., homography matrices with the highest 10 conversion scores) to generate multiple superimposed images.

In some embodiments, the accuracy calculation may include: for each of the homography matrices, performing the following steps: a first group and a second group may be subjected to a perspective transformation. Then, a first angle difference and a first distance difference between each of the transformed first group and a corresponding vertical field line in the field template may be calculated, and a first score for each of the transformed first group may be calculated based on a plurality of first angle differences and a plurality of first distance differences. Then, a second angle difference and a second distance difference between each of the transformed second group and a corresponding horizontal field line in the field template may be calculated, and a second score for each of the transformed second group may be calculated based on a plurality of second angle differences and a plurality of second distance differences. Then, one of a plurality of transformation scores may be calculated based on the plurality of first scores and the plurality of second scores.

In some embodiments, a quotient between one of the first angle differences and a preset angle may be calculated, and a hyperbolic tangent function of the quotient of one of the first angle differences may be calculated. Then, a quotient between one of the first distance differences and a preset distance may be calculated, and a hyperbolic tangent function of the quotient of one of the first distance differences may be calculated. Then, a first score of one of the first groups of transformations may be calculated based on the angle weight, the distance weight, the hyperbolic tangent function of the quotient of one of the first angle differences, and the hyperbolic tangent functions of the multiple quotients of one of the first distance differences.

In some embodiments, a quotient between one of the second angle differences and a preset angle may be calculated, and a hyperbolic tangent function of the quotient of one of the second angle differences may be calculated. Then, multiple quotients between one of the second distance differences and a preset distance may be calculated, and a hyperbolic tangent function of the quotient of one of the second distance differences may be calculated. Then, a second score of one of the second groups of transformations may be calculated based on the angle weight, the distance weight, the hyperbolic tangent function of the multiple quotients of one of the second angle differences, and the hyperbolic tangent function of the multiple quotients of one of the second distance differences.

The calculation of the conversion score is described below with an actual example. Referring to FIG. 13 , FIG. 13 is a schematic diagram showing a plurality of conversion lines TSL1 - TSL10 on a field template TEM1 in some embodiments of the present disclosure. As shown in Figures 7, 10 and 13, continuing with the examples of Figures 11 and 12, assuming that the homography matrix between the field image IMG1 and the field template TEM1 can be calculated based on the selection of Figures 11 and 12, this homography matrix can be used to convert the fusion lines IL1~IL10 in the field image IMG1 into straight line equations of transformation lines TSL1~TSL10 on the field template TEM1 (i.e., perform the above-mentioned viewing angle conversion), wherein the fusion lines IL6~IL10 are the fusion lines of the first group, the fusion lines IL1~IL5 are the fusion lines of the second group, the transformation lines TSL6~TSL10 are the transformation lines of the transformed first group, and the transformation lines TSL1~TSL5 are the transformation lines of the transformed second group.

Furthermore, since the conversion line TSL1 is converted from the fusion line IL1 in the second group, the corresponding horizontal field line can be found from the horizontal field lines TL1~TL4 in the field template TEM1 according to the conversion line TSL1.

In detail, the minimum distance between the horizontal field lines TL1~TL4 and the origin TO can be calculated as a plurality of field line minimum distances, and the minimum distance between the conversion line TSL1 and the origin TO can be calculated as the conversion line minimum distance of the conversion line TSL1. Then, a plurality of distance differences between the plurality of field line minimum distances and the conversion line minimum distance of the conversion line TSL1 can be calculated, and the horizontal field line corresponding to the minimum distance difference can be selected as the above-mentioned corresponding horizontal field line.

In this embodiment, for the conversion line TSL1, the corresponding horizontal field line can be found to be the horizontal field line TL1 through the above calculation. Similarly, through the above calculation, it can also be identified that the conversion lines TSL2 and TSL4 to TSL10 correspond to the horizontal field lines TL2 to TL4 and the vertical field lines TL5 to TL9, respectively, and it can be identified that the conversion line TSL3 corresponds to the horizontal field line TL2.

Furthermore, the angle difference and the minimum distance between the conversion line TSL1 and the horizontal field line TL1 can be calculated, and the calculated angle difference and the minimum distance can be used as the second angle difference and the second distance difference of the conversion line TSL1. Similarly, through the above calculation, the second angle difference and the second distance difference of the conversion lines TSL2~TSL5 can also be calculated, and the first angle difference and the first distance difference of the conversion lines TSL6~TSL10 can be calculated.

Furthermore, the quotient between the first angle difference of the conversion line TSL6 and a preset angle can be calculated, and the hyperbolic tangent function of the quotient of the first angle difference of the conversion line TSL6 can be calculated. Then, the quotient between the first distance difference of the conversion line TSL6 and a preset distance can be calculated, and the hyperbolic tangent function of the quotient of the first distance difference of the conversion line TSL6 can be calculated. Then, the first score of the conversion line TSL6 is calculated based on the angle weight, the distance weight, the hyperbolic tangent function of the quotient of the first angle difference of the conversion line TSL6, and the hyperbolic tangent functions of multiple quotients of the first distance differences of the conversion line TSL6. By analogy, the first scores of the conversion lines TSL7~TSL10 can also be calculated by the same calculation.

Furthermore, the quotient between the second angle difference of the conversion line TSL1 and a preset angle can be calculated, and the hyperbolic tangent function of the quotient of the second angle difference of the conversion line TSL1 can be calculated. Then, the quotient between the second distance difference of the conversion line TSL1 and a preset distance can be calculated, and the hyperbolic tangent function of the quotient of the second distance difference of the conversion line TSL1 can be calculated. Then, the second score of the conversion line TSL1 is calculated based on the angle weight, the distance weight, the hyperbolic tangent function of the quotient of the second angle difference of the conversion line TSL1, and the hyperbolic tangent functions of multiple quotients of the second distance differences of the conversion line TSL1. By analogy, the second scores of the conversion lines TSL2~TSL5 can also be calculated by the same calculation.

In some embodiments, the hyperbolic tangent function of the quotient of the above-mentioned angle difference (ie, the first angle difference or the second angle difference) is as shown in the following formula (1). …Formula 1)

Wherein angleDiff is the hyperbolic tangent function of the quotient of the angle difference, a1 is the inclination of one of the fusion lines, a2 is the inclination of the field line (ie, vertical field line or horizontal field line) corresponding to the fusion line in the field template TEM1, and ā is a preset angle (eg, set by the user).

In some embodiments, the hyperbolic tangent function of the quotient of the above distance difference (ie, the first distance difference or the second distance difference) is as shown in the following formula (2). …Formula (2)

Wherein originDiff is the hyperbolic tangent function of the quotient between the distance difference and the preset distance, d1 is the minimum distance between one of the fusion lines and the origin of the field template TEM1 (i.e., the origin TO), d2 is the minimum distance between the field line (i.e., the vertical field line or the horizontal field line) corresponding to this fusion line in the field template and the origin of the field template, and đ is the preset distance (for example, set by the user).

In some embodiments, the first score is as shown in the following formula (3). …Formula (3)

Where SC1 is the first score, W1 is the angle weight, angleDiff1 is the first angle difference of one of the transformation lines in the first group of transformations, W2 is the distance weight, and originDiff1 is the first distance difference of this transformation line.

In some embodiments, the second score is as shown in the following formula (4). …Formula (4)

SC2 is the second score, angleDiff1 is the second angle difference of one of the conversion lines in the second group of conversions, and originDiff1 is the second distance difference of the conversion line.

Furthermore, one of a plurality of conversion scores is calculated based on the plurality of first scores and the plurality of second scores calculated above. In some embodiments, the conversion score is as shown in the following formula (5). …Formula (5)

Where warpingScore is the transformation score corresponding to this homography matrix. is the sum of the first scores of all conversion lines in the first group of conversions, is the sum of the second scores of all conversion lines in the second group of conversions.

It is worth noting that, as shown in Figures 11 and 12, since the fusion lines IL1, IL5, IL6, and IL9 selected from the field image IMG1 do not completely correspond to the field lines TL1, TL4, TL5, and TL6 selected from the field template TEM1 (i.e., the fusion line IL9 is on the right singles line on the tennis ball, and the field line TL6 is on the left singles line on the tennis ball, and there is no correspondence), the conversion effect of the calculated homography matrix is not good (as shown in Figure 13, only the conversion lines TSL1, TSL5, and TSL6 correspond to the correct field lines).

The following is an actual example to illustrate a homography matrix with a higher conversion score. Referring to FIG. 14, FIG. 14 is a schematic diagram showing the selection of multiple blend lines IL1, IL5, IL6, and IL10 in some embodiments of the present disclosure. As described in FIG. 14, continuing the example of FIG. 7, the blend lines IL1 and IL5 can be randomly selected from the blend lines IL1 to IL5 in the first group in FIG. 7, and the blend lines IL6 and IL10 can be randomly selected from the blend lines IL6 to IL10 in the second group. Then, the four intersection points IP1 to IP4 between the blend lines IL1 and IL5 and the blend lines IL6 and IL10 can be used as one of the first intersection points.

Referring to FIG. 15 , FIG. 15 is a schematic diagram showing the selection of two horizontal field lines IL1 and IL5 and two vertical field lines IL6 and IL9 in some embodiments of the present disclosure. As described in FIG. 15 , following the example of FIG. 10 , horizontal field lines TL1 and IL5 can be randomly selected from horizontal field lines TL1 to TL4 in field template TEM1 in FIG. 10 , and vertical field lines TL6 and TL9 can be randomly selected from vertical field lines TL5 to TL9. Then, four intersection points TP1 to TP4 between horizontal field lines TL1 and IL5 and vertical field lines TL6 and TL9 can be used as one of the second intersection point sets.

Please refer to FIG. 16 , which is a schematic diagram of a plurality of switching lines TSL1 - TSL10 on a field template TEM1 in some embodiments of the present disclosure. As shown in Figures 7, 10 and 13, continuing with the examples of Figures 14 and 15, assuming that the homography matrix between the field image IMG1 and the field template TEM1 can be calculated based on the selection of Figures 14 and 15, this homography matrix can be used to convert the fusion lines IL1~IL10 in the field image IMG1 into straight line equations of transformation lines TSL1~TSL10 on the field template TEM1 (i.e., perform the above-mentioned viewing angle conversion), wherein the fusion lines IL6~IL10 are the fusion lines of the first group, the fusion lines IL1~IL5 are the fusion lines of the second group, the transformation lines TSL6~TSL10 are the transformation lines of the transformed first group, and the transformation lines TSL1~TSL5 are the transformation lines of the transformed second group.

Furthermore, it can be seen from FIG. 16 that the conversion effect of this homography conversion matrix is better (because all conversion lines TSL1~TSL2, TSL4~TSL10 correspond to the correct ground line, and only conversion line TSL3 cannot correspond to the ground line). Then, a homography matrix and the conversion score of this homography matrix can be calculated according to the calculation method of the examples in FIGS. 11 to 13. After actual calculation, it can be found that the conversion score of this homography matrix will be much greater than the conversion score of the examples in FIGS. 11 to 13. Based on the above steps, the present disclosure can select the first intersection set from the first group and the second group multiple times (which can be a preset number of times or all possible selection times), and select the second intersection set from the field template multiple times, and then calculate multiple homography matrices. In this way, the homography matrix with a higher conversion score can be found as the best homography matrix.

In some embodiments, the accuracy calculation may also include: for each of the at least one homography matrix, performing the following steps: a plurality of vertical field lines may be subjected to a perspective transformation, and a plurality of first overlapping pixels and a plurality of first non-overlapping pixels on the plurality of transformed vertical field lines may be calculated. Then, a plurality of horizontal field lines may be subjected to a perspective transformation, and a plurality of second overlapping pixels and a plurality of second non-overlapping pixels on the plurality of transformed horizontal field lines may be calculated. Then, one of a plurality of transformation scores may be calculated based on the plurality of first overlapping pixels, the plurality of first non-overlapping pixels, the plurality of second overlapping pixels, and the plurality of second non-overlapping pixels.

In some embodiments, multiple homography matrices with higher conversion scores (for example, the top ten homography matrices with higher conversion scores) may be selected from at least one homography matrix calculated by the above method using angle difference and distance difference to execute the above method of calculating the number of overlaps and non-overlaps to calculate a new conversion score.

The following uses actual examples to explain overlapping pixels and non-overlapping pixels. Referring to FIG. 17 , FIG. 17 is a schematic diagram showing a plurality of pixel points P of field lines (including horizontal field lines and vertical field lines) on a field template TEM1 in some embodiments of the present disclosure. As shown in FIG. 17 , the coordinates of the plurality of pixel points P can be captured from the field lines in the field template TEM1. In some embodiments, the capture (or sampling) can be performed every two pixels on the field lines in the field template TEM1.

Next, refer to FIG. 18 , which is a schematic diagram of multiple transition points TSP on the field image IMG1 in some embodiments of the present disclosure. As shown in FIG. 18 , following the example of FIG. 17 , the coordinates of multiple pixel points P of the field lines on the field template TEM1 can be converted into the coordinates of multiple transition points TSP on the field image IMG1 using a homography matrix calculated as described above, and the coordinates of these transition points TSP are marked on the field image IMG1. Then, it can be determined whether these transition points TSP are located on the white pixels of the black and white image generated as described above. When the transition point TSP is located on the white pixels of the black and white image generated as described above, it can be identified as a superimposed pixel, wherein the superimposed pixel converted by the vertical field line is the first superimposed pixel, and the superimposed pixel converted by the horizontal field line is the second superimposed pixel.

On the contrary, non-overlapping pixels (i.e., pixels on a plurality of field line segments OG1-OG3) can be identified, wherein the non-overlapping pixels converted by the vertical field lines are first non-overlapping pixels, and the non-overlapping pixels converted by the horizontal field lines are second non-overlapping pixels. Finally, one of a plurality of conversion scores can be calculated according to the number of the first overlapping pixels, the number of the first non-overlapping pixels, the number of the second overlapping pixels, and the number of the second non-overlapping pixels.

In some embodiments, the conversion score is as shown in the following formula (6). …Formula (6)

Where projectScore is the conversion score, sum(,) is the sum calculation function, numSpoint is the sum of the number of all first overlapping pixels and the number of all second overlapping pixels, W3 is the unoverlapping weight, and numUSpoint is the sum of the number of all first unoverlapping pixels and the number of all second unoverlapping pixels.

It is worth noting that the above method of calculating the number of overlaps and non-overlaps can be executed after executing the method of angle difference and distance difference, or can be executed directly after executing the above step S250, without any special restrictions. In addition, such a method of calculating the number of overlaps and non-overlaps can also correctly track the boundary line on the field of the field image (for example, it is used to determine whether the ball on the field is out of bounds).

In some embodiments, after generating at least one homography matrix, the template to be superimposed may be converted into at least one transformation map using at least one homography matrix, and the at least one transformation map may be superimposed on the field image IMG1. In some embodiments, when a specific image (e.g., an advertisement or graphic provided by a manufacturer) needs to be superimposed on the field image IMG1, the specific image may be overlaid on the field template TEM1 to generate the template to be superimposed.

The following describes the superposition of the transformation graph using an actual example. Referring to FIG. 19, FIG. 19 is a schematic diagram of the template TEM2 to be superimposed in some embodiments of the present disclosure. As shown in FIG. 19, the template TEM2 to be superimposed has an advertising marker AD. Thus, the template TEM2 to be superimposed can be transformed using the homography matrix to overlay the generated transformation graph on the field image IMG1. Referring to FIG. 20, FIG. 20 is a schematic diagram of the superimposed image SIMG in some embodiments of the present disclosure. As shown in FIG. 20, when the transformation graph is overlaid on the field image to generate the superimposed image SIMG, a deformed advertising marker AD can be generated on the superimposed image SIMG, and the deformed advertising marker AD can be completely in accordance with the deformation of the captured field image. In this way, when viewing the scene image, the user will not feel that the shape of the advertisement mark AD is strange. In other words, the image processing device and method disclosed in the present invention can place the advertisement mark AD in a proper position, reduce the abruptness of the advertisement mark AD, and avoid deformation of the advertisement mark AD.

It is worth noting that, in terms of application, the image processing device and method of the disclosed document can be used in various fields with machine vision, image classification or data classification in addition to the above-mentioned application of placing advertisements. For example, the image processing device and method can also be used to place virtual objects on field images. On the other hand, the image processing device and method can also be used to determine whether the ball on the sports field is out of bounds. In addition, the image processing device and method can also be used in the shooting of panoramic images or the application of virtual reality (augmented reality).

It is worth noting that when shooting a scene, the shooting position may change (i.e., moving from the first shooting position to the second shooting position). Since the homography matrix calculated above can only correspond to the first shooting position, this will cause the calculated homography matrix to lose accuracy. Therefore, it is necessary to use the above calculation method to further recalculate the homography matrix corresponding to the second shooting position.

In view of this, a method for saving the amount of calculation for recalculating the homography matrix corresponding to the second shooting position based on the new image is further proposed below. The present disclosure further proposes an image processing device and method for view angle conversion, generating an optical flow map from different frames of different shooting positions, and using the optical flow map to calculate the pixel movement trend between the first shooting position and the second shooting position, and then generating the key points in the image of the second shooting position from the key points in the field template according to the pixel movement trend and the homography matrix corresponding to the first shooting position. In this way, a new homography matrix is further calculated from the key points in the field template and the key points in the image of the second shooting position. In this way, the amount of calculation for regenerating the homography matrix can be further reduced, and a new homography matrix can be generated in real time. The above-mentioned technology of the present disclosure is specifically described by taking the following embodiments as examples.

Referring to FIG. 21, FIG. 21 is a flowchart of an image processing method for perspective conversion in some embodiments of the present disclosure. The image processing method for perspective conversion is applicable to an electronic device, which may be the image processing device 100 of FIG. 1. The components in the image processing device 100 of FIG. 1 are used to execute steps S2110 to S2150 in the image processing method for perspective conversion. As shown in FIG. 21, first, in step S2110, a previous image and a current image are captured.

In some embodiments, the image capture circuit 110 may capture a previous image and a current image. In some embodiments, the previous image may be a frame before the current image among a plurality of images continuously captured by the image capture circuit 110. In some embodiments, the capture position of the image capture circuit 110 may be continuously changed. In some embodiments, a scene may be captured at a first capture position to generate a previous image, and a scene may be captured at a second capture position to generate a current image. In some embodiments, the first capture position and the second capture position correspond to different capture angles.

In step S2120 , feature extraction is performed on the previous image and the current image to calculate an optical flow information matrix from a first feature set of the previous image and a second feature set of the current image.

In some embodiments, the feature extraction operation for the previous image and the current image includes: a plurality of first candidate feature points and a plurality of second candidate feature points may be detected from the previous image and the current image, respectively. Then, a feature matching operation may be performed on the plurality of first candidate feature points and the plurality of second candidate feature points to generate a first feature set of the previous image and a second feature set of the current image.

In some embodiments, a plurality of first candidate feature points and a plurality of second candidate feature points may be detected from a previous image and a current image respectively by using a corner detection operation or an edge detection operation. In some embodiments, a SIFT (scale-invariant feature transform) algorithm, a SURF (speeded up robust features) algorithm, or an ORB (oriented fast and rotated brief) algorithm may be used to perform feature mapping between the plurality of first candidate feature points and the plurality of second candidate feature points and delete the unmatched first candidate feature points and the unmatched second candidate feature points.

In some embodiments, after feature matching, feature filtering may be performed on the plurality of first candidate feature points and the plurality of second candidate feature points to generate a first feature set of the previous image and a second feature set of the current image. In some embodiments, the feature filtering operation may include: calculating a plurality of feature vectors between the plurality of matched first candidate feature points and the plurality of matched second candidate feature points (for example, taking one of the matched first candidate feature points as a starting point and the corresponding matched second candidate feature point as an end point to generate one of the feature vectors), and filtering the plurality of feature vectors using an outlier algorithm. In some embodiments, the step of filtering the plurality of eigenvectors using an outlier algorithm may include: using an outlier algorithm to find at least one outlier vector from the plurality of eigenvectors, and then deleting at least one outlier vector from the plurality of eigenvectors. In some embodiments, the outlier algorithm may be a local outlier factor (LOF) algorithm or a random cut forest (RCF) algorithm.

In some embodiments, the operation of calculating the optical flow information matrix from the first feature set of the previous image and the second feature set of the current image includes: an optical flow map can be generated based on the first feature set of the previous image and the second feature set of the current image. Then, the optical flow information matrix can be calculated based on the optical flow map. In some embodiments, the optical flow map can be generated based on the first feature set of the previous image and the second feature set of the current image using a Lucas-Kanade algorithm or a dense optical flow (Farneback) algorithm. In some embodiments, the optical flow information matrix indicates the pixel movement trend between the previous image and the current image. In some embodiments, an affine transformation operation may be used to generate a transformation matrix (eg, an optical flow information matrix) from the optical flow map, where the transformation matrix indicates the position change trend between pixels of a previous image and pixels of a current image.

The generation of an optical flow map is described below using a practical example. Referring to FIG. 22 , FIG. 22 is a schematic diagram showing the generation of an optical flow map OF_MAP in some embodiments of the present disclosure. As shown in FIG. 22 , a corner detection operation can be used to detect a plurality of first candidate feature points and a plurality of second candidate feature points from a previous image P_IMG and a plurality of second candidate feature points, respectively. Then, a SIFT algorithm and a regional outlier factor algorithm can be used to generate a first feature set of the previous image P_IMG and a second feature set of the current image C_IMG from the plurality of first candidate feature points and the plurality of second candidate feature points. Then, based on the first feature set of the previous image P_IMG and the second feature set of the current image C_IMG, the Lucascanade algorithm can be used to generate an optical flow map OF_MAP.

Furthermore, as shown in FIG. 21, in step S2130, the optical flow information matrix is used to infer the position of the second key point set in the previous image in the current image to store it as the third key point set. In some embodiments, the previous homography matrix can be used to perform a perspective conversion process on the first key point set in the field template to generate the second key point set in the previous image. In some embodiments, this previous homography matrix can be a homography matrix generated from the field template and the previous image by the step of FIG. 2, or a homography matrix pre-stored in the memory 120, or a homography matrix obtained by a traditional calculation method. In some embodiments, the optical flow information matrix may be used to transform the second keypoint set in the previous image into the third keypoint set in the current image.

In some embodiments, the scene template may have a target image and a first key point set. In some embodiments, the target image may be an advertisement marker (e.g., the advertisement marker AD in FIG. 19 above). In some embodiments, the multiple key points in the first key point set may be marking points of multiple intersections between multiple field lines pre-set on the scene template. In some embodiments, the number of these marking points may be any positive integer not less than 4, and there is no particular limitation.

The first set of key points is described below with an actual example. Referring to FIG. 23 , FIG. 23 is a schematic diagram of a field template TEM in some other embodiments of the present disclosure. As shown in FIG. 23 , the field template TEM has a plurality of field lines, and a plurality of key points KP are marked on a plurality of intersections between the field lines.

The second key point set and the third key point set are explained below with an actual example. Referring to FIG. 24, FIG. 24 is a schematic diagram showing the second key point set and the third key point set in some embodiments of the present disclosure. As shown in FIG. 24, the multiple key points KP in the field template TEM of FIG. 23 can be converted into multiple key points KP in the previous image P_IMG as the second key point set using the previous homography matrix, and the multiple key points KP in the previous image P_IMG can be converted into multiple key points KP in the current image C_IMG as the third key point set using the optical flow information matrix.

Furthermore, as shown in FIG. 21 , in step S2140 , the current homography matrix between the first key point set and the third key point set is calculated.

In some embodiments, the third key point set in the current image may be updated based on the second key point set and the fundamental matrix in the previous image, wherein the fundamental matrix indicates the corresponding view angle conversion relationship between the previous image and the current image. In some embodiments, the fundamental matrix may be calculated based on the starting point and the end point indicated by the multiple optical flows in the optical flow map. In some embodiments, the fundamental matrix may be calculated based on the first feature set of the previous image and the second feature set of the current image. In other words, the fundamental matrix can be calculated by using the feature points of the previous image and the corresponding feature points of the current image (i.e., the starting point and the end point of the optical flow), and the calculation of the fundamental matrix is a calculation commonly used by technicians in this field, so it will not be repeated here.

In some embodiments, the step of updating the third key point may include: using a base matrix to convert the second key point set in the previous image into a plurality of straight lines in the current image. Then, based on the plurality of straight lines, the third key point set in the current image may be updated.

In some embodiments, it may be determined whether each of the third key point set in the current image is located on one of a plurality of corresponding straight lines in the current image. Then, at least one noise key point in the third key point set may be deleted to update the third key point set in the current image, wherein each of the at least one noise key point is not located on a corresponding straight line in the current image. In some embodiments, the key points corresponding to the at least one noise key point in the first key point set may be deleted.

The principle of the basic matrix is explained below with an actual example. Referring to FIG. 25 , FIG. 25 is a schematic diagram showing the viewing angle conversion relationship in some embodiments of the present disclosure. As shown in FIG. 25 , the field can be photographed from the optical center O of the first shooting position and the optical center O’ of the second shooting position to generate image IMG_O and image IMG_O’. Pixel point x is generated by field point X in the shooting field. However, since the depth of image IMG_O is unknown, all field points X on the ray r extending from the optical center O and pixel point x can form pixel point x on image IMG_O.

Next, since it is uncertain which field point X forms the pixel x on the image IMG_O, the ray r can form an antipolar line l’ on the image IMG_O’, and all the pixel points x’ on the antipolar line l’ may correspond to the pixel point x, wherein the pixel point x’ and the pixel point x are generated by photographing the same field point X. In addition, the line between the optical center O and the optical center O’ can generate the antipolar point e and the antipolar point e’ on the image IMG_O and the image IMG_O’, respectively. The antipolar point e’ is located on the antipolar line l’, and the antipolar point e and the pixel point x can form another antipolar line l. All the pixels on the antipolar line l will correspond to the antipolar line l’. In other words, all the pixels on the epipolar line l can be converted to the pixels on the epipolar line l' using the basic matrix, and the basic matrix can be calculated based on the corresponding view angle conversion relationship between these pixels. In this way, any pixel on the image IMG_O can be converted to the epipolar line on the image IMG_O' using the basic matrix. In this way, the key point of the second key point set can be converted to a straight line on the current image using the same method, and it is determined whether the corresponding key point of this key point in the third key point set is on this straight line. When it is determined that it is not on this straight line, the corresponding key point is deleted.

The updated third key point set is explained below with an actual example. Referring to FIG. 26, FIG. 26 is a schematic diagram of the updated third key point set in some embodiments of the present disclosure. As shown in FIG. 26, the multiple key points KP of the current image C_IMG in FIG. 24 can be filtered using the base matrix and the multiple key points KP of the previous image P_IMG in FIG. 24 to generate multiple filtered key points FKP of the current image C_IMG as the updated third key point set. It is worth noting that, since some key points KP of the current image C_IMG in FIG. 24 (i.e., key points on the center line of the court and key points overlapping with characters) have been filtered out, the number of multiple filtered key points FKP of the current image C_IMG is less than the number of multiple key points KP of the current image C_IMG in FIG. 24 .

Furthermore, as shown in FIG. 21, in step S2150, the target image is projected to the corresponding position on the current image using the current homography matrix. For example, the method of FIG. 19 and FIG. 20 can be used to transform the pixels of the target image (e.g., the advertisement mark AD in FIG. 19) in the field template using the current homography matrix to project them onto the current image.

Through the above steps, users will not feel that the shape of the advertising mark is strange when viewing the scene image. In addition, this approach only uses the four arithmetic operations and differentials in mathematical calculations, and the number of calculations of feature points and key points is not large. This approach will greatly reduce the amount of calculations and improve the real-time tracking. The use of optical flow calculations will also prevent the target image from being stuck when projected onto the current image.

In summary, the image processing device for perspective conversion proposed in the present disclosure can use the optical flow method to update the homography matrix to greatly reduce the amount of calculation required to regenerate the homography matrix. In addition, the homography matrix can be updated between different frames in real time. Regardless of how the shooting position changes, a new homography matrix can be calculated immediately. On the other hand, the general method of calculating the homography matrix may cause a sense of stuttering when the target image is projected onto the current image, while the image processing device for perspective conversion disclosed in the present disclosure can greatly increase the sense of smoothness after projecting onto continuous images.

Although specific embodiments of the present disclosure have been disclosed with respect to the above-mentioned embodiments, these embodiments are not intended to limit the present disclosure. Various substitutions and improvements may be performed in the present disclosure by a person skilled in the art without departing from the principles and spirit of the present disclosure. Therefore, the scope of protection of the present disclosure is determined by the scope of the attached patent application.

100: Image processing device 110: Image acquisition circuit 120: Memory 130: Processor IMG1: Field image S210~S250, S2110~S2150: Steps BIMG: Binary image LS1~LS12: Line segment L1, L2: Line segment EL: Extension line θ: Line segment angle d: Maximum distance IL1~IL10, VL1~VL5, HL1~HL5: Fusion line IMG2, IMG3: Image LO, UO: Reference point LD1~LD5, UD1~UD5: Minimum distance TEM1, TEM: Field template TO: Origin TL1~TL4: Horizontal field line TL5~TL9: Vertical field line IP1~IP4, TP1~TP4: Intersection point TSL1~TSL10: Transition line P: Pixel point TSP: Transition point OG1~OG3: Field line segment TEM2: Template to be superimposed AD: Advertising marker SIMG: Superimposed image C_IMG: Current image P_IMG: Previous image OF_MAP: Optical flow map KP: Key point IMG_O, IMG_O’: Image O, O’: Optical center x, x’: Pixel point X: Field point r: Ray l, l’: Antipolar line e, e’: Antipolar point l, l’: Antipolar line FKP: Filter key point

FIG. 1 is a schematic diagram of an image processing device in some embodiments of the present disclosure. FIG. 2 is a flow chart of an image processing method in some embodiments of the present disclosure. FIG. 3 is a schematic diagram of the generation of a binary image in some embodiments of the present disclosure. FIG. 4 is a schematic diagram of generating multiple line segments from a field image in some embodiments of the present disclosure. FIG. 5 is a schematic diagram of the angle of a line segment in some embodiments of the present disclosure. FIG. 6 is a schematic diagram of the maximum distance in some embodiments of the present disclosure. FIG. 7 is a schematic diagram of multiple fusion lines in a field image in some embodiments of the present disclosure. FIG. 8 is a schematic diagram of the sorting of the first group in some embodiments of the present disclosure. FIG. 9 is a schematic diagram of the sorting of the second group in some embodiments of the present disclosure. FIG. 10 is a schematic diagram of a field template in some embodiments of the present disclosure. FIG. 11 is a schematic diagram of selecting multiple fusion lines in some embodiments of the present disclosure. FIG. 12 is a schematic diagram of selecting two horizontal field lines and two vertical field lines in some embodiments of the present disclosure. FIG. 13 is a schematic diagram of multiple conversion lines on a field template in some embodiments of the present disclosure. FIG. 14 is a schematic diagram of selecting multiple fusion lines in some embodiments of the present disclosure. FIG. 15 is a schematic diagram of selecting two horizontal field lines and two vertical field lines in some embodiments of the present disclosure. FIG. 16 is a schematic diagram of multiple conversion lines on a field template in some embodiments of the present disclosure. FIG. 17 is a schematic diagram of multiple pixel points of a field line on a field template in some embodiments of the present disclosure. FIG. 18 is a schematic diagram of multiple conversion points on a field image in some embodiments of the present disclosure. FIG. 19 is a schematic diagram of a superimposed template in some embodiments of the present disclosure. FIG. 20 is a schematic diagram of a superimposed image in some embodiments of the present disclosure. FIG. 21 is a flow chart of an image processing method for perspective conversion in some embodiments of the present disclosure. FIG. 22 is a schematic diagram of the generation of an optical flow map in some embodiments of the present disclosure. FIG. 23 is a schematic diagram of a field template in other embodiments of the present disclosure. FIG. 24 is a schematic diagram of a second key point set and a third key point set in some embodiments of the present disclosure. FIG. 25 is a schematic diagram of a perspective conversion relationship in some embodiments of the present disclosure. FIG. 26 is a schematic diagram of an updated third key point set in some embodiments of the present disclosure.

S2110~S2150: Steps

Claims (14)

An image processing device for perspective conversion includes: an image capture circuit for capturing a previous image and a current image; a memory for storing a plurality of instructions, a target image in a field template, and a first key point set in the field template; and a processor for processing the instructions to perform the following operations: performing a feature extraction operation on the previous image and the current image to calculate an optical flow information matrix from a first feature set of the previous image and a second feature set of the current image; using the optical flow information matrix to infer the position of a second key point set in the previous image in the current image to store it as a third key point set; calculating a current homography matrix between the first key point set and the third key point set; and The target image is projected to the corresponding position on the current image using the current homography matrix. The image processing device for view angle conversion as described in claim 1, wherein the second key point set is converted from the first key point set on the scene template by a previous homography matrix. An image processing device for view angle conversion as described in claim 1, wherein the previous image is a previous frame of the current image, and the operation of performing the feature extraction operation on the previous image and the current image includes: Detecting a plurality of first candidate feature points and a plurality of second candidate feature points from the previous image and the current image respectively; and Performing a feature matching operation on the first candidate feature points and the second candidate feature points to generate the first feature set of the previous image and the second feature set of the current image. The image processing device for view angle conversion as described in claim 3, wherein the processor is further used to perform the following operations: Based on the second key point set in the previous image and a base matrix, the third key point set in the current image is updated, wherein the base matrix indicates a corresponding view angle conversion relationship between the previous image and the current image. The image processing device for view angle conversion as described in claim 4, wherein the base matrix is generated by the following operation: Based on the first feature set of the previous image and the second feature set of the current image, the base matrix is calculated. The image processing device for view angle conversion as described in claim 4, wherein the operation of updating the third key point set includes: Using the base matrix to convert the second key point set in the previous image into a plurality of straight lines in the current image; and Based on the straight lines, updating the third key point set in the current image. The image processing device for view angle conversion as described in claim 6, wherein the operation of updating the third key point set in the current image based on the straight lines includes: Determining whether each of the third key point set in the current image is located on one of the corresponding straight lines in the current image; and Deleting at least one noise key point in the third key point set to update the third key point set in the current image, wherein each of the at least one noise key point is not located on the corresponding straight line in the current image. An image processing method for perspective conversion includes: Shooting a previous image and a current image, and storing a target image in a field template and a first key point set in the field template; Performing a feature extraction operation on the previous image and the current image to calculate an optical flow information matrix from a first feature set of the previous image and a second feature set of the current image; Using the optical flow information matrix to infer the position of a second key point set in the previous image in the current image to store it as a third key point set; Calculating a current homography matrix between the first key point set and the third key point set; and Using the current homography matrix to project the target image to a corresponding position on the current image. The view-transformed image processing method as described in claim 8, wherein the second key point set is obtained by transforming the first key point set on the scene template by a prior homography matrix. The image processing method for view angle conversion as described in claim 8, wherein the previous image is the previous frame of the current image, and the step of extracting the feature of the previous image and the current image includes: Detecting a plurality of first candidate feature points and a plurality of second candidate feature points from the previous image and the current image respectively; and Performing a feature matching operation on the first candidate feature points and the second candidate feature points to generate the first feature set of the previous image and the second feature set of the current image. The image processing method for view angle conversion as described in claim 10 further includes: Updating the third key point set in the current image based on the second key point set in the previous image and a base matrix, wherein the base matrix indicates a corresponding view angle conversion relationship between the previous image and the current image. The image processing method for view angle conversion as described in claim 11, wherein the base matrix is generated by the following steps: Based on the first feature set of the previous image and the second feature set of the current image, the base matrix is calculated. The image processing method for view angle conversion as described in claim 11, wherein the step of updating the third key point set includes: Using the base matrix to convert the second key point set in the previous image into a plurality of straight lines in the current image; and Based on the straight lines, updating the third key point set in the current image. The image processing method for view angle conversion as described in claim 13, wherein the step of updating the third key point set in the current image based on the straight lines includes: Determining whether each of the third key point set in the current image is located on one of the corresponding straight lines in the current image; and Deleting at least one noise key point in the third key point set to update the third key point set in the current image, wherein each of the at least one noise key point is not located on the corresponding straight line in the current image.