TW202330067A - Calculating method of ball trajectory based on image and radar - Google Patents

Abstract

The purpose of the present invention is to provide a ball trajectory calculation method based on images and radar, wherein, when calculating the trajectory of a moving ball hit by a player's golf shot, the entire trajectory of the ball can be calculated by using a camera and a radar sensor, which are provided separately or in the same sensing device, to match ball trajectory information, calculated through the analysis of images acquired by the camera, and ball trajectory information, calculated through the analysis of radar sensing data collected by the radar sensor.

Description

Calculation Method of Ball Trajectory Based on Image and Radar

The present invention belongs to a method of calculating the trajectory of the ball by using the image captured by the camera of the ball moving as the golfer hits the ball with a golf club and the sensing data of the radar sensor.

In the case of ball sporting events, especially golf, attempts have been made to accurately sense the physical properties of the ball being struck by the golfer and use the sensed values for play analysis or to Realized as images for application in the field of analog golf such as so-called screen golf.

As representative devices that sense the moving ball after being hit by a golfer to calculate the motion characteristic information of the ball, there are camera sensing devices based on image analysis and radar sensing devices based on radar signals.

As a device that separately analyzes the image of the moving ball acquired by the camera to calculate the position information of the ball, and uses the calculated position information to generate the trajectory information of the moving ball, the sensing device based on image analysis Although it has the advantage of being able to detect the position of the ball quite accurately within the shooting angle of the camera, due to the performance of the camera and the surrounding brightness and other environmental effects and image processing performance, there are cases where the shooting angle is out of the way of the camera and the ball is far away from the camera. The problem of detecting the ball.

In addition, the problem with the way the camera senses is that since it cannot sense the entire flight range of the ball being hit and moved, it is also completely unable to sense when the ball is flying, such as a fade or a draw. The sphericity of a ball that curves after the middle, so that its trajectory cannot be calculated.

As a device capable of transmitting a radar signal toward a moving ball and receiving a signal reflected from the ball, and by analyzing the received signal to calculate information on various characteristics of the motion of the ball, the radar sensing device is advantageous in that due to the The sensing range is wide, the ball can be sensed even if the distance between the device and the ball is long to some extent, and difficult-to-sense information such as the spin of the ball can be sensed with considerable accuracy by image analysis , but is susceptible to peripheral interference to the radar signal, may contain noise, and thus has the problem of not being able to accurately calculate information such as the trajectory of a ball.

In particular, in the initial section of the predetermined distance from the position where the ball is hit, noise is likely to be generated in the radar signal due to the interference of people and golf clubs, and the distortion of the signal is serious, so there is a possibility that it is difficult to calculate the movement of the ball. track problem.

In order to calculate the trajectory of the ball moving as the golfer hits the ball with the golf club, people are trying to make up for each other's shortcomings by mixing the camera sensing method and the radar sensing method, but for the camera sensing method, due to the acquisition of two-dimensional Image, the data about trajectory is also 2D data, and for the radar sensing method, since the position information of the ball in the calculation space is basically 3D data in the three-dimensional space, there are limitations in using 2D data and 3D data at the same time , and whether the camera and radar are implemented as one device or used independently, there is a problem that it is difficult to fuse the data processing results of the two due to the different sensing positions.

Existing patent documents: Korea Authorized Patent Publication No. 10-1848864; Korean Patent Publication No. 2019-0085152; Japanese Granted Patent Publication No. 4773176; Korea Granted Patent Publication No. 10-1845503.

technical problem The present invention aims to provide a method for calculating the movement trajectory of a ball hit by a golfer's golf shot, which can utilize a camera and a radar sensor provided in one sensing device or independently of each other, and will pass through the camera. The ball trajectory calculation method based on the image and radar is used to match the ball trajectory information calculated by the analysis and calculation of the acquired image with the ball trajectory information calculated by the analysis and calculation of the radar sensing data collected by the radar sensor to calculate the entire trajectory of the ball movement.

Technical solutions The method for calculating ball trajectory based on image and radar according to an embodiment of the present invention includes: acquiring and analyzing the image of the moving ball by the camera, and calculating the image processing trajectory as the moving trajectory of the ball; The radar signal sent by the radar sensor is reflected by the moving ball and the radar sensing data about the movement of the ball is calculated by the received signal, and a plurality of radar ball trajectory candidates are respectively generated; the image processing trajectory and the plurality of radars are calculated a matching rate of each of the ball track candidates; and a step of calculating a ball track by integrating the radar ball track candidate corresponding to the highest matching rate among the matching rates and the image processing track.

In addition, preferably, the step of calculating the image processing trajectory includes: calculating, as the image processing trajectory, a trajectory from the hitting position of the ball until the ball object corresponding to the ball can be detected from the acquired image.

Furthermore, preferably, the step of calculating the ball trajectory by the integration includes the step of extracting a trajectory from a position where the image processing trajectory ends on the radar ball trajectory candidate corresponding to the highest matching rate and connecting to the image processing trajectory.

In addition, preferably, the step of calculating the image processing trajectory includes: the step of collecting the image captured by the camera based on the hitting time point of the ball; and detecting the corresponding The calculation of the ball object of the ball and the line connecting the detected ball objects until the ball object cannot be detected from the image is the step of the image processing trajectory.

In addition, preferably, the camera is a single camera that captures a two-dimensional image from the perspective of the direction in which the ball is traveling, and the step of calculating the image processing trajectory includes: connecting the hitting position of the ball to the detected and acquired The step of calculating the image processing trajectory by detecting the position of the ball object in the image and displaying it on the two-dimensional image corresponding to the shooting angle of the camera.

In addition, preferably, the step of separately calculating the plurality of radar ball trajectory candidates includes: using the collected radar sensing data to calculate ball position coordinate information; using the ball position coordinate information to calculate a basic ball trajectory as a radar-based ball trajectory the step of trajectory; and the step of changing the viewing angle and scaling of the basic ball trajectory to generate the plurality of radar ball trajectory candidates respectively.

In addition, preferably, the step of calculating the matching rate includes: comparing the image processing trajectory with the plurality of radar ball trajectory candidates to quantify the similarity of the trajectory, and the step of calculating the ball trajectory includes: The step of determining the radar ball trajectory candidate with the highest numerical value of the similarity of the trajectory as the radar ball trajectory candidate corresponding to the highest matching rate; The trajectory from position is connected to the image processing trajectory to calculate the ball trajectory step.

In addition, preferably, the step of calculating the plurality of radar ball trajectory candidates respectively includes: a step of calculating a basic ball trajectory based on the collected radar sensing data; Each of the radar ball trajectory candidates is respectively generated as a two-dimensional view form corresponding to the viewing angle of the two-dimensional image of the camera, and the step of calculating the matching rate includes: combining the image processing trajectory as a two-dimensional image with the two A plurality of radar ball track candidates in the dimensional view form are respectively compared to the step of numericalizing the similarity of the track, and the step of calculating the ball track includes: a two-dimensional view of the radar ball track candidate with the highest numerical value of the similarity of the track The trajectory on is connected to the image processing trajectory to calculate the ball trajectory step.

On the other hand, the ball trajectory calculation method based on image and radar according to another embodiment of the present invention includes: by acquiring and analyzing the image of the moving ball by the camera, calculating the image processing trajectory as the moving trajectory of the ball to generate a corresponding The step of two-dimensional image at the angle of view of the camera; the radar sensing data about the ball movement calculated by collecting and analyzing the radar signal sent by the radar sensor reflected by the moving ball and the received signal, will be based on the radar The step of generating the ball trajectory as a two-dimensional view form corresponding to the angle of view of the two-dimensional image of the camera; image processing on the two-dimensional image corresponding to the angle of view of the camera between the radar-based ball trajectory of the two-dimensional view form adjusting the angle of view and scaling of the radar-based ball trajectory in the two-dimensional view form by trajectory matching; and calculating the ball trajectory by matching the image processing trajectory with the radar-based ball trajectory.

In addition, preferably, the step of calculating the image processing trajectory to generate a two-dimensional image corresponding to the viewing angle of the camera includes: a step of collecting images acquired by the camera based on the hitting time of the ball; A ball object corresponding to the ball is detected in the image acquired at the hitting position of the ball, and a line connecting the detected ball objects until the ball object cannot be detected from the image is generated on the two-dimensional image as the image processing The step of trajectory, the step of generating the radar-based ball trajectory into a two-dimensional view form corresponding to the viewing angle of the two-dimensional image of the camera includes: using the collected radar sensing data to calculate the coordinate information of the ball position in three-dimensional space Steps: using the ball position coordinate information to calculate a radar-based ball trajectory in three-dimensional space; and generating the radar-based ball trajectory in three-dimensional space on a two-dimensional view corresponding to the viewing angle of the two-dimensional image of the camera A step of.

The effect of the invention The ball trajectory calculation method based on image and radar of the present invention has the ability to use a camera and radar installed in one sensing device or independently of each other when calculating the trajectory of the ball being hit by the golfer's golf shot. The sensor matches the ball trajectory information calculated by the analysis and calculation of the image acquired by the camera with the ball trajectory information calculated by the analysis and calculation of the radar sensing data collected by the radar sensor to calculate the effect of the entire trajectory of the ball movement.

The specific content of the ball trajectory calculation method using the radar sensing data of the hit ball and the radar sensing device using the same will be described below with reference to the accompanying drawings.

The method for calculating the ball trajectory based on images and radars of the present invention will be described in detail below with reference to the accompanying drawings.

A ball trajectory calculation method according to an embodiment of the present invention belongs to a method of calculating the ball trajectory that is hit when a golfer hits a ball with a golf club in a field such as a golf course by using the results sensed by a camera sensing device and a radar sensing device. A method of ball trajectory in the overall interval of ball movement.

1 is a diagram illustrating a state in which a camera device 100 and a radar sensor 200 disposed behind a golfer 10 sense a state in which a ball 20 moves in a golf course as the golfer 10 makes a golf shot.

The ball trajectory calculation method based on image and radar according to an embodiment of the present invention not only includes the camera device 100 and the radar sensor 200 which are independently set up separately as shown in FIG. It may also include the case where the camera and the radar sensor are configured together in one sensing device housing to perform image analysis and radar analysis respectively.

In addition, the camera may be a single camera that acquires a two-dimensional image from its angle of view, or may be a camera device that consists of two cameras stereoscopically like a launch monitor. The stereoscopic camera device can calculate the three-dimensional position coordinates of the ball by using the data on the two-dimensional image captured by each camera.

Regardless of whether it is a single camera or a stereoscopic camera device, when the ball is far away from the camera device, the ball cannot be detected from the image, so the camera device can only sense the ball from the initial starting position of the ball. Measure the moving ball and calculate the trajectory of the ball up to the interval.

The basic feature of the present invention is that the trajectory of the ball is calculated by analyzing the image captured by the camera from the first start of the ball to the period when the ball can be detected from the image captured by the camera, and the subsequent trajectory of the ball is calculated using the radar sensing device. The sensory radar sensing data is calculated, and the image-based trajectory is matched with the radar-based trajectory to calculate the entire ball trajectory.

The ball trajectory calculation method based on image and radar in an embodiment of the present invention can be connected to the camera device and the radar sensor to use the image analysis results of the camera device and the radar analysis results of the radar sensor to calculate the ball trajectory. The execution of the client device may also be executed by a component used for information calculation in a sensing device equipped with a camera and a radar.

As mentioned above, the main body of the ball trajectory calculation method of the present invention can be realized through various forms of hardware settings or software configurations. In the following, the main body that executes the ball trajectory calculation method of the present invention is named "trajectory calculation device" for description.

FIG. 1 shows a trajectory calculation device 300 connected to a camera device 100 and a radar sensor 200 (communicatively connected by wire or wirelessly) as a device for calculating a ball trajectory calculation method according to an embodiment of the present invention. . However, the present invention is not limited thereto, and as described above, the form of the trajectory calculation device can be variously realized.

For example, the above-mentioned trajectory calculation device can be implemented as a separate terminal (a terminal that connects the camera and the radar sensor in a wired/wireless manner to perform calculations for trajectory calculation using data received from the camera and radar sensor), It can also be implemented in smartphones that drive applications for trajectory calculations by connecting to cameras and radar sensors and performing calculations.

FIG. 2 shows an example of images captured by the camera at a predetermined shooting speed in the state shown in FIG. 1 .

(a) of FIG. 2 is the image 100a of the nth frame captured by the camera at its viewing angle, and (b) of FIG. 2 is the image 200b of the n+mth frame. There are multiple frames of images between the image 100a and the image 100b.

The image processing unit provided in the camera device or the trajectory calculation device for calculating the trajectory of the ball can detect the object corresponding to the ball in the image for each frame of the image to specify the position of the ball object in the image.

As shown in (a) of FIG. 2 , the camera device can find and detect the object corresponding to the ball in the image 100 a , that is, the ball object BO, and can store the detected position of the ball object BO on the two-dimensional image.

Figure 2(a) shows the state where the ball object moves a predetermined distance from the initial starting position PO. Although it cannot be seen from the two-dimensional image, since the ball is actually flying forward, the actual ball is gradually moving away from the camera device. As the camera captures the image, the size of the spherical object on the image becomes smaller and smaller.

In this way, the size of the spherical object gradually decreases, and then in the image of the n+mth frame as shown in (b) of Figure 2, its size becomes so small that it is difficult to judge whether the detected object Ob is a spherical object, and finally The spherical object will not be detected from the image.

The trajectory calculation device can use the result of detecting the spherical object in each image captured by the camera (using the detection result of the spherical object detected in the image until the spherical object cannot be detected), and calculate the connection as shown in (c) of Figure 2 The trajectory of the position P0, P1, P2 ... Pf of each spherical object is the image processing trajectory TRI.

Since the camera that acquires the image is a single camera, the images captured by the camera (100a, 100b, etc.) are all two-dimensional images corresponding to the viewing angle of the camera, so the image processing trajectory TRI shown in (c) of Figure 2 can be connected by two-dimensional A line is created of the position of the spherical object on the image 110 .

Since the above-mentioned image processing trajectory TRI is a trajectory until the ball object can be detected in the image captured by the camera, it is impossible to export the movement of the ball from when the actual ball is far away from the camera and the ball object can no longer be detected in the image captured by the camera. Therefore, the trajectory of the ball in the interval from when the ball object cannot be detected can be obtained by using the radar sensing data collected by the radar sensor.

3 and 4 show that the radar sensor calculates the position coordinate information of the moving ball through the analysis of the radar signal.

A radar sensor is a device that basically uses the Doppler Effect of radar (Radar) to calculate various information such as the movement characteristics of a golf club shot. As shown in Figure 3, it may include signal transmission part 210 , signal receiving part 220 , signal analyzing part 230 and information calculating part 240 .

The radar sensor may be placed on or near the ground behind a predetermined distance from a position of a ball to be hit by a user, and may be configured to transmit a signal of a specific frequency toward the direction of motion of the ball to be hit to move at the set position. radar signal, and track the ball as it is struck while moving while receiving and analyzing the reflected waves from the ball.

The signal transmission part 210 may be configured to transmit a specific radar (Radar) signal in an aimed direction, and, although not shown in the drawing, may include a transmission antenna for transmitting the radar signal.

The signal receiving part 220 is configured to receive the reflected wave signal returned by the radar signal sent by the signal sending part 210 after being reflected from the ball. Due to the Doppler effect, the frequency of the signal transmitted by the signal transmission unit 210 changes with respect to the reflected wave signal transmitted by the signal transmission unit 210 and reflected from the ball, thereby causing a Doppler shift (Doppler shift). That is, the signal receiving unit 120 receives a Doppler shifted signal.

The signal receiving unit 220 may be configured to have a plurality of receiving antennas for receiving the reflected wave signal, so as to calculate information such as the position, velocity, trajectory, and direction angle of the moving ball by using the phase differences of the signals received by the plurality of receiving antennas. .

(a) of FIG. 4 schematically shows an example of the structure of the above-mentioned signal receiving unit 220. As shown in the figure, when the signal receiving unit 220 is properly arranged and provided with three or more receiving antennas including RA1, RA2, and RA3, Each receiving antenna RA1, RA2, and RA3 can receive the reflected wave signal received from the ball B, and can calculate the trajectory (altitude angle) and direction angle of the moving ball B using the phase difference of the signal between each receiving antenna, respectively.

For example, according to the arrangement of the receiving antennas in the signal receiving unit 220 shown in (a) of FIG. 4 , the phase difference of the signals received by RA1 and RA2 can be used to calculate the trajectory of the moving ball B, and the The phase difference of the signals received by RA3 is used to calculate the direction angle of the moving ball B.

In addition, as the signal receiving unit 220 receives the reflected wave signal, the distance between the ball B and the signal receiving unit 220 can be easily calculated. Therefore, when the signal receiving unit 220 receives the reflected wave signal reflected from the ball B, The distance to the ball B, the trajectory angle of the ball B, and the direction angle information of the ball B can be obtained, and the position coordinate information of the ball B can be calculated by using the above information. When the position coordinate information of the ball B is calculated, it can also be used to calculate the speed of the ball.

As such, the signal analysis part 230 may be configured to calculate phase value information by analyzing the radar signal reflected by the ball at every predetermined time interval when the ball moves, and calculate position coordinate information of the moving ball at predetermined time intervals.

That is, the signal analysis unit 230 may calculate radar sensing data about the movement of the ball by analyzing the reflected wave signal of the radar received as described above, and the radar sensing data may be phase information of the ball calculated at predetermined time intervals, or It may be position coordinate information of the ball calculated based on the phase information.

The information calculation unit 240 can use the position coordinate information of the ball calculated by the signal analysis unit 230 to calculate the movement trajectory of the ball (to distinguish it from the image processing trajectory calculated by image analysis, it is called “ball trajectory based on radar”).

The radar signal is reflected by the ball and received by the signal receiving unit. Distortion and phase jitter of the radar signal may occur due to interference between the user and the club, buckling of the ball surface, etc., and the information calculation unit 240 compensates for the distortion or phase of the signal. jitter, and calculate the trajectory of the ball through statistical analysis of radar sensing data.

However, the signal distortion and phase jitter of the radar signal are more serious, especially in the initial movement interval corresponding to the predetermined distance interval after the ball starts. In the initial movement interval, the ball trajectory calculated by using radar sensing data may be different from the actual The trajectory of the ball is different, so the trajectory calculated by image analysis may be more accurate in the initial range of the ball's motion.

In the present invention, the method of calculating the radar-based ball trajectory by using radar sensing data can be realized through various methods.

For example, it is possible to analyze the reliability of each radar sensing data, assign a weight value to each data according to the reliability, and reflect the weight value, and calculate the ball trajectory through statistical analysis such as RANSAC algorithm, or based on the corresponding radar Partial trajectories are obtained from highly reliable data in the ball position coordinate range of the sensing data, and the radar-based ball trajectory is calculated by fitting the partial trajectories to the radar sensing data through the physics engine.

In the present invention, the specific method of calculating the radar-based ball trajectory using radar sensing data is not characteristic in itself, but is characterized in that, regardless of the method, the radar-based ball trajectory calculated by any method is combined with image processing. Trajectories are combined to calculate the entire trajectory of the ball's movement.

The specific process of the ball trajectory calculation method based on image and radar according to an embodiment of the present invention will be described below.

FIG. 5 is a flow chart illustrating the process of the ball trajectory calculation method based on imagery and radar according to an embodiment of the present invention.

When the camera device and the radar sensor are placed at a predetermined distance based on the user's hitting position (or, the sensing device that realizes the camera and the radar sensor as one device is placed at a predetermined distance), and the target When the hit ball is in place, the camera device senses the ball, so that it may become ball-ready (meaning that the camera device and the radar sensor are ready to sense the ball) (S100).

As the ball is hit (S110), the image is acquired by the camera (S121), and the signal analysis unit of the radar sensor can collect radar sensing data on the movement of the ball by analyzing the signal received by the signal receiving unit (S131) .

The camera device detects an object corresponding to a portion where the actual ball is shot, that is, a ball object, in each of the captured images ( S122 ).

In the detection of the ball object, for example, ball object requirements such as the shape of the golf ball and the roundness of the object can be set in advance, and the object in the image that meets the requirements can be detected as the ball object, or the corresponding object can be stored in advance. A template of the ball, and detect the ball object by comparing the object detected in the image with the template and judging the similarity.

However, it is not possible to detect the ball object in all the images of the entire range of the ball movement. When the distance between the camera and the ball is greater than the predetermined distance, since the position of the camera is fixed, the images acquired by the camera correspond to The size of the part due to the ball becomes so small that the ball object can no longer be detected.

When the detection of the ball object in the image acquired from the camera fails (S123), that is, when the ball object can no longer be detected from the image, the camera device or the trajectory calculation device can use the image detection of each frame until the ball object cannot be detected. The ball object calculates the image processing trajectory and displays it on the camera image (S124).

For example, as shown in (c) of FIG. 2 , the positions P0, P1, P2 ... Pf of the spherical objects respectively detected in the images obtained from the camera can be connected to calculate the image processing trajectory TRI, and it is shown in the two-dimensional image 110 superior.

On the other hand, the radar sensor can use the collected radar sensing data to calculate the coordinate information of the ball position as shown in (b) of FIG. 4 ( S132 ). Then, a radar-based ball trajectory may be calculated using the ball position coordinate information ( S133 ). Here, the radar-based ball trajectory calculated in step S133 is referred to as a "basic ball trajectory".

As mentioned above, the ball position coordinates calculated by using the radar sensing data collected by the radar sensor are 3D coordinate information, so the basic ball trajectory can be a trajectory in a 3D space calculated by using the 3D coordinate information.

The radar sensor or the trajectory calculation device may respectively generate a plurality of radar ball trajectory candidates from the basic ball trajectory as described above ( S134 ). The plurality of radar ball trajectory candidates may respectively be trajectories generated with different viewing angles and/or scaling ratios for the same basic ball trajectory.

For example, the basic ball trajectory can be generated as a two-dimensional view form corresponding to the viewing angle of the two-dimensional image of the camera, and a plurality of different viewing angles and/or scaling ratios can be generated by changing the viewing angle and zoom ratio of the two-dimensional view. Radar ball trajectory candidates for 2D view morphologies.

The trajectory calculation device implementing the method of the present invention can calculate the matching rate of the above-mentioned image processing trajectory and each of the plurality of radar ball trajectory candidates (S140), and select the image processing trajectory with the highest matching rate, wherein, it can be obtained by Selecting a radar ball trajectory candidate combination ( S150 ), and integrating the image processing trajectory and the radar ball trajectory candidate with the highest matching rate to calculate a final ball trajectory ( S160 ).

The calculation of the matching rate of the image processing trajectory and the radar ball trajectory candidate through the above steps S134 to S160 and the calculation of the final ball trajectory through the corresponding matching will be described in more detail with reference to FIGS. 6 and 7 .

(a) of Fig. 6 shows the image processing trajectory TRI calculated by the analysis and calculation of the image acquired by the camera on the two-dimensional image 110 corresponding to the angle of view of the camera, and (b) to (f) of Fig. Radar ball trajectory candidates generated from basic ball trajectories calculated from sensing data.

The above-mentioned image processing trajectory TRI is the information on the two-dimensional image 110, so it is 2D data, and the basic ball trajectory calculated by using the radar sensing data is information on the three-dimensional space, which is 3D data, so it is difficult to directly combine 2D data and 3D data , so it is preferable to convert the radar-based basic ball trajectory as 3D data into 2D data to generate radar ball trajectory candidates, and, for the matching of image processing trajectory and radar ball trajectory candidates, it is preferable to generate radar ball trajectory candidates such as two The form of the two-dimensional view 251, 252 etc. of the form of the three-dimensional image 110.

As shown in (b) to (f) of Figure 6, the radar ball trajectory candidates show the basic ball trajectory as a two-dimensional view modality, and each radar ball trajectory candidate (two-dimensional views 251, 252, 253, 254, 255, etc.) It can be generated according to different viewing angles and/or scaling ratios while adjusting the viewing angle and scaling ratio within a preset range, and the generated number can also be preset.

The more the number of radar ball trajectory candidates, the more accurate the ball trajectory can be exported, but it may take more time to calculate the matching rate with the image processing trajectory. An appropriate amount of time would be required to generate radar ball trajectory candidates.

Each radar ball trajectory candidate generated as a 2D view modality is generated from the same basic ball trajectory, but due to being generated at a different 2D view angle and/or zoom, each 2D view Although the trajectories within are generated from the same basic ball trajectories, they may be displayed in different forms as a 2D view.

Assume that (b) of Figure 6 is the first radar ball trajectory candidate TRr1 in the form of the two-dimensional view 251, (c) in Figure 6 is the second radar ball trajectory candidate TRr2 in the form of the two-dimensional view 252, and (d) in Figure 6 is The third radar ball trajectory candidate TRr3 in the form of the two-dimensional view 253, (e) of FIG. 6 is the fourth radar ball trajectory candidate TRr4 in the form of the two-dimensional view 254, and (f) of FIG. The radar ball trajectory candidate TRr5, in addition to (b)~(f) in Figure 6, can also generate a larger number of radar ball trajectory candidates.

Hereinafter, the description will be made in the case of five radar ball trajectory candidates in (b)-(f) of Fig. 6 .

As described above, when the image processing trajectory TRI and the plurality of radar ball trajectory candidates TRr1˜TRr5 are provided, the trajectory calculation means calculates the matching rate of the image processing trajectory TRI and each of the plurality of radar ball trajectory candidates TRr1˜TRr5 to decide the matching Higher rate image processing trajectory-radar ball trajectory candidate combination.

That is, the 2D image 110 and the 2D views 251 to 255 are in the same form and are easy to compare with each other, and each of the image processing trajectory TRI as the 2D image and the radar ball trajectory candidates TRr1 to TRr5 in the form of the 2D view can be The comparison is performed to quantify the similarity of the trajectory, and the image processing trajectory-radar ball trajectory candidate combination with the highest numerical value of the trajectory similarity can be determined as the combination with the highest matching rate.

The similarity of the trajectory between the image processing trajectory and the radar ball trajectory candidate can be expressed by numericalizing the similarity in the shape and geometric characteristics of the trajectory. At this time, in addition to the morphological characteristics of the trajectory, when judging the ball on the trajectory When the quality is high, it can be judged together with the spherical quality, so as to take into account the similarity in the spherical quality of the image processing trajectory and the radar ball trajectory candidate.

In the example shown in Figure 6, it is possible to calculate the matching rate of the image processing trajectory TRI and the first radar ball trajectory candidate TRr1, and then calculate the matching rate of the image processing trajectory TRI and the second radar ball trajectory candidate TRr2, and then calculate the image The matching rate of the processing trajectory TRI and the third radar ball trajectory candidate TRr3, and then calculating the matching rate of the image processing trajectory TRI and the fourth radar ball trajectory candidate TRr4 compares each radar ball trajectory one-to-one with the image processing trajectory And analyze the similarity of the trajectory to find the combination with the highest matching rate, that is, the highest similarity.

When it is assumed that the image processing trajectory TRI in FIG. 6 has the highest matching rate with the fourth radar ball trajectory candidate TRr4 shown in (e) of FIG. 6, as shown in FIG. 7, the trajectory calculation device can integrate the ( The image processing trajectory TRI shown in a) and the fourth radar ball trajectory candidate TRr4 shown in (b) of FIG. 7 are used to calculate the final ball trajectory fTR shown in (c) of FIG. 7 .

That is, the final ball trajectory can be calculated by determining the image processing trajectory TRI-radar ball trajectory candidate TRr4 combination with the highest trajectory similarity value and connecting the radar ball trajectory candidate TRr4 to the image processing trajectory TRI.

When connecting the radar ball trajectory candidate TRr4 to the image processing trajectory TRI, it can be calculated by connecting the trajectory part TRr from the last position of the image processing trajectory TRI in the radar ball trajectory candidate TRr4 to the last position of the image processing trajectory TRI as shown in Fig. 7 (c) shows the final ball trajectory fTR on the two-dimensional image 330 .

On the other hand, an image and radar based ball trajectory calculation method according to another embodiment of the present invention will be described with reference to the flowchart shown in FIG. 8 .

Compared with the ball trajectory calculation method of the embodiment shown in FIG. 5 mentioned above, the ball trajectory calculation method of the embodiment shown in FIG. Aspects of trajectory are common, but differences exist in the process of matching radar-based ball trajectories with image processing trajectories to calculate final ball trajectories.

First, when the camera device and the radar sensor are placed at a predetermined distance based on the user's hitting position (or the sensing device that realizes the camera and radar sensor as one device is placed at a predetermined distance), and the When the hit ball is in place, the camera device senses the ball, so that it may become ball ready (meaning that the camera device and the radar sensor are ready to sense the ball) (S200).

As the ball is hit (S210), the image is acquired by the camera (S221), and the signal analysis unit of the radar sensor can collect radar sensing data on the movement of the ball by analyzing the signal received by the signal receiving unit (S231) .

The camera device detects the object corresponding to the part where the actual ball is photographed in each captured image, that is, the ball object (S222), and when the detection of the ball object in the image obtained from the camera fails (S223), that is, when When the ball object is detected in the frame, the camera device or the trajectory calculation device can calculate the image processing trajectory using the ball object detected from each frame of image until the ball object cannot be detected, and display it on the camera image (S224).

The process of the above-mentioned steps S221 to S224 is substantially the same as the process of steps S121 to S124 in Figure 5 mentioned above. As shown in (c) of Figure 2, the position of the ball object detected by the image obtained from each camera can be ( P0 , P1 , P2 . . . Pf ) are connected to calculate the image processing trajectory TRI, and display it on the two-dimensional image 110 .

On the other hand, the radar sensor can use the collected radar sensing data to calculate the coordinate information of the ball position in the three-dimensional space ( S232 ).

Then, the radar-based ball trajectory can be calculated by using the coordinate information of the ball position in the three-dimensional space ( S233 ).

The trajectory calculation device generates the radar-based ball trajectory in the three-dimensional space as a two-dimensional view form corresponding to the viewing angle of the two-dimensional image of the camera ( S234 ). For example, a two-dimensional view form of one of (b) to (f) of FIG. 6 may be generated.

The trajectory calculating means adjusts the viewing angle and the radar-based ball trajectory of the two-dimensional viewing form in such a manner that the radar-based ball trajectory in the two-dimensional viewing form matches the image processing trajectory on the two-dimensional image corresponding to the viewing angle of the camera. /or scaling (S240). This adjustment can continue until the matching is complete.

When the viewing angle and/or scaling of the radar-based ball trajectory in the two-dimensional view form is thus adjusted to complete the matching between the image processing trajectory and the radar-based ball trajectory (S250), the trajectory calculation device matches the radar-based ball trajectory with the image processing trajectory trajectory matching to calculate the final ball trajectory (S260).

For example, the final ball trajectory can be calculated by extracting and concatenating the portion of the trajectory from the last position of the image processing trajectory to the last position of the image processing trajectory for the radar-based ball trajectory of the two-dimensional view modality described above.

As described above, the ball trajectory calculation method based on images and radars of the present invention can use the camera and radar installed in one sensing device or separately when calculating the moving trajectory of the ball hit by the golfer's golf shot. The sensor matches the ball trajectory information calculated by the analysis and calculation of the image acquired by the camera with the ball trajectory information calculated by the analysis and calculation of the radar sensing data collected by the radar sensor to calculate the entire trajectory of the ball movement, so it has the ability to pass the camera The trajectory of a part of the interval generated by the image analysis of the radar sensor is extended to the entire interval to generate the characteristics of the trajectory.

10: golfer 100: camera device 100a, 100b: image 110, 330: 2D image 20: ball 200: Radar sensor 210: Signal sending department 220: Signal receiving department 230: Signal Analysis Department 240: Department of Information Computing 251~255: Two-dimensional view 300: trajectory calculation device B: ball BO: Ball Object Ob: object P0~Pf: the position of the ball object RA1~RA3: receiving antenna Sr: St: TRI: image processing trace TRr1~TRr5: Candidates for radar ball trajectory fTR: final ball trajectory rP1, rP2, rP3 t1, t2, t3 x1, x2, x3 y1, y2, y3 z1, z2, z3

FIG. 1 is a diagram illustrating a state in which a camera device and a radar sensor disposed behind a golfer sense the movement of a ball on a golf course as the golfer makes a golf shot. FIG. 2 is a diagram illustrating an example of an image captured by a camera at a predetermined shooting speed in the state shown in FIG. 1 and an example of an image processing trace generated using the same. FIG. 3 and FIG. 4 are used to illustrate that the radar sensor calculates the position coordinate information of the moving ball through the analysis of the radar signal. FIG. 5 is a flow chart showing a ball trajectory calculation method based on imagery and radar according to an embodiment of the present invention. FIG. 6 is a diagram for explaining generation and matching of a video processing trajectory and a plurality of radar ball trajectory candidates according to the flowchart of FIG. 5 . 7 is a diagram illustrating calculation of a final ball trajectory by combining a radar ball trajectory candidate having the highest matching rate with an image processing trajectory among the plurality of radar ball trajectory candidates shown in FIG. 5 and the image processing trajectory. FIG. 8 is a flow chart showing a ball trajectory calculation method based on imagery and radar according to another embodiment of the present invention.

10: golfer

100: camera device

20: ball

200: Radar sensor

300: trajectory calculation device

Claims (10)

A ball trajectory calculation method based on images and radar, characterized in that it includes: A step of calculating an image processing trajectory as the moving trajectory of the ball by acquiring and analyzing the image of the moving ball by the camera; A step of generating a plurality of radar ball trajectory candidates respectively by collecting and analyzing radar sensing data about the movement of the ball calculated by the radar signal transmitted by the radar sensor and received by the signal reflected by the moving ball; calculating a matching rate between the image processing trajectory and each of the plurality of radar ball trajectory candidates; and A step of calculating a ball trajectory by integrating the radar ball trajectory candidate corresponding to the highest matching rate and the image processing trajectory. The ball trajectory calculation method based on image and radar according to claim 1, characterized in that, The steps of calculating the image processing trajectory include: The step of calculating the image processing trajectory is the trajectory from the hit position of the ball until the ball object corresponding to the ball can be detected from the acquired image. The ball trajectory calculation method based on image and radar according to claim 1, characterized in that, The steps to calculate the ball trajectory through this integration include: A step of extracting a trajectory from the position where the image processing trajectory ends on the radar ball trajectory candidate corresponding to the highest matching rate and connecting to the image processing trajectory. The ball trajectory calculation method based on image and radar according to claim 1, characterized in that, The steps of calculating the image processing trajectory include: the step of collecting the image captured by the camera based on the hitting time of the ball; and Detecting a ball object corresponding to the ball from the image captured at the hitting position of the ball, and calculating a line connecting the detected ball objects until the ball object cannot be detected from the image is the step of the image processing trajectory. The ball trajectory calculation method based on image and radar according to claim 1, characterized in that, The camera is a single camera that captures a two-dimensional image from a perspective looking in the direction the ball is traveling. The step of calculating the image processing trajectory includes: displaying the two-dimensional image corresponding to the shooting angle of the camera by connecting the hitting position of the ball to the position of the ball object detected until the ball object can be detected from the acquired image Steps to calculate the image processing trajectory are described above. The ball trajectory calculation method based on image and radar according to claim 1, characterized in that, The steps of calculating the plurality of radar ball track candidates respectively include; A step of calculating the coordinate information of the ball position by using the collected radar sensing data; the step of calculating a base ball trajectory as a radar-based ball trajectory using the ball position coordinate information; and A step of generating the plurality of radar ball trajectory candidates by changing the viewing angle and scaling of the basic ball trajectory. The ball trajectory calculation method based on image and radar according to claim 6, characterized in that, The steps to calculate this match rate include: comparing the image processing trajectory with the plurality of radar ball trajectory candidates respectively to quantify the similarity of the trajectory, The steps to calculate the ball trajectory include: a step of determining the radar ball trajectory candidate with the highest numerical value of the similarity of the trajectory as the radar ball trajectory candidate corresponding to the highest matching rate; and A step of calculating the ball trajectory by connecting a trajectory from a position corresponding to the last position of the image processing trajectory among the determined radar ball trajectory candidates to the image processing trajectory. The ball trajectory calculation method based on image and radar according to claim 1, characterized in that, The steps of calculating the plurality of radar ball track candidates respectively include: the step of calculating a basic ball trajectory based on the collected radar sensing data; and a step of changing the angle of view and zoom ratio of the basic ball trajectory to generate each of the plurality of radar ball trajectory candidates as a two-dimensional view form corresponding to the angle of view of the two-dimensional image of the camera, The steps to calculate this match rate include: comparing the image processing trajectory as a two-dimensional image with the plurality of radar ball trajectory candidates in the two-dimensional view form to quantify the similarity of the trajectory, The steps to calculate the ball trajectory include: A step of calculating the ball trajectory by connecting the trajectory on the two-dimensional view of the radar ball trajectory candidate having the highest similarity value to the image processing trajectory. A ball trajectory calculation method based on images and radar, characterized in that it includes: A step of generating a two-dimensional image corresponding to the viewing angle of the camera by acquiring and analyzing the image of the moving ball by the camera, and calculating an image processing trajectory as the trajectory of the ball; Radar-based trajectory of the ball is generated as a 2D image corresponding to the camera The steps of the two-dimensional view form of the angle of view; Adjusting the viewing angle and scaling of the radar-based ball trajectory of the 2D view modality in such a way that the radar-based ball trajectory of the 2D view modality matches the image processing trajectory on the 2D image corresponding to the viewing angle of the camera steps; and The step of calculating a ball trajectory by matching the image processing trajectory with the radar-based ball trajectory. The ball trajectory calculation method based on image and radar according to claim 9, characterized in that, The step of calculating the image processing trajectory to generate a two-dimensional image corresponding to the viewing angle of the camera includes: the step of collecting the image captured by the camera based on the hitting time of the ball; and A ball object corresponding to the ball is detected from the image acquired at the hit position of the ball, and a line connecting the detected ball objects until the ball object cannot be detected from the image is generated on the two-dimensional image as the The steps of the image processing trajectory, The step of generating the radar-based ball trajectory into a 2D view form corresponding to the viewing angle of the 2D image of the camera includes: A step of calculating coordinate information of the ball position in three-dimensional space by using the collected radar sensing data; using the ball position coordinate information to calculate a radar-based ball trajectory in three dimensions; and A step of generating the radar-based ball trajectory in the three-dimensional space on a two-dimensional view corresponding to the viewing angle of the two-dimensional image of the camera.