JP7083334B2 - Image processing equipment, image processing methods, and programs - Google Patents

Description

The present invention relates to an image processing apparatus, an image processing method, and a program.

In recent years, it has become widely practiced to capture a moving image of a shot with a mobile terminal such as a smartphone and check the form while playing golf.

Japanese Unexamined Patent Publication No. 2008-158984

However, the current situation is that although the form can be confirmed from such a video, the trajectory of the ball is not displayed.

The present invention has been made in view of the above circumstances, and provides an image processing device, an image processing method, and a program capable of obtaining information about a shot by a simple operation and diagnosing a shot of a player. That is one of the purposes.

One aspect of the present invention that solves the problems of the above-mentioned conventional example is an image processing device, which is a means for acquiring moving image data including a series of N still image data obtained by capturing a golf shot, and the acquisition. A ball detecting means for detecting an image portion of a golf ball from each of the still image data included in the moving image data, and a flight distance estimating means for estimating a flight distance using information on the position of the detected image portion of the golf ball. And includes.

According to the present invention, information about a shot can be obtained by a simple operation, and a player's shot can be diagnosed.

It is a block diagram which shows the structural example of the image processing apparatus which concerns on embodiment of this invention. It is a functional block diagram which shows the example of the image processing apparatus which concerns on embodiment of this invention. It is explanatory drawing which shows the operation example of the image processing apparatus which concerns on one aspect of Embodiment of this invention. It is a functional block diagram which shows the example of the image preprocessing part of the image processing apparatus which concerns on one aspect of embodiment of this invention. It is a flowchart which shows the operation example of the image processing apparatus which concerns on one aspect of Embodiment of this invention. It is a functional block diagram which shows the example of the image preprocessing part of the image processing apparatus which concerns on another aspect of embodiment of this invention. It is a flowchart which shows the operation example of the image processing apparatus which concerns on another aspect of embodiment of this invention. It is explanatory drawing which shows the operation example of the image processing apparatus which concerns on another aspect of embodiment of this invention. It is a functional block diagram which shows the example of the display information generation part of the image processing apparatus which concerns on one aspect of Embodiment of this invention. It is explanatory drawing which shows the example of the method of estimating an elevation angle in one aspect of the image processing apparatus which concerns on embodiment of this invention. It is explanatory drawing which shows the example of the calculation method of the flight distance in one aspect of the image processing apparatus which concerns on embodiment of this invention. It is another explanatory diagram which shows the example of the method of calculating the flight distance in one aspect of the image processing apparatus which concerns on embodiment of this invention. It is still another explanatory diagram which shows the example of the method of calculating the flight distance in one aspect of the image processing apparatus which concerns on embodiment of this invention. It is a functional block diagram which shows the example of the display information generation part of the image processing apparatus which concerns on another aspect of embodiment of this invention. It is explanatory drawing which shows the example of the method of estimating the meet rate in the image processing apparatus which concerns on another aspect of embodiment of this invention.

An embodiment of the present invention will be described with reference to the drawings. As an example of the image processing device 10 according to the embodiment of the present invention, as illustrated in FIG. 1, the control unit 11, the storage unit 12, the operation unit 13, the display unit 14, and the image pickup unit 15 communicate with each other. It is a mobile terminal such as a smartphone provided with the unit 16.

Here, the control unit 11 is a program control device such as a CPU, and operates according to a program stored in the storage unit 12. In an example of this embodiment, the control unit 11 controls the image pickup unit 15 to capture moving image data. Further, the control unit 11 receives input of moving image data of a player's shot during sports play using a ball, such as an imaged golf shot, from the image pickup unit 15. Further, the control unit 11 detects the position of the ball from the input moving image data, and applies the information on the position of the detected ball to a predetermined process. The details of the operation of the control unit 11 will be described later.

The storage unit 12 is a memory device or the like, and holds a program executed by the control unit 11. The storage unit 12 also operates as a work memory of the control unit 11.

The operation unit 13 is, for example, a touch panel, a button, or the like, and outputs the content of the operation to the control unit 11 based on the user's operation. The display unit 14 includes a display and the like arranged so as to be superimposed on the touch panel. The display unit 14 displays an image according to an instruction input from the control unit 11.

The image pickup unit 15 is a camera and captures image data according to an instruction input from the control unit 11. In one example of the present embodiment, the imaging unit 15 continuously captures still image data for a plurality of frames at predetermined timings, and outputs data including the still image data for the plurality of frames as moving image data. do.

The communication unit 16 is an interface or the like for transmitting / receiving information to / from a communication line such as a mobile phone communication network or a network. The communication unit sends data via the communication line according to the instructions input from the control unit 11, and outputs the data received through the communication line to the control unit 11.

Next, an operation example of the control unit 11 will be described. The control unit 11 according to an example of the present embodiment executes the program stored in the storage unit 12, and as illustrated in FIG. 2, the image pickup control unit 21, the image preprocessing unit 22, and the pre-detection unit. A configuration that functionally includes a processing unit 23, a detection processing unit 24, a display information generation unit 25, and a display control unit 26 is realized.

[First aspect of the embodiment]
Upon receiving the user's instruction, the image pickup control unit 21 controls the image pickup unit 15 to capture moving image data. In the following description, golf will be taken as an example as a sport. Further, the user places the image processing device 10 behind the golf player to capture still image data for a plurality of frames before and after including the time of the shot. Here, the rear of the player means the side opposite to the direction in which the ball is to be thrown with respect to the position of the player. Therefore, as illustrated in FIG. 3A, the captured image data is located slightly to the left of the center of the frame when the player is right-handed. Further, the ball is initially located near the lower side of the center of the frame, and after the shot, the ball moves upward to the frame and is in a state of being imaged. It is assumed that the angle of view and the like are adjusted so that the ball does not come out of the frame even after the shot.

The image preprocessing unit 22 processes the moving image data captured by the image pickup unit 15.

In the first aspect of the image processing apparatus 10 according to the embodiment of the present invention, the image preprocessing unit 22 includes a correction unit 31 and a conversion unit 32, as illustrated in FIG.

The correction unit 31 performs so-called image stabilization processing on each of the still image data for a plurality of frames included in the moving image data. Since this process can be performed by adopting a widely known method, detailed description thereof is omitted here.

The conversion unit 32 converts each of the still image data for a plurality of frames into grayscale image data and performs smoothing processing. Here, the average of the pixel values in the block is calculated for each of the still image data of a plurality of frames for each block of a predetermined size, and a smoothed image in which all the pixels in the block are set as the average value is generated. You just have to do the processing. The predetermined size may be, for example, a size sufficiently smaller than the number of pixels of the imaged golf ball when the imaged golf ball is relatively close. In the smoothing process, various other widely known methods can be adopted as long as the golf ball does not lose the imaged pixels.

In this aspect of the present embodiment, the converted still image data for a plurality of frames output by the conversion unit 32 is output as the processing target data (FIG. 3B).

The detection pre-processing unit 23 performs the following processing on the processing target data output by the image pre-processing unit 22, for example, still image data for a plurality of frames after conversion. Here, it is assumed that the processing target data includes N still image data or N frame still image data.

The detection preprocessing unit 23 generates difference image data between the still image data of the i-th frame and the still image data of the i + 1th frame. Here, i is a natural number of N-2 or less. In the following description, the frames of the still image data will be numbered 1, 2, ... N in the order of the earliest captured time. This difference image data is image data having the same size as the still image data included in the processing target data.

The detection preprocessing unit 23 sequentially selects each pixel of the i-th difference image data as a pixel of interest, and the pixel in the still image data of the i-th frame corresponding to the pixel of interest and the still image of the i + 1th frame. If the pixels in the data have the same pixel value, or if the difference between the pixel values is less than a predetermined threshold value, it is set to "0", otherwise it is set to "1".

The detection preprocessing unit 23 repeats the above processing to obtain the difference image data between the still image data of the first and second frames and the difference image data between the still image data of the second and third frames. ..., The difference image data for the N-1 frame of the difference image data between the still image data of the N-1st and the Nth frames is generated (FIG. 3 (c)).

In the example of FIG. 3C, the difference image data is a pixel whose pixel value is "0", in other words, a pixel having no difference is black, and a pixel value is "1", in other words, a difference. It is represented as binarized image data in which a certain pixel, that is, a significant pixel is white.

Further, the detection preprocessing unit 23 performs a morphology conversion process for each of the generated difference image data, specifically, an opening process when the difference pixel is white, and a closing process when the difference pixel is black. A process may be applied to remove a mass of significant pixels less than a predetermined number of pixels as noise. The planned number of pixels is set to a size sufficiently smaller than the number of pixels of the imaged golf ball when a relatively nearby golf ball is imaged.

The detection processing unit 24 executes detection processing of the position where the ball is imaged from each of the difference image data processed by the detection preprocessing unit 23. FIG. 5 shows an example of the processing of the detection processing unit 24. The detection processing unit 24 starts the iterative processing of the processes S2 to S6 described below for each k value with k = 1, 2, ... N-2 (S1). Further, the detection processing unit 24 sets a detection target area where the ball is assumed to be imaged for each of the k-th difference image data and the k + 1-th difference image data (S2). In the present embodiment, the k-th difference image is the difference image data between the k-th frame and the k + 1-th frame, and the k + 1-th difference image data is the difference image data between the k + 1-th frame and the k + 2nd frame. And.

Specifically, the detection processing unit 24 detects a group of significant pixels including a group of pixels having a pixel number larger than a predetermined number of pixels as a range in which the player is imaged. Then, when the player is on the left side of the center of the frame, the detection processing unit 24 identifies the range on the right side of the right side of the circumscribed rectangle of the pixel block detected as the player as the detection target area. In FIG. 3C, the detection target region is indicated by reference numeral R. When the player is on the right side of the center of the frame, the detection processing unit 24 identifies the range on the left side of the left side of the circumscribed rectangle of the pixel block detected as the player as the detection target area.

If the area R is specified for the k-th difference image data in the previous iterative process from the processes S2 to S6, the process S2 may process only the k + 1-th difference image data.

The detection processing unit 24 extracts a block of significant pixels in the region R defined in the processing S2 from the k-th and k + 1-th difference image data, respectively. Then, the detection processing unit 24 detects the group of significant pixels including the group of significant pixels in the range of a predetermined number of pixels as a temporary ball candidate among the group of the extracted significant pixels (S3).

The detection processing unit 24 extracts information on the position of the lump of significant pixels for each of the lumps of significant pixels used as temporary ball candidates in the kth difference image data (S4). Further, the detection processing unit 24 has a significant pixel corresponding to a mass of significant pixels extracted at the kth position, which is a temporary ball candidate, at a position in the k + 1th difference image data specified by the information of the extracted position. It is determined whether or not there is a lump (S5). Hereinafter, the mass of significant pixels corresponding to the mass of significant pixels extracted at the kth position, which is a temporary ball candidate, is referred to as a corresponding significant pixel mass.

When it is determined in the process S5 that there is a corresponding significant pixel block (S5: Yes), the detection processing unit 24 obtains coordinate information for specifying the position of the significant pixel block, in other words, the circumscribing rectangle of the significant pixel block. It is the position of the ball candidate in the kth difference image data (S6). If it is determined in the process S4 that there is no corresponding significant pixel block (S5: No), the detection processing unit 24 continues the process assuming that there is no ball candidate in the kth difference image data.

The detection processing unit 24 repeatedly executes the processes of processes S2 to S6 for k = 1, 2, ... N-2. Then, the detection processing unit 24 detects a mass of significant pixels as ball candidates from each of the k = 1, 2, ... N-2nd difference image data.

The detection processing unit 24 repeats the following processing for the still image data of the k = 1, 2, ... N-2nd frame (S7). In this iterative process, the detection processing unit 24 uses the ball candidate newly set as the ball candidate in the k-th difference image data as the position of the golf ball in the still image data of the k-th frame, that is, information for specifying the frame. The position of the golf ball is stored in the storage unit 12 in association with the value of k (S8). If there is no new ball candidate in the k-th difference image data, the detection processing unit 24 assumes that the golf ball could not be detected in the still image data of the k-th frame, and there is no position information. Information is stored in the storage unit 12.

As a result, the detection processing unit 24 generates information indicating the position of the golf ball with respect to the still image data of the frame in which the mass of significant pixels presumed to be a golf ball is detected. The information indicating the position of the golf ball may be a set of coordinate information that specifies the circumscribing rectangle of the mass of significant pixels detected as the golf ball.

The display information generation unit 25 corresponds to at least a part of the still image data of the i-th frame acquired by the image pickup control unit 21 with respect to the position of the golf ball detected by the still image data of the i-1st and earlier frames, in other words. A line segment connecting the center of the circumscribing rectangle of the pixel block and the position of the golf ball detected by the still image data of the i-th frame is generated as display information related to the still image data of the i-th frame. Here, i = 2, 3, ... N-1.

If the golf ball cannot be detected in the still image data of the i-th frame, the display information generation unit 25 extrapolates the golf ball from the position of the golf ball detected in the still image data of the i-1st and earlier frames. The position of the golf ball in the still image data of the i-th frame may be estimated. In this case, the display information generation unit 25 converts the line segment connecting the position of the golf ball detected by the still image data of the i-1st frame and the estimated position into the still image data of the i-th frame. Generated as such display information.

Alternatively, when the golf ball cannot be detected in the still image data of the i-th frame, the display information generation unit 25 includes the still image data of the i-1st and earlier frames and the still image data of the i + 1th and subsequent frames. The position of the golf ball may be estimated from the still image data of the i-th frame by interpolating from the position of the golf ball detected in each of the above.

The display control unit 26 sequentially displays the still image data of each frame acquired by the image pickup control unit 21 in response to the playback instruction of the user. Further, when the display information related to the still image data of the frame being reproduced or displayed is generated, the display control unit 26 superimposes the display information on the still image data and displays the display information. As a result, as illustrated in FIG. 3D, a figure or a line segment A representing the flight trajectory of the golf ball is superimposed and displayed on the image of the golf shot.

[Modified example of detection processing in the first aspect]
Further, in the process of the process S6 of the detection processing unit 24, when there are a plurality of significant pixel clusters that are ball candidates in the kth difference image data, the detection processing unit 24 further performs the following processing to perform the ball candidate. May be narrowed down. Here, k> 2.

That is, when one is detected from each golf ball, the detection processing unit 24 is detected as a point representing the position of each golf ball in the k-2nd and k-1th still image data, for example, a golf ball. The center point of the pixel block is defined as a plurality of points representing the positions of the golf balls detected in each still image data. The detection processing unit 24 acquires information on an extension line connecting points that are supposed to represent the positions of these golf balls. Then, the detection processing unit 24 sets the mass of the significant pixels of the ball candidate existing on the extension line specified by the information in the k-th difference image data as the position of the golf ball in the k-th still image data.

In addition to the above description, other than the k-th still image data, information for specifying a line segment connecting points representing the positions of golf balls detected in two or more still image data or an extension line of the line segment is used. You may get it. Also in this case, in the k-th difference image data, the mass of significant pixels of the ball candidate existing on the extension line specified by the information is set as the position of the golf ball in the k-th still image data.

[Second aspect of the embodiment]
Next, a second aspect of the image processing apparatus 10 according to the embodiment of the present invention will be described. In this second aspect, the control unit 11 functionally realizes the configuration illustrated in FIG. 2 as in the first aspect, but there are differences in the operation of each unit.

Similar to the first aspect, the image pickup control unit 21 controls the image pickup unit 15 to take an image of moving image data in response to a user's instruction. The user places the image processing device 10 behind the golf player on the opposite side of the direction in which the ball flies, and has the user capture still image data for a plurality of frames before and after including the time of the shot. As for the image data to be captured, for example, as illustrated in FIG. 3A, when the player is right-handed, the player is located slightly to the left of the center in the frame. Further, in this image data, the ball is initially located near the lower side of the center of the frame, and after the shot, the ball moves to the upper side of the frame and is imaged. It is assumed that the angle of view and the like are adjusted so that the ball does not come out of the frame even after the shot.

In this second aspect, the image preprocessing unit 22 includes a correction unit 41, a player area specifying unit 42, and a conversion unit 43, as illustrated in FIG. The correction unit 41 performs so-called image stabilization processing on each of the still image data for a plurality of frames included in the moving image data. Since this process can be performed by adopting a widely known method, detailed description thereof is omitted here.

The player area specifying unit 42 identifies the area in which the player is imaged from the still image data of each frame. This process uses, for example, a segmentation network such as segnet; https://arxiv.org/pdf/1511.00561.pdf, and an object detection network such as YOLO; https://arxiv.org/abs/1506.02640 using deep learning. It may be done by using. In addition to this, any method may be adopted as long as it can extract an object having a predetermined feature from the image data.

The player area specifying unit 42 identifies the area in which the player is imaged in the still image data of each frame. The area in which the player is imaged may be a rectangular area circumscribing the pixel group recognized as the player.

The conversion unit 43 obtains the average of the pixel values in the block for each block of a predetermined size for each of the still image data for a plurality of frames, and smoothes all the pixels in the block as the average value. Performs the process of generating an image. As the smoothing process of the conversion unit 43, various other widely known methods can be adopted as long as the golf ball does not lose the imaged pixel. The predetermined size may be a size sufficiently smaller than the number of pixels of the imaged golf ball when the imaged golf ball is relatively close.

The detection pre-processing unit 23 performs the following processing on the processing target data output by the conversion unit 43 of the image pre-processing unit 22, for example, still image data for a plurality of frames after smoothing processing. Here, it is assumed that the processing target data includes N still image data of smoothed images for N pieces or N frames.

The detection preprocessing unit 23 acquires a smoothed image corresponding to the still image data of the i-th frame. In addition, i is a natural number of N-2 or less, and in the following, the smoothed image corresponding to the still image data of the i-th frame is referred to as the i-th smoothed image. In the following description, the frames of the still image data will be numbered 1, 2, ... N in the order of the earliest captured time.

In the detection preprocessing unit 23, of the N-2 smoothed images, p is a predetermined integer of 2 or more, and when i> p, the p + 1 frames from the i-pth to the i-th frame. Calculate the average of still image data and generate average image data.

Then, a smoothed image corresponding to the average image and the still image data of the i + 1st frame, that is, the difference image data between the i + 1th smoothed image is generated. The difference image data is image data having the same size as the captured still image data, and each pixel is specified by performing the following processing.

That is, the detection preprocessing unit 23 sequentially selects each pixel of the average image. In the following, the selected pixel will be referred to as a "pixel of interest" for the sake of explanation. Then, in the detection preprocessing unit 23, when the pixel of the average image corresponding to the pixel of interest and the pixel of the i + 1th smoothed image have the same pixel value or the difference between the pixel values is less than a predetermined threshold value. Is "0", otherwise it is "1". Here, for example, the difference image data is a pixel whose pixel value is "0", in other words, a pixel having no difference is black, a pixel having a pixel value of "1", in other words, a pixel having a difference, that is, a significant pixel. Is represented as white binarized image data.

The detection preprocessing unit 23 repeats this process to obtain the difference image data between the first to p-1th average images and the pth smoothed image, that is, the pth difference image data. , The difference image data between the second to pth average image and the p + 1st smoothed image, that is, the p + 1st difference image data, ..., Npth to N-1th Each difference image data between the average image of the above and the Nth smoothed image is acquired.

The detection preprocessing unit 23 is a pixel block having a size larger than the number of pixels when the golf ball is imaged in the k-th smoothed image instead of the average image, and is a pixel block having a color different from that of the peripheral pixels. , That is, a process of filling the pixel block with the colors of peripheral pixels, that is, a noise removing process may be executed to acquire an image that is substantially a background. In this case, the detection preprocessing unit 23 acquires the difference image data between the substantially background image and the k + 1th smoothed image.

The detection preprocessing unit 23 performs a morphology conversion process for each acquired difference image data, specifically, an opening process when the difference pixel is white, or a closing process when the difference pixel is black. Apply the processing of. Further, the detection processing preprocessing unit 23 may remove a block of significant pixels less than a predetermined number of pixels as noise. The predetermined number of pixels may be a size sufficiently smaller than the number of pixels of the imaged golf ball when the imaged golf ball is relatively close.

The detection processing unit 24 executes the detection processing of the position where the ball is imaged from each of the acquired difference image data by the detection preprocessing unit 23. In this embodiment of the present embodiment, the detection processing unit 24 executes the processing shown in FIG. 7 instead of the processing exemplified in FIG. 5 of the first aspect.

When executing the process shown in FIG. 7, when the area specified by the player area specifying unit 42 exists on the left side of the center of the difference image data, the detection processing unit 24 identifies the area on the right side of the area as the detection target area. do. Further, when the area specified by the player area specifying unit 42 exists on the right side of the center of the difference image data, the range on the left side of the area is identified as the detection target area.

The detection processing unit 24 extracts a block of significant pixels in the detection target region from each difference image data. Then, the detection processing unit 24 detects as a ball candidate a group of significant pixels including a group of significant pixels as many as the number of the number of significant pixels in a predetermined range of the number of pixels among the group of the extracted significant pixels in each difference image data (S11). ).

The detection processing unit 24 determines the position of each ball candidate detected from each difference image data, for example, the coordinate information for specifying the rectangle circumscribing the significant pixel block that has become the ball candidate, in the frame corresponding to the detected difference image data. In association with the information for specifying the still image data, it is recorded as a candidate for a position where the golf ball is presumed to be imaged in the kth still image data, in other words, as a ball position candidate (S12). The detection processing unit 24 may execute the processing using the numerical value k as it is if it is the k-th difference image data.

As a result, the position of the ball candidate recorded in association with the information for identifying the k-th difference image data represents the candidate of the position where the golf ball is presumed to be imaged in the k-th still image data. .. In the following, for the sake of explanation, the candidates for the positions where the golf ball is presumed to be imaged in the kth still image data are referred to as Bk_1, Bk_2, and so on. The position candidates estimated here are referred to as ball position candidates below.

The detection processing unit 24 has one ball from each of the ball position candidate in the k-1st still image data, the ball position candidate in the kth still image data, and the ball position candidate in the k + 1th still image data. The possible order sequences obtained by extracting the position candidates are listed (S13). Here, k is a natural number of 2 or more and N-1 or less, that is, k = 2,3 ... N-1. That is, in this process S13, the order sequence listing of the ball position candidates is performed. In the above description, an example of using three still image data will be given, but any number of still image data may be used as long as it is three or more.

For example, in FIG. 8, there are three ball position candidates in the k-1st still image data, Bk-1_1, Bk-1_2, and Bk-1_3, and the ball position candidates in the kth still image data are Bk_1, Bk_2. There are two, and the case where there are three ball position candidates in the k + 1st still image data of Bk + 1_1, Bk + 1_2, and Bk + 1_3 is shown. In FIG. 8, the kth still image data shows the ball position candidates of Bk-1_1, Bk-1_2, and Bk-1_3 with broken lines, and the k + 1th still image data shows Bk-1_1 and Bk. The ball position candidates of -1_2 and Bk-1_3 and the ball position candidates of Bk_1 and Bk_2 are shown by broken lines.

In this case, the chronological order in which one can be retrieved from each is
Bk-1_1 → Bk_1 → Bk + 1_1
Bk-1_1 → Bk_1 → Bk + 1_2
Bk-1_1 → Bk_1 → Bk + 1_3
Bk-1_1 → Bk_2 → Bk + 1_1
Bk-1_1 → Bk_2 → Bk + 1_2
Bk-1_1 → Bk_2 → Bk + 1_3
Bk-1_2 → Bk_1 → Bk + 1_1
…
Bk-1_3 → Bk_2 → Bk + 1_2
Bk-1_3 → Bk_2 → Bk + 1_3
There are 18 ways. The detection processing unit 24 lists this order in the processing S13. Each of these sequences is a candidate for the movement trajectory of the ball.

In the process S13, it is not always necessary to enumerate all possible sequence sequences. For example, a sequence sequence in which the difference between ball position candidates included adjacent to the sequence sequence is larger than a predetermined threshold value may be excluded from the enumeration. .. The difference between the ball position candidates may be, for example, a difference in either the X-axis or the Y-axis direction.

The detection processing unit 24 selects, among the movement locus candidates, the movement locus candidate that best meets the predetermined conditions (S14). That is, in this process S14, a movement locus candidate that matches the conditions is selected.

Here, the condition is, for example, a condition related to the position represented by the ball position candidate included in the movement locus candidate. In the following example, the vertical direction of the still image data is the Y axis, and the horizontal direction is the X axis. Here, the Y-axis has a positive direction vertically below, and the X-axis has a positive direction to the right. At this time, the specific conditions are, for example:
(1) In three adjacent ball position candidates Bi-1_a → Bi_b → Bi + 1_c in chronological order, the Y coordinate of the ball position candidate at the time after Bi + 1_b is the ball position candidate at the time before Bi_b. When it is smaller than the Y coordinate, the value obtained by subtracting the value of the Y coordinate of Bi_b from the value of the Y coordinate of Bi + 1_c is greater than the value ΔYi obtained by subtracting the value of the Y coordinate of Bi-1_a from the value of the Y coordinate of Bi_b. The value of 1 is small (2) In the three adjacent ball position candidates Bi-1_a → Bi_b → Bi + 1_c in chronological order, Bi + 1_b, that is, the Y coordinate of the ball position candidate at a later time is Bi_b, That is, when it is larger than the Y coordinate of the ball position candidate at the earlier time, Bi_b is obtained from the value of the Y coordinate of Bi + 1_c rather than the value ΔYi obtained by subtracting the value of the Y coordinate of Bi-1_a from the value of the Y coordinate of Bi_b. The value obtained by subtracting the value of the Y coordinate of ΔYi + 1 is large. (3) Three adjacent ball position candidates in chronological order Bi- 1_a → Bi_b → Bi + 1_c The value ΔXi obtained by subtracting the value of the X coordinate of Bi-1_a from the value of the X coordinate and the value ΔXi + 1 obtained by subtracting the value of the X coordinate of Bi_b from the value of the X coordinate of Bi + 1_c are positive or negative. With the same sign, | ΔXi + 1-ΔXi |, which is the absolute value of the difference in the amount of change in the X-axis direction, is as small as possible.
It is a condition of. In other words, (1) shows the case of decelerating, and (2) shows the case of accelerating.

In the example of FIG. 8, by the processing of the detection processing unit 24, for example, ・ Bk-1_1 → Bk_1 → Bk + 1_1 satisfies the condition (1) above, and (3) the amount of change in the X-axis direction. Since the absolute value of the difference between them is relatively small, it is judged to be appropriate as a movement trajectory.
・ In Bk-1_1 → Bk_1 → Bk + 1_2, although it is rising, ΔYk + 1 is larger than ΔYk, that is, it is accelerating, so the condition (1) is not satisfied and the change in the X-axis direction is not satisfied. Since the sign of the quantity has changed, the condition (3) is not satisfied either. Therefore, it is judged that it is not appropriate as a movement trajectory.
-Bk-1_1->Bk_1-> Bk + 1_3 also does not satisfy the condition (3) as in the example of Bk-1_1->Bk_1-> Bk + 1_2.
…
-Bk-1_2->Bk_1-> Bk + 1_1 satisfies the conditions (1) and (3), but the absolute value of the difference in the amount of change in the X-axis direction is Bk-1_1->Bk_1-> Bk + 1_1. Since it is larger, it is judged that it is not suitable as a movement trajectory.
…
As described above, it is determined whether or not each movement locus candidate is appropriate as a movement locus.

In the example of FIG. 8, as in the above example, Bk-1_1 → Bk_1 → Bk + 1_1 is determined as the most appropriate movement locus.

The detection processing unit 24 uses information for identifying the position of the ball position candidate included in the most appropriate movement locus as the position of the golf ball in the still image data of the corresponding frame, in other words, the position where the golf ball is imaged. Is output as information representing.

The display information generation unit 25 detects the position of the golf ball for at least a part of the still image data of the i-th frame acquired by the image pickup control unit 21 and the golf ball detected by the still image data of the i-1st and earlier frames. The line segment connecting the positions of is generated as display information related to the still image data of the i-th frame. Here, i = 2, 3, ... N-1. The position of the golf ball may be the center of the circumscribed rectangle of the corresponding pixel block.

If the golf ball cannot be detected in the still image data of the i-th frame, the display information generation unit 25 extrapolates the golf ball from the position of the golf ball detected in the still image data of the i-1st and earlier frames. The position of the golf ball in the still image data of the i-th frame may be estimated. In this case, the display information generation unit 25 converts the line segment connecting the position of the golf ball detected by the still image data of the i-1st frame and the estimated position into the still image data of the i-th frame. Generated as such display information.

Alternatively, when the golf ball cannot be detected in the still image data of the i-th frame, the display information generation unit 25 includes the still image data of the i-1st and earlier frames and the still image data of the i + 1th and subsequent frames. The position of the golf ball may be estimated from the still image data of the i-th frame by interpolating from the position of the golf ball detected in each of the above.

The display control unit 26 sequentially displays the still image data of each frame acquired by the image pickup control unit 21 in response to the playback instruction of the user, and generates display information related to the still image data of the frame being played or displayed. If so, the display information is superimposed on the still image data and displayed. As a result, similar to the one illustrated in FIG. 3D, a figure or a line segment representing the flight trajectory of the golf ball is superimposed and displayed on the image of the golf shot.

[Example of display information]
The image processing device 10 of the above-described embodiment estimates the position where the golf ball is imaged in each frame of the moving image data obtained by imaging, draws a line segment connecting the estimated positions, and draws golf. It displayed the flight trajectory of the ball. However, the image processing device 10 of the present embodiment is not limited to this, and various information regarding golf shots may be estimated and displayed.

[Estimation of head speed]
In one embodiment of the present embodiment, the control unit 11 of the image processing device 10 detects a range in which the head of the golf club of the player is imaged in the processing of the display information generation unit 25 from each still image data. For the detection of the head part of this golf club, a widely known method such as a method using a general object detection network such as YOLO; https://arxiv.org/abs/1506.02640 can be adopted, so here. Detailed explanation of is omitted. Further, in the following example, as described above, the vertical direction of the still image data is the Y axis, and the horizontal direction is the X axis.

The display information generation unit 25 calculates the ratio between the number of pixels and the actual length, particularly in meters in the present embodiment, by using the result of detection of the area where the image is captured by the player. In the first aspect described above, among the groups of significant pixels in the difference image data, the group of significant pixels including the pixels having a pixel number larger than the predetermined number of pixels is detected as the range in which the player is imaged. Therefore, the display information generation unit 25 has the height of a rectangle circumscribing the block of significant pixels, and the pixel unit is the number of pixels of the height of the player.

Further, in the second aspect, the value representing the height of the area specified by the player area specifying unit 42 in pixel units is taken as the number of pixels of the height of the player. Then, the display information generation unit 25 divides the predetermined height of the player by the number of pixels of the height of the player to obtain the ratio r. The height of the player may be an average value of the heights of adults, or the user may be requested to input information on the height of the player.

The display information generation unit 25 is between the frame when the Y-axis value is the minimum and the frame when the Y-axis value is the maximum among the positions of the golf club heads detected from each frame. Extract the number of frames in. For example, if the frame when the Y-axis value is the minimum is the i-th frame and the frame when the Y-axis value is the maximum is the j-th frame among the positions of the golf club heads. , The display information generation unit 25 extracts (ji) as the number of frames between them. Then, the display information generation unit 25 multiplies the imaging interval between frames by the number of frames obtained here, and sets the time t during which the golf club is swung down from the highest position to the lowest position in seconds. Ask. The detected position of the head of the golf club may be the position of the center of the rectangle circumscribing the portion where the head is photographed. The imaging interval between frames is, for example, 1/30 if the frame speed is 1/30 second.

Further, the display information generation unit 25 includes coordinate information (xi, yi) at the position of the golf club head of the i-th frame and coordinate information (xj, yj) at the position of the golf club head of the j-th frame. The Euclidean distance D is calculated in pixel units on the still image data of. Further, the display information generation unit 25 calculates the head speed s by multiplying the distance D by the ratio r calculated in advance and dividing by the time t obtained above. The formula is as follows. :
s = D · r / t

The image processing device 10 may display information on the head speed calculated based on the above method together with the movement locus of the golf ball.

[Estimation of initial position]
The display information generation unit 25 uses the detection result of the region imaged by the player, is a line segment extending the lower side of the region, and is a straight line representing the trajectory of the flight of the golf ball, and is the Y axis of the still image data. The position where the line segment extending in the positive direction intersects may be estimated as the initial position of the golf ball.

The display control unit 26 indicates that a golf shot was performed at the initial position of the still image data when sequentially displaying the still image data of each frame acquired by the image pickup control unit 21 in response to the playback instruction of the user. The figures may be overlapped and arranged to emphasize the impact. The figure displayed by the display control unit 26 may be a circle, a square, or another polygon.

The display information generation unit 25 may use the information of this initial position in the next processing.

[Estimation of flight distance (first method)]
In one embodiment of the present embodiment, the display information generation unit 25 includes a head speed estimation unit 51, an initial position estimation unit 52, a temporary flight distance calculation unit 53, and an elevation angle estimation unit 54, as illustrated in FIG. , The meet rate estimation unit 55 and the flight distance estimation unit 56 are included.

As described above, the head speed estimation unit 51 has coordinate information (xi, yi) at the position of the golf club head of the i-th frame and coordinate information (xj) at the position of the golf club head of the j-th frame. , Yj) and the Euclidean distance D is calculated in pixel units on the still image data. Further, the display information generation unit 25 estimates the head speed s = D · r / t by multiplying the distance D by the ratio r calculated in advance and dividing by the time t obtained above. The initial position estimation unit 52 estimates the initial position by the above method. In this example of the present embodiment, the initial position estimation unit 52 further extrapolates the time according to the position by using the image pickup time of the frame in which the golf ball is detected and the information of the detection position. Is performed, and the time t0 at the time of impact, which is the last time when the ball was at the previously estimated initial position, is estimated in frame units.

In the present embodiment, the meet rate is tentatively determined to obtain the tentative flight distance, which is the tentative flight distance. The value of this provisional flight distance is later used in the process of estimating the meet rate. For this purpose, the temporary flight distance calculation unit 53 calculates the ball speed bs = 1.25 × s by using the head speed s estimated by the head speed estimation unit 51 with the temporary meet rate as 1.25. The meet rate means the ratio of the initial velocity of the ball divided by the head speed. Then, the temporary flight distance calculation unit 53 sets the value obtained by multiplying the ball speed bs by four as the temporary flight distance.

[Estimation of elevation angle]
The elevation angle estimation unit 54 estimates the elevation angle at the time of launching the ball by the process illustrated in FIG. As illustrated in FIG. 11, the elevation angle estimation unit 54 calculates the height H of the region F imaged by the player using the detection result of the region F. The elevation angle estimation unit 54 is a virtual line segment parallel to the X-axis located at a predetermined ratio (0.9 in FIG. 11) to the height H of the region F from the lower side of the region F. Set λ (S21). In the following processing, it is assumed that this line segment λ is a line segment parallel to the ground at the height of the imaging unit 15.

The elevation angle estimation unit 54 passes through the position B where the golf ball is detected, and sets a virtual line segment λ'parallel to the X axis (S22). Here, the position B may be any time point, but in FIG. 11, the position B is the position where the golf ball is first detected. Then, the elevation angle estimation unit 54 calculates the distance h between the virtual line segment λ and λ'set so far (S23). The distance h between the virtual line segment λ and λ'is the absolute value of the difference between the value of the Y coordinate of λ and the value of the Y coordinate of λ', and the unit is a pixel.

The elevation angle estimation unit 54 calculates the ratio r of the length to the pixel based on the ratio of the information regarding the height of the player and the height H of the region F in which the player is imaged, which has already been described (S24). Further, the elevation angle estimation unit 54 multiplies the distance h by this ratio r to calculate the length information h'= r × h in metric units (S25).

FIG. 12 shows a state in which the traveling direction of the launched golf ball is to the right when taking an image. Assuming that the line segment CB connecting the position C of the image pickup unit 15 and the position B of the golf ball, this length h'is the intersection P of the position C of the image pickup unit 15 and the midline of the player and the midline. It corresponds to the length PQ between the line segment CB and the intersection Q with the line segment CB.

Here, it is assumed that the initial position E of the golf ball is the intersection of the median line and the ground, and the intersection of the line segment extending the line segment CB and the ground is L. Further, the length of the line segment EL is d2, and the length of the line segment EB is d1. Assuming that the angle of elevation to be obtained is α and the angle formed by the line segment CL and the ground is β, as illustrated in FIG. 12, when the ground is assumed to be a plane, the angle LCP is also β from the theorem regarding parallel lines. Therefore, the triangle CPQ and the triangle LEQ have similar figures.

が成立するので、これをαについて整理して、

を得る。 Furthermore, from the law of sines,

Is established, so organize this for α and

To get.

と書くことができる。 On the other hand, since the distance D to the subject when the height of the player is imaged with the number of pixels H can be estimated from the information of the angle of view of the image pickup unit 15, D / h = d2 / 0.9H from the above similar relationship. -H holds,

Can be written.

となる。 Further, d1 can be obtained by multiplying the estimated initial velocity of the ball by the time required for the golf ball to move from the initial position E to the point B. This time is 1 if the frame at point B is the kth frame (kt0) and the time fps between frames (for example, 1/30 second) using the previously obtained t0. Multiply by / 30)
(Kt0) ・ fps
Because it can be asked

Will be.

と表すことができる。 The angle β is

It can be expressed as.

The elevation angle estimation unit 54 has D, H, h', r (ratio of the actual length (meter) to the pixel), k, which are known values according to the equations (2), (3), and (4). , T0, fps, bs to obtain d1, d2, and β (S26). Then, the estimated value α of the elevation angle is obtained by substituting the obtained value into the equation (1) (S27).

If the ratio of the height of the camera to the height is other than 0.9, the constant 0.9 in Eq. (2) is also changed accordingly.

[Estimation of meet rate (first method)]
The meet rate estimation unit 55 estimates the meet rate as follows. Note that FIG. 12 will also be referred to in the following description. Further, in FIG. 12, the intersection of the line drawn vertically downward from the position C of the imaging unit 15 and the ground is defined as O. As already mentioned, the length of this line segment CO is 0.9H (when the ratio of the height of the camera to the height is 0.9. When different values are used, that value is used).

を求める。なお、ここでは地面は平面と仮定する。そしてミート率推定部５５は、この位置Ｂ′に対応する静止画像データ上での画素のＹ座標の値を、プレーヤーの撮像されている領域Ｆの上辺から

だけ下がった、つまりこの値だけＹ軸正の方向に移動した位置ｙとする。以下、この位置ｙを通り、Ｘ軸に平行な仮想的な線分をλ″とする。 The meat rate estimation unit 55 uses the temporary flight distance m'= 4 · bs obtained by the temporary flight distance calculation unit 53, and takes an image assuming that the golf ball is at the point B'on the ground at this flight distance position. As the angle β'when looking at this point B'from the position C of the part 15,

Ask for. Here, it is assumed that the ground is a flat surface. Then, the meet rate estimation unit 55 sets the value of the Y coordinate of the pixel on the still image data corresponding to this position B'from the upper side of the region F in which the player is imaged.

It is assumed that the position y is lowered by, that is, moved in the positive direction of the Y axis by this value. Hereinafter, a virtual line segment that passes through this position y and is parallel to the X axis is defined as λ ″.

Further, the meet rate estimation unit 55 obtains the value of the X coordinate of the position B'of the golf ball as follows. In the following description, FIG. 13 will be referred to. In FIG. 13, the same points as those in FIG. 11 are designated by the same reference numerals. In FIG. 13, the angle formed by the line segment EB and the Y axis is defined as the launch angle φ. Also, when considering the line segment connecting the point B and the position G where the value of the Y-axis of the coordinates is the minimum among the positions of the golf balls detected from the still image data of each frame, the line segments BG and the X-axis are used. Let θ be the angle formed by.

とする。なお、ここで角度αは、円周を３６０度とする角度単位であるディグリーで表すものとし、｜ｚ｜は、ｚの絶対値を表すものとする。また定数である１．３３や、０．０１は経験的に定められている。 The meat rate estimation unit 55 sets another virtual line segment that passes through the point G and has an angle of θ with the X axis, and B'is the intersection of this line segment and the above-mentioned λ ″. Then, the meet rate estimation unit 55 takes the angle BEB'as the bending angle ψ, and multiplies the cosine of the sum of this bending angle ψ and the launch angle φ described above by a predetermined function of the elevation angle α obtained above. Calculate the meet rate M. Here,

And. Here, the angle α is represented by a degree, which is an angle unit having a circumference of 360 degrees, and | z | represents the absolute value of z. Further, the constants 1.33 and 0.01 are empirically determined.

The flight distance estimation unit 56 uses the meet rate M obtained by the meet rate estimation unit 55 and the head speed s obtained earlier to set the initial velocity of the ball as M · s and set the flight distance e.
e = 4 ・ M ・ s
I presume.

When the display control unit 26 sequentially displays the still image data of each frame acquired by the image pickup control unit 21 in response to the playback instruction of the user, the display control unit 26 also displays the flight distance value e and the meet rate obtained here. You may.

[Estimation of flight distance (second method)]
In another aspect of this embodiment, the flight distance is estimated as follows. In this aspect, the display information generation unit 25 includes a head speed estimation unit 61, a meet rate estimation unit 62, and a flight distance estimation unit 63, as illustrated in FIG. Here, the head speed estimation unit 61 estimates the head speed s by the method already described.

[Estimation of meet rate (second method)]
Further, as illustrated in FIG. 15, the meet rate estimation unit 62 determines a region U having a width of a predetermined number of times, for example, 5 times the width of this region from the left end of the region F in which the player is imaged. The region U is divided into a predetermined number of horizontal divisions in the X-axis direction. Further, the meet rate estimation unit 62 is a first determination line DL1 parallel to the X-axis, for example, from the lower side of the region F in which the player is imaged to a position at a height of, for example, 0.8 times the height of the region F. To set. Further, the meet rate estimation unit 62 sets a plurality of second determination lines DL2 parallel to the X-axis at a position having a height of, for example, 1.2 times the height of the region F from the lower side of the region F. The determination line of is set, and the area U is divided into a plurality of areas in the Y-axis direction as well.

Then, the meet rate estimation unit 62 sets the meet rate in advance for each small region obtained by dividing the region U. In the example of FIG. 15, 1.15, 1.2, and 1.15 are set in order from the left to the right of the first row for each small area divided into 9 pieces of 3 × 3. Similarly, the second line is set to 1.35, 1.42, 1.35, and the third line is also set to 1.1, 1.2, 1.1.

The meet rate estimation unit 62 determines, among the still image data of each frame, where in the small area the position of the golf ball belongs in the frame in which the position of the golf ball is last detected. Then, as a result of the determination, the meet rate estimation unit 62 sets a preset value of the meet rate as the estimated value M'of the meet rate in relation to the small area to which the position of the golf ball last detected belongs. Output.

The flight distance estimation unit 63 uses the meet rate M'obtained by the meet rate estimation unit 62 and the head speed s obtained earlier (hence, the initial velocity of the ball is M'· s) to set the flight distance e'.
e ′ = 4 ・ M ′ ・ s
I presume.

In this example, when the display control unit 26 sequentially displays the still image data of each frame acquired by the image pickup control unit 21 in response to the playback instruction of the user, the flight distance value e'obtained here or the like. The meat rate is also displayed.

[Effect of embodiment]
According to the embodiment of the present invention, if a moving image at the time of a golf shot is captured by the imaging unit 15 of the image processing device 10 from the rear side of the player, a series of still images included in the captured moving image data. The position of the golf ball can be detected from the data frame, and the line segment representing the movement trajectory of the golf ball can be superimposed and displayed when the moving image data is reproduced.

In this way, the user can view a moving image in which the flight trajectory of the golf ball is synthesized with a simple operation, and can improve the entertainment of the golf shot image.

Further, in a certain aspect of the present embodiment, as described above, the initial positions of the shots can be combined and displayed, and the entertainment of the golf shot image can be further improved.

Further, in some aspects of the present embodiment, the head speed at the time of impact can be estimated, the meet rate can be estimated to estimate the flight distance, and the estimated meet rate, flight distance information, and the like can be displayed together. Therefore, the user can obtain information about the shot, such as rough flight distance information, with a simple operation, and can easily diagnose the shot of the player.

In addition, based on the estimated meet rate and flight distance, when these exceed a predetermined threshold value, an image for staging such as an image of characters such as "Nice shot!" Will be displayed together. Further, the entertainingness of the shot image of golf can be improved, and the shot of the player can be easily diagnosed.

[Application to sports other than golf]
The image processing device 10 of the present embodiment can also be used in sports other than golf. For example, in the case of hitting at a baseball or batting center, by using an image captured from the back side of the batter, the so-called back of the back net, the image processing device 10 starts detecting the ball until the batter hits the ball. The throwing speed is estimated using the time and the information of about 18 m, which is the distance d from the batter to the mound.

That is, since the ball is detected when the ball is separated from the pitcher's hand, the pitching speed can be determined by dividing the distance d by the time T until the batter hits.

In baseball, the swing speed of the bat and the impact position when the bat launches the ball affect the hitting speed in addition to the above-mentioned throwing speed. Therefore, the image processing device 10 detects the position of the bat by the same method as the method for detecting the golf club described above, and the speed of the swing, the last detected position of the ball before hitting, and the region where the bat is detected are determined. The impact position is estimated based on the range occupied in the image data.

Then, the image processing device 10 estimates the hitting speed based on the throwing speed, the swing speed of the bat, and the impact position obtained above, with the coefficient of restitution of the ball as a constant. Further, the image processing device 10 identifies the frame after hitting from the detection result of the area of the bat, detects the position of the ball after hitting from the frame after hitting, and estimates the angle of the hit ball. As this estimation method, the same method as the above-mentioned method for estimating the launch angle of a golf ball can be adopted.

Then, the image processing device 10 estimates and displays the flight distance of the hit ball based on the above hitting speed and the angle of the hit ball.

Further, in sports using other balls, the image processing device 10 of the present embodiment is used to estimate the trajectory of the ball thrown, launched, kicked, that is, shot by the player, and the trajectory thereof is estimated. The information can be used for processing such as superimposing the information on the captured image of the player and the ball and displaying the information.

10 Image processing device, 11 Control unit, 12 Storage unit, 13 Operation unit, 14 Display unit, 15 Imaging unit, 16 Communication unit, 21 Imaging control unit, 22 Image preprocessing unit, 23 Pre-detection processing unit, 24 Detection processing unit , 25 Display information generation unit, 26 Display control unit, 31,41 Correction unit, 32,43 Conversion unit, 42 Player area identification unit, 51,61 Head speed estimation unit, 52 Initial position estimation unit, 53 Temporary flight distance calculation unit , 54 elevation angle estimation unit, 55,62 meat rate estimation unit, 56,63 flight distance estimation unit.

Claims (6)

A means for acquiring moving image data including a series of N still image data obtained by capturing a golf shot from a direction substantially parallel to the launch direction of a golf ball .
A ball detecting means for detecting an image portion of a golf ball from each of the still image data included in the acquired moving image data, and a ball detecting means.
A flight distance estimation means for estimating the flight distance using the information on the position of the detected image portion of the golf ball, and
Including
The flight distance estimating means estimates the launch angle of the still image data on a plane based on the position of the image portion of the golf ball detected from the plurality of still image data, and is based on the estimated launch angle. An image processing device that estimates the flight distance . The image processing apparatus according to claim 1.
Further, the head detecting means for trying to detect the image portion of the head of the golf club from each of the still image data included in the acquired moving image data is included.
An image processing device in which the flight distance estimating means estimates a flight distance using the detected information on the position of an image portion of a golf ball and information on the position of an image portion of a golf club head. The image processing apparatus according to claim 2.
The flight distance estimation means is
The speed of the head at the time of impact is estimated based on the position of the image portion detected by the head detecting means detected from a plurality of still image data, and the speed of the head is estimated.
Estimate the meet rate based on the position of the image part of the golf ball detected from multiple still image data,
An image processing device that estimates the flight distance based on the estimated head speed and meet rate. The image processing apparatus according to claim 3.
The flight distance estimation means is
When estimating the meet rate,
The provisional flight distance is obtained using the estimated head speed and the provisional meet rate determined in advance, and the provisional flight distance and the information on the position of the image portion of the golf ball detected from a plurality of still image data are obtained. Based on this, an image processing device that estimates the meet rate. Using a computer
The acquisition means acquires moving image data including a series of N still image data obtained by capturing a golf shot from a direction substantially parallel to the launch direction of the golf ball .
The ball detecting means detects an image portion of a golf ball from each of the still image data included in the acquired moving image data.
The flight distance estimation means is an image processing method for estimating the flight distance using the information on the position of the detected image portion of the golf ball.
The flight distance estimating means estimates the launch angle of the still image data on a plane based on the position of the image portion of the golf ball detected from the plurality of still image data, and is based on the estimated launch angle. An image processing method that estimates the flight distance . Computer,
A means for acquiring moving image data including a series of N still image data obtained by capturing a golf shot from a direction substantially parallel to the launch direction of a golf ball .
A ball detecting means for detecting an image portion of a golf ball from each of the still image data included in the acquired moving image data, and a ball detecting means.
A flight distance estimation means for estimating the flight distance using the information on the position of the detected image portion of the golf ball, and
To function as
When functioning as the flight distance estimation means, a computer is made to estimate the launch angle of the still image data in a plane based on the position of an image portion of a golf ball detected from the plurality of still image data, and the estimation is performed. A program that estimates the flight distance based on the launch angle .

Images