JP2020098441A - Arrival order determination method, arrival order determination device and program - Google Patents

Abstract

To provide an arrival order determination method, an arrival order determination device and a program capable of automatically determining the arrival order of athletes from a goal scene image acquired by a slit camera with high accuracy.SOLUTION: An arrival order determination device 100a includes: an image storage unit 30 for storing a goal scene image obtained by a slit camera installed at a goal position; a clothing color region recognition unit 70 for recognizing a color region of each athlete's clothing color in the goal scene image as a body part of the athlete; a goal determination target region extraction unit 80 for extracting a goal determination target region including a specific part of an athlete boarding object from the goal scene image based on the position of the body part of each athlete; an image matching unit 90 for obtaining specific position coordinates subjected to goal determination by matching the extracted image of the goal determination target region with a reference image subjected to goal determination; and an arrival order output unit 95 for determining arrival orders of the plurality of athletes based on the specific position coordinates, and for outputting them.SELECTED DRAWING: Figure 1

Description

The present invention relates to a method of determining an order of arrival in a competition, an apparatus for determining an order of arrival and a program.

Generally, in order to determine the order of arrival in public competitions such as bicycle races and horse racing, visual judgment as well as visual judgment by the referee are used.

In Patent Document 1, when the passing order determined by the referee during a horse racing race is displayed on the arrival order display panel, the visual judgment by the referee is made to reduce the erroneous input of the passing order by the referee. In order to assist, a technique is disclosed in which a video analysis unit analyzes a video image captured by an imaging unit and causes the display unit to display horse number data of passage ranks up to the top three.
Patent Document 2 provides a slit camera, which has a problem of reliably and simplifying an output operation of a measurement result of a landing order determination device that electronically determines the landing order in athletics, ice competitions, boat competitions, and the like. Disclosed is a technique for obtaining a goal time (passing time) by overlaying a cursor that can move along the time axis on a slit camera image.

In Patent Document 3, in a competition in which the order of arrival and time are competed such as a bicycle race, it is an object to make it possible to simultaneously check the entire recorded image and make a determination work, and a color video image captured on a goal line by a line sensor. When displaying it on a monitor and judging the order of arrival and time in a bicycle race etc. by looking at it, a partial image for judgment including the first runner (bicycle) who first arrived at the goal line and the whole of multiple runners are shown. An electronic arrival order and time determination device is disclosed in which a reduced image of the entire recorded image is displayed at the same time, and a frame indicating a partial image is displayed in the reduced image.

JP, 2013-192871, A Utility model registration No. 3030178 JP 2008-22963 A

Although the above-described photographic determination and the arrival order determination in Patent Documents 2 and 3 use the output image of the slit camera, in essence, a person visually determines the arrival order.

In this regard, the prior art described in Patent Document 1 is for the purpose of supporting the referee, but determines the order of arrival by image analysis by the video analysis unit. In this prior art, a relatively large number indicating the horse number in Arabic numerals is attached to the side of the horse body, and basically the number is image-recognized to identify the horse number. Patent Document 1 mentions that the recognition/judgment of the trunk pattern and sleeve pattern (ground color and pattern color as required), hat color, and horse hair color of a player's competition clothes is used together. However, not only these pieces of information do not directly correspond to horse numbers, but their judgment processing is relatively difficult, and highly accurate judgment results cannot be expected.

On the other hand, in bicycle races, the number attached to the clothes worn by the athlete (clothing color or competition clothes) is relatively small, and the image is tilted, distorted, or only partially visible on the captured image. Accurate recognition of numbers is not always easy. Further, in Patent Document 1, immediately after the image recognition processing of the processor in the passage order determination support device is started, “n-th image cutout” (starting from n=1 and sequentially incrementing) is performed. Although it is difficult for humans to determine which horse is "nth", it is necessary to perform photo determination, but how "next image cropping" is performed automatically and accurately by image processing. It is not specified whether it will be executed.

The present invention has been made in such a background, and an object thereof is a finish order determination method and a finish order determination method capable of automatically determining the finish order of athletes from a goal scene image acquired by a slit camera with high accuracy. To provide a device and a program.

In the competition order determination method according to the present invention, in a competition in which a plurality of athletes wear clothes of different colors, the goal scene image obtained by the slit camera installed at the goal position is subjected to image processing by a computer, thereby Is a method of determining the order of arrival when a player of (1) makes a goal. This method of determining the order of arrival is the step of recognizing the color region of the clothing color of each player as the body of the player based on the goal scene image, and the boarding of the player based on the position of the body of each player. The step of extracting the goal determination target area including the specific portion of the boarding object, and the specific position coordinates of the goal determination target by matching the image of the extracted goal determination target area with the reference image of the goal determination target It is provided with a step of acquiring and a step of determining the order of arrival of the plurality of players based on the specific position coordinates obtained for the plurality of players.

In this arrival order determination method, a step of generating a background image corresponding to the background of the goal scene image based on the goal scene image, and whether or not it is a moving body part including each player based on the goal scene image The method further comprises a step of generating a binary image and a step of generating a moving body partial image based on the goal scene image and the binary image, wherein the step of recognizing the color region of the clothing color includes the moving body. You may make it recognize a color area based on a partial image.

In one aspect of the step of generating the background image, the background image can be generated by copying an image of one line of the goal scene image, which is guaranteed not to include the moving body portion.

In one aspect thereof, the step of extracting the goal determination target area may include the step of defining a rectangular area of a predetermined size in a predetermined positional relationship from the body as the goal determination target area of the player.

In another aspect, the step of recognizing the color area of the clothing color can recognize the color area based on the goal scene image.

The step of recognizing the color region of the clothing color is, in one aspect thereof, changing the color of each pixel in the image to each clothing color of the pixel in consideration of a predetermined range setting parameter at the time of clothing color determination. It may include a step of determining whether the pixel is applicable or not, and a step of removing noise from the pixel group determined to be applicable.

The arrival order determination device according to the present invention is a arrival order determination device that determines the arrival order when a plurality of players wear clothes of different colors, and is installed at the goal position. An image storage unit for storing the goal scene image obtained by the slit camera, and a clothing color region recognition unit for recognizing the color region of the clothing color of each player as the body of the player in the goal scene image; A goal determination target area extraction unit that extracts a goal determination target area including a specific portion of the boarding object on which the player is riding from the goal scene image based on the position of the body of the player, and the extracted goal An image matching unit that obtains the specific position coordinates of the goal determination target by matching the image of the determination target region with the reference image of the goal determination target, and the plurality of players based on the specific position coordinates obtained for the plurality of players. And an arrival order output unit that determines and outputs the arrival order of the.

In one aspect thereof, the incoming call determination device includes a background image generation unit that generates an image corresponding to the background of the goal scene image from the goal scene image, and a moving body part including each player based on the goal scene image. The clothing further includes a binary image generation unit that generates a binary image indicating whether or not there is a goal determination target region extraction unit that generates a moving body partial image based on the goal scene image and the binary image. The color area recognition unit may recognize the color area based on the moving body partial image.

As one mode thereof, the background image generation unit can generate the background scene image by copying an image of one line in which it is guaranteed that the moving body portion is not included.

As one aspect of the goal determination target area extraction unit, a rectangular area of a predetermined size in a predetermined positional relationship from the body portion can be set as the goal determination target area of the player.

As one mode, the clothing color area recognition unit can recognize a color area based on the goal scene image.

As one mode thereof, the clothing color area recognition unit determines whether the pixel corresponds to each clothing color or not by comparing the color of the pixel with a predetermined parameter for setting an allowable range at the time of determining the clothing color. , Noise can be removed from the pixel group determined to be applicable.

The present invention is also understood as a program. This program uses a computer to image-process a goal scene image obtained by a slit camera installed at the goal position in a competition in which multiple players wear clothing of different colors, and the multiple players score a goal. A program for determining the order of arrival at the time, the step of recognizing the color area of the clothing color of each player as the body part of the player based on the goal scene image, and the position of the body part of the player. Based on this, a step of extracting a goal determination target area including a specific portion of the boarding object on which the player is boarding, and a goal by matching the extracted image of the goal determination target area with the reference image of the goal determination target The step of acquiring the specific position coordinates of the determination target and the step of determining the order of arrival of the plurality of players based on the specific position coordinates obtained for the plurality of players are executed.

According to the arrival order determination method, the arrival order determination device, and the program of the present invention, it is possible to automatically determine the arrival order of the athlete with high accuracy from the goal scene image acquired by the slit camera.

It is a functional block diagram showing various functions of the 1st arrival order judging device by an embodiment of the present invention, and their mutual relation. It is a functional block diagram of the 2nd arrival order judging device of an embodiment of the present invention. It is a functional block diagram of the 3rd arrival order judging device of an embodiment of the present invention. It is a block diagram which shows the hardware structural example of the data processing apparatus for implement|achieving each functional block of the arrival order determination apparatus shown in FIGS. 5 is a first flow chart of arrival order determination processing for the data processing apparatus shown in FIG. 4 to execute each function of the functional block diagram of the arrival order determination apparatus shown in FIG. 1. 4 is a second flowchart of the arrival order determination process for the data processing device illustrated in FIG. 4 to execute each function of the functional block diagram of the arrival order determination device illustrated in FIG. 2. 5 is a third flowchart of the arrival order determination process for the data processing device illustrated in FIG. 4 to execute each function of the functional block diagram of the arrival order determination device illustrated in FIG. 3. FIG. 8A is a diagram showing an example of a GS image, and FIG. 8B is a diagram showing an example of a BG image. FIG. 9A is a diagram showing an example of RCn candidate pixels, FIG. 9B is a diagram showing an image example after the erode processing, and FIG. 9C is an image example of RCn after the dilate processing. FIG. It is a figure which shows the example which set the goal determination object area|region HZn based on the gravity center coordinate CCn. It is a figure which shows the example of comparison of the image of the area|region HZn of the front wheel corresponding to the nth player, and the reference|standard image RI. 12A is a diagram showing an example of an MP image corresponding to the GS image of FIG. 8, and FIG. 12B is a diagram showing an example of an RB image corresponding to the GS image of FIG. 8A. .. It is a figure which shows the example of comparison of the image of the area|region HZn of the front wheel corresponding to the nth player, and the reference|standard image RI.

Hereinafter, embodiments of the present invention will be described in detail.

Conventionally, a photographing device called a slit camera is used for the photo determination. A slit camera uses an imaging device such as a line sensor to capture an image of an individual passing through a reference line corresponding to a goal, capture an image for each slit-shaped element that matches the reference line, and use each element as the time axis. It is an apparatus for arranging along and combining a slit camera image.

In order to determine the order of arrival in a competition (race) based on the slit camera image, the part of the moving body that is judged to have reached the goal, that is, the part to be judged for the goal (for example, in the case of a bicycle race or an auto race, the tip of the front wheel of the vehicle, the boat). In the race, it is necessary to recognize the area of the boat tip, etc.) from the slit camera image. However, it is not always easy to directly separate such a moving body portion (area) by image processing. Even if they can be separated, the order of arrival cannot be determined unless the number of the moving body is known. As described above, even if the numbers (numerals) are displayed on the clothing, it cannot be expected that all the numbers will be correctly recognized based on the captured image.

Therefore, the inventor of the present application paid attention to the fact that the athletes participating in the competition wear clothes (for example, hoodies) of different colors as so-called competition clothes (for example, for each car number in a bicycle race etc.). That is, by identifying the color of each player's clothing (clothing color), it is possible to recognize the number (vehicle number, order number, etc.) written on the clothing of the athlete.

However, even if the clothing color area is detected based on the color of the pixel in the slit camera image, the contour of the player is vague, and in competitions such as bicycle races, the goal judgment target is not the player himself but the boarding target. It is a thing. Therefore, it is not possible to immediately determine the order of finish based on the detected player area itself. Therefore, first, in the slit camera image, the color region of the clothing color of each player is recognized as the body of the player. This allows the approximate position of each player to be identified. Next, an image area of a predetermined size and shape that includes a predetermined portion of the boarding object on which the athlete is riding, that is, a specific portion to be the goal determination target (for example, the front end portion of the front wheel of the bicycle in a bicycle race) is extracted. Since it is known that the position of the specific portion of the boarding object has a predetermined positional relationship with the body of the player, it is possible to reliably cut out the image region including the specific portion. Furthermore, the coordinates of the specific location of the goal determination target can be obtained by matching the cut-out image area with a known reference image. By comparing the coordinates of the specific location of each player obtained in this way, the order of arrival can be determined without having to read the number (number) of the player.

Hereinafter, specific embodiments based on such an idea will be presented. A bicycle race will be described below as an example of a competition.

<First arrival order determination device>
FIG. 1 is a functional block diagram showing various functions of the first arrival order determination device 100a according to the present embodiment and their mutual relationship.

The wearing order determination device 100a includes a color collecting unit 10, an allowable range setting parameter generation unit 20, an image storage unit 30, a clothing color region recognition unit 70, a goal determination target region extraction unit 80, an image matching unit 90, and a wearing order. The output unit 95 is provided.

The color sampling unit 10 receives the images obtained by the start position cameras (described later in FIG. 4) that capture the still images or moving images of all the players in the competition aligned at the start position of the competition, and assigns the number to each player. It is a color sampling means for sampling the corresponding clothing color data. The clothing color is the color of the clothing of the athlete. A player's clothing is, for example, a hoodie (a jacket with a hood). The clothing color is preferably a single color (plain) without a pattern or pattern, but a part of the clothing color may be different from the background color. As an example of the color classification, for example, the correspondence between the number (vehicle number) of a moving body in a bicycle race and the color is 1: white, 2: black, 3: red, 4: blue, 5: yellow, 6: green, 7: orange. , 8: peach, 9: purple.

In this example, the color data of the clothing color of each player collected is RGB 8 bits/clothing color in this example. The color area is preferably specified automatically, but may be executed with the support of the operator (for example, designation of points in the area).

In this example, the clothing color RGB data is converted into PCn (nth ParkaColor), which is HSV data that is a kind of data format convenient for subsequent processing, and then stored (registered) in an image storage unit described later. Here, n represents the participating player number, and PCn represents the HSV clothing color of the nth player. Further, H represents hue (Hue), S represents saturation (V), and V represents lightness (Value).
PCn (Hn, Sn, Vn) (each value is a hexadecimal value from 00 to FF)

If the clothing color does not change for each competition and is constant, the clothing color data that has been acquired and stored in advance may be used regardless of the start position camera. Further, the format of color data is not necessarily limited to HSV.

The allowable range setting parameter generation unit 20 generates a parameter for determining whether or not each pixel in the image of the goal scene GS (Goal Scene) obtained by the slit camera corresponds to the color of each color data PCn. It is a means. The image of the goal scene GS is an image of a predetermined size, 1000 x 1000 pixels in this example, and 8 bits/pixel for each of RGB. This parameter is a parameter for setting an allowable range at the time of determining the clothing color, and for each pixel forming the image, the color of each color data PCn is given a predetermined width, and its approximate color is also recognized as the corresponding color. To set. That is, it is determined whether or not the pixel is applicable to each clothing color according to the parameter.

A specific processing example of the allowable range setting parameter generation unit 20 will be described later.

The image storage unit 30 is a storage unit such as a memory that receives and temporarily stores the slit camera image output from the slit camera.

The clothing color area recognizing unit 70 is a means for recognizing each clothing color area in the image. More specifically, the clothing color area recognizing unit 70 compares each clothing color (approximate color) of the clothing color with the allowable range setting parameter. This is a means for generating a clothing color cluster RCn (nth RacerCluster) composed of pixel groups (in this example, a region having an approximate color to the clothing color of the nth player). The clothing color area recognition unit 70 also obtains the coordinates CCn of the center of gravity of each clothing color cluster RCn. Here, CCn (nth Cluster Center) is the coordinate of the center of gravity of the area having the approximate color of the clothing color of the nth player. Alternatively, the center of gravity may be obtained by the goal determination target area extraction unit 80 in the subsequent stage.

In this example, this process is performed directly from the GS image.

The goal determination target area extraction unit 80 determines the goal determination target area HZn (nth HeadZone: in this example, the front wheel (front wheel) of the nth player's bicycle based on the clothing color cluster RCn (the center of gravity CCn of the clothing color cluster CCn) and the goal scene GS. ) A rectangular area including the tip). More specifically, in the region HZn, the relative coordinates (x, y) with the barycentric coordinate CCn as the origin are (A, B), (C, B), (A, D), and (C, D). It is a rectangular area surrounded by coordinates. Here, the x-axis represents the horizontal (horizontal) direction of the goal scene GS image, and the y-axis represents the vertical (vertical) direction thereof.

The image matching unit 90 is a unit that compares the image of the goal determination target area HZn of each player with the reference image RI and obtains each specific position coordinate WPn (nth Wheel Position). Here, the specific position coordinates WPn are the coordinates of the image portion that is the goal determination target of each player, and in this example are the position coordinates (Xnh, Ynh) of the front wheel tip of the bicycle of the nth player. These coordinates are absolute coordinates with the origin (0, 0) at the upper left of the rectangle of the goal scene GS. Further, RI (Reference Image) is a reference image as a typical image of a goal determination target prepared in advance. In this example, the reference image RI is an image including the tip of the bicycle front wheel, and is, for example, an image of the tip half of the front wheel (or the right half or the upper right ¼). A specific example of this processing will be described later.

The arrival order output unit 95 is a part that determines and outputs a sequence of vehicle numbers ordered based on the magnitude of the X coordinate of the specific position coordinates WPn as the arrival order.

<Second arrival order determination device>
FIG. 2 shows a functional block diagram of the second arrival order determination device 100b of the present embodiment. The same elements as those shown in FIG. 1 are designated by the same reference numerals, and duplicated description will be omitted. The second arrival order determination device 100b further includes a background image generation unit 40 and a binary image generation unit 50 in addition to the constituent elements of the arrival order determination device 100a.

The background image generation unit 40 is means for generating a background image BG (BackGround) corresponding to the background of the GS image, based on the GS image (slit camera image) obtained from the image storage unit 30. This background image BG has the same size as the goal scene GS, for example, generated by duplicating a required vertical region (1 line) of one pixel at the beginning (usually the right end) of the goal scene GS to the rear side. It is an image. Although one line that is the source of the duplication is the head of the goal scene GS, it may not be the head as long as it is a line that does not include a moving object.

The binary image generation unit 50 is a unit that generates a binary image by separating the area of the moving body part MP (MovingPart) existing in the GS image from the GS image. The moving body portion MP is a foreground with respect to the background and is a portion that moves in the competition. For example, in the case of a bicycle race, it is an area where a player and a bicycle are combined. The MP image obtained in this example is a binary image in which the area where the moving body part MP exists is “true (“1”)” and the area where it does not exist is “false (“0”)”. And the background BG are inconsistent.

In the second wearing order determination device 100b, the goal determination target area extraction unit 80 is based on the MP image, which is a binary image in place of the image of the goal scene GS, and the clothing color cluster RCn (the barycentric coordinate CCn of the clothing color cluster RCn). Then, the goal determination target region HZn (in this example, a rectangular region including the front wheel tip of the nth player's bicycle) is extracted. Further, in the second arrival order determination device 100b, preferably, the reference image RI used in the image matching unit 90 is a binary image. Other configurations are similar to those of the first arrival order determination device 100a.

<Third arrival order determination device>
FIG. 3 shows a functional block diagram of the third arrival order determination apparatus 100c of this embodiment. The same elements as those shown in FIGS. 1 and 2 are designated by the same reference numerals, and the duplicated description will be omitted. The third arrival order determination device 100c further includes a moving body partial image generation unit 60 in addition to the constituent elements of the second arrival order determination device 100b.

The moving body partial image generation unit 60 is means for generating a moving body partial image RB (Racer+Bicicle) in which only the moving body portion in the goal scene GS is cut out based on the GS image and the MP image. In other words, by masking the GS image with the MP image, an image obtained by cutting out the moving body partial image RB from the GS image is generated.

Although the MP image and the RB image are similar to each other, the MP image is a monochrome binary image, whereas the RB image is a color image.

In the third wearing order determination device 100c, the clothing color area recognition unit 70 receives the moving body partial image RB output from the moving body partial image generating unit 60 instead of the goal scene GS image, and receives the clothing color cluster RCn(. The center of gravity coordinates CCn) of the are generated. By performing this processing based on the RB image in which the background image is removed from the GS image, the processing accuracy can be improved.

Further, in the third incoming call determination device 100c, the goal determination target area extraction unit 80 uses the goal scene GS or the binary image MP as an image to be input together with (the center of gravity coordinates CCn of the clothing color cluster RCn). When using the goal scene GS, the reference image RI used in the image matching unit 90 is preferably a color image. When using the binary image MP, the reference image RI used in the image matching unit 90 is preferably a binary image. Other configurations are the same as those of the arrival order determination device 100b.

<Hardware configuration example>
FIG. 4 shows a hardware configuration example of the data processing device 200 for realizing each functional block of the arrival order determination devices 100a to 100c shown in FIGS. 1 to 3 of the present embodiment.

The data processing device 200 corresponds to a computer, and typically includes a central processing unit (CPU) 210, a memory 220, a display control unit 230, a storage 250, and an input interface (I/F) unit 260. The central processing unit (CPU) 210 is a processor that controls each unit of the data processing device 200 via the bus 205 and controls its operation. The memory 220 is a storage device (storage unit) including areas 221 and 223 for storing an operating system (OS) executed by the CPU, programs and data, and a work area 225. The image storage unit 30 shown in FIGS. 1 to 3 can be configured by the memory 220. The display control unit 230 is a display control unit that displays display information such as an image, video, and text on the display screen of the display unit 310 under the control of the CPU 210. The storage 250 is a large-capacity storage unit that stores programs and data in a nonvolatile manner. The input interface (I/F) unit 260 is a means for connecting an external input device to the bus 205, and in this example, a slit camera 320 installed at a goal point, a still image or a moving image installed at a start position. A camera 330 for picking up an image and an operation unit 340 for accepting an operation of an operator are connected. A communication interface unit may be provided instead of or in addition to the input interface unit 260.

<First flowchart>
FIG. 5 is a first flowchart of the arrival order determination process for the data processing apparatus 200 illustrated in FIG. 4 to execute each function of the functional block diagram of the arrival order determination device 100a illustrated in FIG. Hereinafter, an example of an image will be described by taking a bicycle race as an example.

This process is started at the start of the competition, and first, the clothing color PCn of each athlete participating in the competition is collected based on the image captured by the camera 330 installed at the start position (predetermined alignment position) (S1). This step corresponds to the operation of the color sampling unit 10 shown in FIG. As described above, when the fixed clothing color PCn is used, this step S1 and the next step S2 can be omitted.

Next, a permissible range in which it can be determined that the color is close to the clothing color PCn of each player is set (S2). This step corresponds to the operation of the allowable range setting parameter generation unit 20 shown in FIG.

For example, the following is a specific example of the setting of the permissible range in which it can be determined that the clothing color PCn of each player is an approximate color.

First, the thresholds (PCn max, Cn min) for determining the same color are determined from the measured values of PCn by the following equation.
PCn max = [Hn + Δh, Sn + Δs, Vn + Δv]
PCn min = [Hn-Δh', Sn-Δs', Vn-Δv']
Δh and Δh′ are set narrow, and Δs, Δs′, Δv and Δv′ are set wide.

These parameters are used to detect the position of each player in the next stage, but by appropriately setting PCn max and PCn min, it is possible to obtain resistance to disturbances such as weather changes, light source changes, and shadows.

Each Δ value that defines the allowable range can be determined from the histogram of each color gamut. Preferably, Δh and Δh′ are set narrow, and Δs, Δs′, Δv and Δv′ are set wide. This is expected to have the effect of reducing the possibility of false detection of clothing color. For example, Δh and Δh′ are about 1σ above and below the peak of the histogram, and Δs, Δs′, Δv and Δv′ are about 2σ above and below the allowable range. Δh and Δh′ may be the same value or may be slightly different values. The same applies to Δs and Δs′ and Δv and Δv′. Further, the saturation S is determined by a predetermined threshold value, and when the saturation S is smaller than the threshold value, the allowable range of the hue H is widened to improve the determination accuracy when the clothing color is white and black. Good.

After that, the GS image obtained from the slit camera 320 is acquired and stored in the image storage unit 30 (S3). FIG. 8A shows an example of the GS image.

Next, the clothing color cluster RCn (and its barycentric coordinate CCn) of each player is extracted from the goal scene GS (S7). This processing may be performed by a method of extracting the maximum continuous area after erod/dilate, or alternatively, a method of extracting the maximum continuous area after the binarization processing after the Blur processing. This step corresponds to the processing of the clothing color area recognition unit 70 shown in FIG.

The erode process is a kind of morphology process, and is a contraction process performed to remove white noise (noise) in an image or to divide a plurality of connected objects. Dilate processing is a kind of morphology processing, and is expansion processing that increases the size of the foreground object, contrary to contraction processing. By using it in combination with the erode process, it becomes possible to effectively remove noise.

The Blur process is a process for blurring an image and can be used for noise removal. For example, the binary image is subjected to blur processing, and then a threshold value is set to perform binarization again to drastically reduce discrete minute areas, and it is easy to obtain a continuous maximum area in the binarized image. Become. According to the erode/dilate processing and the Blur processing, it is possible to reduce the calculation load and avoid erroneously recognizing a region that does not belong to the target clothing color cluster RCn.

Below, the specific example of process S7 is shown.

(When extracting the largest continuous area after erode/dilate)
When the color (HSV value) of each pixel of the goal scene GS falls within the range of the above threshold value (upper and lower limit values), these pixels are set as pixel points belonging to each clothing color cluster RCn, and points belonging to each clothing color cluster. Find the center of gravity of the group. However, since pixels that fall within the threshold value may exist at positions other than the position of the corresponding player, the image of each clothing color cluster RCn consisting of n pieces is subjected to 3-8 pixel erode processing, and further the same amount of dilate processing is performed. Top, the center of gravity (coordinates) is calculated. Do this for all clothing color clusters.

In addition, when there are clothes colors that are quite similar to each other, it may be difficult to separate them individually. In such a case, even if erode/dilate processing is performed, a plurality of point groups belonging to one clothing color cluster may remain. In order to avoid such a problem, the point group belonging to each clothing color cluster may be subjected to a process of narrowing it down to a continuous region having the widest area. (For example, the largest one is selected after extracting a continuous region with cv2.connectedComponents() of OpenCV.)

(When extracting the largest continuous region after Blur processing)
The point group belonging to each clothing color cluster RCn is selected as the point group belonging to each clothing color cluster RCn without performing the above-mentioned eode/dilate processing. The coordinates CCn of the center of gravity of the selected area are obtained for each clothing color cluster RCn. (OpenCV cv2.connectedComponents() etc.)

FIG. 9A shows an example of candidate pixels of the clothing color cluster RCn, FIG. 9B shows an image example after eode processing, and FIG. 9C shows an image example of RCn after dilate processing. When the images of FIGS. 9A to 9C are compared with each other, noise in the image of FIG. 9A is reduced in the image of FIG. 9B, and the once reduced main region is restored in FIG. 9C. You can see that

Next, an image of the goal determination target area HZn, that is, an area HZn image including the front wheel tip of each player's vehicle is extracted (S8). In this example, a GS image is used as the extraction source. This step corresponds to the processing of the goal determination target area extraction unit 80 shown in FIG.

Specific examples are shown below.

A barycenter coordinate CCn of each clothing color cluster RCn is (Xn, Yn), and a region HZn surrounded by (Xn+A, Yn+B), (Xn+C, Yn+B), (Xn+A, Yn+D), (Xn+C, Yn+D) is the goal scene GS. Crop (cut out) from. The values of the variables A, B, C, D are selected so that this rectangular area HZn is an area in which the front wheel tip of each player is surely included. Examples of the values of the variables A, B, C, and D are around 30, 30, 100, and 110 (when the goal scene GS is a 1000 x 1000 pixel image).

FIG. 10 shows an example in which the goal determination target area HZn is set based on the barycentric coordinates CCn.

Next, image matching is performed by comparing the image of each goal determination target region HZn with the reference image RI to obtain the specific position coordinates Wpn(Xnh, Ynh) (S9). This processing may be based on expansion/contraction template matching or may be based on feature amount matching. This step corresponds to the processing of the image matching unit 90 shown in FIG.

Specific examples are shown below.

(When acquiring the position of the front wheel tip by expansion/contraction template matching)
The absolute position (Xnh, Ynh) of the front wheel tip portion is specified by performing template matching on the image of the region HZn including the front wheel tip portion using the reference image RI as a template without using a feature amount such as AKAZE. .. For that purpose, first, an image region corresponding to the reference image RI in the HZn image is obtained. Since the wheel tip position within the frame of the reference image RI is known at the stage of defining the reference image RI, if the image frame in the HZn image corresponding to the reference image RI is determined, the wheel tip in the HZn image is determined from the mapping relationship between both images. The position of the section is determined.

More specifically, while comparing the reference image RI and a region of the same shape as RI in the HZn image (for the time being referred to as Z), the region Z is sequentially moved until the entire region of HZn is covered, and the moving position with the highest degree of approximation ( Find the position where the sum of the squared errors of the corresponding pixel values is the minimum). Then, the distance between the point in the region HZn where the matching with the reference image RI (maximum cross-correlation point) is obtained and the upper left corner point of the region HZn (the origin in this region) is obtained. When this distance vector is (Xnc, Ync), the absolute position (Xnh, Ynh) of the front wheel tip portion is (Xn+A+Xnc, Yn+B+Ync). This absolute position is a numerical value obtained in relation to the reference image RI and is not the X position itself of the wheel tip portion (to be exact, at the X position that matches RI, the left end of the image RI and the wheels in the image RI). It is the sum of the distance to the right edge). However, since the relative positional relationship among all the players is expressed by the above (Xn+A+Xnc, Yn+B+Ync), the order of finish can be determined based on this.

The rotation vulnerability of template matching is not a problem in this case, but scaling tolerance is added because it is necessary to consider the distance difference between players. That is, with respect to the region image HZn including the front wheel tip of each player, the wheel tip position is specified by performing template matching with the reference image RI while scaling (expanding and contracting) by changing the scale factor, and the mutual comparison between the players is performed. Go and determine the order of arrival. The above process is repeated while scaling HZn, and the target coordinates are obtained from the moving position and the scaling factor when the image RI and the region Z have the highest degree of approximation.

FIG. 11 shows an example of comparison between the image of the front wheel region HZn corresponding to the nth player and the reference image RI. The reference image RI of the present example is an image of a rectangular area including a substantially right front quarter of the front wheel. The image of the region HZn in this example is obtained from the GS image. In the figure, in the HZn image, a region F detected as an image portion corresponding to the reference image RI is shown by a frame F. The lower part of the front wheel may be obstructed by the player in front, and by setting "upper right 1/4", it is possible to avoid the erroneous determination in such a case.

(When acquiring the position of the front wheel tip by feature amount matching)
As the feature amount, for example, AKAZE, SIFT, or ORB can be used. In this example, the reference image RI is, for example, an image including the upper right quarter of the front wheel of the bicycle. In any case, the reference image RI is sufficient as long as it includes the front wheel tip portion and ensures that proper matching is performed. The feature amount matching with the reference image RI is performed for each rectangular region HZn including the front wheel tip of each athlete's bicycle, and the corresponding point cloud is extracted from both images. More specifically, the feature points extracted from the reference image RI and the feature points extracted from the HZn image are compared with each other, and when sufficient corresponding points are obtained, the reference image RI and the front wheel tip region HZn are compared. At, a projective transformation of point groups that are close to each other by a certain amount or more, that is, a transformation matrix into an image of the reference image RI and the front wheel tip region HZn is obtained (for example, by cv2.findHomography() of OpenCV). The absolute position (Xnh, Ynh) of the front wheel tip portion of the vehicle is specified based on this matrix and the barycentric coordinates CCn (Xn, Yn) of each clothing color cluster RCn. In this way, the region that best fits the reference image RI within the rectangular region HZn is specified, and the x component of the coordinates of the absolute position of the front wheel tip is determined.

Finally, the ranking (order of arrival) of each player is determined and output (S10). Specifically, the rank of each player is numbered in descending order of Xnh. The output mode of the order of arrival may include displaying text and/or images on the display unit 310 and outputting data to the outside via the communication interface unit. This step corresponds to the processing of the arrival order output unit 95 shown in FIG.

In addition, the order of performing the above-described steps may be changed before or after the step does not hinder. Further, there may be a case where the image matching with the reference image RI does not succeed for any of the rectangular regions HZn obtained in the goal scene GS, for example, the frontmost part of the front wheels is completely hidden by the player in front. To solve this problem, a mirror is provided on the opposite side of the slit camera at the goal position, the slit camera image is obtained for the reflection image reflected on the mirror, the same processing as above is executed in parallel, and the non-reflection It is possible to supplement the image processing result. In this embodiment, since the numbers in the image are not recognized, there is no particular problem even if the image is reversed. Further, as seen from the slit camera, the mirror shows an image of the back side of the player, so that the same reference image RI can be used. When image matching is successful for both the reversed image and the non-reversed image and the arrival order determination results of the both are different, a means for notifying the operator of the arrival order determination device or the like may be provided. A separate slit camera may be provided instead of the mirror, and the same processing as the above may be changed for the output of this slit camera. In this case, the reference image RI is inverted or the output image of the slit camera is inverted.

<Second flowchart>
FIG. 6 shows a second flowchart of the arrival order determination process for the data processing apparatus 200 shown in FIG. 4 to execute each function of the functional block diagram of the arrival order determination device 100b shown in FIG. The same steps as those shown in FIG. 1 are designated by the same reference numerals, and the duplicated description will be omitted.

In the second flowchart, after step S3, the background image BG of the GS image is generated (S4). Specifically, as described above, an image in which the first vertical line of the slit camera image is duplicated in the horizontal direction is created and used as the background BG. FIG. 8B shows an example of a BG image. This step corresponds to the operation of the background image generation unit 40 shown in FIG.

Next, a binary image MP in which the moving body part (player+bicycle) is separated from the goal scene GS image is generated (S5). This step corresponds to the operation of the binary image generation unit 50 shown in FIG. FIG. 12A shows an example of an MP image corresponding to the GS image of FIG.

After this step S5, the process proceeds to step S8 via step S7.

In step S8 of this example, the region HZn is cropped from the MP image based on the barycentric coordinates CCn of each clothing color cluster RCn. The variables A, B, C and D that determine the size of the region HZn are as described above.

FIG. 13 shows an example of a comparison between the image of the front wheel region HZn corresponding to the nth player and the reference image RI. In the present example, the reference image RI is a binary image of a rectangular area including a substantially front half area of the front wheel. The image of the region HZn in this example is obtained from the MP image. Other steps are the same as those in the first flowchart of FIG.

<Third flowchart>
FIG. 7 shows a third flowchart of the arrival order determination process for the data processing apparatus 200 shown in FIG. 4 to execute each function of the functional block diagram of the arrival order determination device 100c shown in FIG. The same steps as those shown in FIGS. 1 and 2 are designated by the same reference numerals, and a duplicate description will be omitted.

In the third flowchart, after the step S5 of the second flowchart, the moving body partial image RB is generated based on the goal scene GS and the binary image MP (player + bicycle) of the moving body portion (S6). .. In other words, the GS image is masked with the MP image, and the moving body partial image RB is generated from the GS image. This step corresponds to the process of the moving body partial image generating unit 60 shown in FIG. FIG. 12B shows an example of an RB image corresponding to the GS image of FIG. 8A.

The process in this step may be performed by comparing all the pixels or by comparing a plurality of predetermined pixel units (for example, horizontal 3 pixel units).

Specific examples are shown below.

(When comparing all pixels)
Both the GS image and the BG image are compared pixel by pixel, and a binary image MP is obtained in which pixels in which the MSBs of hue Hn match are set to "1" and pixels which do not are set to "0".

(When comparing in units of 3 horizontal pixels)
For each line of the GS image, the region of the rightmost 3 pixels and the pixels on the left side line are compared in units of 3 pixels, and the portion where the degree of coincidence is less than or equal to the threshold is extracted. That is, the GS image is divided into horizontal images each having a height of 1 pixel, the rightmost 1x3 pattern corresponding to each horizontal image is set as a kernel K, and the norm of the kernel K and 1x3 extracted in order from the left horizontal image one pixel at a time. A binary image MP having "1" when the difference between the norms of the patterns is equal to or larger than the threshold value and "0" in other cases is generated.

In step S7a following step S6, the clothing color cluster RCn (and its center of gravity) of each player is extracted from the moving body portion RB. Further, in step S8 of this example, the barycentric coordinates CCn of each clothing color cluster RCn are set to (Xn, Yn), and (Xn+A, Yn+B), (Xn+C, Yn+B), (Xn+A, Yn+D), (Xn+C, Yn+D) 4 are obtained. The area HZn surrounded by the dots is cropped from the MP image or the GS image. Other steps are the same as those in the second flowchart in FIG.

<Modification>
The embodiments described above are examples for the purpose of description, and the present invention is not necessarily limited to the above configurations and operations, and various modifications and changes can be made.

For example, the present invention is not necessarily limited to bicycle races, but can be applied to any competition in which an athlete (competition participant) moves with a boarded object. Such races include, for example, auto racing and boat racing, in addition to bicycle races. In the case of auto racing, the loaded object is a motorcycle, and the target of the goal determination is the front end portion of the front wheel as in the race. In the case of a boat race, the boarding object is a boat, and the goal determination target is the tip of the boat. In the case of boat racing, the reference image is the image of the tip of the boat. Although it is different from the current situation, it is applicable to horse racing if the jockey wears different colored clothing corresponding to the horse number. In that case, the determination target is a typical horse profile (nose). Furthermore, the competition is not limited to public competitions.

10 Color collection unit 20 Allowable range setting parameter generation unit 30 Image storage unit 40 Background image generation unit 50 Binary image generation unit 60 Moving body partial image generation unit 70 Clothing color area recognition unit 80 Goal determination target area extraction unit 90 Image matching Unit 95 Arrival order output unit 100a to 100c Arrival order determination device 200 Data processing device 200 Data processing device 205 Bus 220 Memory 221 OS/Program area 223 Data area 225 Work area 230 Display control section 250 Storage 260 Input/output I/F section ( Communication I/F part)
310 Display unit 320 Slit camera 330 Camera 340 Operation unit BG Background CCn nth player's clothing color GS Goal scene HZn nth player's goal determination target region (front wheel tip region, rectangular region)
MP moving body part (player + bicycle, foreground, binary image)
PCn nth player's clothing color RB Image cropped out of moving body part MP in goal scene GS RCn Region RI having approximate color of nth player's clothing color Reference image WPn nth player's specific position coordinates (front wheel tip Position coordinates)

Claims (13)

In a competition in which multiple athletes wear clothing of different colors, the goal scene image obtained by the slit camera installed at the goal position is processed by a computer, and the order of arrival when the athletes finish A method of judging the order of arrival,
Based on the goal scene image, the step of recognizing the color region of each player's clothing color as the body of the player,
Based on the position of the body of each player, a step of extracting a goal determination target region including a specific portion of the boarding object on which the player is boarding,
A step of obtaining specific position coordinates of the goal determination target by matching the image of the extracted goal determination target region with a reference image of the goal determination target;
A step of determining the order of arrival of the plurality of players based on the specific position coordinates obtained for the plurality of players. Generating a background image corresponding to the background of the goal scene image based on the goal scene image;
A step of generating a binary image representing whether or not a moving body part including each player based on the goal scene image,
Further comprising a step of generating a moving body partial image based on the goal scene image and the binary image,
The wearing order determination method according to claim 1, wherein the step of recognizing the color area of the clothing color recognizes the color area based on the moving body partial image. The background image is generated by copying the one-line image of the goal scene image that is guaranteed not to include the moving body portion in the step of generating the background image. Order judgment method. 4. The step of extracting the goal determination target area includes the step of defining a rectangular area of a predetermined size in a predetermined positional relationship from the body as a goal determination target area of the player. How to determine the arrival order. The wearing order determination method according to claim 1, wherein the step of recognizing the color area of the clothing color recognizes the color area based on the goal scene image. The step of recognizing the color region of the clothing color includes
A step of judging whether the color of each pixel in the image corresponds to each clothing color of the pixel by referring to a predetermined allowable range setting parameter at the time of judging the clothing color;
6. The arrival order determination method according to claim 1, further comprising the step of removing noise from the pixel group determined to be applicable. In a competition in which a plurality of athletes wear clothes of different colors, it is an arrival order determination device that determines the arrival order when the plurality of athletes make a goal,
An image storage unit that stores the goal scene image obtained by the slit camera installed at the goal position,
In the goal scene image, a clothing color region recognition unit for recognizing the color region of each player's clothing color as the body of the player,
Based on the position of the body of each player, a goal determination target area extraction unit that extracts a goal determination target area including a specific portion of the boarding object on which the player is riding from the goal scene image,
An image matching unit that obtains specific position coordinates of the goal determination target by matching the extracted image of the goal determination target region with a reference image of the goal determination target,
A finish order determination device comprising: a finish order output unit that determines and outputs a finish order of the plurality of players based on the specific position coordinates obtained for the plurality of players. A background image generation unit that generates an image corresponding to the background of the goal scene image from the goal scene image,
A binary image generation unit that generates a binary image indicating whether or not a moving body portion including each player based on the goal scene image,
A goal determination target area extraction unit that generates a moving body partial image based on the goal scene image and the binary image,
The wearing order determination device according to claim 7, wherein the clothing color area recognition unit recognizes a color area based on the moving body partial image. The arrival order determination device according to claim 8, wherein the background image generation unit generates the background scene image by copying an image of one line in which it is guaranteed that the moving body portion is not included. 10. The finish order determination device according to claim 7, wherein the goal determination target area extraction unit determines a rectangular area of a predetermined size in a predetermined positional relationship from the body portion as a goal determination target area of the player. .. The wearing order determination device according to claim 8, wherein the clothing color area recognition unit recognizes a color area based on the goal scene image. The clothing color area recognizing unit determines whether the color of each pixel in the image corresponds to each clothing color of the pixel by referring to a predetermined parameter for allowable range setting at the time of clothing color determination, The arrival order determination device according to any one of claims 7 to 11, wherein noise is removed from a pixel group determined to be applicable. In a competition in which multiple athletes wear clothing of different colors, the goal scene image obtained by the slit camera installed at the goal position is processed by a computer, and the order of arrival when the athletes finish It is a program that makes a judgment,
On the computer,
Based on the goal scene image, the step of recognizing the color region of each player's clothing color as the body of the player,
Based on the position of the body of each player, a step of extracting a goal determination target region including a specific portion of the boarding object on which the player is boarding,
A step of obtaining specific position coordinates of the goal determination target by matching the image of the extracted goal determination target region with a reference image of the goal determination target;
And a step of determining the order of arrival of the plurality of players based on the specific position coordinates obtained for the plurality of players.