JP4697923B2 - Counting system and counting method for moving object in water or water surface - Google Patents

Description

  The present invention relates to a counting system and a counting method for moving objects in water or on the surface of water. More specifically, the present invention relates to an improved system and method for automatically counting quantities of moving objects such as fish swimming underwater or moving leaves on a river, by image processing.

  As a dam location measure by the electric power company accompanying the construction of the dam, there is a work to set up a fishway in the river, count fish such as ayu (fish) going up the fishway, and report it to the fishery cooperative. However, the current measurement of the number of sweetfish is carried out by visual observation, such as counting the number of fish by visually observing the captured fishway images. Improvements are also desired in terms of measurement efficiency.

  Therefore, in order to improve such a problem, a plurality of methods such as using ultrasonic waves have been proposed as a method for measuring the number of fish such as sweetfish going up the fishway. The conventional techniques for measuring the number of fish and the conventional techniques that are intended for other than fish but are considered to be related to the present application are as follows.

  (1) A method for determining the presence / absence of fish using background differences, and a method for determining fish species using recorded images, decision trees, neuros, human eyes, and the like have been proposed (for example, see Patent Document 1). .

  (2) A method has been proposed in which a local bright area is extracted based on the difference between the image smoothed for each block and the image before smoothing, and the number of fish is counted by detecting the belly of the fish and the splashing at the time of landing ( For example, see Patent Document 2).

  (3) A method has been proposed in which a target passing through a region having a width d perpendicular to a moving object is extracted by background difference, and the number of passing objects is counted in the time direction (see, for example, Patent Document 3). .

  (4) Extract the change area by background difference or inter-frame difference, calculate the correlation between the background and the change area, determine that it is a moving target object when the correlation is low, perform area integration and tracking, and move objects such as people Has been proposed (see, for example, Patent Document 4). As a method for updating the background data, two types of methods, partial update and total update, have been proposed.

  (5) A method to detect moving objects such as people by dividing the screen into square blocks, estimating the inter-frame movement direction of each block, and statistically processing the movement direction and amount has been proposed ( For example, see Patent Document 5).

Japanese Patent Publication No. 8-7069 JP 2001-323445 A JP-A-8-123935 JP 2001-8189 A JP-A-9-252467

  However, in the case of the conventional method for measuring the number of run-ups such as sweetfish, a special device is required, so there is a problem that it takes a lot of installation cost and labor, and cost reduction is an issue. Furthermore, there is a practical problem that the measurement accuracy is as low as about 70% according to the conventional method. In addition to these problems, the conventional techniques (1) to (5) described above have the following problems.

  First, the conventional method shown in (1) is primarily a method that is intended to assist humans in counting rather than using natural counting, and features related to the moving direction and speed of the measurement object. May not be used, and it is difficult to automatically count. In addition, since noise countermeasures are hardly taken, it is not suitable for automatic processing in that respect. Furthermore, since the background data creation is an integration type, an error becomes large when a large number of fish shadows pass all at once.

  In the case of (2), there is a problem that many noises are detected and accurate counting is difficult. Moreover, since this difference method is a method of extracting a bright part, reflection of the water surface or the like may be spread as noise. Furthermore, since there is no fish tracking process, there is a problem that if the fish is seen over a plurality of frames, it may be counted more than the actual number. In addition, the number of fish moving in the water is calculated by estimation, and there is a problem that the counting accuracy is poor because it is not an actual measurement.

  Furthermore, in the case of (3), for example, there is a problem that a measurement error may occur because no countermeasure is taken when there is reflection in water or on the water surface. Moreover, since it is necessary to increase the width d for a fast moving object, while it is necessary to reduce the width d for a small object, for example, a small and fast counting object such as a fry of sweetfish. It is difficult to find d having an appropriate size, and it is difficult to accurately detect such a moving body. Furthermore, since there is no idea about the method for creating and updating the background data, if the background changes or shadows or fluctuations occur, accurate counting cannot be performed as it is. Further, since the object is detected and measured only in a thin area, the size and speed of the object become an estimated value, and there is a possibility that the error may increase depending on the speed and direction of the moving body.

  In the case of (4), two types of background data update methods, partial update and overall update, have been proposed as described above, but the former (partial update of background data) absorbs fluctuations in the water surface. Since it is not possible, it is easy to lead to erroneous measurement, and the latter (overall update of background data) is updated only when the change area becomes a certain amount, for example, 70% or more. Each has its own problems. In other words, in the former (partial update of background data), a process is performed in which an area with little change is added to the background. However, this condition occurs because a background change occurs in an area that fluctuates like the water surface. Is not added to the background, and valid background data is not updated. In the latter (overall background data update), the background data update is triggered by whether the change area exceeds a certain amount, for example, more than 70% of the change area. Even if there is an area that continues, the trigger is not triggered and the update is not performed. In this case, old background data is used, and there is a problem that over-measurement is performed in an area where the change continues. In addition, the method of correlating with the background as in (4) is suitable for accurate measurement of the moving area, but as the number of measurement objects increases, a large amount of calculation is required. It is not suitable for measurement. Furthermore, when tracking fish, the closest area that does not overlap with the candidate area of the previous frame is selected, but with this method, low-speed moving objects at high frame rates and low frame rates There is a problem that the high-speed moving object cannot be tracked.

  Furthermore, in the case of (5), since the movement direction of the block is estimated while relying on the existing method, the estimation accuracy regarding this point is not necessarily high. Also, it is suitable for measuring relatively large objects such as people, but not for measuring small objects, and when used for counting fish moving on the fishway or objects floating on the water surface. There is a problem that accuracy is inferior due to lack of accuracy. In other words, in this conventional method, the screen is divided into blocks, a plurality of blocks are compared with the previous and next frames, and the movement speed is calculated by so-called block matching in which it is determined whether or not they are the same object. Therefore, it is necessary to make the block fine to calculate the moving speed of the small object. However, this causes a problem that a huge amount of calculation is required for block matching. Further, in order to measure a small object that moves at high speed, it is necessary to handle a wide measurement target region, and handling such a wide measurement target also contributes to increase the amount of calculation for matching. Furthermore, when block matching is assumed as described above, the measurement object needs to have a certain characteristic color or pattern. Therefore, for example, ayu, which is a light shadow area on the screen. This is disadvantageous when dealing with small objects that cannot guarantee the same color and pattern.

  SUMMARY OF THE INVENTION Accordingly, an object of the present invention is to provide a counting system and a counting method for moving objects in water or on the water surface, which can automatically measure fish and the like with high accuracy and more accurately, and can also reduce installation costs and labor. .

  In order to achieve this object, the present inventor has conducted various studies. If the measurement accuracy is simply improved, for example, there is a measure of simply preparing a high-precision measurement device or environment, but taking into account the installation cost and labor, measurement is carried out using existing measurement devices. It is desirable. In view of this, the present inventor paid attention to a technique of performing counting with high accuracy while performing cost reduction by performing specific image processing on image data of a frame image that has been conventionally used. I came to know the image processing method.

  The present invention is based on such knowledge, and the invention according to claim 1 obtains a camera image obtained by continuously photographing a certain measurement target region through which a moving object that is a counting object passes, In the counting system for moving objects on the water or on the water surface, the camera images are processed and the moving objects that have passed through the measurement target area within the predetermined time are counted from the frame image obtained by the image processing. The pixel having the brightest value or the darkest value is extracted from each pixel having the same coordinate in the frame image of the frame image, and the brightest image or the darkest pixel set including the set of the brightest pixels is extracted. Create the darkest image and use this brightest image or darkest image as the base data that is the basis of the counting process. A base data creation unit that re-creates the base data when there is a change exceeding the value, and a moving object candidate pixel extraction unit that sequentially creates a binary image that is a background difference between the camera image and the base data image; In the binary image created by the moving object candidate pixel extraction unit, a set of pixels whose values are within a certain distance from each other is selected, and the selected set of pixels is extracted as a connected region. Compare the labeling unit, the group of connected regions in the latest frame image extracted by the labeling unit, and the group of connected regions in the frame image at the previous stage, and move from the comparison result in the frame image A tracking / counting unit that extracts a body, detects a shadow of a moving body that is actually moving, and counts the moving body is provided.

  Further, the invention according to claim 2 is to obtain a camera image obtained by continuously capturing a certain measurement target region through which a moving object to be counted passes for a certain period of time, image-processing the camera image, In the method of counting moving objects in water or on the water surface that counts moving objects that have passed through the measurement target area within the predetermined time from the frame image obtained by the image processing, the same coordinates in a plurality of consecutive frame images A step of extracting a pixel having the brightest value or the darkest value from each pixel, and creating a darkest image including a set of the brightest pixels or a darkest image including a set of the darkest pixels. When the brightest image or darkest image is used as the base data that is the basis of the counting process, and changes in the luminance distribution are periodically measured with respect to changes in the entire screen, and there is a change that exceeds a certain threshold Includes a step of re-creating the base data, a step of sequentially creating a binary image that is a background difference between the camera video and the image of the base data, and a constant mutual distance in the sequentially created binary image. A step of selecting a group of pixels having a value of 1 within the distance, a step of extracting the selected group of pixels as a connected region, a group of connected regions in the extracted latest frame image, and one stage thereof The step of comparing the connected region group in the frame image at the previous time point, and the moving object is extracted from the frame image from the comparison result, and the shadow of the moving object actually moving is detected to detect the moving object And a step of counting.

In the counting system or counting method described above, even when the frame image to be processed is the same as the conventional one, the following specific processing is performed on the frame image (see FIGS. 1 and 2). ). That is, a plurality of continuous frame images (for example, 15 frame images, but 8 continuous frame images are shown in FIGS. 1 and 2) are extracted, and the first frame image is first extracted. The luminance i ₁ (x, y) of the pixel at the coordinates (x, y) in I ₁ is extracted and taken on the graph (i ₁ shown in FIG. ₁ ). Subsequently, the luminance i ₂ (x, y) of the pixel at the same coordinate (x, y) in the frame image I ₂ is extracted, and this is also taken as i ₂ on the graph. Similarly, the luminance i _k (x, y) of the same coordinates located in a row in a skewered manner for each successive frame (I ₁ , I ₂ ,..., I _n ) is extracted and taken on the graph. In addition, a luminance range that is identified as a moving object in the frame image is set based on the color and darkness of the moving object (that is, the counting target) displayed in the image. For example, when the moving object is a fish that is darker than the background image, a luminance band A that is a moving object detection range is set in a range corresponding to the lower luminance, for example, i _{y to} i _x (see FIG. 1). Furthermore, although setting the luminance to be a reference for detecting a moving object, it calculates the average brightness i _av of i ₁ through i ₈ as indicated by the two-dot chain line in FIG. If the prior referenced to this In contrast to the general case, in the present invention, the darkest pixel i _min (i _{4 in} FIG. 1 corresponds to i ₄ ) is extracted from i _{1 to} i ₈ and used as a reference luminance ( FIG. 1). Conventionally, the difference between the reference luminance and the luminance band A is the B width shown in FIG. 1, whereas in the present invention in which such a reference is provided, the difference is an extremely narrow C width. These B width and C width are arbitrarily set values that can be appropriately adjusted according to the brightness value of the moving body that is the detection target. For example, when the B width is set to a smaller value, about i ₄ May be misidentified as a moving object, even if the luminance is about the darkest pixel in the background. However, when the reference luminance is the darkest i _min as in the present invention, the luminance is about i ₄ , that is, the darkest pixel in the background, as long as the C width is set based on this. There is very little risk of mistaking that the brightness is the brightness of a moving object. The darkest background image formed by the synthesis operation of extracting the darkest pixel and creating the background data in this way for all the pixels in the continuous frame image is the base data that is the basis of the counting process in the present invention. Become. Contrary to the above case, when the moving body is brighter than the background image, the reference brightness is set based on the brightness of the brightest coordinate among the same coordinates, and compared with the brightest image. A moving object will be extracted.

  Furthermore, in the present invention, in addition to creating suitable base data based on the darkest pixel by the above-described procedure, it is also considered that the brightness of the background image changes afterwards due to changes in the weather, etc. On the other hand, the change in luminance distribution is periodically measured, and when there is a change exceeding a certain threshold, the base data is recreated. Also, for binary images that are sequentially created by taking the difference between the latest frame image and the base data, a plurality of pixels that are densely packed within a certain range are selected from the difference data, and this collection is used as one connected region. Extract. Further, a small independent area having a certain number of pixels or less belongs to so-called noise data that is not a moving body and is deleted to remove noise. The connected region group created as described above is compared with the connected region group in the previous frame image, and when a moving object is extracted based on the difference between the two, it is counted as a counting target.

Furthermore, the present inventor has repeatedly studied and, based on the above-described technique, has come to know the technique of obtaining the base data by obtaining the average value and standard deviation of the luminance of the pixels at the same coordinates (x, y). . The inventions according to claims 3 and 4 are based on such knowledge, and the invention according to claim 3 continuously photographs a certain measurement target region through which a moving object as a counting target passes for a certain period of time. A moving body in water or on the water surface that obtains a camera image obtained by performing image processing on the camera image and counts the moving body that has passed through the measurement target area within the predetermined time from the frame image obtained by the image processing. in the counting system, the luminance i ₁ for each pixel in the same coordinates in the plurality of consecutive frame images _{(x, y), i 2} (x, y), ..., i n (x, y) of the average v ( x, y) and standard deviation σ (x, y)
b (x, y) = v (x, y) + kσ (x, y) (where k is any number)
An image consisting of a set of pixels represented by x, y coordinate values b (x, y) of the image is created and used as the base data that is the basis of the counting process. When the change is periodically measured and there is a change that exceeds a certain threshold, a base data creation unit that re-creates the base data and a binary image that is a background difference between the camera video and the base data image are sequentially In the binary image created by the moving object candidate pixel extraction unit and the moving object candidate pixel extraction unit to be created, a set of pixels whose values are within a certain distance from each other is selected and selected. Compare the labeling unit that extracts a collection of pixels as a connected region, the connected region group in the latest frame image extracted by the labeling unit, and the connected region group in the frame image at the previous stage. Comparison Extracting mobile from fruit in the frame image, and characterized by having an actually detects the shadow of the moving body is moving track-counting unit for counting the moving object.

Further, the invention according to claim 4 obtains a camera image obtained by continuously capturing a certain measurement target region through which a moving object to be counted passes for a certain period of time, performs image processing on the camera image, In the method of counting moving objects in water or on the water surface that counts moving objects that have passed through the measurement target area within the predetermined time from the frame image obtained by the image processing, the same coordinates in a plurality of consecutive frame images Step of obtaining σ (x, y) as an average v (x, y) and standard deviation of luminance i ₁ (x, y), i ₂ (x, y), ..., i _n (x, y) of each pixel And the formula
b (x, y) = v (x, y) + kσ (x, y) (where k is any number)
A step of creating an image consisting of a set of pixels represented by x, y coordinate values b (x, y) and making it the base data that is the basis of the counting process, and changes in the luminance distribution with respect to changes in the entire screen The base data is periodically measured and when there is a change exceeding a certain threshold, the base data is recreated, and the binary image that is the background difference between the camera image and the base data image is sequentially created; Selecting a group of pixels having a value of 1 that are within a certain distance from each other in the sequentially created binary image, extracting the selected group of pixels as a connected region, The step of comparing the connected region group in the latest extracted frame image and the connected region group in the frame image at the previous stage is extracted, and the moving object is extracted in the frame image from the comparison result. In By detecting the shadow of the moving body is moving and is characterized in that comprising the step of counting the mobile.

In the counting system according to claim 3 or the counting method according to claim 4, a plurality of continuous frame images (for example, 15 frame images, but in FIG. 1 and FIG. First, the luminance i ₁ (x, y) of the pixel at the coordinate (x, y) in the _first frame image I ₁ is extracted and taken on the graph (in FIG. 1). I ₁ ). Subsequently, the luminance i ₂ (x, y) of the pixel at the same coordinate (x, y) in the frame image I ₂ is extracted, and this is also taken as i ₂ on the graph. Similarly, the luminance i _k (x, y) of the same coordinates located in a row in a skewered manner for each successive frame (I ₁ , I ₂ ,..., I _n ) is extracted and taken on the graph. In addition, a luminance range that is identified as a moving object in the frame image is set based on the color and darkness of the moving object (that is, the counting target) displayed in the image. For example, when the moving object is a fish that is darker than the background image, a luminance band A that is a moving object detection range is set in a range corresponding to the lower luminance, for example, i _{y to} i _x (see FIG. 1). The process up to this point is the same as in the first or second aspect. Furthermore, the reference luminance for detecting the moving body is set. Here, the luminance i ₁ (x, y), i ₂ (x, y),..., I _n (x, y) of each pixel having the same coordinates is set. The average v (x, y) of is obtained. Further, σ (x, y) is obtained as the standard deviation, and this is multiplied by an arbitrary numerical value k, and finally added to obtain b (x, y). In the first and second aspects, the darkest pixel i _min (or the brightest pixel i _max ) extracted is used as the reference luminance as it is, but in the third and fourth aspects, the luminance of the moving object to be detected is determined. Since the reference value can be appropriately adjusted according to the value of the image, the luminance of the darkest pixel (or the brightest pixel) in the background is set to be the same as in the case of the first or second aspect. There is very little risk of being mistaken for brightness.

  According to the measurement system of claim 1 described above, the background data for background difference is generated based on the darkest image or the brightest image obtained by combining the plurality of frame images, so that shadows and water surfaces in the shooting environment are generated. It is possible to realize accurate automatic counting while performing high-speed processing by suppressing the influence of fluctuations and the like, and making the highest priority on the characteristic part of the moving speed and direction of the moving object to be counted. In addition, this measurement system has a feature in the procedure of image processing, and can perform measurement by diverting an existing measurement device as it is, so that installation cost and labor can be reduced.

  Similarly, according to the measurement method of claim 2, the background data for background difference is generated based on the darkest image or the brightest image obtained by the composition calculation of a plurality of frame images. It is possible to realize accurate automatic counting while performing high-speed processing by suppressing the influence of fluctuation of the water surface and the like, and making the highest priority on the characteristic part of the moving speed and direction of the moving object to be counted. In addition, this measurement method has a feature in the procedure of image processing, and can make use of an existing measurement device as it is, thereby making it possible to reduce installation costs and labor.

  Furthermore, according to the measurement system of claim 3, the background data for background difference is obtained by using the average v (x, y) of the luminance of the frame images and the standard deviation σ (x, y). By generating, while suppressing the influence of shadows and water surface fluctuations in the shooting environment, and using the characteristic part of the moving speed and direction of the moving object to be counted as the highest priority, while being high-speed processing Accurate automatic counting can be realized. In addition, this measurement system has a feature in the procedure of image processing, and can perform measurement by diverting an existing measurement device as it is, so that installation cost and labor can be reduced.

  Similarly, according to the measurement method of claim 4, the background data for background difference is obtained by using the average v (x, y) of the luminance of the frame images and the standard deviation σ (x, y). It is a high-speed process that suppresses the influence of shadows and water surface fluctuations in the shooting environment, and uses the characteristic part of the moving speed and direction of the moving object to be counted with the highest priority. However, accurate automatic counting can be realized. In addition, this measurement method has a feature in the procedure of image processing, and can make use of an existing measurement device as it is, thereby making it possible to reduce installation costs and labor.

  In addition, according to the above measurement system or measurement method, automatic measurement is possible by utilizing the feature points related to the moving direction and speed of the counting target object to the problem of the conventional method (1). Measures against noise by devising the procedure of image processing and generation of background data, and moreover, it is possible to cope with a large number of counting objects by adopting a method of updating background data generation discontinuously rather than integrating type. Therefore, this problem can be solved. For example, in the conventional method of (1) above, new background data is created using the previous background data (accumulation type), but in this application, new background data is used each time without using the previous background data. (Discontinuous update type). For example, considering a large amount of sweetfish going up, in the accumulation type, sweetfish creates a background that includes these sweetfish even though it is not a background. In the case of molds, such problems do not occur because the new background is recreated when the sweetfish is gone.

  In addition, for the above problem (2), it is difficult to count accurately because it avoids unnecessary over-counting by directly measuring fish in the water and also performing tracking processing. It can solve such a problem.

  To solve the problem (3) above, when the moving object is measured based on the background data created by performing the synthesis operation for all the pixels in the frame image, the background changes or shadows or fluctuations occur. However, since it is possible to perform accurate counting, it is possible to solve such a problem that accurate counting is difficult.

  To solve the problem (4) above, the moving object is counted based on the background data generated by the synthesis operation to eliminate the influence of reflection and fluctuation before extracting the moving object candidate pixel, and the luminance distribution. Periodic changes are measured and background data is recreated to avoid miscounting before updating, and at the high frame rate, emphasis is placed on characteristic features of movement such as the moving speed and direction of moving objects. Such a low-speed moving object and a high-speed moving object at a low frame rate can be measured, so that this problem can be solved.

  Furthermore, the above problem (5) can be solved because accurate counting is possible by a counting method that does not depend on the block movement speed and the block movement direction, which are difficult to calculate accurately. .

  Hereinafter, the configuration of the present invention will be described in detail based on embodiments shown in the drawings.

  3 to 11 show an embodiment of the present invention. The counting system for a moving object in water or on the surface of water according to the present invention obtains a camera image obtained by continuously capturing a certain measurement target region through which a moving object as a counting target passes for a certain period of time. Image processing is performed, and moving objects that have passed through the measurement target region within the predetermined time are counted from the frame image obtained by the image processing. The counting system according to the present embodiment includes a base data creation unit that performs synthesis calculation on all pixels of a frame image to create background data as a base and periodically creates the data, a camera video, a base data image, A moving object candidate pixel extraction unit that creates a binary image using the background difference of the image, a labeling unit that selects and extracts a set of pixels having a value of 1 in the binary image, and the latest extracted by the labeling unit A tracking / counting unit that extracts a moving body by comparing a group of connected regions in a frame image with a group of connected regions in a frame image at a point before that one stage. In this embodiment, fry of sweetfish going back to the river is used as a moving object to be counted, and the number of sweetfish passing through the fishway 6 is measured after a fishway 6 is provided in the river as a measurement target region (see FIG. 10). ). Hereinafter, an embodiment in which the moving body counting system having the above-described configuration is applied in the measurement will be described. In the counting system of the present embodiment, a sheet-like object 7 (hereinafter simply referred to as “seat”) having a color brighter than at least a sweetfish swimming in water is installed in the water to provide a fishway 6, and the sweetfish is placed on the sheet 7. When passing, the image showing the sweetfish is made dark in the image. Of course, the shape, pattern, etc. of the underwater sheet 7 must not prevent the sweetfish from going up. A fixed camera (not shown) is installed in the vicinity of the fishway 6 so that at least the measurement target range in the fishway 6 is included in the fishway 6 provided with the underwater sheet 7 in this way.

  Hereinafter, a method for automatically counting the number of sweetfish going up by image processing according to the present embodiment will be described. First, the overall flow of the processing is outlined, and then the contents of each processing block are described in detail.

  First, the entire processing flow will be described below (see FIG. 3). In the present embodiment, measurement is performed using a video camera and a computer. In actual measurement, the video may be recorded on a tape or the like and played off-line with a computer while playing it, or on-line processing measured in real time at the measurement site. Also good. Image processing is performed on video data captured by the video camera.

  The image processing performed in this embodiment is based on the classic background subtraction method of taking out an image of an object (or image data representing an object) reflected on a fixed camera, and storing background data in advance. In addition, the difference between the current camera image and the background data is calculated and the part that is not reflected in the background is extracted. Not only this classic method but also the following two points are added. Is. In other words, one is the improvement of the conventional background subtraction method to deal with noise elements such as reflection and refraction of the water surface, and the other is detection of the target of ayu moving at a constant speed. Therefore, tracking and counting are performed by utilizing characteristic parts of the operation such as the moving speed. By performing these characteristic processes, a technique has been constructed that can count up the ayu like a light shadow moving at high speed in the screen with high accuracy and in real time in the presence of fluctuations in the water surface. These feature points will be described later when the contents of each processing block are described in detail.

  First, the processing procedure will be described with reference to the processing flow shown in FIG. The base data creation unit 1 that performs the first process creates and updates the background data (step 100). Next, a moving fish candidate pixel extraction unit (hereinafter referred to as “fish shadow candidate pixel extraction unit” in this embodiment) 2 takes out ayu fish shadow candidate pixels. Here, pixels having image features different from the background are detected. However, since only a pixel different from the background is taken out, such an image contains various noises. Subsequently, apparent noise of about several pixels is removed by the labeling unit 3, the areas that have been blurred by the background difference processing and divided into a plurality of areas are combined, and an identification number is assigned to each area. In the present specification, these processes by the labeling unit 3 are referred to as “labeling”. Then, the last tracking / counting unit 4 compares the result of labeling of the image taken immediately before and examines the movement of the fish shadow while considering the sweetfish characteristics such as the moving speed and the moving direction. Detect and count the number of fish. When the last processing (fourth processing counted from the first base data creation unit 1) in the tracking / counting unit 4 is finished, the process returns to the base data creation unit 1 that performs the first processing by tracing the loop (FIG. 3). reference).

  In the present embodiment, the above-described four processes are continuously performed on each frame of video data captured by the video camera, and the count result is recorded in a file together with the measurement time. For example, in the case of NTSC video, the frame rate is 29.97 fps (frames / second).

[Base data creation unit 1]
Next, the contents of each processing block will be described in detail. First, the base data creation unit 1 will be described. As described above, the base data creation unit 1 creates and updates the base data (indicated by reference symbol B in FIG. 5) as a background. The processing target when performing such image processing is the entire video image. There is no need, and a limited portion where the sweetfish to be counted passes and is not easily disturbed, for example, the portion obtained by masking an unnecessary portion is sufficient. In this specification, this portion is referred to as a measurement target region, and is indicated by a symbol S in FIG. By presuming such mask processing, the amount of calculation can be reduced and the mixing of noise can be restricted, so that high-speed and high-precision processing is possible. For example, in the case of DV-NTSC, the number of pixels of the video is 720 × 480 pixels, but the measurement target region S is limited by performing mask processing.

  Here, the background data creation method in the base data creation unit 1 of this embodiment will be compared with the conventional background data creation method. In the case of the conventional type in which a simple background data is created by extracting an image of an arbitrary frame from the video of the video camera and using it as the base data B, if the background after the base data B is created changes The background of time and the base data B may be different, and accurate measurement may be hindered. By the way, there are two types of background changes here, one is the change in lighting of the entire measurement environment, the change in the entire screen due to the white balance and exposure change of the video camera, and the other is the fluctuation of the water surface and the trees. It is a partial change due to fluctuations in the image.

  On the other hand, in this embodiment, regarding the change of the entire screen, the luminance and the luminance distribution are measured regularly or constantly, and the base data B is recreated when there is a change exceeding a certain threshold. Measures are taken (see the processing flow shown in FIG. 4). That is, after the base data creation unit 1 starts processing (step 101), the average brightness of the measurement target region S is calculated (step 102), and the difference from the average brightness when the previous base data B is recorded is calculated (step 102). Step 103) As a result of determining whether or not this difference exceeds a certain value (Step 104), if it exceeds the certain value, the base data B is re-created and updated (Step 105). The process of the base data creation unit 1 is terminated without updating B (step 106).

  On the other hand, partial changes in the measurement target region S due to fluctuations in the water surface, fluctuations in the trees, and the like are dealt with by creating a base data B by synthesizing a plurality of images. . By performing the synthesis operation in this way, it is possible to create the base data B in consideration of fluctuations in the time direction (changes with time) that cannot be handled by a single image. In the present embodiment, the base data B is created when the apparatus is activated, when instructed by the user, and when an instruction to regenerate the base data B is given due to the above change.

Next, the concept of the base data creation method will be described (see FIG. 5). A base image is created from successive frame images (I ₁ , I ₂ ,..., I _n ). I ₁ is a frame image at the start creation based data, I ₂ is the next frame image, the I _n an image after (n-1) frame of I _1. In this case, if n is increased, a long-term change can be absorbed. However, since the extreme change that occurs infrequently is included and the detection accuracy may be lowered, the value of n is measured. It should be determined by testing in the environment. In the program in this embodiment, n = 15 and 15 frames are taken in 0.5 seconds, and base data B is created from ₁₅ frame images (I ₁ , I ₂ ,..., I ₁₅ ).

A specific example of creating the base data B is as follows (see FIG. 5). That is, first, let the luminance value of the x, y coordinates of the frame image I _{k be} i _k (x, y). When the background is brighter than the counting target as in the present embodiment where the sweetfish is counted, i ₁ (x, y), i ₂ ( The darkest value among x, y),..., i _k (x, y) is set as the x, y coordinate value b (x, y) of the base data B.
(Formula 1)
b (x, y) = min (i ₁ (x, y), i ₂ (x, y),…, i _n (x, y))
For example, in the present embodiment, the lowest luminance value among the 15 coordinates at the same position in each of the 15 frame images is set as the darkest value, and such darkest value calculation is performed in the frame image. Repeat for all coordinates. In contrast to the case of the present embodiment, when the background is dark and the counting target is bright (Equation 2)
b (x, y) = max (i ₁ (x, y), i ₂ (x, y),…, i _n (x, y))
As described above, the darkest value is the x, y coordinate value b (x, y) of the base data B. For example, when the fishway 6 is bright and the fish shadow is darker than that as in the experimental video described later, the darkest image in which all coordinates are the darkest values is obtained by Equation 1. On the other hand, depending on the situation of the fishway 6 provided in the river and its surroundings and the type of fish to be counted, the fishway 6 is dark and the fish shadow passing there may be brighter. It is necessary to obtain the brightest image in which all coordinates are the brightest values according to Equation 2 (see FIG. 5).

The method described so far is the method of calculating the brightest value or the darkest value based on the maximum or minimum luminance value, but instead of simply calculating the maximum or minimum value in this way, the average and standard are calculated. By calculating the deviation and setting a value such as k times the standard deviation, it is possible to maintain robustness against instantaneous noise when creating the base image. An example of a method using the mean and standard deviation is, for example, the average of i ₁ (x, y), i ₂ (x, y), ..., i _n (x, y) is v (x, y), When the standard deviation is σ (x, y), the x, y coordinate value b (x, y) of the base data B can be calculated using the following Equation 3.
(Formula 3)
b (x, y) = v (x, y) + kσ (x, y)
For example, if k is 1, a bright image in a fluctuation range of 1σ is created. Of course, if the background is bright and the counting target is dark, k <0. Conversely, if the background is dark and the counting target is bright, k> 0.

  In addition, when creating the base data B as described above, these changes are also taken into consideration so that a partial area with fluctuations such as when the water surface changes is not mistakenly identified as a fish shadow. It is desirable to do. “Considering change” corresponds to, for example, changing a difference amount serving as a threshold. For example, the part that is caught by the threshold value is displayed in red on the user interface screen, and the user who sees it does not turn red by moving the threshold value up and down with a slider so that the fluctuation of the water surface does not turn red. For example, designing software so that it can be reconfigured. However, when “change” is considered in this way, the difference amount decreases when ayu passes through a partial area with large changes, and one fish shadow is divided into multiple parts, or a small fish shadow is hidden. In this embodiment, the fish shadow division is prevented from having a plurality of fish shadows by performing a labeling process. On the other hand, since the fish shadow is hidden, the movement of sweetfish is relatively fast, and the region with fluctuations can be limited to a narrow range, so it can be said that the frequency of hiding is low. In the form, no special measures are taken.

[About Fish Shadow Candidate Pixel Extraction Unit 2]
Next, the fish shadow candidate pixel extraction process in the fish shadow candidate pixel extraction unit 2 will be described in detail.

The fish shadow candidate pixel extraction unit 2 always compares the current frame image with the generated base data B except when the base data B is generated, and creates an image as a material for detecting sweetfish (see FIG. 6) In addition, in FIG. 6, the fish shadow of the sweetfish extracted by the background difference is shown with the code | symbol 5). Specifically, when extracting the sweetfish candidate pixels, assuming that the luminance image of the current frame image is I, pixels that satisfy the following conditional expression are extracted. However, the case where the base data B is generated as the darkest image based on the above-described Equation 1 is described here.
(Formula 4)
b (x, y) -i (x, y)> t _b
T _{b in} Equation 4 is a threshold value representing the difference between the brightness of the detection target and the darkest pixel value of the base data B, and is set by a human (operator) via the user interface. In this case, when the sample video is obtained in advance, the threshold t _b may be set by measuring the luminance of the detection target in the sample video. In this way, a binary image is calculated with 1 as the pixel that satisfies the condition and 0 as the pixel that does not satisfy the condition. In this case, when a pixel of the candidate pixel extraction image S is expressed, s (x, y) = 1 or 0. In general, base image groups B _1u , B _1l , B _2u , B _2l ,..., B _mu , B _ml and a set of target image features (I ₁ , I ₂ ,..., _Im ) In this case, for all k = l, ..., m (Equation 5)
b _ku (x, y) −i _k (x, y)> t _ku ∧ i _k (x, y) −b _kl (x, y)> t _kl
It can be generalized as a process of extracting pixels satisfying the above. Note that the subscript u represents upper (upper limit), and l represents lower (lower limit). Specifically, the upper (upper limit) indicates the max value obtained by Equation 2, and the lower (lower limit) indicates the min value obtained by Equation 1.

[About labeling unit 3]
Next, the labeling process in the labeling unit 3 will be described in detail.

  In the binary image S created by the fish shadow candidate pixel extraction unit 2, most of the pixels with s (x, y) = 1 are shadows of sweetfish (see Fig. 6). It may not be reflected in One cause is when, for example, a fluctuation that is not a target of counting but similar in brightness to the target of counting (similar to the target of counting) appears on the screen, such as a fluctuation that occurs when water increases. Another cause is when the counting object is only partially reflected by some blurring factor. Examples of blurring factors include changes in the lighting environment such as fog, rain, and cloudiness, the presence of a part with an image feature similar to the counting target, and the case of counting underwater objects For example, turbidity that lowers the transparency of water and generation of suspended matters are generated.

  On the other hand, the labeling unit 3 performs two image processes to take measures against these problems. That is, one is noise removal by deleting small independent connected areas, and the other is reliable extraction of the counting object by combining adjacent areas. This will be specifically described below.

Here, first, the connection region C is defined as follows. That is, the connected region C is a region formed by overlapping circles having a plurality of radii d _p (here, the radius (distance) is the Euclidean distance). In this case, one circle needs to include the center of at least one other circle within the region (circumference). “A pixel is included in the connected region C” means that the pixel is inside any of the circles that form the connected region C. For example, in the present embodiment, when creating a concatenation region C from a binary image (candidate pixel extraction image) S, a circle with a radius d _p is formed around all points having a pixel value of 1 in the binary image S. The process of drawing and taking out what satisfies the definition of the connected region C described above is performed.

Here, a supplementary explanation will be given for the definition of the Euclidean distance. The distance between the coordinates (a, b) and the coordinates (c, d) is the distance between the coordinates of these two pixels and is expressed as the “Euclidean distance”, that is, the square root of (ac) ² + (bd) ² . Next, the distance between a certain pixel (hereinafter referred to as “s” for convenience) and the connected region C is set as the distance between the pixel s and all other pixels included in the connected region C (all the circles constituting the same). The minimum value of the distance to the center). Further, the distance between the connected regions C is a set (c ₁ , c ₂ ) ∀c of all pixels (centers of all the circles) included in each of the two connected regions C ₁ and C ₂ . _The distance is calculated for ₁ ∈ C ₁ and ∀c ₂ ∈ C ₂ and defined as the minimum value.

The labeling unit 3 creates a plurality of connected regions (C ₁ , C ₂ ,...) Having a mutual distance larger than the radius d _p from the binary image S. All the pixels (value 1) of the binary image S are included in any of the created connected regions. Here, the simplest creation method for creating a connected region group is shown. Starting from a connected region C including only one pixel, pixels having a distance of d _p or less are connected to the connected region C. There is a method in which the combination is terminated when there is no pixel less than d _p , and the same processing is continued using the remaining pixels as seeds. For example, when the number of pixels having a value of 1 in the binary image S is small, or when the computer is extremely fast, the connected region C can be created by this method. In this case, although the calculation amount is large, the calculation amount can be greatly reduced by using the distance between the center of gravity and the distance between the rectangular regions including the pixel group while slightly reducing the accuracy of creating the connected region C. . For example, the connected region C is expressed by simplifying with a rectangular region including a pixel group, and the distance between the pixel s and the connected region C is the minimum value of the distance between the pixel s and the four sides, and the distance between the connected regions C. Uses the minimum distance between edges.

  Here, in the labeling unit 3 in the present embodiment, a two-stage connected region creation including a [pixel registration step] and a [connected region integration step] is performed, which is shown (see FIGS. 7 and 8). ). Although not particularly illustrated, the connection region C of the present embodiment is expressed as a rectangular region including a pixel group, and the distance between the pixel and the connection region C is expressed by the minimum value of the distance between the pixel and the four sides, The distance between the connected regions C is represented by the minimum distance between the sides.

[Pixel registration step]
The pixel registration process is started (step 301), and the list of connected regions C is initialized (step 302). Here, one pixel having a value of 1 belonging to the binary image S is selected (step 303), one registered connection region C is further selected (step 304), and the value in the image S is 1. The following is performed for all pixels. That is, the distance between the connected region C and the target pixel is calculated (step 305), and it is determined whether the distance is within d _p1 (step 306). If it is within d _p1 , the process of adding the pixel to the connected region C is performed, and the process for the pixel is terminated (steps 307 and 309). On the other hand, if the distance exceeds d _p1 , the distance to the target pixel is calculated after selecting another connected region C unless the comparison with all the connected regions C is completed (step 308). Steps are repeated (step 304, step 305). If the pixel does not belong to any connected region C, a new connected region is created, and this pixel belongs to it (step 308a). When the processing for all the pixels is finished (step 309), the pixel registration processing is finished (step 310). As can be seen from this example, “d _p1 ” is the distance used when pixels are attached to each other, and is smaller than the size on the screen to be counted, and pixels representing the same target are separated on the screen. The value needs to be larger than the distance.

[Integration step of connected area]
Subsequently, the integration process of the connected area C is started (step 311). The following is performed for all connected regions C. That is, two connected regions (for example, C ₁ and C ₂ ) are selected (step 312), and the distance between the connected regions C ₁ and C ₂ is calculated (steps 313 and 314). If this distance is within d _p2 , these two connected regions C ₁ and C ₂ are integrated and registered as a new connected region C ₃ (step 315). At this time, the original two connected regions C ₁ and C ₂ are deleted from the list. On the other hand, when the distance exceeds d _p2 , the two connected regions C are not integrated / deleted. When the calculation is completed for the distances of all the connected regions C (step 316), the integrated processing of the connected regions C is ended (step 318). On the other hand, if the distance calculation has not been completed for all the connected regions C, another set of connected regions C is selected (step 317), and the distance between the two connected regions C is calculated (step 313). Repeat the steps.

  Finally, for the purpose of noise removal, a connected region having a certain number of pixels or less is deleted from all the connected regions C. The constant value in this case is a threshold value that depends on the size of the counting target in the screen. This completes the labeling process.

[About the tracking / counting unit 4]
Next, the tracking process and the counting process in the tracking / counting unit 4 will be described in detail.

  The tracking / counting unit 4 compares the connected region group of the current frame extracted by the labeling unit 3 with the connected region group of the previous frame, and detects a moving object. Here is a brief description of the characteristics of Ayu being counted. In the case of Ayu, the speed of the fish is so fast that it can be shot on the screen while changing the shape of the slight fish shadow. In order to detect objects that move slowly (for example, intruders in a building) on the premise that there are many common features in the image characteristics. If the conventional moving body detection method used for the processing is attempted, many processing omissions and excessive detection may occur. On the other hand, in the present embodiment, the high-speed but high-speed detection is performed by detecting the moving object without performing comparison of image features by taking advantage of the nature of ayu that moves at almost the same speed. Is realized. Hereinafter, the contents of tracking and counting in the present embodiment will be described.

First, the articulation region of the current image _{C c1, C c2, ...,} C cn, the articulation region of the previous frame C _p1, C _p2, ..., and C _pm. The distance between C _ck (k = 1, 2,..., N) and C _pj (j = 1, 2,..., _M ) is compared for all combinations (C _ck , C _pj ). When this distance is not less than a certain lower limit value and not more than a certain upper limit value, the two areas are determined to be the same area that has moved. In other words, when the distance is d, the upper limit is d _max , and the lower limit is d _min , C _ck is generated by the movement of C _pj when there is a pair of C _ck and C _pj that satisfies d _min ≦ d ≦ d _max. It is determined that these are connected areas for the same object. The purpose of setting the lower limit d _min here is to delete a noise-like region with a small amount of movement. The noise-like area mainly includes reflection and refraction of the water surface or a shaking shadow. On the other hand, the upper limit d _max is set so as not to confuse the connected region C with another target. When this value is set to be large to some extent, a target that moves greatly between frames is detected without omission. However, when multiple objects appear on the screen at the same time, they may be confused. Therefore, the upper limit d _max is not set too large, and it is appropriate based on prior knowledge about the moving speed of the counting object. It is desirable to set to. For example, if the average moving distance in the image in 1/30 seconds is 10 pixels and the standard deviation σ is 3 pixels, 7 pixels are set to d _min and 13 pixels are set to d _max as 1σ. If there are multiple pairs that satisfy the condition, select the closest one.

In addition, when the moving direction is almost constant, such as a fish going up the fishway 6 or an object flowing on the river surface, it can be considered as a restriction of operation by adding the feature points of the moving object with respect to the moving direction. By providing such a restriction, it becomes possible to detect quickly and accurately. For example it has river flows down from the top of the screen, in a case of detecting the fish swim upstream the river divides the screen vertically long area, if it is determined for the set of C _ck and C _pj in the region It ’s enough. In this case, the distance between C _ck (k = 1, 2,..., N) and C _pj (j = 1, 2,..., _M ) is compared for all combinations (C _ck , C _pj ). Therefore, it is possible to detect more quickly and accurately.

  Furthermore, if the features of the image are consistent between frames, such as colors and textures (the textures here are "textures" and patterns on the surface), use histograms and correlations. It is possible to improve the accuracy by performing the similarity determination and limiting the combination targets. However, since such a similarity determination generally increases the amount of calculation, it is preferable to appropriately select whether or not the similarity determination is performed in accordance with the purpose of counting / measurement.

When it is determined by the above method that the connected region C in the current image and the connected region C in the previous frame represent the same shadow of the sweetfish, 1 is added to the number of measurements to be counted. With respect to the counted object, the tracking is continued after the counting until it disappears from the measurement target area S, but the counting is not performed again. Note that “tracking” as used in this specification means that the same count is applied to those having d satisfying the above-mentioned distance condition d _min ≦ d ≦ d _max between consecutive frames (for example, I _k and I _{k + 1} ). Consecutive two or more frames, such as the target (in this case, Ayu), and the next consecutive frame (I _{k + 1} and I _{k + 2} ) are considered to be the same counting target (Ayu) This means that the object to be counted is continuously identified in the frame to be performed. In this embodiment, counting is performed at the time of determination as described above, but instead, a method of tracking to the measurable limit in the image and counting when it disappears from the image can be employed. In short, it suffices if one count can be reliably performed for one connected region to be counted.

  Further, how to process when fish shadows overlap or overlap will be described with reference to FIGS. If the fish shadow overlaps or separates in the middle, it is difficult to determine what the articulated area representing the fish shadow (hereinafter simply referred to as “area” until the end of this explanation) is a moving object. According to the embodiment, in such a case, more accurate counting can be performed by performing a certain process. In FIGS. 9 to 11, a region that has not been determined as a moving body is represented by a white rectangular frame, and a region that has been determined as a moving body is represented by a black rectangular frame.

  First, when an area appears in an image and is separated immediately (see FIG. 9), each of the areas after separation is incremented by +1 and counted as a total of two animals. On the contrary, if they overlap immediately after appearing in the image and do not separate thereafter (see FIG. 10), they are counted as one animal. Furthermore, when the image overlaps immediately after appearing in the image and then is separated into two (see FIG. 11), the separated region becomes an undetermined region, but when it is confirmed that the region has moved. Further increase the count. By these methods, a plurality of similar objects can be simultaneously photographed on the screen, and even when the number of objects is increased or decreased, it is possible to count stably and quickly. When the areas are separated, it is not necessary to determine which one is separated. It is only necessary that one of them is separated and the other one is swimming from the beginning. In this case, which one is to be separated is determined by the order of processing, and whichever comes first, the counting result does not change.

  After performing the processing described so far, the tracking / counting unit 4 displays the counting result and the current number of moving objects on the screen of the output device. Also, the appearance time of the moving object, the number of pixels in the connected area, the moving speed, the route, etc. are displayed on the screen or stored in a file.

  As described above, the automatic counting using the counting system of the present embodiment can automatically measure the number of fly-ups with high accuracy by using image processing technology only from the video taken by the video camera with the fishway 6 fixed. It is what. In addition, since the measuring device can be configured only with a commercially available or existing video camera and a PC (personal computer), the introduction is simple, and the installation cost and the time and effort during installation can be reduced.

  The above-described embodiment is an example of a preferred embodiment of the present invention, but is not limited thereto, and various modifications can be made without departing from the scope of the present invention. For example, in the above-described embodiment, the case where the fry of ayu that goes up is targeted for counting has been described, but it is possible to target fish other than ayu, as well as organisms other than fish, and for example, the surface of water. It is also possible to automatically measure the number of passing through the measurement target region S within a certain time by using the above-described measurement procedure for objects other than fish such as flowing fallen leaves.

In addition, color video is captured by the video camera, and there is a color feature in the counting target (for example, when measuring the fallen leaves flowing on the river surface, the river surface is gray while the fallen leaves are reddish. If the background is dark and the object to be counted is bright, the respective maximum values are taken into consideration. Conversely, if the background is bright and the object to be counted is dark, the respective minimum values are taken into consideration. It is also possible to measure and to make the base data B three, B _r , B _g , B _b . Alternatively, another color space such as HSV (hue, saturation, value) can be used.

  In addition, in the measurement target region S, when the counting target has an intermediate brightness with respect to the background (when the background includes a bright portion and a dark portion, and the counting target has an intermediate brightness. When measuring, the river surface is a dark color, there is a bright part due to the reflection of sunlight on the water surface, and the fallen leaves to be counted are between the two) It is also possible to create two base data B for the threshold value, set the threshold values to the upper and lower sides, and set a certain range between them as the detection condition. In this case, different detection ranges can be set by changing the threshold value.

Here, an example of a measurement technique when using upper and lower threshold values relating to luminance will be briefly shown. First, base images representing upper and lower threshold values are denoted as B _u and B _l , respectively. When the luminance image of the current frame image is I, based on Equation 5 described above (Equation 6)
b _u (x, y) −i (x, y)> t _bu ∧ i (x, y) −b _l (x, y)> t _bl
Pixels that satisfy are extracted. Here, t _bu and t _bl are thresholds arbitrarily set by the operator. For example, if the video is a color video, base images representing the upper and lower threshold values of red, green, and blue are B _ru , B _rl , B _gu , B _gl , B _bu , and B _bl , respectively. Assuming that the R, G, and B decomposition images of the current frame image are I _r , I _g , and I _b , pixels that satisfy the following conditional expression are extracted to detect the moving object.
(Formula 7)
b _ru (x, y) −i _r (x, y)> t _ru ∧ i _r (x, y) −b _rl (x, y)> t _rl ∧
b _gu (x, y) −i _g (x, y)> t _gu ∧ i _g (x, y) −b _gl (x, y)> t _gl ∧
b _bu (x, y) −i _b (x, y)> t _bu ∧i _b (x, y) −b _bl (x, y)> t _bl

  An evaluation experiment for confirming the effect of the sweetfish counting system described in this embodiment was performed. Here, an experiment was conducted for the purpose of evaluating the measurement (counting) accuracy of the counting system according to the present invention and searching for the cause of the occurrence of a measurement error, using an image of the actual fishway 6 taken. .

  The subject of the experiment was an image of the fishway 6 for 1 hour from 8:15 am to 9:15 am on April 26, 2002 (see FIG. 12). This video was taken with a Sony (registered trademark) video camera DCR-PC3. FIG. 12 shows an image taken in a form of looking down at the left half portion from the upper center of the fishway 6 toward the downstream side. The river flows from the front of the screen to the back, and the sweetfish that goes up from the downstream on the plate-like sheet 7 visible in the lower half of the screen goes up to the lower side of the screen, that is, toward the upstream side. Further, FIG. 13 shows an image of the fishway 6 when four ayus that are going up are located on the seat 7. The sheet 7 on the bottom surface of the fishway 6 was brightly colored so that it could be distinguished. However, it was difficult to detect because it was small in size and became a shabby fish shadow because it passed underwater.

  As a computer for evaluation, a Windows (registered trademark) personal computer equipped with Pentium (registered trademark) 4 having an operating frequency of 2.4 GHz was used. In addition, the program used for this evaluation (hereinafter referred to as “Ayu run-up image measurement software”) captures video images using Microsoft (registered trademark) Visual (registered trademark) C ++ (6.0 and .Net) as the development environment. Developed using Direct Show, a Direct (registered trademark) X library. An example of the user interface screen is shown in FIG. Further, the measurement target region S is a trapezoidal region as indicated by a long and narrow frame in FIG. This is an area in which the sweetfish can be confirmed on the screen, and the black wall at the left end is not reflected in the frame due to fluctuations in the water surface. However, the measurement target region S needs to be set so as to cover the region through which the sweetfish passes. Note that the trapezoidal shape here is only an example, and if the area in the screen is specified, it is counted by the same method whether it is a polygonal area, a curved area, or even multiple areas. Can do.

  Also, as a preparatory work, the video file recorded on the DVD provided by the video image photographer is ripped, that is, the digital audio data is extracted and converted into a file format that can be processed by a personal computer and saved. Was performed, and the process of writing back to the DV tape was performed. This is because the developed measurement program is designed to process video images that are directly connected to a computer via IEEE 1394 so that it can actually measure the upstream of the sweetfish.

  In addition, data measured by human eyes was used as an evaluation standard for detection accuracy. This is the total number of sweetfish that passed every minute and was provided with the DVD by the director of Fishway 6. The visual measurement is performed by a plurality of people based on the slow-played video for the sake of accuracy, and is considered sufficient as a measurement standard.

  Various settings used in the experiment are shown in Table 1. The luminance-related threshold values (1-A, 2-A) mean values in a range where 0 is the minimum value and 65,535 is the maximum. The pixel-related threshold values (3-A, 3-B, 3-C, 4-A, 4-B) and the setting value (1-B) related to the number of frames are NTSC-DV (720 × 480 pixels, 29.97 fps). ) Means the number of pixels and the number of frames.

  The number of human visual counts over the entire target video for 1 hour was 225. In contrast, the counting system according to the present invention counted 226 animals. The error was 0.4% (= (226-225) / 225). However, in actuality, this counting result includes measurement omissions and over-measurements, and as a result, it was considered that the error was offset and the total value was close. Therefore, in order to calculate the recall and accuracy, the data is counted every 5 minutes and every minute, and the count values are compared. The results are shown in FIGS. 16 and 17, respectively.

  The horizontal axis is the time interval from the start time of the sample video. “0 to 5” in the graph totaled every 5 minutes shown in FIG. 16 means a time of 5 minutes from the start of video shooting to 5 minutes later. The vertical axis is the number of sweetfish that passed during that time interval. From the figure, it can be seen that the program overcounts a relatively large amount in the time interval of 20 to 35 minutes, and that the program causes measurement omission in the time interval of 45 to 50 minutes. The number of excess counts by the program in this 5 minute count is 24, the shortage is 23, the recall is 90% (= (226-24) / 225), and the accuracy is 89% (= (226-24) ) / 226).

  In the graph totaled every minute shown in FIG. 17, the measurement characteristics of automatic measurement can be seen more finely. The horizontal axis means a time interval divided every minute from the start time of the sample video. For example, “10” on the horizontal axis means a period from 10 minutes to 11 minutes after the start of the video. It can be seen that a relatively large number of overcounting is performed in the 31 minute and 33 minute time intervals, and that there is a measurement omission for a large amount of run-up in the 47 minute time interval. The number of excess counts by the program in this one minute count is 37, the shortage count is 36, the recall is 84% (= (226-37) / 225), and the accuracy is 84% (= (226-37). ) / 226).

  As the experimental results show, it has been found that the measurement program used in this example can perform high-precision counting in real time on the actual fly-up video. However, as described above, with respect to the fact that there were about 10% overcounting and undercounting with respect to the number serving as the measurement standard, some of the causes of erroneous measurement have been clarified from the investigations so far. That is, as a cause of overcounting, there is a case where the black wall surface at the left end of the fishway 6 becomes a fish shadow shape due to the fluctuation of the water surface, and it is recognized that this is regarded as a fish shadow. It was thought that other measures such as painting the sides of the fishway were necessary.

  Further, in FIG. 17, the cause of the shortage in the 47-minute time interval in which the most deficient counts of 13 were counted is that the base data B was updated while Ayu was going up in large quantities. In the experiment, the base data was updated for 0.5 seconds, but during this time period, a large amount of sweetfish went up in a very short time, resulting in 13 count deficiencies in 0.5 seconds. On the other hand, it is considered that improvement can be achieved by a method of waiting for the update while the moving body is confirmed or a method of continuing the counting using the previous base data B during the update.

  As explained so far, in the present embodiment, the sweetfish that goes up the fishway 6 was photographed with a video camera, and the number of going up was counted in real time. This counting system for moving objects in water or on the surface of water improves the background subtraction method to avoid erroneous measurement due to fluctuations in the water surface, etc., and takes advantage of the property of the counting object to move at a constant speed. By tracking and counting with precision, quick and highly accurate counting is realized. An evaluation experiment was conducted on a 1 hour sample image of the fishway 6, and 226 animals were automatically counted against the actual run-up number of 225 animals, achieving a recall and accuracy of 84% even when counting every minute. It was confirmed that it had high counting accuracy.

FIG. 6 is a graph for explaining a characteristic concept of the counting process by the moving body counting system according to the present invention, in which pixel luminances i _k (in the same coordinates (x, y) in each frame image having 1 to 8 frames); x, y). It is a figure which shows a continuous several frame image notionally easy to understand. In the flowchart which shows one Embodiment of this invention, the whole process sequence at the time of automatically counting the number of sweetfish going up by a moving body counting system is represented. It is a flowchart which shows the process sequence of a base data preparation part. It is a figure which shows simply the processing content at the time of creating base data from a continuous frame image. It is a schematic diagram which shows simply the method of extracting a fish shadow by background difference. It is a flowchart which shows the process sequence of the pixel registration step by a labeling part. It is a flowchart which shows the process sequence of the pixel integration step by a labeling part. It is a figure for demonstrating the example of a process when a mobile body overlaps, and the case where a mobile body appears and leaves | separates immediately is shown. It is a figure for demonstrating the example of a process when a mobile body overlaps, and the case where a mobile body appears and overlaps immediately is shown. It is a figure for demonstrating the example of a process when a mobile body overlaps, and the case where a mobile body overlaps immediately after appearing, and shows the case where it leaves | separates after that. A photographed image of the fishway used in this example is shown. This is a picture of the fishway taken when Ayu passes. It is a figure which shows an example of the user interface screen of the moving body counting system to which this invention is applied. It is a figure which shows an example of the measurement object area | region set to the fishway. It is a graph which shows the comparison result of the automatic measurement by the counting program to which this invention is applied, and the automatic measurement by visual observation in the time interval for every 5 minutes. It is a graph which shows the comparison result of the automatic measurement by the counting program to which this invention is applied, and the automatic measurement by visual observation in the time interval for every minute.

Explanation of symbols

1 Base data creation unit 2 Fish shadow candidate pixel extraction unit (moving object candidate pixel extraction unit)
3 Labeling unit 4 Tracking / counting unit B Base data C Concatenated area I Frame image S Measurement target area

Claims (4)

  Obtain a camera image obtained by continuously capturing a certain measurement target region through which a moving object to be counted passes for a certain time, perform image processing on the camera image, and from a frame image obtained by performing the image processing In an underwater or water surface counting system that counts the moving body that has passed through the measurement target area within the predetermined time, the brightness is the brightest among the pixels at the same coordinate in a plurality of consecutive frame images. Value or the darkest value is extracted, and the brightest image consisting of the set of the brightest pixels or the darkest image consisting of the set of the darkest pixels is created. In addition to the base data that is the basis of the counting process, the brightness distribution changes are periodically measured with respect to changes in the entire screen, and if there is a change that exceeds a certain threshold, the base data A base data creation unit, a moving object candidate pixel extraction unit that sequentially creates a binary image that is a background difference between the camera image and the base data image, and the moving object candidate pixel extraction unit. In the binary image, a group of pixels having a value of 1 that is within a certain distance from each other is selected, a labeling unit that extracts the selected group of pixels as a concatenated region, and the labeling unit Compare the connected region group in the latest frame image with the connected region group in the frame image at the previous stage, extract the moving body from the comparison result in the frame image, and actually move And a tracking / counting unit for detecting a shadow of the moving object and counting the moving object.   Obtain a camera image obtained by continuously capturing a certain measurement target region through which a moving object to be counted passes for a certain time, perform image processing on the camera image, and from a frame image obtained by performing the image processing In the method of counting the moving object in water or on the water surface that counts the moving object that has passed through the measurement target area within the predetermined time, the brightness is the brightest among the pixels at the same coordinate in the plurality of consecutive frame images. A value or a darkest value, and a brightest image consisting of a set of the brightest pixels or a darkest image consisting of a set of the darkest pixels is created and the brightest image or darkest The step of making the image into base data that is the basis of the counting process, and the change in the luminance distribution with respect to the change in the entire screen are regularly measured. A step of re-creating a data, a step of sequentially creating a binary image that is a background difference between the camera image and the image of the base data, and a distance between each other within a certain distance in the sequentially created binary image Selecting a set of pixels having a value of 1 in the above, a step of extracting the selected set of pixels as a connected region, a group of connected regions in the extracted latest frame image, and a step before that The step of comparing the connected region group in the frame image at the time point, and extracting the moving body from the frame image from the comparison result, detecting the shadow of the moving body actually moving, A counting method of a moving body in water or on the water surface. Obtain a camera image obtained by continuously capturing a certain measurement target region through which a moving object to be counted passes for a certain time, perform image processing on the camera image, and from a frame image obtained by performing the image processing In an underwater or water surface counting system that counts the moving object that has passed through the measurement target area within the predetermined time, the luminance i ₁ (x of each pixel at the same coordinate in a plurality of consecutive frame images , y), i ₂ (x, y), ..., i _n (x, y) mean v (x, y) and standard deviation σ (x, y)
b (x, y) = v (x, y) + kσ (x, y) (where k is any number)
An image consisting of a set of pixels represented by x, y coordinate values b (x, y) of the image is created and used as the base data that is the basis of the counting process. A base data creation unit that re-creates the base data when the change is periodically measured and exceeds a certain threshold, and a binary image that is a background difference between the camera video and the base data image A moving object candidate pixel extraction unit that sequentially generates a pixel, and a binary image generated by the moving object candidate pixel extraction unit. A labeling unit that extracts a group of the extracted pixels as a connected region, a connected region group in the latest frame image extracted by the labeling unit, and a connected region group in the frame image at the previous stage , Or a tracking / counting unit for extracting the moving object from the frame image from the comparison result, detecting a shadow of the moving object that is actually moving, and counting the moving object. A moving object counting system on the surface of the water. Obtain a camera image obtained by continuously capturing a certain measurement target region through which a moving object to be counted passes for a certain time, perform image processing on the camera image, and from a frame image obtained by performing the image processing In the method of counting the moving object in water or on the water surface that counts the moving object that has passed through the measurement target region within the predetermined time, the luminance i ₁ (x of each pixel at the same coordinate in a plurality of successive frame images , y), i ₂ (x, y), ..., i _n (x, y) mean v (x, y) and standard deviation to obtain σ (x, y),
b (x, y) = v (x, y) + kσ (x, y) (where k is any number)
A step of creating an image consisting of a set of pixels represented by x, y coordinate values b (x, y) and making it the base data that is the basis of the counting process, and changes in the luminance distribution with respect to changes in the entire screen Is periodically measured, and when there is a change exceeding a certain threshold, the step of re-creating the base data and the binary image that is the background difference between the camera image and the image of the base data are sequentially created A step of selecting a group of pixels having a value of 1 that is within a certain distance from each other in the sequentially generated binary image, and a step of extracting the selected group of pixels as a connected region A step of comparing the extracted connected region group in the latest frame image with the connected region group in the frame image at the previous stage, and extracting the moving body in the frame image from the comparison result. And counting method for a mobile body in the water or the water surface, characterized by comprising a step of counting the detected and the mobile shadows of a moving body that is actually moving.