JP7462265B2 - Automatic plankton detection method - Google Patents

Description

Article 30, paragraph 2 of the Patent Act applied. Oral presentation and exhibition at CEATEC 2019 held from October 15 to 18, 2019

The present invention relates to an automatic plankton detection method.

In the current oyster farming method, at the beginning of summer when the oyster attachment stage larvae start floating, attachment substrates such as scallop shells are suspended in the sea and the larvae are allowed to attach to the substrate to secure seedlings. Therefore, if a location with an abundance of oyster larvae can be identified, it will be possible to efficiently secure oyster seedlings and begin farming. However, in reality, local government officials and fishermen themselves count the oysters by magnifying them under a microscope, and this method has a limit to the number of samples that can be detected. In particular, in years when the number of attachment stage larvae is low due to natural conditions, it is a challenge to increase the number of detection samples, and there is a demand for more efficient methods of detecting attachment stage larvae.

The present invention aims to provide a new automatic plankton detection method using a general object detection algorithm.

In one embodiment of the present invention, a computer executes a first step of training a general object detection algorithm using an image of a rectangle of a first predetermined size centered on a bounding box as training data, a second step of dividing an image of the plankton in the container into rectangles of a second predetermined size, and a third step of causing the general object detection algorithm to determine whether the plankton is present in each of the rectangles of the second predetermined size obtained by dividing the image. In the second step, the width of the overlap between adjacent rectangles is determined based on the maximum width of the plankton. The plankton may be a floating larva or a sessile larva of a sessile organism. The plankton may be an sessile larva having an eye spot of an oyster. The general object detection algorithm may be an R-CNN model, an SSD, or a YOLO model. The width of the overlap of the quadrangles may be 0.8 to 1.2 times, 0.9 to 1.1 times, or 0.95 to 1.05 times the maximum width of the plankton.

Another embodiment of the present invention is a program for causing a computer to execute any of the above methods. A storage medium storing this program in a computer-readable manner is also an embodiment of the present invention.

The present invention provides a novel automatic plankton detection method using a general object detection algorithm.

FIG. 1 is a schematic diagram of a system for managing plankton distribution information according to one embodiment of the present invention. 13 is an interface of a user application according to an embodiment of the present invention. 1 is an interface of a user application on a first terminal according to an embodiment of the present invention. 13 is an interface (area mode) of a user application on a second terminal according to an embodiment of the present invention. 13 is an interface (point mode) of a user application on a second terminal according to an embodiment of the present invention; An example of an image used in the embodiment is shown below. In addition to the images used in the examples, images of oyster larvae identified by visual inspection are shown below.

The objectives, features, advantages, and ideas of the present invention will be clear to those skilled in the art from the description in this specification, and those skilled in the art will be able to easily reproduce the present invention from the description in this specification. The embodiments and specific examples of the invention described below show preferred embodiments of the present invention, and are shown for illustrative or explanatory purposes, and do not limit the present invention thereto. It will be clear to those skilled in the art that various changes and modifications can be made based on the description in this specification within the intent and scope of the present invention disclosed in this specification.

==Server for managing plankton distribution information==
The server 11 of the present disclosure is a server 11 for managing distribution information of one or more types of plankton, communicably connected to a first user terminal 21 and a second user terminal 31, and includes a first receiving means 12 for receiving an image of the plankton from the first user terminal 21, an analyzing means 14 for detecting the plankton from the image and analyzing the number of plankton, a storage means 16 for storing the analysis result by linking it to an ID indicating the result, and a transmitting means 13 for transmitting the analysis result together with the ID to the second terminal. The ID may be received from the first user terminal 21 together with the image of the plankton, or the server 11 may assign the ID to the image after receiving the image of the plankton from the first user terminal 21.

The type of plankton is not particularly limited, and may be zooplankton or phytoplankton, but it is preferable that the plankton be in the attached stage larvae, such as, but not limited to, oysters, barnacles, mussels, sea cucumbers, and hydra. The number of types may be one or more.

This server 11 is communicatively connected to a first user terminal 21, and includes a receiving means 12 for receiving images of plankton from the first user terminal 21. Along with the images of the plankton, it may receive information on the time when the plankton was collected and the location where the plankton was collected. The server 11 may be connected to a plurality of first user terminals 21, and the first user terminals 21 may send data from various locations, making it possible to know the population of plankton at various locations.

The server 11 receives the image of the plankton and analyzes the image using the analysis means 14 to calculate how many plankton are present in the field of view. There are no particular limitations on the program used for this analysis, but it is preferable to use the program described below.

The obtained plankton count is linked to the ID assigned to the image and stored in the storage means 16. There are no particular limitations on the storage means 16, but a non-volatile storage device is preferable, and examples of the storage means 16 include ROM, flash memory, magnetic storage devices (hard disk drives, floppy disks, magnetic tapes, etc.), and optical disks.

This server 11 is communicably connected to the second user terminal 31, and includes a transmission means 13 for linking the analysis results with an ID and transmitting them to the second user terminal 31. At the same time as the analysis results, the image of the plankton transmitted from the first user terminal 21, information on the time when the plankton was collected, and/or location information on the location where the plankton was collected may be transmitted. This server 11 may receive a request for the analysis results from the second user terminal 31 via the receiving means 12, and transmit the analysis results to the second terminal 31 accordingly. In this case, the location information and time information may be transmitted according to a request received for them, or the location information and time information linked to the ID may be transmitted automatically.

The content of the time information received from the first user terminal 21 is not particularly limited, and may be just the day or the date and time.

The content of the location information received from the first user terminal 21 is not particularly limited, and may be numbers such as latitude and longitude, or a location on a map. In the case of a location on a map, the first user terminal 21 and the server 11 share the same map information, and the information may specify a location within that map, or the location information may include information indicating the map and the location within it.

The content of the time information sent to the second user terminal 31 is not particularly limited, and may be just the day or the date and time.

The content of the location information to be sent to the second user terminal 31 is not particularly limited, and may be numbers such as latitude and longitude, or a location on a map. In the case of a location on a map, the second user terminal 31 and the server 11 share the same map information, and the information may specify a location within that map, or the location information may include information indicating the map and the location within it.

The location information received from the first user terminal 21 and the location information transmitted to the second user terminal 31 may be the same data or different data, but when the data contained in the received location information and the transmitted location information are different, for example when latitude and longitude are received from the first user terminal 21 and transmitted to the second user terminal 31 as a position on a map, the server 11 has a conversion means 15 (e.g., a converter) that converts the received location information into the transmitted location information.

The server of the present disclosure may be a cloud server.

First User Terminal 21
The first user terminal 21 is communicably connected to the above-mentioned server 11, and includes an imaging device 22 for capturing an image of the plankton, and a transmission means 23 for transmitting the image of the plankton to the server 11. The transmission means 23 may transmit, together with the image of the plankton, time information at which the plankton was collected and location information of the location at which the plankton was collected. Details of the time information and location information are as described above.

The imaging device 22 may be a camera or a video. The imaging device 22 must have a resolution that allows the server 11 to identify plankton. It is preferable for the imaging device 22 to have a zoom function, for example when the lens aperture is small. The mechanism of the zoom function is not particularly limited, but an optical zoom is preferable to a digital zoom.

The first user terminal 21 may be a computer, but considering outdoor use, a tablet or smartphone with mobile phone functionality is preferable, and a mobile phone is also acceptable.

The location information transmitted by the first user terminal 21 to the server 11 may be input by the user, or the first user terminal 21 may have a location identification means such as a GPS function or Wi-Fi receiving means, and automatically identify the current location of the user terminal. The original location information generated by the user input or identification by the location identification means, and the first user terminal 21 may have a conversion means (e.g., a converter) that converts the original location information into location information to be transmitted to the server 11, and the location information to be transmitted to the server 11 may be data different from the original location information.

The image, time information, and location information transmitted to the server 11 may be stored in association with each other in the storage device 24 of the first user terminal 21.
Second User Terminal 31
The second user terminal 31 includes a receiving means 32 for receiving the results of the analysis of the plankton image by the server 11 together with the ID, and a display device 33 for displaying the analysis results together with the ID. The receiving means 32 may receive the plankton image, time information when the plankton was collected, and/or location information of the location where the plankton was collected together with the ID, along with the results of the analysis of the plankton image. Details of the time information and location information are as described above.

The second user terminal 31 may be a mobile phone, but considering that various display methods will be used when displaying, a tablet or smartphone with computer functions is preferable, but a PC may also be used. The second user terminal 31 may be the same device as the first user terminal 21 or a different device.

The location information that the second user terminal 31 receives from the server 11 may be the first location information that the first user terminal 21 sent to the server 11, or may be the second location information converted from the first location information by the server 11. When the second user terminal 31 receives the first location information, it may display the first location information on the display means 33, or may convert the first location information into second location information using a conversion device 35 (such as a converter) and display the first location information on the display means 33. When the second user terminal 31 receives the second location information, it may display the second location information on the display means 33, or may convert the second location information into third location information using the conversion device 35 and display the third location information on the display means 33.

For example, the second user terminal 31 may obtain latitude and longitude information from the server 11, and display the analysis results and the latitude and longitude values together with the ID on the display means 33. The latitude and longitude values may also be converted to positions on a map, and the analysis results and the positions on the map may be displayed on the display means 33. Alternatively, the latitude and longitude values may be converted to positions on a map, and the analysis results and the latitude and longitude values may be displayed on the display means 33, and the positions on the map may be displayed when the latitude and longitude values are clicked. Conversely, the position where the plankton was collected may be shown on the map first, and the latitude and longitude may be displayed when the position on the map is clicked. The display method on the display means 33 is not particularly limited, and the ID, analysis results, time when the plankton was collected, the latitude and route of the position where the plankton was collected, etc. may be displayed in a table.

The plankton images, analysis results, time information, and/or location information received by the second user terminal 31 from the server 11 may be associated and stored in the storage device 34 of the second user terminal 31.

==System for managing plankton distribution information==
The system according to the present embodiment is a system for managing plankton distribution information, and includes a server, a first user terminal, and a second user terminal. The details of the server, the first user terminal, and the second user terminal are as described above.

In this system, multiple first user terminals and multiple second user terminals may be communicatively connected to a server. This makes it possible to simultaneously manage plankton distribution information at multiple locations uploaded from multiple first user terminals, and to download plankton distribution information from anywhere if the conditions of multiple second user terminals are met.

==Utilization of plankton distribution information==
As an example of a method for utilizing plankton distribution information, a method using the system shown in FIG. 1B will be described in detail below. In this embodiment, the functions of the first user terminal 21 and the second user terminal 31 are executed by a single user application. The interface is, for example, as shown in FIG. 2, and allows the user to select between providing information and acquiring information.

First, a user providing plankton distribution information uses the first user terminal 21 to capture an image of the plankton with the imaging device 22. Then, using the user app, the user presses the "Acquire collection point data and send data to cloud" button to provide information. Figure 3 shows the next screen of the user app. When the "Acquire collection point data" button is pressed on the screen, the GPS function measures the latitude and longitude of the collection point and the coordinates are automatically entered. At the same time, the clock function enters the date and time of the operation. Press the "Register image" button, press the "Browse" button to select an image, and press the "Register" button to register the image. Finally, check the checkbox to the left of the "Registered" point ID and press the "Send to cloud" button at the bottom of the screen to upload this information to the server.

When the server receives an image of plankton from the first user terminal 21, it uses a plankton detection program to detect plankton from the image and counts the number of plankton. The number, along with the time information and location information received from the first user terminal 21, are linked to the transmitted ID and stored in the storage means 16.

A user who wants to obtain plankton distribution information uses the user application on the second user terminal 31 to press the "display detection results" button in FIG. 2 to perform an information request operation. The user application obtains the number of plankton, time information, and location information stored in the server, analyzes the data, and provides the data in a display format as described below. On the next screen, select the area for which you want to obtain plankton distribution information, and if you want to obtain information on a specific area within that area, press the area mode button and the button for the date and time for which you want information (FIG. 4A). The points where information exists are displayed as dots, and the selectable areas are displayed (FIG. 4B). When you click on the area for which you want information, the information entered in that area is displayed as a list (FIG. 4C). If you want to obtain accumulated information from the past, press the "accumulated up to one week ago" button to overlay data from up to one week ago (FIG. 4D). If you want to obtain information on a specific location, press the point mode button (FIG. 5A) and click on a point with information (FIG. 5B). Then, the information entered at that location is obtained (FIG. 5C). If you want to obtain past accumulated information, press the "Accumulated up to one week ago" button to overlay data from up to one week ago (Figure 5D).

==How to detect plankton==
The plankton detection method of this embodiment is a method that performs the following steps: a first step in which a computer trains a general object detection algorithm using an image of a rectangle of a first predetermined size centered on a bounding box (bbox) that represents the position of the object as training data; a second step in which an image of plankton in a container is divided into rectangles of a second predetermined size; and a third step in which the computer determines whether plankton is present in each of the rectangles of the second predetermined size obtained by dividing the image; in the second step, the method detects specific plankton in a container where the overlap width of adjacent rectangles is approximately the same as the maximum width of the plankton.

In the first step, an image is created by cutting out a rectangle of a first predetermined size centered on a bounding box (bbox) that represents the position of the object. Here, in order to accurately detect plankton, the predetermined size may be a size that is equal to or larger than the size that the entire plankton can be contained in, and equal to or smaller than the size that the general object detection algorithm can process. The rectangle of the first predetermined size is preferably a square that is larger than 1.3 times the maximum width of the plankton, more preferably a square that is larger than 1.5 times, and even more preferably a square that is larger than 2.0 times, and preferably a square that is smaller than 6.0 times the maximum width of the plankton, more preferably a square that is smaller than 4.5 times, and even more preferably a square that is smaller than 4.0 times. Alternatively, it is a rectangle whose short side is larger than 1.0 times the maximum width of the plankton, preferably a rectangle that is larger than 1.3 times, more preferably a rectangle that is larger than 1.5 times, and even more preferably a rectangle that is larger than 2.0 times, and preferably a rectangle whose long side is smaller than 6.0 times the maximum width of the plankton, more preferably a rectangle that is smaller than 4.5 times, and even more preferably a rectangle that is smaller than 4.0 times. In this specification, the maximum width of the plankton refers to the length of the line segment where the plankton overlaps with the line drawn across the plankton on the image, when the line segment overlaps the plankton for the longest time. Additionally, the maximum height of the plankton refers to the length of the line segment where the plankton overlaps with the line perpendicular to the line segment that is the maximum width of the plankton, when the line segment overlaps the plankton for the longest time.

The number of bounding boxes (bboxes) used to learn the plankton to be detected is not particularly limited, but is preferably 1000 to 2000, more preferably 2000 to 3000, and even more preferably 3000 to 4000. It is preferable that the orientation of the plankton is randomly shown in the image.

The type of plankton is not particularly limited, and may be zooplankton or phytoplankton, but is preferably a floating or attached larva of a sessile organism, such as, but not limited to, oysters, barnacles, mussels, sea urchins, and hydra. The number of types may be one or more. However, it is preferable to have eyespots, as this increases the detection power of the general object detection algorithm.

The general object detection algorithm used is not particularly limited, but an R-CNN-based model (e.g., Fast R-CNN, Faster R-CNN, Mask R-CNN), SSD, or a YOLO-based model (e.g., YOLO v1 to v3) is preferred.

In the second step, the image of the plankton in the container to be measured is divided into rectangles of a second predetermined size. The shape of the rectangle is not particularly limited, but it is sufficient that the opposite sides are parallel. It may be a parallelogram or a rhombus, but a rectangle is more preferable, and a square is even more preferable. The length of each side is preferably greater than 1.0 times the maximum width of the plankton, more preferably greater than 1.5 times, and even more preferably greater than 2.0 times, preferably less than 12.0 times the maximum width, more preferably less than 9.0 times, and even more preferably less than 6.0 times. It is preferable to arrange the rectangles by shifting them along the parallel sides, and the width of the overlap between adjacent rectangles on any of the four sides is determined based on the maximum width of the plankton and is preferably approximately the same as the maximum width. As a result, for all plankton, the entire plankton is contained in some rectangle. If the overlap width of adjacent rectangles is smaller than the maximum width of the plankton, there will be more plankton that do not entirely fit into any rectangle, and if it is larger than the maximum width of the plankton, there will be more plankton that fit into multiple rectangles. Here, the overlap width of adjacent rectangles refers to the length of shift when the rectangles are shifted along their parallel sides. Specifically, the overlap width of adjacent rectangles is preferably 0.8 to 1.2 times the maximum width of the plankton, more preferably 0.9 to 1.1 times, and even more preferably 0.95 to 1.05 times.

In the third step, the previously trained general object detection algorithm is made to determine whether plankton is present within each of the second predetermined size rectangles obtained by division. In this way, it becomes possible to have the computer measure the number of plankton. When multiple types of plankton are present, the number of each type of plankton can be measured.

Plankton were collected from the waters around Etajima using a plankton net, tap water was added to stop their activity, and the fractions were collected in the order of their sinking. Because oyster larvae are heavier than other plankton, by collecting the fraction of plankton that sank quickly, almost only oyster larvae were collected. They were placed in a petri dish and photographed with a digital camera, enlarging with optical zoom until the eyespots could be seen with the naked eye. An example is shown in Figure 6.

As shown in Figure 7, 160 images measuring 2992 x 2992px (7.6 x 7.6mm) were visually checked into three groups: (0) oyster larvae with eyespots and a maximum width of 270μm or more, (1) oyster larvae without eyespots and a maximum width of 270μm or more, and (2) oyster larvae with a maximum width of 210-270μm or more. (0) 1561 individuals, (1) 1432 individuals, and (2) 2536 individuals were identified. The image files were shifted by 15° and then inverted to make the number of images 7680, and 264640 image files were created by cutting out 300 x 300px images from each image file with the bbox as the base point. This was used as learning data for the SSD300, and 32 images were input 24000 times, one cycle, for three cycles of input. The learning time was about 6.6 days using a GPU.

For images of the same size that were not used for learning, 300 x 300 px squares were cut out, shifted by 150 px each, and 361 images were created for each image. For each image, oyster larvae with eyespots and a maximum width of 270 μm or more were detected (thresholds were Confidence: 0.6, non_maximum_suppression: 0.4). The precision was 0.97 and the recall was 0.96. Note that precision and recall are both evaluation indices for object detection. Precision is the proportion of oyster larvae with eyespots and a maximum width of 270 μm or more that were correctly detected, and recall is the proportion of oyster larvae with eyespots and a maximum width of 270 μm or more that were correctly detected, among those detected.

In this way, the method disclosed herein makes it possible to measure the number of target plankton with sufficient accuracy.

REFERENCE SIGNS LIST 11 Server 12 Receiving means 13 Transmitting means 14 Analyzing means 15 Converting means 16 Storage means 21 First user terminal 22 Imaging means 23 Transmitting means 24 Storage means 31 Second user terminal 32 Receiving means 33 Display means 34 35 Conversion means

Claims (8)

The computer
A first step of training a generic object detection algorithm using a rectangular image of a first predetermined size centered on a bounding box as training data;
A second step of dividing the image of the plankton in the container into rectangles of a second predetermined size;
A third step of causing the generic object detection algorithm to determine whether the plankton is present within each of the quadrangles of the second predetermined size obtained by the division;
Do the following:
In the second step, the width of the overlap between the adjacent rectangles is determined based on the maximum width of the plankton ;
A method for detecting specific plankton in a container , wherein the width of the overlap of the rectangles is greater than or equal to 0.8 times and less than or equal to 1.2 times the maximum width of the plankton. The method according to claim 1, wherein the plankton is a planktonic or attached larva of a sessile organism. The method according to claim 1 or 2, wherein the plankton is oyster larvae in the attachment stage. The method according to any one of claims 1 to 3, wherein the general object detection algorithm is an R-CNN-based model, an SSD, or a YOLO-based model. The method according to any one of claims 1 to 4, wherein the width of the overlap of the quadrangles is 0.9 to 1.1 times the maximum width of the plankton. The method according to any one of claims 1 to 4, wherein the width of the overlap of the quadrangles is 0.95 to 1.05 times the maximum width of the plankton. A program for causing a computer to execute the method according to any one of claims 1 to 6 . A storage medium storing the program according to claim 7 in a computer-readable manner.

Images